[GH-ISSUE #1688] ollama serve crushing after 0.1.17 #950

Closed
opened 2026-04-12 10:38:59 -05:00 by GiteaMirror · 6 comments
Owner

Originally created by @Cybervet on GitHub (Dec 23, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1688

Originally assigned to: @BruceMacD on GitHub.

After pulling the new 0.1.17 and changing the source for AVX=off , compilation went fine with no errors.
ollama serve starts fine until I am trying to run a model and then it crushes with the following msg (LONG).
Any help aw to what is happening will be appreciated.

CUDA error 2 at /root/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9133: out of memory
current device: 0
GGML_ASSERT: /root/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9133: !"CUDA error"
SIGABRT: abort
PC=0x7fc589bcad3c m=5 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 12 [syscall]:
runtime.cgocall(0xa12b00, 0xc0002ca7a8)
/usr/local/go/src/runtime/cgocall.go:157 +0x5c fp=0xc0002ca780 sp=0xc0002ca748 pc=0x44df3c
github.com/jmorganca/ollama/llm._Cfunc_dynamic_shim_llama_server_init({0x7fc550001070, 0x7fc518eb7250, 0x7fc518eb07e0, 0x7fc518eb1160, 0x7fc518eb9c20, 0x7fc518eb4cd0, 0x7fc518eb1ce0, 0x7fc518eb0850, 0x7fc518eb9d10, 0x7fc518eba230, ...}, ...)
cgo_gotypes.go:291 +0x45 fp=0xc0002ca7a8 sp=0xc0002ca780 pc=0x80f2e5
github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init.func1(0x47d93f?, 0x80?, 0x80?)
/root/ollama/llm/shim_ext_server.go:43 +0xec fp=0xc0002ca898 sp=0xc0002ca7a8 pc=0x814a8c
github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init(0xc86fe1?, 0x4000?, 0x0?)
/root/ollama/llm/shim_ext_server.go:43 +0x19 fp=0xc0002ca8c0 sp=0xc0002ca898 pc=0x814959
github.com/jmorganca/ollama/llm.newExtServer({0x2313d270, 0xc0004363c0}, {
, }, {, _, }, {0x0, 0x0, 0x0}, ...)
/root/ollama/llm/ext_server.go:192 +0x6ff fp=0xc0002cab00 sp=0xc0002ca8c0 pc=0x8113bf
github.com/jmorganca/ollama/llm.newDynamicShimExtServer({0xc000512060, 0x27}, {
, }, {, _, }, {0x0, 0x0, 0x0}, ...)
/root/ollama/llm/shim_ext_server.go:95 +0x4af fp=0xc0002cacd8 sp=0xc0002cab00 pc=0x815e4f
github.com/jmorganca/ollama/llm.newLlmServer({0xc74877, 0xb}, {
, }, {, _, }, {0x0, 0x0, 0x0}, ...)
/root/ollama/llm/llm.go:94 +0x18b fp=0xc0002cae20 sp=0xc0002cacd8 pc=0x80e8cb
github.com/jmorganca/ollama/llm.New({0x0?, 0xc000110d20?}, {
, }, {, _, _}, {0x0, 0x0, 0x0}, ...)
/root/ollama/llm/llm.go:84 +0x2a6 fp=0xc0002cb010 sp=0xc0002cae20 pc=0x80e4e6
github.com/jmorganca/ollama/server.load(0x0?, {0xc00052a620?, 0x23133f60?}, 0xc00005a120?, 0xc343c0?)
/root/ollama/server/routes.go:99 +0x4ce fp=0xc0002cb290 sp=0xc0002cb010 pc=0x9f5dee
github.com/jmorganca/ollama/server.GenerateHandler(0xc0004ba000)
/root/ollama/server/routes.go:172 +0x377 fp=0xc0002cb6d0 sp=0xc0002cb290 pc=0x9f6437
github.com/gin-gonic/gin.(*Context).Next(...)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/jmorganca/ollama/server.(*Server).GenerateRoutes.func1(0xc0004ba000)
/root/ollama/server/routes.go:834 +0x6b fp=0xc0002cb708 sp=0xc0002cb6d0 pc=0x9ff44b
github.com/gin-gonic/gin.(*Context).Next(...)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc0004ba000)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +0x82 fp=0xc0002cb758 sp=0xc0002cb708 pc=0x9d9fa2
github.com/gin-gonic/gin.(*Context).Next(...)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc0004ba000)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0xe7 fp=0xc0002cb908 sp=0xc0002cb758 pc=0x9d90c7
github.com/gin-gonic/gin.(*Context).Next(...)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc000137ba0, 0xc0004ba000)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x66b fp=0xc0002cba90 sp=0xc0002cb908 pc=0x9d812b
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc000137ba0, {0x23138910?, 0xc0003fc0e0}, 0xc00053a300)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1dd fp=0xc0002cbad0 sp=0xc0002cba90 pc=0x9d78dd
net/http.serverHandler.ServeHTTP({0x231365c0?}, {0x23138910, 0xc0003fc0e0}, 0xc00053a300)
/usr/local/go/src/net/http/server.go:2936 +0x316 fp=0xc0002cbb80 sp=0xc0002cbad0 pc=0x7231d6
net/http.(*conn).serve(0xc00049eb40, {0x23138f88, 0xc00050e1e0})
/usr/local/go/src/net/http/server.go:1995 +0x612 fp=0xc0002cbfb8 sp=0xc0002cbb80 pc=0x71e6f2
net/http.(*Server).Serve.func3()
/usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc0002cbfe0 sp=0xc0002cbfb8 pc=0x723b2e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cbfe8 sp=0xc0002cbfe0 pc=0x4b18e1
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3089 +0x5ed

goroutine 1 [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0001478c8 sp=0xc0001478a8 pc=0x481f76
runtime.netpollblock(0x7fc5606a7968?, 0x44d5cf?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000147900 sp=0xc0001478c8 pc=0x47a977
internal/poll.runtime_pollWait(0x7fc56072e018, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000147920 sp=0xc000147900 pc=0x4abd29
internal/poll.(*pollDesc).wait(0xc00040df80?, 0x4?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000147948 sp=0xc000147920 pc=0x5389f2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00040df80)
/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0001479f0 sp=0xc000147948 pc=0x53e2fd
net.(*netFD).accept(0xc00040df80)
/usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000147aa8 sp=0xc0001479f0 pc=0x5b3d95
net.(*TCPListener).accept(0xc000012f18)
/usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000147ad0 sp=0xc000147aa8 pc=0x5ca125
net.(*TCPListener).Accept(0xc000012f18)
/usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000147b00 sp=0xc000147ad0 pc=0x5c921d
net/http.(*onceCloseListener).Accept(0xc00049eb40?)
:1 +0x2a fp=0xc000147b18 sp=0xc000147b00 pc=0x7484aa
net/http.(*Server).Serve(0xc00035eff0, {0x23138700, 0xc000012f18})
/usr/local/go/src/net/http/server.go:3059 +0x385 fp=0xc000147c48 sp=0xc000147b18 pc=0x723745
github.com/jmorganca/ollama/server.Serve({0x23138700, 0xc000012f18})
/root/ollama/server/routes.go:914 +0x2d0 fp=0xc000147cf8 sp=0xc000147c48 pc=0x9ff770
github.com/jmorganca/ollama/cmd.RunServer(0xc000415800?, {0x2366aac8?, 0x0?, 0x0?})
/root/ollama/cmd/cmd.go:1077 +0x169 fp=0xc000147d70 sp=0xc000147cf8 pc=0xa0f769
github.com/spf13/cobra.(*Command).execute(0xc000415800, {0x2366aac8, 0x0, 0x0})
/root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 fp=0xc000147ea8 sp=0xc000147d70 pc=0x7bf6c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc000414c00)
/root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd fp=0xc000147f60 sp=0xc000147ea8 pc=0x7bff3d
github.com/spf13/cobra.(*Command).Execute(...)
/root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
/root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
/root/ollama/main.go:11 +0x54 fp=0xc000147f80 sp=0xc000147f60 pc=0xa11c34
runtime.main()
/usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc000147fe0 sp=0xc000147f80 pc=0x481b47
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000147fe8 sp=0xc000147fe0 pc=0x4b18e1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000044fb0 sp=0xc000044f90 pc=0x481f76
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:387
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000044fe0 sp=0xc000044fb0 pc=0x481db0
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000044fe8 sp=0xc000044fe0 pc=0x4b18e1
created by runtime.init.6
/usr/local/go/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000045780 sp=0xc000045760 pc=0x481f76
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
/usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000457c8 sp=0xc000045780 pc=0x46e29e
runtime.gcenable.func1()
/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000457e0 sp=0xc0000457c8 pc=0x463506
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000457e8 sp=0xc0000457e0 pc=0x4b18e1
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x38f9409e3ef0?, 0x3b9f4ff3?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000045f70 sp=0xc000045f50 pc=0x481f76
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x23574640)
/usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000045fa0 sp=0xc000045f70 pc=0x46c173
runtime.bgscavenge(0x0?)
/usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000045fc8 sp=0xc000045fa0 pc=0x46c765
runtime.gcenable.func2()
/usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000045fe0 sp=0xc000045fc8 pc=0x4634a6
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000045fe8 sp=0xc000045fe0 pc=0x4b18e1
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x1a0?, 0x23575040?, 0x60?, 0x78?, 0xc000044770?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000044628 sp=0xc000044608 pc=0x481f76
runtime.runfinq()
/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000447e0 sp=0xc000044628 pc=0x462547
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000447e8 sp=0xc0000447e0 pc=0x4b18e1
created by runtime.createfing
/usr/local/go/src/runtime/mfinal.go:163 +0x45

goroutine 6 [select, locked to thread]:
runtime.gopark(0xc0000467a8?, 0x2?, 0xf2?, 0x22?, 0xc0000467a4?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000046618 sp=0xc0000465f8 pc=0x481f76
runtime.selectgo(0xc0000467a8, 0xc0000467a0, 0x0?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000046758 sp=0xc000046618 pc=0x49181e
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal_unix.go:1000 +0x1af fp=0xc0000467e0 sp=0xc000046758 pc=0x4a96ef
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000467e8 sp=0xc0000467e0 pc=0x4b18e1
created by runtime.ensureSigM
/usr/local/go/src/runtime/signal_unix.go:983 +0xbd

goroutine 18 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
/usr/local/go/src/runtime/lock_futex.go:236 +0x34 fp=0xc0000407a0 sp=0xc000040768 pc=0x456134
os/signal.signal_recv()
/usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc0000407c0 sp=0xc0000407a0 pc=0x4ae10f
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc0000407e0 sp=0xc0000407c0 pc=0x74b319
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000407e8 sp=0xc0000407e0 pc=0x4b18e1
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 34 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000508700 sp=0xc0005086e0 pc=0x481f76
runtime.chanrecv(0xc0001b3500, 0x0, 0x1)
/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000508790 sp=0xc000508700 pc=0x450cfd
runtime.chanrecv1(0x0?, 0x0?)
/usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0005087b8 sp=0xc000508790 pc=0x4507f8
github.com/jmorganca/ollama/server.Serve.func1()
/root/ollama/server/routes.go:896 +0x28 fp=0xc0005087e0 sp=0xc0005087b8 pc=0x9ff828
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005087e8 sp=0xc0005087e0 pc=0x4b18e1
created by github.com/jmorganca/ollama/server.Serve
/root/ollama/server/routes.go:895 +0x219

goroutine 35 [IO wait]:
runtime.gopark(0x75?, 0xb?, 0x0?, 0x0?, 0xc?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000141900 sp=0xc0001418e0 pc=0x481f76
runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000141938 sp=0xc000141900 pc=0x47a977
internal/poll.runtime_pollWait(0x7fc56072df28, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000141958 sp=0xc000141938 pc=0x4abd29
internal/poll.(*pollDesc).wait(0xc00050c080?, 0xc000536000?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000141980 sp=0xc000141958 pc=0x5389f2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00050c080, {0xc000536000, 0x1000, 0x1000})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000141a18 sp=0xc000141980 pc=0x539dd9
net.(*netFD).Read(0xc00050c080, {0xc000536000?, 0x538ee6?, 0x0?})
/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000141a60 sp=0xc000141a18 pc=0x5b1c09
net.(*conn).Read(0xc00051c020, {0xc000536000?, 0x0?, 0xc00050e2d8?})
/usr/local/go/src/net/net.go:183 +0x45 fp=0xc000141aa8 sp=0xc000141a60 pc=0x5c1045
net.(*TCPConn).Read(0xc00050e2d0?, {0xc000536000?, 0x0?, 0xc000141ac8?})
:1 +0x29 fp=0xc000141ad8 sp=0xc000141aa8 pc=0x5d3c09
net/http.(*connReader).Read(0xc00050e2d0, {0xc000536000, 0x1000, 0x1000})
/usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc000141b28 sp=0xc000141ad8 pc=0x7188b1
bufio.(*Reader).fill(0xc00051e060)
/usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000141b60 sp=0xc000141b28 pc=0x6a5c5f
bufio.(*Reader).Peek(0xc00051e060, 0x4)
/usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc000141b80 sp=0xc000141b60 pc=0x6a5dbd
net/http.(*conn).serve(0xc000532000, {0x23138f88, 0xc00050e1e0})
/usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc000141fb8 sp=0xc000141b80 pc=0x71e85c
net/http.(*Server).Serve.func3()
/usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc000141fe0 sp=0xc000141fb8 pc=0x723b2e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000141fe8 sp=0xc000141fe0 pc=0x4b18e1
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3089 +0x5ed

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x38f93b6df672?, 0x1?, 0x4e?, 0x14?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000508f50 sp=0xc000508f30 pc=0x481f76
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000508fe0 sp=0xc000508f50 pc=0x465271
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000508fe8 sp=0xc000508fe0 pc=0x4b18e1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x38f9390eb146?, 0x3?, 0x1?, 0xb2?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000046f50 sp=0xc000046f30 pc=0x481f76
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000046fe0 sp=0xc000046f50 pc=0x465271
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x4b18e1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x38f93b5b21f1?, 0x3?, 0xf5?, 0x68?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000047750 sp=0xc000047730 pc=0x481f76
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000477e0 sp=0xc000047750 pc=0x465271
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x4b18e1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x38f93b6def59?, 0x3?, 0x66?, 0x58?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000047f50 sp=0xc000047f30 pc=0x481f76
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000047fe0 sp=0xc000047f50 pc=0x465271
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x4b18e1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 51 [IO wait]:
runtime.gopark(0x7?, 0xb?, 0x0?, 0x0?, 0xe?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0004d5900 sp=0xc0004d58e0 pc=0x481f76
runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0004d5938 sp=0xc0004d5900 pc=0x47a977
internal/poll.runtime_pollWait(0x7fc56072dd48, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0004d5958 sp=0xc0004d5938 pc=0x4abd29
internal/poll.(*pollDesc).wait(0xc00013e000?, 0xc0004ce000?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0004d5980 sp=0xc0004d5958 pc=0x5389f2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00013e000, {0xc0004ce000, 0x1000, 0x1000})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0004d5a18 sp=0xc0004d5980 pc=0x539dd9
net.(*netFD).Read(0xc00013e000, {0xc0004ce000?, 0x538ee6?, 0x0?})
/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0004d5a60 sp=0xc0004d5a18 pc=0x5b1c09
net.(*conn).Read(0xc00051c000, {0xc0004ce000?, 0x0?, 0xc00050e0c8?})
/usr/local/go/src/net/net.go:183 +0x45 fp=0xc0004d5aa8 sp=0xc0004d5a60 pc=0x5c1045
net.(*TCPConn).Read(0xc00050e0c0?, {0xc0004ce000?, 0x0?, 0xc000593ac8?})
:1 +0x29 fp=0xc0004d5ad8 sp=0xc0004d5aa8 pc=0x5d3c09
net/http.(*connReader).Read(0xc00050e0c0, {0xc0004ce000, 0x1000, 0x1000})
/usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc0004d5b28 sp=0xc0004d5ad8 pc=0x7188b1
bufio.(*Reader).fill(0xc000096060)
/usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc0004d5b60 sp=0xc0004d5b28 pc=0x6a5c5f
bufio.(*Reader).Peek(0xc000096060, 0x4)
/usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc0004d5b80 sp=0xc0004d5b60 pc=0x6a5dbd
net/http.(*conn).serve(0xc000532090, {0x23138f88, 0xc00050e1e0})
/usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc0004d5fb8 sp=0xc0004d5b80 pc=0x71e85c
net/http.(*Server).Serve.func3()
/usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc0004d5fe0 sp=0xc0004d5fb8 pc=0x723b2e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0004d5fe8 sp=0xc0004d5fe0 pc=0x4b18e1
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3089 +0x5ed

goroutine 10 [IO wait]:
runtime.gopark(0x7?, 0xb?, 0x0?, 0x0?, 0xd?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000145900 sp=0xc0001458e0 pc=0x481f76
runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000145938 sp=0xc000145900 pc=0x47a977
internal/poll.runtime_pollWait(0x7fc56072de38, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000145958 sp=0xc000145938 pc=0x4abd29
internal/poll.(*pollDesc).wait(0xc00050c100?, 0xc000588000?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000145980 sp=0xc000145958 pc=0x5389f2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00050c100, {0xc000588000, 0x1000, 0x1000})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000145a18 sp=0xc000145980 pc=0x539dd9
net.(*netFD).Read(0xc00050c100, {0xc000588000?, 0x538ee6?, 0x0?})
/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000145a60 sp=0xc000145a18 pc=0x5b1c09
net.(*conn).Read(0xc0004a8008, {0xc000588000?, 0x0?, 0xc0004a21e8?})
/usr/local/go/src/net/net.go:183 +0x45 fp=0xc000145aa8 sp=0xc000145a60 pc=0x5c1045
net.(*TCPConn).Read(0xc0004a21e0?, {0xc000588000?, 0x0?, 0xc000597ac8?})
:1 +0x29 fp=0xc000145ad8 sp=0xc000145aa8 pc=0x5d3c09
net/http.(*connReader).Read(0xc0004a21e0, {0xc000588000, 0x1000, 0x1000})
/usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc000145b28 sp=0xc000145ad8 pc=0x7188b1
bufio.(*Reader).fill(0xc00048e120)
/usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000145b60 sp=0xc000145b28 pc=0x6a5c5f
bufio.(*Reader).Peek(0xc00048e120, 0x4)
/usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc000145b80 sp=0xc000145b60 pc=0x6a5dbd
net/http.(*conn).serve(0xc00049e120, {0x23138f88, 0xc00050e1e0})
/usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc000145fb8 sp=0xc000145b80 pc=0x71e85c
net/http.(*Server).Serve.func3()
/usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc000145fe0 sp=0xc000145fb8 pc=0x723b2e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000145fe8 sp=0xc000145fe0 pc=0x4b18e1
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3089 +0x5ed

goroutine 53 [IO wait]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0xf?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0005045a0 sp=0xc000504580 pc=0x481f76
runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0005045d8 sp=0xc0005045a0 pc=0x47a977
internal/poll.runtime_pollWait(0x7fc56072dc58, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0005045f8 sp=0xc0005045d8 pc=0x4abd29
internal/poll.(*pollDesc).wait(0xc00050c300?, 0xc00050f181?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000504620 sp=0xc0005045f8 pc=0x5389f2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00050c300, {0xc00050f181, 0x1, 0x1})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0005046b8 sp=0xc000504620 pc=0x539dd9
net.(*netFD).Read(0xc00050c300, {0xc00050f181?, 0xbdaba0?, 0xc000504748?})
/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000504700 sp=0xc0005046b8 pc=0x5b1c09
net.(*conn).Read(0xc0004a8140, {0xc00050f181?, 0xc0005380d0?, 0xc0005165a0?})
/usr/local/go/src/net/net.go:183 +0x45 fp=0xc000504748 sp=0xc000504700 pc=0x5c1045
net.(*TCPConn).Read(0xc00050e0c0?, {0xc00050f181?, 0xc0005165a0?, 0x0?})
:1 +0x29 fp=0xc000504778 sp=0xc000504748 pc=0x5d3c09
net/http.(*connReader).backgroundRead(0xc00050f170)
/usr/local/go/src/net/http/server.go:674 +0x3f fp=0xc0005047c8 sp=0xc000504778 pc=0x71843f
net/http.(*connReader).startBackgroundRead.func2()
/usr/local/go/src/net/http/server.go:670 +0x26 fp=0xc0005047e0 sp=0xc0005047c8 pc=0x718366
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005047e8 sp=0xc0005047e0 pc=0x4b18e1
created by net/http.(*connReader).startBackgroundRead
/usr/local/go/src/net/http/server.go:670 +0xca

rax 0x0
rbx 0x3ec
rcx 0x7fc589bcad3c
rdx 0x6
rdi 0x3ec
rsi 0x3ec
rbp 0x7fc5617a86c0
rsp 0x7fc5617a6120
r8 0x7fffb3d98faa
r9 0x7fc5617a60f0
r10 0x8
r11 0x246
r12 0x6
r13 0x0
r14 0x4000000
r15 0x0
rip 0x7fc589bcad3c
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
SIGABRT: abort
PC=0x7fc589bcad3c m=5 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 12 [syscall]:
runtime.cgocall(0xa12b00, 0xc0002ca7a8)
/usr/local/go/src/runtime/cgocall.go:157 +0x5c fp=0xc0002ca780 sp=0xc0002ca748 pc=0x44df3c
github.com/jmorganca/ollama/llm._Cfunc_dynamic_shim_llama_server_init({0x7fc550001070, 0x7fc518eb7250, 0x7fc518eb07e0, 0x7fc518eb1160, 0x7fc518eb9c20, 0x7fc518eb4cd0, 0x7fc518eb1ce0, 0x7fc518eb0850, 0x7fc518eb9d10, 0x7fc518eba230, ...}, ...)
cgo_gotypes.go:291 +0x45 fp=0xc0002ca7a8 sp=0xc0002ca780 pc=0x80f2e5
github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init.func1(0x47d93f?, 0x80?, 0x80?)
/root/ollama/llm/shim_ext_server.go:43 +0xec fp=0xc0002ca898 sp=0xc0002ca7a8 pc=0x814a8c
github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init(0xc86fe1?, 0x4000?, 0x0?)
/root/ollama/llm/shim_ext_server.go:43 +0x19 fp=0xc0002ca8c0 sp=0xc0002ca898 pc=0x814959
github.com/jmorganca/ollama/llm.newExtServer({0x2313d270, 0xc0004363c0}, {
, }, {, _, }, {0x0, 0x0, 0x0}, ...)
/root/ollama/llm/ext_server.go:192 +0x6ff fp=0xc0002cab00 sp=0xc0002ca8c0 pc=0x8113bf
github.com/jmorganca/ollama/llm.newDynamicShimExtServer({0xc000512060, 0x27}, {
, }, {, _, }, {0x0, 0x0, 0x0}, ...)
/root/ollama/llm/shim_ext_server.go:95 +0x4af fp=0xc0002cacd8 sp=0xc0002cab00 pc=0x815e4f
github.com/jmorganca/ollama/llm.newLlmServer({0xc74877, 0xb}, {
, }, {, _, }, {0x0, 0x0, 0x0}, ...)
/root/ollama/llm/llm.go:94 +0x18b fp=0xc0002cae20 sp=0xc0002cacd8 pc=0x80e8cb
github.com/jmorganca/ollama/llm.New({0x0?, 0xc000110d20?}, {
, }, {, _, _}, {0x0, 0x0, 0x0}, ...)
/root/ollama/llm/llm.go:84 +0x2a6 fp=0xc0002cb010 sp=0xc0002cae20 pc=0x80e4e6
github.com/jmorganca/ollama/server.load(0x0?, {0xc00052a620?, 0x23133f60?}, 0xc00005a120?, 0xc343c0?)
/root/ollama/server/routes.go:99 +0x4ce fp=0xc0002cb290 sp=0xc0002cb010 pc=0x9f5dee
github.com/jmorganca/ollama/server.GenerateHandler(0xc0004ba000)
/root/ollama/server/routes.go:172 +0x377 fp=0xc0002cb6d0 sp=0xc0002cb290 pc=0x9f6437
github.com/gin-gonic/gin.(*Context).Next(...)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/jmorganca/ollama/server.(*Server).GenerateRoutes.func1(0xc0004ba000)
/root/ollama/server/routes.go:834 +0x6b fp=0xc0002cb708 sp=0xc0002cb6d0 pc=0x9ff44b
github.com/gin-gonic/gin.(*Context).Next(...)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc0004ba000)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +0x82 fp=0xc0002cb758 sp=0xc0002cb708 pc=0x9d9fa2
github.com/gin-gonic/gin.(*Context).Next(...)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc0004ba000)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0xe7 fp=0xc0002cb908 sp=0xc0002cb758 pc=0x9d90c7
github.com/gin-gonic/gin.(*Context).Next(...)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc000137ba0, 0xc0004ba000)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x66b fp=0xc0002cba90 sp=0xc0002cb908 pc=0x9d812b
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc000137ba0, {0x23138910?, 0xc0003fc0e0}, 0xc00053a300)
/root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1dd fp=0xc0002cbad0 sp=0xc0002cba90 pc=0x9d78dd
net/http.serverHandler.ServeHTTP({0x231365c0?}, {0x23138910, 0xc0003fc0e0}, 0xc00053a300)
/usr/local/go/src/net/http/server.go:2936 +0x316 fp=0xc0002cbb80 sp=0xc0002cbad0 pc=0x7231d6
net/http.(*conn).serve(0xc00049eb40, {0x23138f88, 0xc00050e1e0})
/usr/local/go/src/net/http/server.go:1995 +0x612 fp=0xc0002cbfb8 sp=0xc0002cbb80 pc=0x71e6f2
net/http.(*Server).Serve.func3()
/usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc0002cbfe0 sp=0xc0002cbfb8 pc=0x723b2e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cbfe8 sp=0xc0002cbfe0 pc=0x4b18e1
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3089 +0x5ed

goroutine 1 [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0001478c8 sp=0xc0001478a8 pc=0x481f76
runtime.netpollblock(0x7fc5606a7968?, 0x44d5cf?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000147900 sp=0xc0001478c8 pc=0x47a977
internal/poll.runtime_pollWait(0x7fc56072e018, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000147920 sp=0xc000147900 pc=0x4abd29
internal/poll.(*pollDesc).wait(0xc00040df80?, 0x4?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000147948 sp=0xc000147920 pc=0x5389f2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00040df80)
/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0001479f0 sp=0xc000147948 pc=0x53e2fd
net.(*netFD).accept(0xc00040df80)
/usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000147aa8 sp=0xc0001479f0 pc=0x5b3d95
net.(*TCPListener).accept(0xc000012f18)
/usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000147ad0 sp=0xc000147aa8 pc=0x5ca125
net.(*TCPListener).Accept(0xc000012f18)
/usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000147b00 sp=0xc000147ad0 pc=0x5c921d
net/http.(*onceCloseListener).Accept(0xc00049eb40?)
:1 +0x2a fp=0xc000147b18 sp=0xc000147b00 pc=0x7484aa
net/http.(*Server).Serve(0xc00035eff0, {0x23138700, 0xc000012f18})
/usr/local/go/src/net/http/server.go:3059 +0x385 fp=0xc000147c48 sp=0xc000147b18 pc=0x723745
github.com/jmorganca/ollama/server.Serve({0x23138700, 0xc000012f18})
/root/ollama/server/routes.go:914 +0x2d0 fp=0xc000147cf8 sp=0xc000147c48 pc=0x9ff770
github.com/jmorganca/ollama/cmd.RunServer(0xc000415800?, {0x2366aac8?, 0x0?, 0x0?})
/root/ollama/cmd/cmd.go:1077 +0x169 fp=0xc000147d70 sp=0xc000147cf8 pc=0xa0f769
github.com/spf13/cobra.(*Command).execute(0xc000415800, {0x2366aac8, 0x0, 0x0})
/root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 fp=0xc000147ea8 sp=0xc000147d70 pc=0x7bf6c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc000414c00)
/root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd fp=0xc000147f60 sp=0xc000147ea8 pc=0x7bff3d
github.com/spf13/cobra.(*Command).Execute(...)
/root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
/root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
/root/ollama/main.go:11 +0x54 fp=0xc000147f80 sp=0xc000147f60 pc=0xa11c34
runtime.main()
/usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc000147fe0 sp=0xc000147f80 pc=0x481b47
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000147fe8 sp=0xc000147fe0 pc=0x4b18e1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000044fb0 sp=0xc000044f90 pc=0x481f76
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:387
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000044fe0 sp=0xc000044fb0 pc=0x481db0
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000044fe8 sp=0xc000044fe0 pc=0x4b18e1
created by runtime.init.6
/usr/local/go/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000045780 sp=0xc000045760 pc=0x481f76
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
/usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000457c8 sp=0xc000045780 pc=0x46e29e
runtime.gcenable.func1()
/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000457e0 sp=0xc0000457c8 pc=0x463506
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000457e8 sp=0xc0000457e0 pc=0x4b18e1
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x38f9409e3ef0?, 0x3b9f4ff3?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000045f70 sp=0xc000045f50 pc=0x481f76
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x23574640)
/usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000045fa0 sp=0xc000045f70 pc=0x46c173
runtime.bgscavenge(0x0?)
/usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000045fc8 sp=0xc000045fa0 pc=0x46c765
runtime.gcenable.func2()
/usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000045fe0 sp=0xc000045fc8 pc=0x4634a6
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000045fe8 sp=0xc000045fe0 pc=0x4b18e1
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x1a0?, 0x23575040?, 0x60?, 0x78?, 0xc000044770?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000044628 sp=0xc000044608 pc=0x481f76
runtime.runfinq()
/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000447e0 sp=0xc000044628 pc=0x462547
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000447e8 sp=0xc0000447e0 pc=0x4b18e1
created by runtime.createfing
/usr/local/go/src/runtime/mfinal.go:163 +0x45

goroutine 6 [select, locked to thread]:
runtime.gopark(0xc0000467a8?, 0x2?, 0xf2?, 0x22?, 0xc0000467a4?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000046618 sp=0xc0000465f8 pc=0x481f76
runtime.selectgo(0xc0000467a8, 0xc0000467a0, 0x0?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000046758 sp=0xc000046618 pc=0x49181e
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal_unix.go:1000 +0x1af fp=0xc0000467e0 sp=0xc000046758 pc=0x4a96ef
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000467e8 sp=0xc0000467e0 pc=0x4b18e1
created by runtime.ensureSigM
/usr/local/go/src/runtime/signal_unix.go:983 +0xbd

goroutine 18 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
/usr/local/go/src/runtime/lock_futex.go:236 +0x34 fp=0xc0000407a0 sp=0xc000040768 pc=0x456134
os/signal.signal_recv()
/usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc0000407c0 sp=0xc0000407a0 pc=0x4ae10f
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc0000407e0 sp=0xc0000407c0 pc=0x74b319
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000407e8 sp=0xc0000407e0 pc=0x4b18e1
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 34 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000508700 sp=0xc0005086e0 pc=0x481f76
runtime.chanrecv(0xc0001b3500, 0x0, 0x1)
/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000508790 sp=0xc000508700 pc=0x450cfd
runtime.chanrecv1(0x0?, 0x0?)
/usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0005087b8 sp=0xc000508790 pc=0x4507f8
github.com/jmorganca/ollama/server.Serve.func1()
/root/ollama/server/routes.go:896 +0x28 fp=0xc0005087e0 sp=0xc0005087b8 pc=0x9ff828
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005087e8 sp=0xc0005087e0 pc=0x4b18e1
created by github.com/jmorganca/ollama/server.Serve
/root/ollama/server/routes.go:895 +0x219

goroutine 35 [IO wait]:
runtime.gopark(0x75?, 0xb?, 0x0?, 0x0?, 0xc?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000141900 sp=0xc0001418e0 pc=0x481f76
runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000141938 sp=0xc000141900 pc=0x47a977
internal/poll.runtime_pollWait(0x7fc56072df28, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000141958 sp=0xc000141938 pc=0x4abd29
internal/poll.(*pollDesc).wait(0xc00050c080?, 0xc000536000?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000141980 sp=0xc000141958 pc=0x5389f2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00050c080, {0xc000536000, 0x1000, 0x1000})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000141a18 sp=0xc000141980 pc=0x539dd9
net.(*netFD).Read(0xc00050c080, {0xc000536000?, 0x538ee6?, 0x0?})
/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000141a60 sp=0xc000141a18 pc=0x5b1c09
net.(*conn).Read(0xc00051c020, {0xc000536000?, 0x0?, 0xc00050e2d8?})
/usr/local/go/src/net/net.go:183 +0x45 fp=0xc000141aa8 sp=0xc000141a60 pc=0x5c1045
net.(*TCPConn).Read(0xc00050e2d0?, {0xc000536000?, 0x0?, 0xc000141ac8?})
:1 +0x29 fp=0xc000141ad8 sp=0xc000141aa8 pc=0x5d3c09
net/http.(*connReader).Read(0xc00050e2d0, {0xc000536000, 0x1000, 0x1000})
/usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc000141b28 sp=0xc000141ad8 pc=0x7188b1
bufio.(*Reader).fill(0xc00051e060)
/usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000141b60 sp=0xc000141b28 pc=0x6a5c5f
bufio.(*Reader).Peek(0xc00051e060, 0x4)
/usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc000141b80 sp=0xc000141b60 pc=0x6a5dbd
net/http.(*conn).serve(0xc000532000, {0x23138f88, 0xc00050e1e0})
/usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc000141fb8 sp=0xc000141b80 pc=0x71e85c
net/http.(*Server).Serve.func3()
/usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc000141fe0 sp=0xc000141fb8 pc=0x723b2e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000141fe8 sp=0xc000141fe0 pc=0x4b18e1
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3089 +0x5ed

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x38f93b6df672?, 0x1?, 0x4e?, 0x14?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000508f50 sp=0xc000508f30 pc=0x481f76
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000508fe0 sp=0xc000508f50 pc=0x465271
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000508fe8 sp=0xc000508fe0 pc=0x4b18e1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x38f9390eb146?, 0x3?, 0x1?, 0xb2?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000046f50 sp=0xc000046f30 pc=0x481f76
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000046fe0 sp=0xc000046f50 pc=0x465271
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x4b18e1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x38f93b5b21f1?, 0x3?, 0xf5?, 0x68?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000047750 sp=0xc000047730 pc=0x481f76
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000477e0 sp=0xc000047750 pc=0x465271
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x4b18e1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x38f93b6def59?, 0x3?, 0x66?, 0x58?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000047f50 sp=0xc000047f30 pc=0x481f76
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000047fe0 sp=0xc000047f50 pc=0x465271
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x4b18e1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 51 [IO wait]:
runtime.gopark(0x7?, 0xb?, 0x0?, 0x0?, 0xe?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0004d5900 sp=0xc0004d58e0 pc=0x481f76
runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0004d5938 sp=0xc0004d5900 pc=0x47a977
internal/poll.runtime_pollWait(0x7fc56072dd48, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0004d5958 sp=0xc0004d5938 pc=0x4abd29
internal/poll.(*pollDesc).wait(0xc00013e000?, 0xc0004ce000?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0004d5980 sp=0xc0004d5958 pc=0x5389f2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00013e000, {0xc0004ce000, 0x1000, 0x1000})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0004d5a18 sp=0xc0004d5980 pc=0x539dd9
net.(*netFD).Read(0xc00013e000, {0xc0004ce000?, 0x538ee6?, 0x0?})
/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0004d5a60 sp=0xc0004d5a18 pc=0x5b1c09
net.(*conn).Read(0xc00051c000, {0xc0004ce000?, 0x0?, 0xc00050e0c8?})
/usr/local/go/src/net/net.go:183 +0x45 fp=0xc0004d5aa8 sp=0xc0004d5a60 pc=0x5c1045
net.(*TCPConn).Read(0xc00050e0c0?, {0xc0004ce000?, 0x0?, 0xc000593ac8?})
:1 +0x29 fp=0xc0004d5ad8 sp=0xc0004d5aa8 pc=0x5d3c09
net/http.(*connReader).Read(0xc00050e0c0, {0xc0004ce000, 0x1000, 0x1000})
/usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc0004d5b28 sp=0xc0004d5ad8 pc=0x7188b1
bufio.(*Reader).fill(0xc000096060)
/usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc0004d5b60 sp=0xc0004d5b28 pc=0x6a5c5f
bufio.(*Reader).Peek(0xc000096060, 0x4)
/usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc0004d5b80 sp=0xc0004d5b60 pc=0x6a5dbd
net/http.(*conn).serve(0xc000532090, {0x23138f88, 0xc00050e1e0})
/usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc0004d5fb8 sp=0xc0004d5b80 pc=0x71e85c
net/http.(*Server).Serve.func3()
/usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc0004d5fe0 sp=0xc0004d5fb8 pc=0x723b2e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0004d5fe8 sp=0xc0004d5fe0 pc=0x4b18e1
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3089 +0x5ed

goroutine 10 [IO wait]:
runtime.gopark(0x7?, 0xb?, 0x0?, 0x0?, 0xd?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000145900 sp=0xc0001458e0 pc=0x481f76
runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000145938 sp=0xc000145900 pc=0x47a977
internal/poll.runtime_pollWait(0x7fc56072de38, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000145958 sp=0xc000145938 pc=0x4abd29
internal/poll.(*pollDesc).wait(0xc00050c100?, 0xc000588000?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000145980 sp=0xc000145958 pc=0x5389f2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00050c100, {0xc000588000, 0x1000, 0x1000})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000145a18 sp=0xc000145980 pc=0x539dd9
net.(*netFD).Read(0xc00050c100, {0xc000588000?, 0x538ee6?, 0x0?})
/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000145a60 sp=0xc000145a18 pc=0x5b1c09
net.(*conn).Read(0xc0004a8008, {0xc000588000?, 0x0?, 0xc0004a21e8?})
/usr/local/go/src/net/net.go:183 +0x45 fp=0xc000145aa8 sp=0xc000145a60 pc=0x5c1045
net.(*TCPConn).Read(0xc0004a21e0?, {0xc000588000?, 0x0?, 0xc000597ac8?})
:1 +0x29 fp=0xc000145ad8 sp=0xc000145aa8 pc=0x5d3c09
net/http.(*connReader).Read(0xc0004a21e0, {0xc000588000, 0x1000, 0x1000})
/usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc000145b28 sp=0xc000145ad8 pc=0x7188b1
bufio.(*Reader).fill(0xc00048e120)
/usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000145b60 sp=0xc000145b28 pc=0x6a5c5f
bufio.(*Reader).Peek(0xc00048e120, 0x4)
/usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc000145b80 sp=0xc000145b60 pc=0x6a5dbd
net/http.(*conn).serve(0xc00049e120, {0x23138f88, 0xc00050e1e0})
/usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc000145fb8 sp=0xc000145b80 pc=0x71e85c
net/http.(*Server).Serve.func3()
/usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc000145fe0 sp=0xc000145fb8 pc=0x723b2e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000145fe8 sp=0xc000145fe0 pc=0x4b18e1
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3089 +0x5ed

goroutine 53 [IO wait]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0xf?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0005045a0 sp=0xc000504580 pc=0x481f76
runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0005045d8 sp=0xc0005045a0 pc=0x47a977
internal/poll.runtime_pollWait(0x7fc56072dc58, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0005045f8 sp=0xc0005045d8 pc=0x4abd29
internal/poll.(*pollDesc).wait(0xc00050c300?, 0xc00050f181?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000504620 sp=0xc0005045f8 pc=0x5389f2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00050c300, {0xc00050f181, 0x1, 0x1})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0005046b8 sp=0xc000504620 pc=0x539dd9
net.(*netFD).Read(0xc00050c300, {0xc00050f181?, 0xbdaba0?, 0xc000504748?})
/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000504700 sp=0xc0005046b8 pc=0x5b1c09
net.(*conn).Read(0xc0004a8140, {0xc00050f181?, 0xc0005380d0?, 0xc0005165a0?})
/usr/local/go/src/net/net.go:183 +0x45 fp=0xc000504748 sp=0xc000504700 pc=0x5c1045
net.(*TCPConn).Read(0xc00050e0c0?, {0xc00050f181?, 0xc0005165a0?, 0x0?})
:1 +0x29 fp=0xc000504778 sp=0xc000504748 pc=0x5d3c09
net/http.(*connReader).backgroundRead(0xc00050f170)
/usr/local/go/src/net/http/server.go:674 +0x3f fp=0xc0005047c8 sp=0xc000504778 pc=0x71843f
net/http.(*connReader).startBackgroundRead.func2()
/usr/local/go/src/net/http/server.go:670 +0x26 fp=0xc0005047e0 sp=0xc0005047c8 pc=0x718366
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005047e8 sp=0xc0005047e0 pc=0x4b18e1
created by net/http.(*connReader).startBackgroundRead
/usr/local/go/src/net/http/server.go:670 +0xca

rax 0x0
rbx 0x3d6
rcx 0x7fc589bcad3c
rdx 0x6
rdi 0x3d2
rsi 0x3d6
rbp 0x7fc5617a86c0
rsp 0x7fc5617a6120
r8 0x0
r9 0x64
r10 0x8
r11 0x246
r12 0x6
r13 0x0
r14 0x4000000
r15 0x0
rip 0x7fc589bcad3c
rflags 0x246
cs 0x33
fs 0x0
gs 0x0

Originally created by @Cybervet on GitHub (Dec 23, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1688 Originally assigned to: @BruceMacD on GitHub. After pulling the new 0.1.17 and changing the source for AVX=off , compilation went fine with no errors. ollama serve starts fine until I am trying to run a model and then it crushes with the following msg (LONG). Any help aw to what is happening will be appreciated. CUDA error 2 at /root/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9133: out of memory current device: 0 GGML_ASSERT: /root/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9133: !"CUDA error" SIGABRT: abort PC=0x7fc589bcad3c m=5 sigcode=18446744073709551610 signal arrived during cgo execution goroutine 12 [syscall]: runtime.cgocall(0xa12b00, 0xc0002ca7a8) /usr/local/go/src/runtime/cgocall.go:157 +0x5c fp=0xc0002ca780 sp=0xc0002ca748 pc=0x44df3c github.com/jmorganca/ollama/llm._Cfunc_dynamic_shim_llama_server_init({0x7fc550001070, 0x7fc518eb7250, 0x7fc518eb07e0, 0x7fc518eb1160, 0x7fc518eb9c20, 0x7fc518eb4cd0, 0x7fc518eb1ce0, 0x7fc518eb0850, 0x7fc518eb9d10, 0x7fc518eba230, ...}, ...) _cgo_gotypes.go:291 +0x45 fp=0xc0002ca7a8 sp=0xc0002ca780 pc=0x80f2e5 github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init.func1(0x47d93f?, 0x80?, 0x80?) /root/ollama/llm/shim_ext_server.go:43 +0xec fp=0xc0002ca898 sp=0xc0002ca7a8 pc=0x814a8c github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init(0xc86fe1?, 0x4000?, 0x0?) /root/ollama/llm/shim_ext_server.go:43 +0x19 fp=0xc0002ca8c0 sp=0xc0002ca898 pc=0x814959 github.com/jmorganca/ollama/llm.newExtServer({0x2313d270, 0xc0004363c0}, {_, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...) /root/ollama/llm/ext_server.go:192 +0x6ff fp=0xc0002cab00 sp=0xc0002ca8c0 pc=0x8113bf github.com/jmorganca/ollama/llm.newDynamicShimExtServer({0xc000512060, 0x27}, {_, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...) /root/ollama/llm/shim_ext_server.go:95 +0x4af fp=0xc0002cacd8 sp=0xc0002cab00 pc=0x815e4f github.com/jmorganca/ollama/llm.newLlmServer({0xc74877, 0xb}, {_, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...) /root/ollama/llm/llm.go:94 +0x18b fp=0xc0002cae20 sp=0xc0002cacd8 pc=0x80e8cb github.com/jmorganca/ollama/llm.New({0x0?, 0xc000110d20?}, {_, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...) /root/ollama/llm/llm.go:84 +0x2a6 fp=0xc0002cb010 sp=0xc0002cae20 pc=0x80e4e6 github.com/jmorganca/ollama/server.load(0x0?, {0xc00052a620?, 0x23133f60?}, 0xc00005a120?, 0xc343c0?) /root/ollama/server/routes.go:99 +0x4ce fp=0xc0002cb290 sp=0xc0002cb010 pc=0x9f5dee github.com/jmorganca/ollama/server.GenerateHandler(0xc0004ba000) /root/ollama/server/routes.go:172 +0x377 fp=0xc0002cb6d0 sp=0xc0002cb290 pc=0x9f6437 github.com/gin-gonic/gin.(*Context).Next(...) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/jmorganca/ollama/server.(*Server).GenerateRoutes.func1(0xc0004ba000) /root/ollama/server/routes.go:834 +0x6b fp=0xc0002cb708 sp=0xc0002cb6d0 pc=0x9ff44b github.com/gin-gonic/gin.(*Context).Next(...) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc0004ba000) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +0x82 fp=0xc0002cb758 sp=0xc0002cb708 pc=0x9d9fa2 github.com/gin-gonic/gin.(*Context).Next(...) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc0004ba000) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0xe7 fp=0xc0002cb908 sp=0xc0002cb758 pc=0x9d90c7 github.com/gin-gonic/gin.(*Context).Next(...) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc000137ba0, 0xc0004ba000) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x66b fp=0xc0002cba90 sp=0xc0002cb908 pc=0x9d812b github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc000137ba0, {0x23138910?, 0xc0003fc0e0}, 0xc00053a300) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1dd fp=0xc0002cbad0 sp=0xc0002cba90 pc=0x9d78dd net/http.serverHandler.ServeHTTP({0x231365c0?}, {0x23138910, 0xc0003fc0e0}, 0xc00053a300) /usr/local/go/src/net/http/server.go:2936 +0x316 fp=0xc0002cbb80 sp=0xc0002cbad0 pc=0x7231d6 net/http.(*conn).serve(0xc00049eb40, {0x23138f88, 0xc00050e1e0}) /usr/local/go/src/net/http/server.go:1995 +0x612 fp=0xc0002cbfb8 sp=0xc0002cbb80 pc=0x71e6f2 net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc0002cbfe0 sp=0xc0002cbfb8 pc=0x723b2e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cbfe8 sp=0xc0002cbfe0 pc=0x4b18e1 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:3089 +0x5ed goroutine 1 [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0001478c8 sp=0xc0001478a8 pc=0x481f76 runtime.netpollblock(0x7fc5606a7968?, 0x44d5cf?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000147900 sp=0xc0001478c8 pc=0x47a977 internal/poll.runtime_pollWait(0x7fc56072e018, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000147920 sp=0xc000147900 pc=0x4abd29 internal/poll.(*pollDesc).wait(0xc00040df80?, 0x4?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000147948 sp=0xc000147920 pc=0x5389f2 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc00040df80) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0001479f0 sp=0xc000147948 pc=0x53e2fd net.(*netFD).accept(0xc00040df80) /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000147aa8 sp=0xc0001479f0 pc=0x5b3d95 net.(*TCPListener).accept(0xc000012f18) /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000147ad0 sp=0xc000147aa8 pc=0x5ca125 net.(*TCPListener).Accept(0xc000012f18) /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000147b00 sp=0xc000147ad0 pc=0x5c921d net/http.(*onceCloseListener).Accept(0xc00049eb40?) <autogenerated>:1 +0x2a fp=0xc000147b18 sp=0xc000147b00 pc=0x7484aa net/http.(*Server).Serve(0xc00035eff0, {0x23138700, 0xc000012f18}) /usr/local/go/src/net/http/server.go:3059 +0x385 fp=0xc000147c48 sp=0xc000147b18 pc=0x723745 github.com/jmorganca/ollama/server.Serve({0x23138700, 0xc000012f18}) /root/ollama/server/routes.go:914 +0x2d0 fp=0xc000147cf8 sp=0xc000147c48 pc=0x9ff770 github.com/jmorganca/ollama/cmd.RunServer(0xc000415800?, {0x2366aac8?, 0x0?, 0x0?}) /root/ollama/cmd/cmd.go:1077 +0x169 fp=0xc000147d70 sp=0xc000147cf8 pc=0xa0f769 github.com/spf13/cobra.(*Command).execute(0xc000415800, {0x2366aac8, 0x0, 0x0}) /root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 fp=0xc000147ea8 sp=0xc000147d70 pc=0x7bf6c2 github.com/spf13/cobra.(*Command).ExecuteC(0xc000414c00) /root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd fp=0xc000147f60 sp=0xc000147ea8 pc=0x7bff3d github.com/spf13/cobra.(*Command).Execute(...) /root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) /root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985 main.main() /root/ollama/main.go:11 +0x54 fp=0xc000147f80 sp=0xc000147f60 pc=0xa11c34 runtime.main() /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc000147fe0 sp=0xc000147f80 pc=0x481b47 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000147fe8 sp=0xc000147fe0 pc=0x4b18e1 goroutine 2 [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000044fb0 sp=0xc000044f90 pc=0x481f76 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:387 runtime.forcegchelper() /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000044fe0 sp=0xc000044fb0 pc=0x481db0 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000044fe8 sp=0xc000044fe0 pc=0x4b18e1 created by runtime.init.6 /usr/local/go/src/runtime/proc.go:293 +0x25 goroutine 3 [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000045780 sp=0xc000045760 pc=0x481f76 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:387 runtime.bgsweep(0x0?) /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000457c8 sp=0xc000045780 pc=0x46e29e runtime.gcenable.func1() /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000457e0 sp=0xc0000457c8 pc=0x463506 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000457e8 sp=0xc0000457e0 pc=0x4b18e1 created by runtime.gcenable /usr/local/go/src/runtime/mgc.go:178 +0x6b goroutine 4 [GC scavenge wait]: runtime.gopark(0x38f9409e3ef0?, 0x3b9f4ff3?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000045f70 sp=0xc000045f50 pc=0x481f76 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:387 runtime.(*scavengerState).park(0x23574640) /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000045fa0 sp=0xc000045f70 pc=0x46c173 runtime.bgscavenge(0x0?) /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000045fc8 sp=0xc000045fa0 pc=0x46c765 runtime.gcenable.func2() /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000045fe0 sp=0xc000045fc8 pc=0x4634a6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000045fe8 sp=0xc000045fe0 pc=0x4b18e1 created by runtime.gcenable /usr/local/go/src/runtime/mgc.go:179 +0xaa goroutine 5 [finalizer wait]: runtime.gopark(0x1a0?, 0x23575040?, 0x60?, 0x78?, 0xc000044770?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000044628 sp=0xc000044608 pc=0x481f76 runtime.runfinq() /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000447e0 sp=0xc000044628 pc=0x462547 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000447e8 sp=0xc0000447e0 pc=0x4b18e1 created by runtime.createfing /usr/local/go/src/runtime/mfinal.go:163 +0x45 goroutine 6 [select, locked to thread]: runtime.gopark(0xc0000467a8?, 0x2?, 0xf2?, 0x22?, 0xc0000467a4?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000046618 sp=0xc0000465f8 pc=0x481f76 runtime.selectgo(0xc0000467a8, 0xc0000467a0, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000046758 sp=0xc000046618 pc=0x49181e runtime.ensureSigM.func1() /usr/local/go/src/runtime/signal_unix.go:1000 +0x1af fp=0xc0000467e0 sp=0xc000046758 pc=0x4a96ef runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000467e8 sp=0xc0000467e0 pc=0x4b18e1 created by runtime.ensureSigM /usr/local/go/src/runtime/signal_unix.go:983 +0xbd goroutine 18 [syscall]: runtime.notetsleepg(0x0?, 0x0?) /usr/local/go/src/runtime/lock_futex.go:236 +0x34 fp=0xc0000407a0 sp=0xc000040768 pc=0x456134 os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc0000407c0 sp=0xc0000407a0 pc=0x4ae10f os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc0000407e0 sp=0xc0000407c0 pc=0x74b319 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000407e8 sp=0xc0000407e0 pc=0x4b18e1 created by os/signal.Notify.func1.1 /usr/local/go/src/os/signal/signal.go:151 +0x2a goroutine 34 [chan receive]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000508700 sp=0xc0005086e0 pc=0x481f76 runtime.chanrecv(0xc0001b3500, 0x0, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000508790 sp=0xc000508700 pc=0x450cfd runtime.chanrecv1(0x0?, 0x0?) /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0005087b8 sp=0xc000508790 pc=0x4507f8 github.com/jmorganca/ollama/server.Serve.func1() /root/ollama/server/routes.go:896 +0x28 fp=0xc0005087e0 sp=0xc0005087b8 pc=0x9ff828 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005087e8 sp=0xc0005087e0 pc=0x4b18e1 created by github.com/jmorganca/ollama/server.Serve /root/ollama/server/routes.go:895 +0x219 goroutine 35 [IO wait]: runtime.gopark(0x75?, 0xb?, 0x0?, 0x0?, 0xc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000141900 sp=0xc0001418e0 pc=0x481f76 runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000141938 sp=0xc000141900 pc=0x47a977 internal/poll.runtime_pollWait(0x7fc56072df28, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000141958 sp=0xc000141938 pc=0x4abd29 internal/poll.(*pollDesc).wait(0xc00050c080?, 0xc000536000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000141980 sp=0xc000141958 pc=0x5389f2 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00050c080, {0xc000536000, 0x1000, 0x1000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000141a18 sp=0xc000141980 pc=0x539dd9 net.(*netFD).Read(0xc00050c080, {0xc000536000?, 0x538ee6?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000141a60 sp=0xc000141a18 pc=0x5b1c09 net.(*conn).Read(0xc00051c020, {0xc000536000?, 0x0?, 0xc00050e2d8?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000141aa8 sp=0xc000141a60 pc=0x5c1045 net.(*TCPConn).Read(0xc00050e2d0?, {0xc000536000?, 0x0?, 0xc000141ac8?}) <autogenerated>:1 +0x29 fp=0xc000141ad8 sp=0xc000141aa8 pc=0x5d3c09 net/http.(*connReader).Read(0xc00050e2d0, {0xc000536000, 0x1000, 0x1000}) /usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc000141b28 sp=0xc000141ad8 pc=0x7188b1 bufio.(*Reader).fill(0xc00051e060) /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000141b60 sp=0xc000141b28 pc=0x6a5c5f bufio.(*Reader).Peek(0xc00051e060, 0x4) /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc000141b80 sp=0xc000141b60 pc=0x6a5dbd net/http.(*conn).serve(0xc000532000, {0x23138f88, 0xc00050e1e0}) /usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc000141fb8 sp=0xc000141b80 pc=0x71e85c net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc000141fe0 sp=0xc000141fb8 pc=0x723b2e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000141fe8 sp=0xc000141fe0 pc=0x4b18e1 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:3089 +0x5ed goroutine 36 [GC worker (idle)]: runtime.gopark(0x38f93b6df672?, 0x1?, 0x4e?, 0x14?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000508f50 sp=0xc000508f30 pc=0x481f76 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000508fe0 sp=0xc000508f50 pc=0x465271 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000508fe8 sp=0xc000508fe0 pc=0x4b18e1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25 goroutine 7 [GC worker (idle)]: runtime.gopark(0x38f9390eb146?, 0x3?, 0x1?, 0xb2?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000046f50 sp=0xc000046f30 pc=0x481f76 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000046fe0 sp=0xc000046f50 pc=0x465271 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x4b18e1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25 goroutine 8 [GC worker (idle)]: runtime.gopark(0x38f93b5b21f1?, 0x3?, 0xf5?, 0x68?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000047750 sp=0xc000047730 pc=0x481f76 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000477e0 sp=0xc000047750 pc=0x465271 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x4b18e1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25 goroutine 9 [GC worker (idle)]: runtime.gopark(0x38f93b6def59?, 0x3?, 0x66?, 0x58?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000047f50 sp=0xc000047f30 pc=0x481f76 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000047fe0 sp=0xc000047f50 pc=0x465271 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x4b18e1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25 goroutine 51 [IO wait]: runtime.gopark(0x7?, 0xb?, 0x0?, 0x0?, 0xe?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0004d5900 sp=0xc0004d58e0 pc=0x481f76 runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0004d5938 sp=0xc0004d5900 pc=0x47a977 internal/poll.runtime_pollWait(0x7fc56072dd48, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0004d5958 sp=0xc0004d5938 pc=0x4abd29 internal/poll.(*pollDesc).wait(0xc00013e000?, 0xc0004ce000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0004d5980 sp=0xc0004d5958 pc=0x5389f2 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00013e000, {0xc0004ce000, 0x1000, 0x1000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0004d5a18 sp=0xc0004d5980 pc=0x539dd9 net.(*netFD).Read(0xc00013e000, {0xc0004ce000?, 0x538ee6?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0004d5a60 sp=0xc0004d5a18 pc=0x5b1c09 net.(*conn).Read(0xc00051c000, {0xc0004ce000?, 0x0?, 0xc00050e0c8?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0004d5aa8 sp=0xc0004d5a60 pc=0x5c1045 net.(*TCPConn).Read(0xc00050e0c0?, {0xc0004ce000?, 0x0?, 0xc000593ac8?}) <autogenerated>:1 +0x29 fp=0xc0004d5ad8 sp=0xc0004d5aa8 pc=0x5d3c09 net/http.(*connReader).Read(0xc00050e0c0, {0xc0004ce000, 0x1000, 0x1000}) /usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc0004d5b28 sp=0xc0004d5ad8 pc=0x7188b1 bufio.(*Reader).fill(0xc000096060) /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc0004d5b60 sp=0xc0004d5b28 pc=0x6a5c5f bufio.(*Reader).Peek(0xc000096060, 0x4) /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc0004d5b80 sp=0xc0004d5b60 pc=0x6a5dbd net/http.(*conn).serve(0xc000532090, {0x23138f88, 0xc00050e1e0}) /usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc0004d5fb8 sp=0xc0004d5b80 pc=0x71e85c net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc0004d5fe0 sp=0xc0004d5fb8 pc=0x723b2e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0004d5fe8 sp=0xc0004d5fe0 pc=0x4b18e1 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:3089 +0x5ed goroutine 10 [IO wait]: runtime.gopark(0x7?, 0xb?, 0x0?, 0x0?, 0xd?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000145900 sp=0xc0001458e0 pc=0x481f76 runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000145938 sp=0xc000145900 pc=0x47a977 internal/poll.runtime_pollWait(0x7fc56072de38, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000145958 sp=0xc000145938 pc=0x4abd29 internal/poll.(*pollDesc).wait(0xc00050c100?, 0xc000588000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000145980 sp=0xc000145958 pc=0x5389f2 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00050c100, {0xc000588000, 0x1000, 0x1000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000145a18 sp=0xc000145980 pc=0x539dd9 net.(*netFD).Read(0xc00050c100, {0xc000588000?, 0x538ee6?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000145a60 sp=0xc000145a18 pc=0x5b1c09 net.(*conn).Read(0xc0004a8008, {0xc000588000?, 0x0?, 0xc0004a21e8?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000145aa8 sp=0xc000145a60 pc=0x5c1045 net.(*TCPConn).Read(0xc0004a21e0?, {0xc000588000?, 0x0?, 0xc000597ac8?}) <autogenerated>:1 +0x29 fp=0xc000145ad8 sp=0xc000145aa8 pc=0x5d3c09 net/http.(*connReader).Read(0xc0004a21e0, {0xc000588000, 0x1000, 0x1000}) /usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc000145b28 sp=0xc000145ad8 pc=0x7188b1 bufio.(*Reader).fill(0xc00048e120) /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000145b60 sp=0xc000145b28 pc=0x6a5c5f bufio.(*Reader).Peek(0xc00048e120, 0x4) /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc000145b80 sp=0xc000145b60 pc=0x6a5dbd net/http.(*conn).serve(0xc00049e120, {0x23138f88, 0xc00050e1e0}) /usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc000145fb8 sp=0xc000145b80 pc=0x71e85c net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc000145fe0 sp=0xc000145fb8 pc=0x723b2e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000145fe8 sp=0xc000145fe0 pc=0x4b18e1 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:3089 +0x5ed goroutine 53 [IO wait]: runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0xf?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0005045a0 sp=0xc000504580 pc=0x481f76 runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0005045d8 sp=0xc0005045a0 pc=0x47a977 internal/poll.runtime_pollWait(0x7fc56072dc58, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0005045f8 sp=0xc0005045d8 pc=0x4abd29 internal/poll.(*pollDesc).wait(0xc00050c300?, 0xc00050f181?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000504620 sp=0xc0005045f8 pc=0x5389f2 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00050c300, {0xc00050f181, 0x1, 0x1}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0005046b8 sp=0xc000504620 pc=0x539dd9 net.(*netFD).Read(0xc00050c300, {0xc00050f181?, 0xbdaba0?, 0xc000504748?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000504700 sp=0xc0005046b8 pc=0x5b1c09 net.(*conn).Read(0xc0004a8140, {0xc00050f181?, 0xc0005380d0?, 0xc0005165a0?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000504748 sp=0xc000504700 pc=0x5c1045 net.(*TCPConn).Read(0xc00050e0c0?, {0xc00050f181?, 0xc0005165a0?, 0x0?}) <autogenerated>:1 +0x29 fp=0xc000504778 sp=0xc000504748 pc=0x5d3c09 net/http.(*connReader).backgroundRead(0xc00050f170) /usr/local/go/src/net/http/server.go:674 +0x3f fp=0xc0005047c8 sp=0xc000504778 pc=0x71843f net/http.(*connReader).startBackgroundRead.func2() /usr/local/go/src/net/http/server.go:670 +0x26 fp=0xc0005047e0 sp=0xc0005047c8 pc=0x718366 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005047e8 sp=0xc0005047e0 pc=0x4b18e1 created by net/http.(*connReader).startBackgroundRead /usr/local/go/src/net/http/server.go:670 +0xca rax 0x0 rbx 0x3ec rcx 0x7fc589bcad3c rdx 0x6 rdi 0x3ec rsi 0x3ec rbp 0x7fc5617a86c0 rsp 0x7fc5617a6120 r8 0x7fffb3d98faa r9 0x7fc5617a60f0 r10 0x8 r11 0x246 r12 0x6 r13 0x0 r14 0x4000000 r15 0x0 rip 0x7fc589bcad3c rflags 0x246 cs 0x33 fs 0x0 gs 0x0 SIGABRT: abort PC=0x7fc589bcad3c m=5 sigcode=18446744073709551610 signal arrived during cgo execution goroutine 12 [syscall]: runtime.cgocall(0xa12b00, 0xc0002ca7a8) /usr/local/go/src/runtime/cgocall.go:157 +0x5c fp=0xc0002ca780 sp=0xc0002ca748 pc=0x44df3c github.com/jmorganca/ollama/llm._Cfunc_dynamic_shim_llama_server_init({0x7fc550001070, 0x7fc518eb7250, 0x7fc518eb07e0, 0x7fc518eb1160, 0x7fc518eb9c20, 0x7fc518eb4cd0, 0x7fc518eb1ce0, 0x7fc518eb0850, 0x7fc518eb9d10, 0x7fc518eba230, ...}, ...) _cgo_gotypes.go:291 +0x45 fp=0xc0002ca7a8 sp=0xc0002ca780 pc=0x80f2e5 github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init.func1(0x47d93f?, 0x80?, 0x80?) /root/ollama/llm/shim_ext_server.go:43 +0xec fp=0xc0002ca898 sp=0xc0002ca7a8 pc=0x814a8c github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init(0xc86fe1?, 0x4000?, 0x0?) /root/ollama/llm/shim_ext_server.go:43 +0x19 fp=0xc0002ca8c0 sp=0xc0002ca898 pc=0x814959 github.com/jmorganca/ollama/llm.newExtServer({0x2313d270, 0xc0004363c0}, {_, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...) /root/ollama/llm/ext_server.go:192 +0x6ff fp=0xc0002cab00 sp=0xc0002ca8c0 pc=0x8113bf github.com/jmorganca/ollama/llm.newDynamicShimExtServer({0xc000512060, 0x27}, {_, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...) /root/ollama/llm/shim_ext_server.go:95 +0x4af fp=0xc0002cacd8 sp=0xc0002cab00 pc=0x815e4f github.com/jmorganca/ollama/llm.newLlmServer({0xc74877, 0xb}, {_, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...) /root/ollama/llm/llm.go:94 +0x18b fp=0xc0002cae20 sp=0xc0002cacd8 pc=0x80e8cb github.com/jmorganca/ollama/llm.New({0x0?, 0xc000110d20?}, {_, _}, {_, _, _}, {0x0, 0x0, 0x0}, ...) /root/ollama/llm/llm.go:84 +0x2a6 fp=0xc0002cb010 sp=0xc0002cae20 pc=0x80e4e6 github.com/jmorganca/ollama/server.load(0x0?, {0xc00052a620?, 0x23133f60?}, 0xc00005a120?, 0xc343c0?) /root/ollama/server/routes.go:99 +0x4ce fp=0xc0002cb290 sp=0xc0002cb010 pc=0x9f5dee github.com/jmorganca/ollama/server.GenerateHandler(0xc0004ba000) /root/ollama/server/routes.go:172 +0x377 fp=0xc0002cb6d0 sp=0xc0002cb290 pc=0x9f6437 github.com/gin-gonic/gin.(*Context).Next(...) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/jmorganca/ollama/server.(*Server).GenerateRoutes.func1(0xc0004ba000) /root/ollama/server/routes.go:834 +0x6b fp=0xc0002cb708 sp=0xc0002cb6d0 pc=0x9ff44b github.com/gin-gonic/gin.(*Context).Next(...) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc0004ba000) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +0x82 fp=0xc0002cb758 sp=0xc0002cb708 pc=0x9d9fa2 github.com/gin-gonic/gin.(*Context).Next(...) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc0004ba000) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0xe7 fp=0xc0002cb908 sp=0xc0002cb758 pc=0x9d90c7 github.com/gin-gonic/gin.(*Context).Next(...) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc000137ba0, 0xc0004ba000) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x66b fp=0xc0002cba90 sp=0xc0002cb908 pc=0x9d812b github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc000137ba0, {0x23138910?, 0xc0003fc0e0}, 0xc00053a300) /root/root/ollama/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1dd fp=0xc0002cbad0 sp=0xc0002cba90 pc=0x9d78dd net/http.serverHandler.ServeHTTP({0x231365c0?}, {0x23138910, 0xc0003fc0e0}, 0xc00053a300) /usr/local/go/src/net/http/server.go:2936 +0x316 fp=0xc0002cbb80 sp=0xc0002cbad0 pc=0x7231d6 net/http.(*conn).serve(0xc00049eb40, {0x23138f88, 0xc00050e1e0}) /usr/local/go/src/net/http/server.go:1995 +0x612 fp=0xc0002cbfb8 sp=0xc0002cbb80 pc=0x71e6f2 net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc0002cbfe0 sp=0xc0002cbfb8 pc=0x723b2e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0002cbfe8 sp=0xc0002cbfe0 pc=0x4b18e1 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:3089 +0x5ed goroutine 1 [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0001478c8 sp=0xc0001478a8 pc=0x481f76 runtime.netpollblock(0x7fc5606a7968?, 0x44d5cf?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000147900 sp=0xc0001478c8 pc=0x47a977 internal/poll.runtime_pollWait(0x7fc56072e018, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000147920 sp=0xc000147900 pc=0x4abd29 internal/poll.(*pollDesc).wait(0xc00040df80?, 0x4?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000147948 sp=0xc000147920 pc=0x5389f2 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc00040df80) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0001479f0 sp=0xc000147948 pc=0x53e2fd net.(*netFD).accept(0xc00040df80) /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000147aa8 sp=0xc0001479f0 pc=0x5b3d95 net.(*TCPListener).accept(0xc000012f18) /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000147ad0 sp=0xc000147aa8 pc=0x5ca125 net.(*TCPListener).Accept(0xc000012f18) /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000147b00 sp=0xc000147ad0 pc=0x5c921d net/http.(*onceCloseListener).Accept(0xc00049eb40?) <autogenerated>:1 +0x2a fp=0xc000147b18 sp=0xc000147b00 pc=0x7484aa net/http.(*Server).Serve(0xc00035eff0, {0x23138700, 0xc000012f18}) /usr/local/go/src/net/http/server.go:3059 +0x385 fp=0xc000147c48 sp=0xc000147b18 pc=0x723745 github.com/jmorganca/ollama/server.Serve({0x23138700, 0xc000012f18}) /root/ollama/server/routes.go:914 +0x2d0 fp=0xc000147cf8 sp=0xc000147c48 pc=0x9ff770 github.com/jmorganca/ollama/cmd.RunServer(0xc000415800?, {0x2366aac8?, 0x0?, 0x0?}) /root/ollama/cmd/cmd.go:1077 +0x169 fp=0xc000147d70 sp=0xc000147cf8 pc=0xa0f769 github.com/spf13/cobra.(*Command).execute(0xc000415800, {0x2366aac8, 0x0, 0x0}) /root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 fp=0xc000147ea8 sp=0xc000147d70 pc=0x7bf6c2 github.com/spf13/cobra.(*Command).ExecuteC(0xc000414c00) /root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd fp=0xc000147f60 sp=0xc000147ea8 pc=0x7bff3d github.com/spf13/cobra.(*Command).Execute(...) /root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) /root/root/ollama/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985 main.main() /root/ollama/main.go:11 +0x54 fp=0xc000147f80 sp=0xc000147f60 pc=0xa11c34 runtime.main() /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc000147fe0 sp=0xc000147f80 pc=0x481b47 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000147fe8 sp=0xc000147fe0 pc=0x4b18e1 goroutine 2 [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000044fb0 sp=0xc000044f90 pc=0x481f76 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:387 runtime.forcegchelper() /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000044fe0 sp=0xc000044fb0 pc=0x481db0 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000044fe8 sp=0xc000044fe0 pc=0x4b18e1 created by runtime.init.6 /usr/local/go/src/runtime/proc.go:293 +0x25 goroutine 3 [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000045780 sp=0xc000045760 pc=0x481f76 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:387 runtime.bgsweep(0x0?) /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000457c8 sp=0xc000045780 pc=0x46e29e runtime.gcenable.func1() /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000457e0 sp=0xc0000457c8 pc=0x463506 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000457e8 sp=0xc0000457e0 pc=0x4b18e1 created by runtime.gcenable /usr/local/go/src/runtime/mgc.go:178 +0x6b goroutine 4 [GC scavenge wait]: runtime.gopark(0x38f9409e3ef0?, 0x3b9f4ff3?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000045f70 sp=0xc000045f50 pc=0x481f76 runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:387 runtime.(*scavengerState).park(0x23574640) /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000045fa0 sp=0xc000045f70 pc=0x46c173 runtime.bgscavenge(0x0?) /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000045fc8 sp=0xc000045fa0 pc=0x46c765 runtime.gcenable.func2() /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000045fe0 sp=0xc000045fc8 pc=0x4634a6 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000045fe8 sp=0xc000045fe0 pc=0x4b18e1 created by runtime.gcenable /usr/local/go/src/runtime/mgc.go:179 +0xaa goroutine 5 [finalizer wait]: runtime.gopark(0x1a0?, 0x23575040?, 0x60?, 0x78?, 0xc000044770?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000044628 sp=0xc000044608 pc=0x481f76 runtime.runfinq() /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000447e0 sp=0xc000044628 pc=0x462547 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000447e8 sp=0xc0000447e0 pc=0x4b18e1 created by runtime.createfing /usr/local/go/src/runtime/mfinal.go:163 +0x45 goroutine 6 [select, locked to thread]: runtime.gopark(0xc0000467a8?, 0x2?, 0xf2?, 0x22?, 0xc0000467a4?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000046618 sp=0xc0000465f8 pc=0x481f76 runtime.selectgo(0xc0000467a8, 0xc0000467a0, 0x0?, 0x0, 0x0?, 0x1) /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000046758 sp=0xc000046618 pc=0x49181e runtime.ensureSigM.func1() /usr/local/go/src/runtime/signal_unix.go:1000 +0x1af fp=0xc0000467e0 sp=0xc000046758 pc=0x4a96ef runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000467e8 sp=0xc0000467e0 pc=0x4b18e1 created by runtime.ensureSigM /usr/local/go/src/runtime/signal_unix.go:983 +0xbd goroutine 18 [syscall]: runtime.notetsleepg(0x0?, 0x0?) /usr/local/go/src/runtime/lock_futex.go:236 +0x34 fp=0xc0000407a0 sp=0xc000040768 pc=0x456134 os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc0000407c0 sp=0xc0000407a0 pc=0x4ae10f os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc0000407e0 sp=0xc0000407c0 pc=0x74b319 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000407e8 sp=0xc0000407e0 pc=0x4b18e1 created by os/signal.Notify.func1.1 /usr/local/go/src/os/signal/signal.go:151 +0x2a goroutine 34 [chan receive]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000508700 sp=0xc0005086e0 pc=0x481f76 runtime.chanrecv(0xc0001b3500, 0x0, 0x1) /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000508790 sp=0xc000508700 pc=0x450cfd runtime.chanrecv1(0x0?, 0x0?) /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0005087b8 sp=0xc000508790 pc=0x4507f8 github.com/jmorganca/ollama/server.Serve.func1() /root/ollama/server/routes.go:896 +0x28 fp=0xc0005087e0 sp=0xc0005087b8 pc=0x9ff828 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005087e8 sp=0xc0005087e0 pc=0x4b18e1 created by github.com/jmorganca/ollama/server.Serve /root/ollama/server/routes.go:895 +0x219 goroutine 35 [IO wait]: runtime.gopark(0x75?, 0xb?, 0x0?, 0x0?, 0xc?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000141900 sp=0xc0001418e0 pc=0x481f76 runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000141938 sp=0xc000141900 pc=0x47a977 internal/poll.runtime_pollWait(0x7fc56072df28, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000141958 sp=0xc000141938 pc=0x4abd29 internal/poll.(*pollDesc).wait(0xc00050c080?, 0xc000536000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000141980 sp=0xc000141958 pc=0x5389f2 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00050c080, {0xc000536000, 0x1000, 0x1000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000141a18 sp=0xc000141980 pc=0x539dd9 net.(*netFD).Read(0xc00050c080, {0xc000536000?, 0x538ee6?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000141a60 sp=0xc000141a18 pc=0x5b1c09 net.(*conn).Read(0xc00051c020, {0xc000536000?, 0x0?, 0xc00050e2d8?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000141aa8 sp=0xc000141a60 pc=0x5c1045 net.(*TCPConn).Read(0xc00050e2d0?, {0xc000536000?, 0x0?, 0xc000141ac8?}) <autogenerated>:1 +0x29 fp=0xc000141ad8 sp=0xc000141aa8 pc=0x5d3c09 net/http.(*connReader).Read(0xc00050e2d0, {0xc000536000, 0x1000, 0x1000}) /usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc000141b28 sp=0xc000141ad8 pc=0x7188b1 bufio.(*Reader).fill(0xc00051e060) /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000141b60 sp=0xc000141b28 pc=0x6a5c5f bufio.(*Reader).Peek(0xc00051e060, 0x4) /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc000141b80 sp=0xc000141b60 pc=0x6a5dbd net/http.(*conn).serve(0xc000532000, {0x23138f88, 0xc00050e1e0}) /usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc000141fb8 sp=0xc000141b80 pc=0x71e85c net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc000141fe0 sp=0xc000141fb8 pc=0x723b2e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000141fe8 sp=0xc000141fe0 pc=0x4b18e1 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:3089 +0x5ed goroutine 36 [GC worker (idle)]: runtime.gopark(0x38f93b6df672?, 0x1?, 0x4e?, 0x14?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000508f50 sp=0xc000508f30 pc=0x481f76 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000508fe0 sp=0xc000508f50 pc=0x465271 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000508fe8 sp=0xc000508fe0 pc=0x4b18e1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25 goroutine 7 [GC worker (idle)]: runtime.gopark(0x38f9390eb146?, 0x3?, 0x1?, 0xb2?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000046f50 sp=0xc000046f30 pc=0x481f76 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000046fe0 sp=0xc000046f50 pc=0x465271 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x4b18e1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25 goroutine 8 [GC worker (idle)]: runtime.gopark(0x38f93b5b21f1?, 0x3?, 0xf5?, 0x68?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000047750 sp=0xc000047730 pc=0x481f76 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000477e0 sp=0xc000047750 pc=0x465271 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x4b18e1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25 goroutine 9 [GC worker (idle)]: runtime.gopark(0x38f93b6def59?, 0x3?, 0x66?, 0x58?, 0x0?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000047f50 sp=0xc000047f30 pc=0x481f76 runtime.gcBgMarkWorker() /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000047fe0 sp=0xc000047f50 pc=0x465271 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x4b18e1 created by runtime.gcBgMarkStartWorkers /usr/local/go/src/runtime/mgc.go:1199 +0x25 goroutine 51 [IO wait]: runtime.gopark(0x7?, 0xb?, 0x0?, 0x0?, 0xe?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0004d5900 sp=0xc0004d58e0 pc=0x481f76 runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0004d5938 sp=0xc0004d5900 pc=0x47a977 internal/poll.runtime_pollWait(0x7fc56072dd48, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0004d5958 sp=0xc0004d5938 pc=0x4abd29 internal/poll.(*pollDesc).wait(0xc00013e000?, 0xc0004ce000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0004d5980 sp=0xc0004d5958 pc=0x5389f2 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00013e000, {0xc0004ce000, 0x1000, 0x1000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0004d5a18 sp=0xc0004d5980 pc=0x539dd9 net.(*netFD).Read(0xc00013e000, {0xc0004ce000?, 0x538ee6?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0004d5a60 sp=0xc0004d5a18 pc=0x5b1c09 net.(*conn).Read(0xc00051c000, {0xc0004ce000?, 0x0?, 0xc00050e0c8?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0004d5aa8 sp=0xc0004d5a60 pc=0x5c1045 net.(*TCPConn).Read(0xc00050e0c0?, {0xc0004ce000?, 0x0?, 0xc000593ac8?}) <autogenerated>:1 +0x29 fp=0xc0004d5ad8 sp=0xc0004d5aa8 pc=0x5d3c09 net/http.(*connReader).Read(0xc00050e0c0, {0xc0004ce000, 0x1000, 0x1000}) /usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc0004d5b28 sp=0xc0004d5ad8 pc=0x7188b1 bufio.(*Reader).fill(0xc000096060) /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc0004d5b60 sp=0xc0004d5b28 pc=0x6a5c5f bufio.(*Reader).Peek(0xc000096060, 0x4) /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc0004d5b80 sp=0xc0004d5b60 pc=0x6a5dbd net/http.(*conn).serve(0xc000532090, {0x23138f88, 0xc00050e1e0}) /usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc0004d5fb8 sp=0xc0004d5b80 pc=0x71e85c net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc0004d5fe0 sp=0xc0004d5fb8 pc=0x723b2e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0004d5fe8 sp=0xc0004d5fe0 pc=0x4b18e1 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:3089 +0x5ed goroutine 10 [IO wait]: runtime.gopark(0x7?, 0xb?, 0x0?, 0x0?, 0xd?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000145900 sp=0xc0001458e0 pc=0x481f76 runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000145938 sp=0xc000145900 pc=0x47a977 internal/poll.runtime_pollWait(0x7fc56072de38, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000145958 sp=0xc000145938 pc=0x4abd29 internal/poll.(*pollDesc).wait(0xc00050c100?, 0xc000588000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000145980 sp=0xc000145958 pc=0x5389f2 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00050c100, {0xc000588000, 0x1000, 0x1000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000145a18 sp=0xc000145980 pc=0x539dd9 net.(*netFD).Read(0xc00050c100, {0xc000588000?, 0x538ee6?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000145a60 sp=0xc000145a18 pc=0x5b1c09 net.(*conn).Read(0xc0004a8008, {0xc000588000?, 0x0?, 0xc0004a21e8?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000145aa8 sp=0xc000145a60 pc=0x5c1045 net.(*TCPConn).Read(0xc0004a21e0?, {0xc000588000?, 0x0?, 0xc000597ac8?}) <autogenerated>:1 +0x29 fp=0xc000145ad8 sp=0xc000145aa8 pc=0x5d3c09 net/http.(*connReader).Read(0xc0004a21e0, {0xc000588000, 0x1000, 0x1000}) /usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc000145b28 sp=0xc000145ad8 pc=0x7188b1 bufio.(*Reader).fill(0xc00048e120) /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000145b60 sp=0xc000145b28 pc=0x6a5c5f bufio.(*Reader).Peek(0xc00048e120, 0x4) /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc000145b80 sp=0xc000145b60 pc=0x6a5dbd net/http.(*conn).serve(0xc00049e120, {0x23138f88, 0xc00050e1e0}) /usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc000145fb8 sp=0xc000145b80 pc=0x71e85c net/http.(*Server).Serve.func3() /usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc000145fe0 sp=0xc000145fb8 pc=0x723b2e runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000145fe8 sp=0xc000145fe0 pc=0x4b18e1 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:3089 +0x5ed goroutine 53 [IO wait]: runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0xf?) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0005045a0 sp=0xc000504580 pc=0x481f76 runtime.netpollblock(0x4c1cc5?, 0x44d5cf?, 0x0?) /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0005045d8 sp=0xc0005045a0 pc=0x47a977 internal/poll.runtime_pollWait(0x7fc56072dc58, 0x72) /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0005045f8 sp=0xc0005045d8 pc=0x4abd29 internal/poll.(*pollDesc).wait(0xc00050c300?, 0xc00050f181?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000504620 sp=0xc0005045f8 pc=0x5389f2 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc00050c300, {0xc00050f181, 0x1, 0x1}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc0005046b8 sp=0xc000504620 pc=0x539dd9 net.(*netFD).Read(0xc00050c300, {0xc00050f181?, 0xbdaba0?, 0xc000504748?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000504700 sp=0xc0005046b8 pc=0x5b1c09 net.(*conn).Read(0xc0004a8140, {0xc00050f181?, 0xc0005380d0?, 0xc0005165a0?}) /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000504748 sp=0xc000504700 pc=0x5c1045 net.(*TCPConn).Read(0xc00050e0c0?, {0xc00050f181?, 0xc0005165a0?, 0x0?}) <autogenerated>:1 +0x29 fp=0xc000504778 sp=0xc000504748 pc=0x5d3c09 net/http.(*connReader).backgroundRead(0xc00050f170) /usr/local/go/src/net/http/server.go:674 +0x3f fp=0xc0005047c8 sp=0xc000504778 pc=0x71843f net/http.(*connReader).startBackgroundRead.func2() /usr/local/go/src/net/http/server.go:670 +0x26 fp=0xc0005047e0 sp=0xc0005047c8 pc=0x718366 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005047e8 sp=0xc0005047e0 pc=0x4b18e1 created by net/http.(*connReader).startBackgroundRead /usr/local/go/src/net/http/server.go:670 +0xca rax 0x0 rbx 0x3d6 rcx 0x7fc589bcad3c rdx 0x6 rdi 0x3d2 rsi 0x3d6 rbp 0x7fc5617a86c0 rsp 0x7fc5617a6120 r8 0x0 r9 0x64 r10 0x8 r11 0x246 r12 0x6 r13 0x0 r14 0x4000000 r15 0x0 rip 0x7fc589bcad3c rflags 0x246 cs 0x33 fs 0x0 gs 0x0
GiteaMirror added the bug label 2026-04-12 10:38:59 -05:00
Author
Owner

@Cybervet commented on GitHub (Dec 23, 2023):

Ok so far I found that in the older version when a cuda error was found then the ollama server was starting again in cpu only mode. In this version if the same error is found then it breaks out and stops while if it's started in cpu only mode it works fine.
Any ideas?

<!-- gh-comment-id:1868390529 --> @Cybervet commented on GitHub (Dec 23, 2023): Ok so far I found that in the older version when a cuda error was found then the ollama server was starting again in cpu only mode. In this version if the same error is found then it breaks out and stops while if it's started in cpu only mode it works fine. Any ideas?
Author
Owner

@BruceMacD commented on GitHub (Dec 24, 2023):

Are you running the same model when testing the newer version? The error seems to be an out of memory problem.

<!-- gh-comment-id:1868528236 --> @BruceMacD commented on GitHub (Dec 24, 2023): Are you running the same model when testing the newer version? The error seems to be an out of memory problem.
Author
Owner

@Cybervet commented on GitHub (Dec 24, 2023):

Yeap exact same models . I tried different models and even with small models like deepseek-coder it is doing the same thing.
This problem started when a change happened in /llm/llama.cpp/ and the gen_linux.sh was added.

<!-- gh-comment-id:1868529459 --> @Cybervet commented on GitHub (Dec 24, 2023): Yeap exact same models . I tried different models and even with small models like deepseek-coder it is doing the same thing. This problem started when a change happened in /llm/llama.cpp/ and the gen_linux.sh was added.
Author
Owner

@Cybervet commented on GitHub (Dec 24, 2023):

Ok here is the differences in the errors b/w the 2 versions, first is the one that works ok second is the newer.
llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: mem required = 257.78 MiB
llm_load_tensors: offloading 30 repeating layers to GPU
llm_load_tensors: offloaded 30/33 layers to GPU
llm_load_tensors: VRAM used: 1268.85 MiB
...........................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1

CUDA error 2 at /home/cybervet/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9132: out of memory
current device: 0
GGML_ASSERT: /home/cybervet/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9132: !"CUDA error"
2023/12/24 14:51:31 llama.go:451: 2 at /home/cybervet/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9132: out of memory
current device: 0
GGML_ASSERT: /home/cybervet/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9132: !"CUDA error"
2023/12/24 14:51:31 llama.go:459: error starting llama runner: llama runner process has terminated
2023/12/24 14:51:31 llama.go:525: llama runner stopped successfully
2023/12/24 14:51:31 llama.go:436: starting llama runner
2023/12/24 14:51:31 llama.go:494: waiting for llama runner to start responding
{"timestamp":1703429491,"level":"WARNING","function":"server_params_parse","line":2160,"message":"Not compiled with GPU offload support, --n-gpu-layers option will be ignored. See main README.md for information on enabling GPU BLAS support","n_gpu_layers":-1}

llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: mem required = 257.78 MiB
llm_load_tensors: offloading 30 repeating layers to GPU
llm_load_tensors: offloaded 30/33 layers to GPU
llm_load_tensors: VRAM used: 1268.85 MiB
...........................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: VRAM kv self = 600.00 MB
llama_new_context_with_model: KV self size = 640.00 MiB, K (f16): 320.00 MiB, V (f16): 320.00 MiB
llama_build_graph: non-view tensors processed: 774/774
llama_new_context_with_model: compute buffer total size = 165.19 MiB
llama_new_context_with_model: VRAM scratch buffer: 162.00 MiB
llama_new_context_with_model: total VRAM used: 2030.85 MiB (model: 1268.85 MiB, context: 762.00 MiB)

CUDA error 2 at /root/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9149: out of memory
current device: 0
GGML_ASSERT: /root/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9149: !"CUDA error"
SIGABRT: abort
PC=0x7f33f8ea9d3c m=5 sigcode=18446744073709551610
signal arrived during cgo execution

<!-- gh-comment-id:1868551767 --> @Cybervet commented on GitHub (Dec 24, 2023): Ok here is the differences in the errors b/w the 2 versions, first is the one that works ok second is the newer. llm_load_tensors: using CUDA for GPU acceleration llm_load_tensors: mem required = 257.78 MiB llm_load_tensors: offloading 30 repeating layers to GPU llm_load_tensors: offloaded 30/33 layers to GPU llm_load_tensors: VRAM used: 1268.85 MiB ........................................................................................... llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 CUDA error 2 at /home/cybervet/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9132: out of memory current device: 0 GGML_ASSERT: /home/cybervet/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9132: !"CUDA error" 2023/12/24 14:51:31 llama.go:451: 2 at /home/cybervet/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9132: out of memory current device: 0 GGML_ASSERT: /home/cybervet/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9132: !"CUDA error" 2023/12/24 14:51:31 llama.go:459: error starting llama runner: llama runner process has terminated 2023/12/24 14:51:31 llama.go:525: llama runner stopped successfully 2023/12/24 14:51:31 llama.go:436: starting llama runner 2023/12/24 14:51:31 llama.go:494: waiting for llama runner to start responding {"timestamp":1703429491,"level":"WARNING","function":"server_params_parse","line":2160,"message":"Not compiled with GPU offload support, --n-gpu-layers option will be ignored. See main README.md for information on enabling GPU BLAS support","n_gpu_layers":-1} llm_load_tensors: using CUDA for GPU acceleration llm_load_tensors: mem required = 257.78 MiB llm_load_tensors: offloading 30 repeating layers to GPU llm_load_tensors: offloaded 30/33 layers to GPU llm_load_tensors: VRAM used: 1268.85 MiB ........................................................................................... llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: VRAM kv self = 600.00 MB llama_new_context_with_model: KV self size = 640.00 MiB, K (f16): 320.00 MiB, V (f16): 320.00 MiB llama_build_graph: non-view tensors processed: 774/774 llama_new_context_with_model: compute buffer total size = 165.19 MiB llama_new_context_with_model: VRAM scratch buffer: 162.00 MiB llama_new_context_with_model: total VRAM used: 2030.85 MiB (model: 1268.85 MiB, context: 762.00 MiB) CUDA error 2 at /root/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9149: out of memory current device: 0 GGML_ASSERT: /root/ollama/llm/llama.cpp/gguf/ggml-cuda.cu:9149: !"CUDA error" SIGABRT: abort PC=0x7f33f8ea9d3c m=5 sigcode=18446744073709551610 signal arrived during cgo execution
Author
Owner

@BruceMacD commented on GitHub (Dec 29, 2023):

Thanks, looks like it is not falling back to CPU properly when out of GPU memory. I'll look into this.

<!-- gh-comment-id:1872101849 --> @BruceMacD commented on GitHub (Dec 29, 2023): Thanks, looks like it is not falling back to CPU properly when out of GPU memory. I'll look into this.
Author
Owner

@jmorganca commented on GitHub (Jan 9, 2024):

This should be fixed with #1850 and released with https://github.com/jmorganca/ollama/releases/tag/v0.1.19 which is on track to go out later today. Sorry you hit an error!

<!-- gh-comment-id:1883613935 --> @jmorganca commented on GitHub (Jan 9, 2024): This should be fixed with #1850 and released with https://github.com/jmorganca/ollama/releases/tag/v0.1.19 which is on track to go out later today. Sorry you hit an error!
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#950