Ollama quits when attempting to run anything. #1151

New Issue

GiteaMirror · 2025-11-12T10:02:29-06:00

GiteaMirror commented

2025-11-12 10:02:29 -06:00

Originally created by @Maxwelldoug on GitHub (Jan 14, 2024).

Originally assigned to: @jmorganca on GitHub.

You folks don't have any templates in place, so I apologize in advance.

I've got a server that I recently deployed (non docker) ollama to, and I kept getting empty responses whenever I tried to run something. upon further investigation of the systemd service, it's exiting with status 2.

Here's the last few hundred lines of journalctl:

Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.cgocall(0x9c1470, 0xc00013c6a0)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc00013c678 sp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm._Cfunc_dynamic_shim_llama_server_init({0x7>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         _cgo_gotypes.go:287 +0x45 fp=0xc00013c6a0 sp=0xc00013c678 pc=0x7cd>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init.func1(0>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:40 +0xe>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init(0xc0000>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:40 +0x1>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.newExtServer({0x17842518, 0xc0004667e0}, {>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/llm/ext_server_common.go:146 +>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.newDynamicShimExtServer({0xc00071c000, 0x2>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:93 +0x5>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.newLlmServer({0xc3d801, 0x4}, {0xc00012815>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/llm/llm.go:86 +0x16b fp=0xc000>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.New({0xc0004aa180?, 0x0?}, {0xc000128150, >Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/llm/llm.go:76 +0x233 fp=0xc000>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/server.load(0xc000002000?, 0xc000002000, {{0x0>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/server/routes.go:84 +0x425 fp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/server.ChatHandler(0xc000486600)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/server/routes.go:1057 +0x828 f>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Context).Next(...)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/server.(*Server).GenerateRoutes.func1(0xc00048>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/server/routes.go:876 +0x68 fp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Context).Next(...)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000486600)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Context).Next(...)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000486600)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0x>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Context).Next(...)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc0000ebba0, 0xc0004>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x65b>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc0000ebba0, {0x1783c860?, 0>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1dd>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.serverHandler.ServeHTTP({0x1783ab80?}, {0x1783c860?, 0xc00044e2a0>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/net/http/server.go:2938 +0x8e fp=0xc00013db78 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.(*conn).serve(0xc0000fe240, {0x1783ded8, 0xc000718240})
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/net/http/server.go:2009 +0x5f4 fp=0xc00013dfb8 s>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.(*Server).Serve.func3()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/net/http/server.go:3086 +0x28 fp=0xc00013dfe0 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00013dfe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by net/http.(*Server).Serve in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/net/http/server.go:3086 +0x5cb
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 1 [IO wait]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x4a05b0?, 0xc00053b828?, 0x78?, 0xb8?, 0x5166dd?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0005af808 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.netpollblock(0x48b9d2?, 0x428946?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0005af840 sp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: internal/poll.runtime_pollWait(0x7fa3240b9e80, 0x72)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0005af860 sp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: internal/poll.(*pollDesc).wait(0xc000488000?, 0x4?, 0x0)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: internal/poll.(*pollDesc).waitRead(...)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
Jan 14 20:38:49 tyrannosaurus ollama[39798]: internal/poll.(*FD).Accept(0xc000488000)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc0005af>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net.(*netFD).accept(0xc000488000)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc0005af9e8 sp=0xc0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net.(*TCPListener).accept(0xc0004595a0)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc0005afa10 s>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net.(*TCPListener).Accept(0xc0004595a0)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc0005afa40 sp=0xc0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.(*onceCloseListener).Accept(0xc0000fe240?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         <autogenerated>:1 +0x24 fp=0xc0005afa58 sp=0xc0005afa40 pc=0x711184Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.(*Server).Serve(0xc000398ff0, {0x1783c650, 0xc0004595a0})
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/net/http/server.go:3056 +0x364 fp=0xc0005afb88 s>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/server.Serve({0x1783c650, 0xc0004595a0})
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/server/routes.go:956 +0x389 fp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/cmd.RunServer(0xc000486300?, {0x17d9db40?, 0x4>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/cmd/cmd.go:634 +0x199 fp=0xc00>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/spf13/cobra.(*Command).execute(0xc00041b800, {0x17d9db40, 0x0, >Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x8>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/spf13/cobra.(*Command).ExecuteC(0xc00041ac00)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/spf13/cobra.(*Command).Execute(...)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985
Jan 14 20:38:49 tyrannosaurus ollama[39798]: main.main()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/main.go:11 +0x4d fp=0xc0005aff>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.main()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc0005affe0 sp=0x>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005affe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 2 [force gc (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006efa8 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goparkunlock(...)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:404
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.forcegchelper()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc00006efe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006efe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.init.6 in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:310 +0x1a
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc00006f7e0 sp=0xc0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 5 [finalizer wait]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0xc364c0?, 0x10045f001?, 0x0?, 0x0?, 0x466045?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006e628 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.runfinq()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00006e7e0 sp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006e7e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.createfing in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mfinal.go:163 +0x3d
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 6 [select, locked to thread]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0xc0000707a8?, 0x2?, 0x29?, 0xe1?, 0xc0000707a4?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000070638 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.selectgo(0xc0000707a8, 0xc0000707a0, 0x0?, 0x0, 0x0?, 0x1)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000070758 sp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.ensureSigM.func1()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/signal_unix.go:1014 +0x19f fp=0xc0000707>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000707e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.ensureSigM in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/signal_unix.go:997 +0xc8
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 18 [syscall]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.notetsleepg(0x0?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/lock_futex.go:236 +0x29 fp=0xc00006a7a0 >Jan 14 20:38:49 tyrannosaurus ollama[39798]: os/signal.signal_recv()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc00006a7c0 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: os/signal.loop()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00006a7e>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006a7e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by os/signal.Notify.func1.1 in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/os/signal/signal.go:151 +0x1f
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 7 [chan receive]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000070f18 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.chanrecv(0xc0001a9a40, 0x0, 0x1)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000070f90 sp=0x>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.chanrecv1(0x0?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc000070fb8 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/server.Serve.func1()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/server/routes.go:938 +0x25 fp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000070fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by github.com/jmorganca/ollama/server.Serve in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /go/src/github.com/jmorganca/ollama/server/routes.go:937 +0x285
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 8 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000071750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0000717e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000717e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 34 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8e3a4?, 0x3?, 0xa9?, 0x5f?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000588750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005887e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005887e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 9 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8e426?, 0xc0004627a0?, 0x1a?, 0x14?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000071f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000071fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000071fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 10 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f80822?, 0x3?, 0x6a?, 0x2f?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000584750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005847e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005847e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 11 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x17d9f7a0?, 0x1?, 0xad?, 0x34?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000584f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000584fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000584fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 12 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8e61a?, 0x3?, 0x9f?, 0x27?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000585750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005857e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005857e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 35 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f804a2?, 0x3?, 0xef?, 0x89?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000588f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000588fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000588fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 50 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f928ed?, 0x3?, 0xf?, 0xfb?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000516750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005167e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005167e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 36 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8e6f3?, 0x1?, 0xbc?, 0xe8?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000589750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005897e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005897e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 51 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f9f31b?, 0x1?, 0x11?, 0x70?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000516f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000516fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000516fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 37 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8e74a?, 0x3?, 0x82?, 0x0?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000589f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000589fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000589fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 52 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8ea5c?, 0x1?, 0x4b?, 0x81?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000517750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005177e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005177e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 38 [GC worker (idle)]:
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x17d9f7a0?, 0x3?, 0x50?, 0xf8?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00058a750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00058a7e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00058a7e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]: rbp    0x9c3c
Jan 14 20:38:49 tyrannosaurus ollama[39798]: rsp    0x7fa2d6ffc0e0
Jan 14 20:38:49 tyrannosaurus ollama[39798]: r8     0x7fa2d6ffc1b0
Jan 14 20:38:49 tyrannosaurus ollama[39798]: r9     0x7fa2d6ffc150
Jan 14 20:38:49 tyrannosaurus ollama[39798]: r10    0x8
Jan 14 20:38:49 tyrannosaurus ollama[39798]: r11    0x246
Jan 14 20:38:49 tyrannosaurus ollama[39798]: r12    0x6
Jan 14 20:38:49 tyrannosaurus ollama[39798]: r13    0x16
Jan 14 20:38:49 tyrannosaurus ollama[39798]: r14    0x1b01560400
Jan 14 20:38:49 tyrannosaurus ollama[39798]: r15    0x1bbd588020
Jan 14 20:38:49 tyrannosaurus ollama[39798]: rip    0x7fa36d5699fc
Jan 14 20:38:49 tyrannosaurus ollama[39798]: rflags 0x246
Jan 14 20:38:49 tyrannosaurus ollama[39798]: cs     0x33
Jan 14 20:38:49 tyrannosaurus ollama[39798]: fs     0x0
Jan 14 20:38:49 tyrannosaurus ollama[39798]: gs     0x0
Jan 14 20:38:50 tyrannosaurus systemd[1]: ollama.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 14 20:38:50 tyrannosaurus systemd[1]: ollama.service: Failed with result 'exit-code'.
Jan 14 20:38:50 tyrannosaurus systemd[1]: ollama.service: Consumed 4.330s CPU time.
Jan 14 20:38:53 tyrannosaurus systemd[1]: ollama.service: Scheduled restart job, restart counter is at 1.
Jan 14 20:38:53 tyrannosaurus systemd[1]: Stopped Ollama Service.
Jan 14 20:38:53 tyrannosaurus systemd[1]: ollama.service: Consumed 4.330s CPU time.
Jan 14 20:38:53 tyrannosaurus systemd[1]: Started Ollama Service.
Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 images.go:834: total blobs: 25
Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 images.go:841: total unused blobs removed: 0
Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 routes.go:929: Listening on [::]:11434 (version 0.1.18)Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 shim_ext_server.go:142: Dynamic LLM variants [cuda roc>Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 gpu.go:34: Detecting GPU type
Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 gpu.go:53: Nvidia GPU detected

The server in question is running Ubuntu 22.04.3 LTS, with the following spec:
Host: PowerEdge R730
Kernel: 5.15.0-91-generic
CPU: Intel Xeon E5-2620 v3 (24) @ 2.600GHz
GPU: NVIDIA GeForce GTX 745
Memory: 19597MiB / 96552MiB

Let me know if anything else is needed or if this is a known issue.

Originally created by @Maxwelldoug on GitHub (Jan 14, 2024). Originally assigned to: @jmorganca on GitHub. You folks don't have any templates in place, so I apologize in advance. I've got a server that I recently deployed (non docker) ollama to, and I kept getting empty responses whenever I tried to run something. upon further investigation of the systemd service, it's exiting with status 2. Here's the last few hundred lines of journalctl: ``` Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.cgocall(0x9c1470, 0xc00013c6a0) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc00013c678 sp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm._Cfunc_dynamic_shim_llama_server_init({0x7>Jan 14 20:38:49 tyrannosaurus ollama[39798]: _cgo_gotypes.go:287 +0x45 fp=0xc00013c6a0 sp=0xc00013c678 pc=0x7cd>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init.func1(0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:40 +0xe>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init(0xc0000>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:40 +0x1>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.newExtServer({0x17842518, 0xc0004667e0}, {>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/llm/ext_server_common.go:146 +>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.newDynamicShimExtServer({0xc00071c000, 0x2>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:93 +0x5>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.newLlmServer({0xc3d801, 0x4}, {0xc00012815>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/llm/llm.go:86 +0x16b fp=0xc000>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/llm.New({0xc0004aa180?, 0x0?}, {0xc000128150, >Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/llm/llm.go:76 +0x233 fp=0xc000>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/server.load(0xc000002000?, 0xc000002000, {{0x0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/server/routes.go:84 +0x425 fp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/server.ChatHandler(0xc000486600) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/server/routes.go:1057 +0x828 f>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Context).Next(...) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/server.(*Server).GenerateRoutes.func1(0xc00048>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/server/routes.go:876 +0x68 fp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Context).Next(...) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000486600) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Context).Next(...) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000486600) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0x>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Context).Next(...) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc0000ebba0, 0xc0004>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x65b>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc0000ebba0, {0x1783c860?, 0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1dd>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.serverHandler.ServeHTTP({0x1783ab80?}, {0x1783c860?, 0xc00044e2a0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/net/http/server.go:2938 +0x8e fp=0xc00013db78 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.(*conn).serve(0xc0000fe240, {0x1783ded8, 0xc000718240}) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/net/http/server.go:2009 +0x5f4 fp=0xc00013dfb8 s>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.(*Server).Serve.func3() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/net/http/server.go:3086 +0x28 fp=0xc00013dfe0 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00013dfe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by net/http.(*Server).Serve in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/net/http/server.go:3086 +0x5cb Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 1 [IO wait]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x4a05b0?, 0xc00053b828?, 0x78?, 0xb8?, 0x5166dd?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0005af808 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.netpollblock(0x48b9d2?, 0x428946?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0005af840 sp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: internal/poll.runtime_pollWait(0x7fa3240b9e80, 0x72) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0005af860 sp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: internal/poll.(*pollDesc).wait(0xc000488000?, 0x4?, 0x0) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: internal/poll.(*pollDesc).waitRead(...) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 Jan 14 20:38:49 tyrannosaurus ollama[39798]: internal/poll.(*FD).Accept(0xc000488000) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc0005af>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net.(*netFD).accept(0xc000488000) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/net/fd_unix.go:172 +0x29 fp=0xc0005af9e8 sp=0xc0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net.(*TCPListener).accept(0xc0004595a0) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc0005afa10 s>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net.(*TCPListener).Accept(0xc0004595a0) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc0005afa40 sp=0xc0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.(*onceCloseListener).Accept(0xc0000fe240?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: <autogenerated>:1 +0x24 fp=0xc0005afa58 sp=0xc0005afa40 pc=0x711184Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.(*Server).Serve(0xc000398ff0, {0x1783c650, 0xc0004595a0}) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/net/http/server.go:3056 +0x364 fp=0xc0005afb88 s>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/server.Serve({0x1783c650, 0xc0004595a0}) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/server/routes.go:956 +0x389 fp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/cmd.RunServer(0xc000486300?, {0x17d9db40?, 0x4>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/cmd/cmd.go:634 +0x199 fp=0xc00>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/spf13/cobra.(*Command).execute(0xc00041b800, {0x17d9db40, 0x0, >Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x8>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/spf13/cobra.(*Command).ExecuteC(0xc00041ac00) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/spf13/cobra.(*Command).Execute(...) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/spf13/cobra.(*Command).ExecuteContext(...) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /root/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985 Jan 14 20:38:49 tyrannosaurus ollama[39798]: main.main() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/main.go:11 +0x4d fp=0xc0005aff>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.main() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc0005affe0 sp=0x>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005affe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 2 [force gc (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006efa8 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goparkunlock(...) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:404 Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.forcegchelper() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc00006efe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006efe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.init.6 in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:310 +0x1a Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc00006f7e0 sp=0xc0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 5 [finalizer wait]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0xc364c0?, 0x10045f001?, 0x0?, 0x0?, 0x466045?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006e628 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.runfinq() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00006e7e0 sp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006e7e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.createfing in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mfinal.go:163 +0x3d Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 6 [select, locked to thread]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0xc0000707a8?, 0x2?, 0x29?, 0xe1?, 0xc0000707a4?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000070638 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.selectgo(0xc0000707a8, 0xc0000707a0, 0x0?, 0x0, 0x0?, 0x1) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000070758 sp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.ensureSigM.func1() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/signal_unix.go:1014 +0x19f fp=0xc0000707>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000707e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.ensureSigM in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/signal_unix.go:997 +0xc8 Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 18 [syscall]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.notetsleepg(0x0?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/lock_futex.go:236 +0x29 fp=0xc00006a7a0 >Jan 14 20:38:49 tyrannosaurus ollama[39798]: os/signal.signal_recv() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc00006a7c0 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: os/signal.loop() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00006a7e>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006a7e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by os/signal.Notify.func1.1 in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/os/signal/signal.go:151 +0x1f Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 7 [chan receive]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000070f18 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.chanrecv(0xc0001a9a40, 0x0, 0x1) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000070f90 sp=0x>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.chanrecv1(0x0?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc000070fb8 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: github.com/jmorganca/ollama/server.Serve.func1() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/server/routes.go:938 +0x25 fp=>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000070fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by github.com/jmorganca/ollama/server.Serve in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /go/src/github.com/jmorganca/ollama/server/routes.go:937 +0x285 Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 8 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000071750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0000717e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000717e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 34 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8e3a4?, 0x3?, 0xa9?, 0x5f?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000588750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005887e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005887e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 9 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8e426?, 0xc0004627a0?, 0x1a?, 0x14?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000071f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000071fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000071fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 10 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f80822?, 0x3?, 0x6a?, 0x2f?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000584750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005847e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005847e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 11 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x17d9f7a0?, 0x1?, 0xad?, 0x34?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000584f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000584fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000584fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 12 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8e61a?, 0x3?, 0x9f?, 0x27?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000585750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005857e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005857e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 35 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f804a2?, 0x3?, 0xef?, 0x89?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000588f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000588fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000588fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 50 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f928ed?, 0x3?, 0xf?, 0xfb?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000516750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005167e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005167e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 36 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8e6f3?, 0x1?, 0xbc?, 0xe8?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000589750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005897e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005897e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 51 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f9f31b?, 0x1?, 0x11?, 0x70?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000516f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000516fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000516fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 37 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8e74a?, 0x3?, 0x82?, 0x0?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000589f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000589fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000589fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 52 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f8ea5c?, 0x1?, 0x4b?, 0x81?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000517750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005177e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005177e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[39798]: goroutine 38 [GC worker (idle)]: Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x17d9f7a0?, 0x3?, 0x50?, 0xf8?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00058a750 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00058a7e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.goexit() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00058a7e8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: rbp 0x9c3c Jan 14 20:38:49 tyrannosaurus ollama[39798]: rsp 0x7fa2d6ffc0e0 Jan 14 20:38:49 tyrannosaurus ollama[39798]: r8 0x7fa2d6ffc1b0 Jan 14 20:38:49 tyrannosaurus ollama[39798]: r9 0x7fa2d6ffc150 Jan 14 20:38:49 tyrannosaurus ollama[39798]: r10 0x8 Jan 14 20:38:49 tyrannosaurus ollama[39798]: r11 0x246 Jan 14 20:38:49 tyrannosaurus ollama[39798]: r12 0x6 Jan 14 20:38:49 tyrannosaurus ollama[39798]: r13 0x16 Jan 14 20:38:49 tyrannosaurus ollama[39798]: r14 0x1b01560400 Jan 14 20:38:49 tyrannosaurus ollama[39798]: r15 0x1bbd588020 Jan 14 20:38:49 tyrannosaurus ollama[39798]: rip 0x7fa36d5699fc Jan 14 20:38:49 tyrannosaurus ollama[39798]: rflags 0x246 Jan 14 20:38:49 tyrannosaurus ollama[39798]: cs 0x33 Jan 14 20:38:49 tyrannosaurus ollama[39798]: fs 0x0 Jan 14 20:38:49 tyrannosaurus ollama[39798]: gs 0x0 Jan 14 20:38:50 tyrannosaurus systemd[1]: ollama.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jan 14 20:38:50 tyrannosaurus systemd[1]: ollama.service: Failed with result 'exit-code'. Jan 14 20:38:50 tyrannosaurus systemd[1]: ollama.service: Consumed 4.330s CPU time. Jan 14 20:38:53 tyrannosaurus systemd[1]: ollama.service: Scheduled restart job, restart counter is at 1. Jan 14 20:38:53 tyrannosaurus systemd[1]: Stopped Ollama Service. Jan 14 20:38:53 tyrannosaurus systemd[1]: ollama.service: Consumed 4.330s CPU time. Jan 14 20:38:53 tyrannosaurus systemd[1]: Started Ollama Service. Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 images.go:834: total blobs: 25 Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 images.go:841: total unused blobs removed: 0 Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 routes.go:929: Listening on [::]:11434 (version 0.1.18)Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 shim_ext_server.go:142: Dynamic LLM variants [cuda roc>Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 gpu.go:34: Detecting GPU type Jan 14 20:38:53 tyrannosaurus ollama[40136]: 2024/01/14 20:38:53 gpu.go:53: Nvidia GPU detected ``` The server in question is running Ubuntu 22.04.3 LTS, with the following spec: Host: PowerEdge R730 Kernel: 5.15.0-91-generic CPU: Intel Xeon E5-2620 v3 (24) @ 2.600GHz GPU: NVIDIA GeForce GTX 745 Memory: 19597MiB / 96552MiB Let me know if anything else is needed or if this is a known issue.

GiteaMirror added the bug nvidia labels 2025-11-12 10:02:29 -06:00

GiteaMirror closed this issue

2025-11-12 10:02:29 -06:00

GiteaMirror commented

2025-11-12 10:02:30 -06:00

@jmorganca commented on GitHub (Jan 14, 2024):

Hi @Maxwelldoug, sorry this happened. Do you have the lines above the large "stack trace"? That might contain a CUDA error we can debug. Thanks so much

@jmorganca commented on GitHub (Jan 14, 2024): Hi @Maxwelldoug, sorry this happened. Do you have the lines above the large "stack trace"? That might contain a CUDA error we can debug. Thanks so much

GiteaMirror commented

2025-11-12 10:02:30 -06:00

@Maxwelldoug commented on GitHub (Jan 14, 2024):

I did another crash, here's a paste from the start of the service to (what I can tell) the start of the trace. I think.

Jan 14 20:10:58 tyrannosaurus ollama[2477]: llama_model_loader: - tensor   32:              blk.2.attn_q.weight q4_0   >Jan 14 20:10:58 tyrannosaurus ollama[2477]: llama_model_loader: - tensor  263:          blk.29.attn_norm.weight f32    >Jan 14 20:10:58 tyrannosaurus ollama[2477]: llama_model_loader: - kv   1:                               general.name st>Jan 14 20:10:58 tyrannosaurus ollama[2477]: llama_model_loader: - kv  12:                       tokenizer.ggml.model st>Jan 14 20:10:58 tyrannosaurus ollama[2477]: llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bo>Jan 14 20:10:58 tyrannosaurus ollama[2477]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Jan 14 20:11:09 tyrannosaurus ollama[8486]:         /go/src/github.com/jmorganca/ollama/server/routes.go:1057 +0x828 fp>Jan 14 20:11:09 tyrannosaurus ollama[8486]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +0>Jan 14 20:11:09 tyrannosaurus ollama[8486]:         /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc00061ba40 sp=0xc00>Jan 14 20:11:09 tyrannosaurus ollama[8486]: github.com/spf13/cobra.(*Command).execute(0xc000489500, {0x17d9db40, 0x0, 0>Jan 14 20:11:09 tyrannosaurus ollama[8486]:         /usr/local/go/src/runtime/mgc.go:200 +0x66
Jan 14 20:11:09 tyrannosaurus ollama[8486]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000114fe0 sp=0xc0>Jan 14 20:11:09 tyrannosaurus ollama[8486]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000116fe8 sp=>Jan 14 20:11:09 tyrannosaurus ollama[8486]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000117fe8 sp=>Jan 14 20:11:09 tyrannosaurus ollama[8486]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:11:09 tyrannosaurus ollama[2477]: runtime.goexit()
Jan 14 20:11:09 tyrannosaurus ollama[2477]: runtime.gopark(0x18fc4ed71f?, 0x1?, 0x84?, 0x48?, 0x0?)
Jan 14 20:11:09 tyrannosaurus ollama[2477]: runtime.gopark(0x18fc519179?, 0x3?, 0x57?, 0x98?, 0x0?)
Jan 14 20:11:09 tyrannosaurus ollama[2477]: rsi    0x2019
Jan 14 20:11:09 tyrannosaurus ollama[2477]: gs     0x0
Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor   17:              blk.1.attn_q.weight q4_0   >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor   45:            blk.4.ffn_gate.weight q4_0   >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor   66:              blk.6.attn_k.weight q4_0   >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor   87:              blk.8.attn_v.weight q4_0   >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor  101:           blk.11.ffn_down.weight q4_0   >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor  108:             blk.11.attn_v.weight q4_0   >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor  122:           blk.13.ffn_norm.weight f32    >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor  192:           blk.21.ffn_gate.weight q4_0   >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor  206:           blk.22.ffn_gate.weight q4_0   >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor  213:           blk.23.ffn_norm.weight f32    >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - kv   5:                  llama.feed_forward_length u3>Jan 14 20:11:27 tyrannosaurus ollama[8853]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
Jan 14 20:11:27 tyrannosaurus ollama[8853]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006efe8 sp=>Jan 14 20:11:27 tyrannosaurus ollama[8853]: os/signal.loop()
Jan 14 20:11:27 tyrannosaurus ollama[8853]: runtime.gopark(0x1f52296ab6?, 0x3?, 0x95?, 0x2f?, 0x0?)
Jan 14 20:11:27 tyrannosaurus ollama[8853]: runtime.gcBgMarkWorker()
Jan 14 20:11:27 tyrannosaurus ollama[8853]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00050a7e8 sp=>Jan 14 20:11:27 tyrannosaurus ollama[8853]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:11:27 tyrannosaurus ollama[8853]: rip    0x7f63865d49fc
Jan 14 20:11:27 tyrannosaurus ollama[8523]: github.com/jmorganca/ollama/llm.newLlmServer({0xc3d801, 0x4}, {0xc000552150>Jan 14 20:11:27 tyrannosaurus ollama[8523]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
Jan 14 20:11:27 tyrannosaurus ollama[8523]: net/http.(*conn).serve(0xc0000262d0, {0x1783ded8, 0xc00050e420})
Jan 14 20:11:27 tyrannosaurus ollama[8523]:         <autogenerated>:1 +0x24 fp=0xc000623a58 sp=0xc000623a40 pc=0x711184
Jan 14 20:11:27 tyrannosaurus ollama[8523]: runtime.goexit()
Jan 14 20:11:27 tyrannosaurus ollama[8523]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:11:27 tyrannosaurus ollama[8523]: goroutine 52 [GC worker (idle)]:
Jan 14 20:11:27 tyrannosaurus ollama[8523]: runtime.gopark(0x1f52299d1f?, 0x3?, 0xef?, 0xac?, 0x0?)
Jan 14 20:11:27 tyrannosaurus ollama[8523]: runtime.gopark(0x1f4ed676e4?, 0x3?, 0x2a?, 0x36?, 0x0?)
Jan 14 20:11:27 tyrannosaurus ollama[8523]: runtime.goexit()
Jan 14 20:11:27 tyrannosaurus ollama[8523]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00050afe8 sp=>Jan 14 20:11:27 tyrannosaurus ollama[8523]: runtime.goexit()
Jan 14 20:11:27 tyrannosaurus ollama[8523]: rbp    0x224e
Jan 14 20:11:27 tyrannosaurus ollama[8523]: rsp    0x7f62e6ffc0e0
Jan 14 20:11:27 tyrannosaurus ollama[8523]: r8     0x7f62e6ffc1b0
Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor   38:            blk.3.ffn_norm.weight f32    >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor   52:           blk.5.attn_norm.weight f32    >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor   59:              blk.5.attn_q.weight q4_0   >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor   73:              blk.7.ffn_up.weight q4_0   >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor   80:            blk.8.ffn_down.weight q4_0   >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor   94:         blk.9.attn_output.weight q4_0   >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor  101:           blk.11.ffn_down.weight q4_0   >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor  136:          blk.15.attn_norm.weight f32    >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - kv  12:                       tokenizer.ggml.model st>Jan 14 20:12:31 tyrannosaurus ollama[10129]: net/http.(*onceCloseListener).Accept(0xc0001385a0?)
Jan 14 20:12:31 tyrannosaurus ollama[10129]: runtime.goexit()
Jan 14 20:12:31 tyrannosaurus ollama[10129]: runtime.goexit()
Jan 14 20:12:31 tyrannosaurus ollama[10129]: runtime.goexit()
Jan 14 20:12:31 tyrannosaurus ollama[10129]: bufio.(*Reader).Peek(0xc00050e240, 0x4)
Jan 14 20:12:31 tyrannosaurus ollama[10129]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000584fe8 sp>Jan 14 20:12:31 tyrannosaurus ollama[8909]: net/http.(*Server).Serve.func3()
Jan 14 20:12:31 tyrannosaurus ollama[8909]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006bf50 sp=0xc0>Jan 14 20:12:31 tyrannosaurus ollama[8909]: rsp    0x7f0e5e7fe0e0
Jan 14 20:13:17 tyrannosaurus ollama[10197]: llama_model_loader: - tensor   67:         blk.6.attn_output.weight q4_0  >Jan 14 20:13:18 tyrannosaurus ollama[11067]: net/http.(*conn).serve(0xc0004d61b0, {0x1783ded8, 0xc000718240})
Jan 14 20:13:18 tyrannosaurus ollama[11067]: runtime.goexit()
Jan 14 20:13:18 tyrannosaurus ollama[11067]: runtime.goexit()
Jan 14 20:13:18 tyrannosaurus ollama[11067]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00051c7e8 sp>Jan 14 20:13:18 tyrannosaurus ollama[11067]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000518fe8 sp>Jan 14 20:13:18 tyrannosaurus ollama[11067]: r12    0x6
Jan 14 20:13:18 tyrannosaurus ollama[10197]: goroutine 6 [select, locked to thread]:
Jan 14 20:13:18 tyrannosaurus ollama[10197]: runtime.chanrecv1(0x0?, 0x0?)
Jan 14 20:13:18 tyrannosaurus ollama[10197]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005157e8 sp>Jan 14 20:13:18 tyrannosaurus ollama[10197]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:14:41 tyrannosaurus ollama[11137]: llama_model_loader: - tensor  210:           blk.23.ffn_down.weight q4_0  >Jan 14 20:14:41 tyrannosaurus ollama[11137]: llama_model_loader: - tensor  237:           blk.26.ffn_down.weight q4_0  >Jan 14 20:14:41 tyrannosaurus ollama[11137]: llama_model_loader: - tensor  252:             blk.27.attn_q.weight q4_0  >Jan 14 20:14:41 tyrannosaurus ollama[11137]: llama_model_loader: - kv  13:                      tokenizer.ggml.tokens a>Jan 14 20:14:42 tyrannosaurus ollama[12662]: net/http.(*onceCloseListener).Accept(0xc0000262d0?)
Jan 14 20:14:42 tyrannosaurus ollama[12662]: runtime.goexit()
Jan 14 20:14:42 tyrannosaurus ollama[12662]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000071fe0 sp=0xc>Jan 14 20:14:42 tyrannosaurus ollama[12662]: runtime.goexit()
Jan 14 20:14:42 tyrannosaurus ollama[12662]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:14:42 tyrannosaurus ollama[11137]:         /go/src/github.com/jmorganca/ollama/server/routes.go:876 +0x68 fp=>Jan 14 20:14:42 tyrannosaurus ollama[11137]: goroutine 1 [IO wait, 1 minutes]:
Jan 14 20:14:42 tyrannosaurus ollama[11137]: runtime.goparkunlock(...)
Jan 14 20:14:42 tyrannosaurus ollama[11137]: runtime.gopark(0x4cc4c911bc?, 0x1?, 0xfe?, 0x91?, 0x0?)
Jan 14 20:14:42 tyrannosaurus ollama[11137]: runtime.gopark(0x4cc4c90812?, 0x3?, 0xc?, 0x45?, 0x0?)
Jan 14 20:14:42 tyrannosaurus ollama[11137]: runtime.gopark(0x4cc4c90b0a?, 0x3?, 0xc2?, 0x2b?, 0x0?)
Jan 14 20:14:42 tyrannosaurus ollama[11137]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:14:42 tyrannosaurus ollama[11137]: rax    0x0
Jan 14 20:14:45 tyrannosaurus ollama[12733]: 2024/01/14 20:14:45 images.go:834: total blobs: 25
Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor    3:            blk.0.ffn_gate.weight q4_0  >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor   80:            blk.8.ffn_down.weight q4_0  >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor   95:              blk.9.attn_q.weight q4_0  >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor  109:          blk.12.attn_norm.weight f32   >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor  136:          blk.15.attn_norm.weight f32   >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor  162:             blk.17.attn_v.weight q4_0  >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor  256:           blk.28.ffn_gate.weight q4_0  >Jan 14 20:19:39 tyrannosaurus ollama[18351]: runtime.gcBgMarkWorker()
Jan 14 20:19:39 tyrannosaurus ollama[18351]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00006cfe0 sp=0xc>Jan 14 20:19:39 tyrannosaurus ollama[18351]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:19:39 tyrannosaurus ollama[18351]: rdx    0x6
Jan 14 20:19:39 tyrannosaurus ollama[18054]:         /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc000573a10 s>Jan 14 20:19:39 tyrannosaurus ollama[18054]: runtime.gopark(0x91df06d125?, 0x3?, 0x10?, 0x1c?, 0x0?)
Jan 14 20:19:39 tyrannosaurus ollama[18054]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00006dfe0 sp=0xc>Jan 14 20:19:39 tyrannosaurus ollama[18054]: runtime.goexit()
Jan 14 20:19:39 tyrannosaurus ollama[18054]: created by runtime.ensureSigM in goroutine 1
Jan 14 20:19:39 tyrannosaurus ollama[18054]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00050e5a0 sp=0xc>Jan 14 20:19:39 tyrannosaurus ollama[18054]:         <autogenerated>:1 +0x25 fp=0xc00050e778 sp=0xc00050e748 pc=0x5a9565Jan 14 20:19:39 tyrannosaurus ollama[18054]: rsp    0x7f0eef7fd0e0
Jan 14 20:19:39 tyrannosaurus ollama[18054]: r8     0x7f0eef7fd1b0
Jan 14 20:19:39 tyrannosaurus ollama[18054]: r9     0x7f0eef7fd150
Jan 14 20:19:39 tyrannosaurus ollama[18054]: r10    0x8
Jan 14 20:19:39 tyrannosaurus ollama[18054]: r11    0x246
Jan 14 20:20:32 tyrannosaurus ollama[18423]: llama_model_loader: - tensor  247:           blk.27.ffn_gate.weight q4_0  >Jan 14 20:20:32 tyrannosaurus ollama[18423]: llama_model_loader: - kv   7:                 llama.attention.head_count u>Jan 14 20:20:33 tyrannosaurus ollama[19395]: goroutine 3 [GC sweep wait]:
Jan 14 20:20:33 tyrannosaurus ollama[19395]: runtime.chanrecv1(0x0?, 0x0?)
Jan 14 20:20:33 tyrannosaurus ollama[19395]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 14 20:20:33 tyrannosaurus ollama[19395]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000512f50 sp=0xc>Jan 14 20:20:33 tyrannosaurus ollama[18423]: github.com/gin-gonic/gin.(*Context).Next(...)
Jan 14 20:20:33 tyrannosaurus ollama[18423]: runtime.goexit()
Jan 14 20:20:33 tyrannosaurus ollama[18423]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000511fe0 sp=0xc>Jan 14 20:20:33 tyrannosaurus ollama[18423]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:20:33 tyrannosaurus ollama[18423]: rsp    0x7feb2dffa0e0
Jan 14 20:20:37 tyrannosaurus systemd[1]: ollama.service: Scheduled restart job, restart counter is at 2.
Jan 14 20:26:39 tyrannosaurus ollama[19505]: llama_model_loader: - tensor    5:            blk.0.ffn_norm.weight f32   >Jan 14 20:26:39 tyrannosaurus ollama[19505]: llama_model_loader: - tensor  110:            blk.2.ffn_down.weight q4_0  >Jan 14 20:26:39 tyrannosaurus ollama[19505]: llama_model_loader: - tensor  117:              blk.2.attn_v.weight q4_0  >Jan 14 20:26:39 tyrannosaurus ollama[19505]: llama_model_loader: - tensor  243:             blk.26.attn_q.weight q4_0  >Jan 14 20:26:39 tyrannosaurus ollama[19505]: llm_load_print_meta: n_layer          = 32
Jan 14 20:26:39 tyrannosaurus ollama[19505]: llm_load_tensors: offloading 26 repeating layers to GPU
Jan 14 20:26:49 tyrannosaurus ollama[26424]:         /go/src/github.com/jmorganca/ollama/llm/llm.go:76 +0x233 fp=0xc000>Jan 14 20:26:49 tyrannosaurus ollama[26424]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
Jan 14 20:26:49 tyrannosaurus ollama[26424]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006efa8 sp=0xc>Jan 14 20:26:49 tyrannosaurus ollama[26424]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005027e0 sp=0xc>Jan 14 20:26:49 tyrannosaurus ollama[26424]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000502fe0 sp=0xc>Jan 14 20:26:49 tyrannosaurus ollama[26424]: runtime.goexit()
Jan 14 20:26:49 tyrannosaurus ollama[26424]: net/http.(*connReader).startBackgroundRead.func2()
Jan 14 20:26:49 tyrannosaurus ollama[19505]: goroutine 26 [GC worker (idle)]:
Jan 14 20:26:49 tyrannosaurus ollama[19505]: goroutine 54 [GC worker (idle)]:
Jan 14 20:26:49 tyrannosaurus ollama[19505]: runtime.netpollblock(0x49e718?, 0x428946?, 0x0?)
Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor   24:             blk.10.attn_k.weight q4_0  >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor   38:           blk.12.ffn_down.weight q4_0  >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor   79:        blk.16.attn_output.weight q4_0  >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor   86:           blk.17.ffn_norm.weight f32   >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor   93:           blk.18.ffn_gate.weight q4_0  >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor  156:            blk.3.ffn_gate.weight q4_0  >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor  185:            blk.6.ffn_norm.weight f32   >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor  247:           blk.27.ffn_gate.weight q4_0  >Jan 14 20:27:26 tyrannosaurus ollama[26482]: llm_load_print_meta: n_ctx_train      = 4096
Jan 14 20:27:27 tyrannosaurus ollama[27163]:         /usr/local/go/src/net/http/server.go:2938 +0x8e fp=0xc0000dfb78 sp>Jan 14 20:27:27 tyrannosaurus ollama[27163]: runtime.gopark(0xfed367bbb9?, 0x3?, 0xbc?, 0xb7?, 0x0?)
Jan 14 20:27:27 tyrannosaurus ollama[26482]: runtime.gcenable.func2()
Jan 14 20:29:11 tyrannosaurus ollama[27215]: llama_model_loader: - tensor  233:        blk.25.attn_output.weight q4_0  >Jan 14 20:29:12 tyrannosaurus ollama[29167]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001117e8 sp>Jan 14 20:29:12 tyrannosaurus ollama[27215]:         /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
Jan 14 20:29:12 tyrannosaurus ollama[27215]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00061b808 sp=0xc>Jan 14 20:29:12 tyrannosaurus ollama[27215]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006bf50 sp=0xc>Jan 14 20:29:12 tyrannosaurus ollama[27215]:         /usr/local/go/src/net/http/server.go:679 +0x25 fp=0xc000519fe0 sp=>Jan 14 20:29:12 tyrannosaurus ollama[27215]: rbp    0x719e
Jan 14 20:30:07 tyrannosaurus ollama[30255]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:30:07 tyrannosaurus ollama[30255]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:30:07 tyrannosaurus ollama[30255]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:30:07 tyrannosaurus ollama[30255]: runtime.gcBgMarkWorker()
Jan 14 20:30:07 tyrannosaurus ollama[29222]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
Jan 14 20:30:07 tyrannosaurus ollama[29222]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00061ffe8 sp>Jan 14 20:30:07 tyrannosaurus ollama[29222]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:30:07 tyrannosaurus ollama[29222]: goroutine 21 [GC worker (idle)]:
Jan 14 20:30:07 tyrannosaurus ollama[29222]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:30:07 tyrannosaurus ollama[29222]: runtime.gopark(0x12411ddc63c?, 0x3?, 0xc8?, 0x60?, 0x0?)
Jan 14 20:30:07 tyrannosaurus ollama[29222]:         /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc000621950 sp=>Jan 14 20:30:07 tyrannosaurus ollama[29222]: runtime.goexit()
Jan 14 20:38:48 tyrannosaurus ollama[39798]: llama_model_loader: - tensor  148:             blk.16.ffn_up.weight q4_0  >Jan 14 20:38:48 tyrannosaurus ollama[39798]: llama_model_loader: - type q4_0:  225 tensors
Jan 14 20:38:49 tyrannosaurus ollama[40058]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:38:49 tyrannosaurus ollama[40058]: runtime.gopark(0x19dd1f8e74a?, 0x3?, 0x82?, 0x0?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[40058]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000517f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[40058]: runtime.gcBgMarkWorker()
Jan 14 20:38:49 tyrannosaurus ollama[40058]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000512fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.serverHandler.ServeHTTP({0x1783ab80?}, {0x1783c860?, 0xc00044e2a0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcenable.func2()
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000070fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f847ad?, 0x3?, 0xfe?, 0xce?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005207e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/net/http/server.go:679 +0xba
Jan 14 20:59:25 tyrannosaurus ollama[40136]: llama_model_loader: - tensor   63:            blk.6.ffn_gate.weight q4_0  >Jan 14 20:59:25 tyrannosaurus ollama[40136]: llama_model_loader: - tensor  203:                    output.weight q6_K  >Jan 14 20:59:25 tyrannosaurus ollama[40136]: llm_load_print_meta: n_ff             = 14336
Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /usr/local/go/src/net/http/server.go:3086 +0x28 fp=0xc000133fe0 sp>Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /usr/local/go/src/runtime/proc.go:404
Jan 14 20:59:26 tyrannosaurus ollama[62986]: runtime.goexit()
Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00050a7e0 sp=0xc>Jan 14 20:59:26 tyrannosaurus ollama[62986]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:59:26 tyrannosaurus ollama[62986]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /usr/local/go/src/runtime/mgc.go:1217 +0x1c
Jan 14 20:59:26 tyrannosaurus ollama[62986]: rbx    0x7f47173d2640
Jan 14 20:59:26 tyrannosaurus ollama[40136]:         /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:93 +0x5>Jan 14 20:59:26 tyrannosaurus ollama[40136]:         /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc00011f>Jan 14 20:59:26 tyrannosaurus ollama[40136]: created by runtime.init.6 in goroutine 1
Jan 14 20:59:26 tyrannosaurus ollama[40136]: runtime.gcenable.func1()
Jan 14 20:59:26 tyrannosaurus ollama[40136]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000070f50 sp=0xc>Jan 14 20:59:26 tyrannosaurus ollama[40136]: runtime.gcBgMarkWorker()
Jan 14 20:59:26 tyrannosaurus ollama[40136]: goroutine 11 [GC worker (idle)]:
Jan 14 20:59:26 tyrannosaurus ollama[40136]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000504fe0 sp=0xc>Jan 14 20:59:26 tyrannosaurus ollama[40136]: runtime.goexit()
Jan 14 20:59:26 tyrannosaurus ollama[40136]: runtime.goexit()
Jan 14 20:59:26 tyrannosaurus ollama[40136]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:59:26 tyrannosaurus ollama[40136]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000505f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[40058]: r12    0x6
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         _cgo_gotypes.go:287 +0x45 fp=0xc00013c6a0 sp=0xc00013c678 pc=0x7cd>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.chanrecv1(0x0?, 0x0?)
Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005897e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]:         /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00058afe8 sp>Jan 14 20:59:25 tyrannosaurus ollama[40136]:   Device 0: NVIDIA GeForce GTX 745, compute capability 5.0
Jan 14 20:59:25 tyrannosaurus ollama[40136]: llama_model_loader: - tensor  159:             blk.17.attn_k.weight q4_0  >Jan 14 20:59:25 tyrannosaurus ollama[40136]: llm_load_print_meta: n_vocab          = 32000
Jan 14 20:59:26 tyrannosaurus ollama[62986]: internal/poll.(*pollDesc).waitRead(...)
Jan 14 20:59:26 tyrannosaurus ollama[62986]: goroutine 4 [GC scavenge wait]:
Jan 14 20:59:26 tyrannosaurus ollama[62986]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 14 20:59:26 tyrannosaurus ollama[62986]: goroutine 39 [GC worker (idle)]:
Jan 14 20:59:26 tyrannosaurus ollama[40136]: current device: 0
Jan 14 20:59:26 tyrannosaurus ollama[40136]: Lazy loading /tmp/ollama2596731661/cuda/libext_server.so library
Jan 14 20:59:26 tyrannosaurus ollama[40136]: GGML_ASSERT: /go/src/github.com/jmorganca/ollama/llm/llama.cpp/ggml-cuda.c>Jan 14 20:59:26 tyrannosaurus ollama[62986]: SIGABRT: abort
Jan 14 20:59:26 tyrannosaurus ollama[62986]: PC=0x7f475f0049fc m=4 sigcode=18446744073709551610
Jan 14 20:59:26 tyrannosaurus ollama[62986]: signal arrived during cgo execution
Jan 14 20:59:26 tyrannosaurus ollama[62986]: goroutine 19 [syscall]:
Jan 14 20:59:26 tyrannosaurus ollama[62986]: runtime.cgocall(0x9c1470, 0xc0001326a0)
Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc000132678 sp=>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm._Cfunc_dynamic_shim_llama_server_init({0x7>Jan 14 20:59:26 tyrannosaurus ollama[62986]:         _cgo_gotypes.go:287 +0x45 fp=0xc0001326a0 sp=0xc000132678 pc=0x7cd>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init.func1(0>Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:40 +0xe>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init(0xc0000>Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:40 +0x1>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.newExtServer({0x17842518, 0xc000468b40}, {>Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /go/src/github.com/jmorganca/ollama/llm/ext_server_common.go:146 +>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.newDynamicShimExtServer({0xc000716000, 0x2>Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:93 +0x5>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.newLlmServer({0xc3d801, 0x4}, {0xc0000e615>Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /go/src/github.com/jmorganca/ollama/llm/llm.go:86 +0x16b fp=0xc000>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.New({0xc0004aa180?, 0x0?}, {0xc0000e6150, >Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /go/src/github.com/jmorganca/ollama/llm/llm.go:76 +0x233 fp=0xc000>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/server.load(0xc000002000?, 0xc000002000, {{0x0>Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /go/src/github.com/jmorganca/ollama/server/routes.go:84 +0x425 fp=>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/server.ChatHandler(0xc000486600)
Jan 14 20:59:26 tyrannosaurus ollama[62986]:         /go/src/github.com/jmorganca/ollama/server/routes.go:1057 +0x828 f>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/gin-gonic/gin.(*Context).Next(...)

@Maxwelldoug commented on GitHub (Jan 14, 2024): I did another crash, here's a paste from the start of the service to (what I can tell) the start of the trace. I think. ``` Jan 14 20:10:58 tyrannosaurus ollama[2477]: llama_model_loader: - tensor 32: blk.2.attn_q.weight q4_0 >Jan 14 20:10:58 tyrannosaurus ollama[2477]: llama_model_loader: - tensor 263: blk.29.attn_norm.weight f32 >Jan 14 20:10:58 tyrannosaurus ollama[2477]: llama_model_loader: - kv 1: general.name st>Jan 14 20:10:58 tyrannosaurus ollama[2477]: llama_model_loader: - kv 12: tokenizer.ggml.model st>Jan 14 20:10:58 tyrannosaurus ollama[2477]: llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bo>Jan 14 20:10:58 tyrannosaurus ollama[2477]: llm_load_print_meta: f_norm_eps = 0.0e+00 Jan 14 20:11:09 tyrannosaurus ollama[8486]: /go/src/github.com/jmorganca/ollama/server/routes.go:1057 +0x828 fp>Jan 14 20:11:09 tyrannosaurus ollama[8486]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +0>Jan 14 20:11:09 tyrannosaurus ollama[8486]: /usr/local/go/src/net/tcpsock.go:315 +0x30 fp=0xc00061ba40 sp=0xc00>Jan 14 20:11:09 tyrannosaurus ollama[8486]: github.com/spf13/cobra.(*Command).execute(0xc000489500, {0x17d9db40, 0x0, 0>Jan 14 20:11:09 tyrannosaurus ollama[8486]: /usr/local/go/src/runtime/mgc.go:200 +0x66 Jan 14 20:11:09 tyrannosaurus ollama[8486]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000114fe0 sp=0xc0>Jan 14 20:11:09 tyrannosaurus ollama[8486]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000116fe8 sp=>Jan 14 20:11:09 tyrannosaurus ollama[8486]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000117fe8 sp=>Jan 14 20:11:09 tyrannosaurus ollama[8486]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:11:09 tyrannosaurus ollama[2477]: runtime.goexit() Jan 14 20:11:09 tyrannosaurus ollama[2477]: runtime.gopark(0x18fc4ed71f?, 0x1?, 0x84?, 0x48?, 0x0?) Jan 14 20:11:09 tyrannosaurus ollama[2477]: runtime.gopark(0x18fc519179?, 0x3?, 0x57?, 0x98?, 0x0?) Jan 14 20:11:09 tyrannosaurus ollama[2477]: rsi 0x2019 Jan 14 20:11:09 tyrannosaurus ollama[2477]: gs 0x0 Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor 17: blk.1.attn_q.weight q4_0 >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor 45: blk.4.ffn_gate.weight q4_0 >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor 66: blk.6.attn_k.weight q4_0 >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor 87: blk.8.attn_v.weight q4_0 >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor 101: blk.11.ffn_down.weight q4_0 >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor 108: blk.11.attn_v.weight q4_0 >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor 122: blk.13.ffn_norm.weight f32 >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor 192: blk.21.ffn_gate.weight q4_0 >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor 206: blk.22.ffn_gate.weight q4_0 >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - tensor 213: blk.23.ffn_norm.weight f32 >Jan 14 20:11:25 tyrannosaurus ollama[8523]: llama_model_loader: - kv 5: llama.feed_forward_length u3>Jan 14 20:11:27 tyrannosaurus ollama[8853]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 Jan 14 20:11:27 tyrannosaurus ollama[8853]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006efe8 sp=>Jan 14 20:11:27 tyrannosaurus ollama[8853]: os/signal.loop() Jan 14 20:11:27 tyrannosaurus ollama[8853]: runtime.gopark(0x1f52296ab6?, 0x3?, 0x95?, 0x2f?, 0x0?) Jan 14 20:11:27 tyrannosaurus ollama[8853]: runtime.gcBgMarkWorker() Jan 14 20:11:27 tyrannosaurus ollama[8853]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00050a7e8 sp=>Jan 14 20:11:27 tyrannosaurus ollama[8853]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:11:27 tyrannosaurus ollama[8853]: rip 0x7f63865d49fc Jan 14 20:11:27 tyrannosaurus ollama[8523]: github.com/jmorganca/ollama/llm.newLlmServer({0xc3d801, 0x4}, {0xc000552150>Jan 14 20:11:27 tyrannosaurus ollama[8523]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 Jan 14 20:11:27 tyrannosaurus ollama[8523]: net/http.(*conn).serve(0xc0000262d0, {0x1783ded8, 0xc00050e420}) Jan 14 20:11:27 tyrannosaurus ollama[8523]: <autogenerated>:1 +0x24 fp=0xc000623a58 sp=0xc000623a40 pc=0x711184 Jan 14 20:11:27 tyrannosaurus ollama[8523]: runtime.goexit() Jan 14 20:11:27 tyrannosaurus ollama[8523]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:11:27 tyrannosaurus ollama[8523]: goroutine 52 [GC worker (idle)]: Jan 14 20:11:27 tyrannosaurus ollama[8523]: runtime.gopark(0x1f52299d1f?, 0x3?, 0xef?, 0xac?, 0x0?) Jan 14 20:11:27 tyrannosaurus ollama[8523]: runtime.gopark(0x1f4ed676e4?, 0x3?, 0x2a?, 0x36?, 0x0?) Jan 14 20:11:27 tyrannosaurus ollama[8523]: runtime.goexit() Jan 14 20:11:27 tyrannosaurus ollama[8523]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00050afe8 sp=>Jan 14 20:11:27 tyrannosaurus ollama[8523]: runtime.goexit() Jan 14 20:11:27 tyrannosaurus ollama[8523]: rbp 0x224e Jan 14 20:11:27 tyrannosaurus ollama[8523]: rsp 0x7f62e6ffc0e0 Jan 14 20:11:27 tyrannosaurus ollama[8523]: r8 0x7f62e6ffc1b0 Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor 38: blk.3.ffn_norm.weight f32 >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor 52: blk.5.attn_norm.weight f32 >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor 59: blk.5.attn_q.weight q4_0 >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor 73: blk.7.ffn_up.weight q4_0 >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor 80: blk.8.ffn_down.weight q4_0 >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor 94: blk.9.attn_output.weight q4_0 >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor 101: blk.11.ffn_down.weight q4_0 >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - tensor 136: blk.15.attn_norm.weight f32 >Jan 14 20:12:30 tyrannosaurus ollama[8909]: llama_model_loader: - kv 12: tokenizer.ggml.model st>Jan 14 20:12:31 tyrannosaurus ollama[10129]: net/http.(*onceCloseListener).Accept(0xc0001385a0?) Jan 14 20:12:31 tyrannosaurus ollama[10129]: runtime.goexit() Jan 14 20:12:31 tyrannosaurus ollama[10129]: runtime.goexit() Jan 14 20:12:31 tyrannosaurus ollama[10129]: runtime.goexit() Jan 14 20:12:31 tyrannosaurus ollama[10129]: bufio.(*Reader).Peek(0xc00050e240, 0x4) Jan 14 20:12:31 tyrannosaurus ollama[10129]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000584fe8 sp>Jan 14 20:12:31 tyrannosaurus ollama[8909]: net/http.(*Server).Serve.func3() Jan 14 20:12:31 tyrannosaurus ollama[8909]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006bf50 sp=0xc0>Jan 14 20:12:31 tyrannosaurus ollama[8909]: rsp 0x7f0e5e7fe0e0 Jan 14 20:13:17 tyrannosaurus ollama[10197]: llama_model_loader: - tensor 67: blk.6.attn_output.weight q4_0 >Jan 14 20:13:18 tyrannosaurus ollama[11067]: net/http.(*conn).serve(0xc0004d61b0, {0x1783ded8, 0xc000718240}) Jan 14 20:13:18 tyrannosaurus ollama[11067]: runtime.goexit() Jan 14 20:13:18 tyrannosaurus ollama[11067]: runtime.goexit() Jan 14 20:13:18 tyrannosaurus ollama[11067]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00051c7e8 sp>Jan 14 20:13:18 tyrannosaurus ollama[11067]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000518fe8 sp>Jan 14 20:13:18 tyrannosaurus ollama[11067]: r12 0x6 Jan 14 20:13:18 tyrannosaurus ollama[10197]: goroutine 6 [select, locked to thread]: Jan 14 20:13:18 tyrannosaurus ollama[10197]: runtime.chanrecv1(0x0?, 0x0?) Jan 14 20:13:18 tyrannosaurus ollama[10197]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005157e8 sp>Jan 14 20:13:18 tyrannosaurus ollama[10197]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:14:41 tyrannosaurus ollama[11137]: llama_model_loader: - tensor 210: blk.23.ffn_down.weight q4_0 >Jan 14 20:14:41 tyrannosaurus ollama[11137]: llama_model_loader: - tensor 237: blk.26.ffn_down.weight q4_0 >Jan 14 20:14:41 tyrannosaurus ollama[11137]: llama_model_loader: - tensor 252: blk.27.attn_q.weight q4_0 >Jan 14 20:14:41 tyrannosaurus ollama[11137]: llama_model_loader: - kv 13: tokenizer.ggml.tokens a>Jan 14 20:14:42 tyrannosaurus ollama[12662]: net/http.(*onceCloseListener).Accept(0xc0000262d0?) Jan 14 20:14:42 tyrannosaurus ollama[12662]: runtime.goexit() Jan 14 20:14:42 tyrannosaurus ollama[12662]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000071fe0 sp=0xc>Jan 14 20:14:42 tyrannosaurus ollama[12662]: runtime.goexit() Jan 14 20:14:42 tyrannosaurus ollama[12662]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:14:42 tyrannosaurus ollama[11137]: /go/src/github.com/jmorganca/ollama/server/routes.go:876 +0x68 fp=>Jan 14 20:14:42 tyrannosaurus ollama[11137]: goroutine 1 [IO wait, 1 minutes]: Jan 14 20:14:42 tyrannosaurus ollama[11137]: runtime.goparkunlock(...) Jan 14 20:14:42 tyrannosaurus ollama[11137]: runtime.gopark(0x4cc4c911bc?, 0x1?, 0xfe?, 0x91?, 0x0?) Jan 14 20:14:42 tyrannosaurus ollama[11137]: runtime.gopark(0x4cc4c90812?, 0x3?, 0xc?, 0x45?, 0x0?) Jan 14 20:14:42 tyrannosaurus ollama[11137]: runtime.gopark(0x4cc4c90b0a?, 0x3?, 0xc2?, 0x2b?, 0x0?) Jan 14 20:14:42 tyrannosaurus ollama[11137]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:14:42 tyrannosaurus ollama[11137]: rax 0x0 Jan 14 20:14:45 tyrannosaurus ollama[12733]: 2024/01/14 20:14:45 images.go:834: total blobs: 25 Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor 3: blk.0.ffn_gate.weight q4_0 >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor 80: blk.8.ffn_down.weight q4_0 >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor 95: blk.9.attn_q.weight q4_0 >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor 109: blk.12.attn_norm.weight f32 >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor 136: blk.15.attn_norm.weight f32 >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor 162: blk.17.attn_v.weight q4_0 >Jan 14 20:19:37 tyrannosaurus ollama[18054]: llama_model_loader: - tensor 256: blk.28.ffn_gate.weight q4_0 >Jan 14 20:19:39 tyrannosaurus ollama[18351]: runtime.gcBgMarkWorker() Jan 14 20:19:39 tyrannosaurus ollama[18351]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00006cfe0 sp=0xc>Jan 14 20:19:39 tyrannosaurus ollama[18351]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:19:39 tyrannosaurus ollama[18351]: rdx 0x6 Jan 14 20:19:39 tyrannosaurus ollama[18054]: /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e fp=0xc000573a10 s>Jan 14 20:19:39 tyrannosaurus ollama[18054]: runtime.gopark(0x91df06d125?, 0x3?, 0x10?, 0x1c?, 0x0?) Jan 14 20:19:39 tyrannosaurus ollama[18054]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00006dfe0 sp=0xc>Jan 14 20:19:39 tyrannosaurus ollama[18054]: runtime.goexit() Jan 14 20:19:39 tyrannosaurus ollama[18054]: created by runtime.ensureSigM in goroutine 1 Jan 14 20:19:39 tyrannosaurus ollama[18054]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00050e5a0 sp=0xc>Jan 14 20:19:39 tyrannosaurus ollama[18054]: <autogenerated>:1 +0x25 fp=0xc00050e778 sp=0xc00050e748 pc=0x5a9565Jan 14 20:19:39 tyrannosaurus ollama[18054]: rsp 0x7f0eef7fd0e0 Jan 14 20:19:39 tyrannosaurus ollama[18054]: r8 0x7f0eef7fd1b0 Jan 14 20:19:39 tyrannosaurus ollama[18054]: r9 0x7f0eef7fd150 Jan 14 20:19:39 tyrannosaurus ollama[18054]: r10 0x8 Jan 14 20:19:39 tyrannosaurus ollama[18054]: r11 0x246 Jan 14 20:20:32 tyrannosaurus ollama[18423]: llama_model_loader: - tensor 247: blk.27.ffn_gate.weight q4_0 >Jan 14 20:20:32 tyrannosaurus ollama[18423]: llama_model_loader: - kv 7: llama.attention.head_count u>Jan 14 20:20:33 tyrannosaurus ollama[19395]: goroutine 3 [GC sweep wait]: Jan 14 20:20:33 tyrannosaurus ollama[19395]: runtime.chanrecv1(0x0?, 0x0?) Jan 14 20:20:33 tyrannosaurus ollama[19395]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) Jan 14 20:20:33 tyrannosaurus ollama[19395]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000512f50 sp=0xc>Jan 14 20:20:33 tyrannosaurus ollama[18423]: github.com/gin-gonic/gin.(*Context).Next(...) Jan 14 20:20:33 tyrannosaurus ollama[18423]: runtime.goexit() Jan 14 20:20:33 tyrannosaurus ollama[18423]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000511fe0 sp=0xc>Jan 14 20:20:33 tyrannosaurus ollama[18423]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:20:33 tyrannosaurus ollama[18423]: rsp 0x7feb2dffa0e0 Jan 14 20:20:37 tyrannosaurus systemd[1]: ollama.service: Scheduled restart job, restart counter is at 2. Jan 14 20:26:39 tyrannosaurus ollama[19505]: llama_model_loader: - tensor 5: blk.0.ffn_norm.weight f32 >Jan 14 20:26:39 tyrannosaurus ollama[19505]: llama_model_loader: - tensor 110: blk.2.ffn_down.weight q4_0 >Jan 14 20:26:39 tyrannosaurus ollama[19505]: llama_model_loader: - tensor 117: blk.2.attn_v.weight q4_0 >Jan 14 20:26:39 tyrannosaurus ollama[19505]: llama_model_loader: - tensor 243: blk.26.attn_q.weight q4_0 >Jan 14 20:26:39 tyrannosaurus ollama[19505]: llm_load_print_meta: n_layer = 32 Jan 14 20:26:39 tyrannosaurus ollama[19505]: llm_load_tensors: offloading 26 repeating layers to GPU Jan 14 20:26:49 tyrannosaurus ollama[26424]: /go/src/github.com/jmorganca/ollama/llm/llm.go:76 +0x233 fp=0xc000>Jan 14 20:26:49 tyrannosaurus ollama[26424]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 Jan 14 20:26:49 tyrannosaurus ollama[26424]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006efa8 sp=0xc>Jan 14 20:26:49 tyrannosaurus ollama[26424]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005027e0 sp=0xc>Jan 14 20:26:49 tyrannosaurus ollama[26424]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000502fe0 sp=0xc>Jan 14 20:26:49 tyrannosaurus ollama[26424]: runtime.goexit() Jan 14 20:26:49 tyrannosaurus ollama[26424]: net/http.(*connReader).startBackgroundRead.func2() Jan 14 20:26:49 tyrannosaurus ollama[19505]: goroutine 26 [GC worker (idle)]: Jan 14 20:26:49 tyrannosaurus ollama[19505]: goroutine 54 [GC worker (idle)]: Jan 14 20:26:49 tyrannosaurus ollama[19505]: runtime.netpollblock(0x49e718?, 0x428946?, 0x0?) Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor 24: blk.10.attn_k.weight q4_0 >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor 38: blk.12.ffn_down.weight q4_0 >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor 79: blk.16.attn_output.weight q4_0 >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor 86: blk.17.ffn_norm.weight f32 >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor 93: blk.18.ffn_gate.weight q4_0 >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor 156: blk.3.ffn_gate.weight q4_0 >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor 185: blk.6.ffn_norm.weight f32 >Jan 14 20:27:25 tyrannosaurus ollama[26482]: llama_model_loader: - tensor 247: blk.27.ffn_gate.weight q4_0 >Jan 14 20:27:26 tyrannosaurus ollama[26482]: llm_load_print_meta: n_ctx_train = 4096 Jan 14 20:27:27 tyrannosaurus ollama[27163]: /usr/local/go/src/net/http/server.go:2938 +0x8e fp=0xc0000dfb78 sp>Jan 14 20:27:27 tyrannosaurus ollama[27163]: runtime.gopark(0xfed367bbb9?, 0x3?, 0xbc?, 0xb7?, 0x0?) Jan 14 20:27:27 tyrannosaurus ollama[26482]: runtime.gcenable.func2() Jan 14 20:29:11 tyrannosaurus ollama[27215]: llama_model_loader: - tensor 233: blk.25.attn_output.weight q4_0 >Jan 14 20:29:12 tyrannosaurus ollama[29167]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001117e8 sp>Jan 14 20:29:12 tyrannosaurus ollama[27215]: /root/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 Jan 14 20:29:12 tyrannosaurus ollama[27215]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00061b808 sp=0xc>Jan 14 20:29:12 tyrannosaurus ollama[27215]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006bf50 sp=0xc>Jan 14 20:29:12 tyrannosaurus ollama[27215]: /usr/local/go/src/net/http/server.go:679 +0x25 fp=0xc000519fe0 sp=>Jan 14 20:29:12 tyrannosaurus ollama[27215]: rbp 0x719e Jan 14 20:30:07 tyrannosaurus ollama[30255]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:30:07 tyrannosaurus ollama[30255]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:30:07 tyrannosaurus ollama[30255]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:30:07 tyrannosaurus ollama[30255]: runtime.gcBgMarkWorker() Jan 14 20:30:07 tyrannosaurus ollama[29222]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 Jan 14 20:30:07 tyrannosaurus ollama[29222]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00061ffe8 sp>Jan 14 20:30:07 tyrannosaurus ollama[29222]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:30:07 tyrannosaurus ollama[29222]: goroutine 21 [GC worker (idle)]: Jan 14 20:30:07 tyrannosaurus ollama[29222]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:30:07 tyrannosaurus ollama[29222]: runtime.gopark(0x12411ddc63c?, 0x3?, 0xc8?, 0x60?, 0x0?) Jan 14 20:30:07 tyrannosaurus ollama[29222]: /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc000621950 sp=>Jan 14 20:30:07 tyrannosaurus ollama[29222]: runtime.goexit() Jan 14 20:38:48 tyrannosaurus ollama[39798]: llama_model_loader: - tensor 148: blk.16.ffn_up.weight q4_0 >Jan 14 20:38:48 tyrannosaurus ollama[39798]: llama_model_loader: - type q4_0: 225 tensors Jan 14 20:38:49 tyrannosaurus ollama[40058]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:38:49 tyrannosaurus ollama[40058]: runtime.gopark(0x19dd1f8e74a?, 0x3?, 0x82?, 0x0?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[40058]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000517f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[40058]: runtime.gcBgMarkWorker() Jan 14 20:38:49 tyrannosaurus ollama[40058]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000512fe0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: net/http.serverHandler.ServeHTTP({0x1783ab80?}, {0x1783c860?, 0xc00044e2a0>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gcenable.func2() Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000070fe8 sp>Jan 14 20:38:49 tyrannosaurus ollama[39798]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.gopark(0x19dd1f847ad?, 0x3?, 0xfe?, 0xce?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005207e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/net/http/server.go:679 +0xba Jan 14 20:59:25 tyrannosaurus ollama[40136]: llama_model_loader: - tensor 63: blk.6.ffn_gate.weight q4_0 >Jan 14 20:59:25 tyrannosaurus ollama[40136]: llama_model_loader: - tensor 203: output.weight q6_K >Jan 14 20:59:25 tyrannosaurus ollama[40136]: llm_load_print_meta: n_ff = 14336 Jan 14 20:59:26 tyrannosaurus ollama[62986]: /usr/local/go/src/net/http/server.go:3086 +0x28 fp=0xc000133fe0 sp>Jan 14 20:59:26 tyrannosaurus ollama[62986]: /usr/local/go/src/runtime/proc.go:404 Jan 14 20:59:26 tyrannosaurus ollama[62986]: runtime.goexit() Jan 14 20:59:26 tyrannosaurus ollama[62986]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc00050a7e0 sp=0xc>Jan 14 20:59:26 tyrannosaurus ollama[62986]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:59:26 tyrannosaurus ollama[62986]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:59:26 tyrannosaurus ollama[62986]: /usr/local/go/src/runtime/mgc.go:1217 +0x1c Jan 14 20:59:26 tyrannosaurus ollama[62986]: rbx 0x7f47173d2640 Jan 14 20:59:26 tyrannosaurus ollama[40136]: /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:93 +0x5>Jan 14 20:59:26 tyrannosaurus ollama[40136]: /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc00011f>Jan 14 20:59:26 tyrannosaurus ollama[40136]: created by runtime.init.6 in goroutine 1 Jan 14 20:59:26 tyrannosaurus ollama[40136]: runtime.gcenable.func1() Jan 14 20:59:26 tyrannosaurus ollama[40136]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000070f50 sp=0xc>Jan 14 20:59:26 tyrannosaurus ollama[40136]: runtime.gcBgMarkWorker() Jan 14 20:59:26 tyrannosaurus ollama[40136]: goroutine 11 [GC worker (idle)]: Jan 14 20:59:26 tyrannosaurus ollama[40136]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc000504fe0 sp=0xc>Jan 14 20:59:26 tyrannosaurus ollama[40136]: runtime.goexit() Jan 14 20:59:26 tyrannosaurus ollama[40136]: runtime.goexit() Jan 14 20:59:26 tyrannosaurus ollama[40136]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:59:26 tyrannosaurus ollama[40136]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:59:26 tyrannosaurus ollama[62986]: /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000505f50 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[40058]: r12 0x6 Jan 14 20:38:49 tyrannosaurus ollama[39798]: _cgo_gotypes.go:287 +0x45 fp=0xc00013c6a0 sp=0xc00013c678 pc=0x7cd>Jan 14 20:38:49 tyrannosaurus ollama[39798]: runtime.chanrecv1(0x0?, 0x0?) Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/mgc.go:1293 +0xe5 fp=0xc0005897e0 sp=0xc>Jan 14 20:38:49 tyrannosaurus ollama[39798]: /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00058afe8 sp>Jan 14 20:59:25 tyrannosaurus ollama[40136]: Device 0: NVIDIA GeForce GTX 745, compute capability 5.0 Jan 14 20:59:25 tyrannosaurus ollama[40136]: llama_model_loader: - tensor 159: blk.17.attn_k.weight q4_0 >Jan 14 20:59:25 tyrannosaurus ollama[40136]: llm_load_print_meta: n_vocab = 32000 Jan 14 20:59:26 tyrannosaurus ollama[62986]: internal/poll.(*pollDesc).waitRead(...) Jan 14 20:59:26 tyrannosaurus ollama[62986]: goroutine 4 [GC scavenge wait]: Jan 14 20:59:26 tyrannosaurus ollama[62986]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Jan 14 20:59:26 tyrannosaurus ollama[62986]: goroutine 39 [GC worker (idle)]: Jan 14 20:59:26 tyrannosaurus ollama[40136]: current device: 0 Jan 14 20:59:26 tyrannosaurus ollama[40136]: Lazy loading /tmp/ollama2596731661/cuda/libext_server.so library Jan 14 20:59:26 tyrannosaurus ollama[40136]: GGML_ASSERT: /go/src/github.com/jmorganca/ollama/llm/llama.cpp/ggml-cuda.c>Jan 14 20:59:26 tyrannosaurus ollama[62986]: SIGABRT: abort Jan 14 20:59:26 tyrannosaurus ollama[62986]: PC=0x7f475f0049fc m=4 sigcode=18446744073709551610 Jan 14 20:59:26 tyrannosaurus ollama[62986]: signal arrived during cgo execution Jan 14 20:59:26 tyrannosaurus ollama[62986]: goroutine 19 [syscall]: Jan 14 20:59:26 tyrannosaurus ollama[62986]: runtime.cgocall(0x9c1470, 0xc0001326a0) Jan 14 20:59:26 tyrannosaurus ollama[62986]: /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc000132678 sp=>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm._Cfunc_dynamic_shim_llama_server_init({0x7>Jan 14 20:59:26 tyrannosaurus ollama[62986]: _cgo_gotypes.go:287 +0x45 fp=0xc0001326a0 sp=0xc000132678 pc=0x7cd>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init.func1(0>Jan 14 20:59:26 tyrannosaurus ollama[62986]: /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:40 +0xe>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.(*shimExtServer).llama_server_init(0xc0000>Jan 14 20:59:26 tyrannosaurus ollama[62986]: /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:40 +0x1>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.newExtServer({0x17842518, 0xc000468b40}, {>Jan 14 20:59:26 tyrannosaurus ollama[62986]: /go/src/github.com/jmorganca/ollama/llm/ext_server_common.go:146 +>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.newDynamicShimExtServer({0xc000716000, 0x2>Jan 14 20:59:26 tyrannosaurus ollama[62986]: /go/src/github.com/jmorganca/ollama/llm/shim_ext_server.go:93 +0x5>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.newLlmServer({0xc3d801, 0x4}, {0xc0000e615>Jan 14 20:59:26 tyrannosaurus ollama[62986]: /go/src/github.com/jmorganca/ollama/llm/llm.go:86 +0x16b fp=0xc000>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/llm.New({0xc0004aa180?, 0x0?}, {0xc0000e6150, >Jan 14 20:59:26 tyrannosaurus ollama[62986]: /go/src/github.com/jmorganca/ollama/llm/llm.go:76 +0x233 fp=0xc000>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/server.load(0xc000002000?, 0xc000002000, {{0x0>Jan 14 20:59:26 tyrannosaurus ollama[62986]: /go/src/github.com/jmorganca/ollama/server/routes.go:84 +0x425 fp=>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/jmorganca/ollama/server.ChatHandler(0xc000486600) Jan 14 20:59:26 tyrannosaurus ollama[62986]: /go/src/github.com/jmorganca/ollama/server/routes.go:1057 +0x828 f>Jan 14 20:59:26 tyrannosaurus ollama[62986]: github.com/gin-gonic/gin.(*Context).Next(...) ```

GiteaMirror commented

2025-11-12 10:02:31 -06:00

@Maxwelldoug commented on GitHub (Jan 14, 2024):

y'know what, here, just have the entire output.
https://mmacneill.xyz/assets/ollama-jctl.log

@Maxwelldoug commented on GitHub (Jan 14, 2024): y'know what, here, just have the entire output. https://mmacneill.xyz/assets/ollama-jctl.log

GiteaMirror commented

2025-11-12 10:02:31 -06:00

@dhiltgen commented on GitHub (Jan 26, 2024):

You have an older GPU, which is not currently supported, and that is tracked under issue #1865

In version 0.1.18 we didn't detect the older card correctly, and attempted to run in GPU mode, and that resulted in the crash. If you upgrade to a newer Ollama (ideally 0.1.22 we just shipped) we will correctly detect this GPU is unsupported, and fallback to CPU mode. We do hope to add support for these older GPUs in a future release, which you can track in ticket #1865

@dhiltgen commented on GitHub (Jan 26, 2024): You have an older GPU, which is not currently supported, and that is tracked under issue #1865 In version 0.1.18 we didn't detect the older card correctly, and attempted to run in GPU mode, and that resulted in the crash. If you upgrade to a newer Ollama (ideally 0.1.22 we just shipped) we will correctly detect this GPU is unsupported, and fallback to CPU mode. We do hope to add support for these older GPUs in a future release, which you can track in ticket #1865

GiteaMirror referenced this issue

2025-11-12 15:01:42 -06:00

[PR #1151] [MERGED] install: fix enable contrib on debian 12 #9221

Sign in to join this conversation.

Branches Tags

main

pdevine/manifest-list

hoyyeva/launch-page-update

codex/fix-codex-model-metadata-warning

hoyyeva/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

hoyyeva/opencode-thinking

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

brucemacd/download-before-remove

parth/update-claude-docs

parth-anthropic-reference-images-path

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama-ollama#1151