github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #8972] GPU Offload Non AVX/AVX2 #52335

New Issue
Closed
opened 2026-04-28 23:04:03 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-28 23:04:03 -05:00
Owner
Copy Link

Originally created by @tris203 on GitHub (Feb 9, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8972

What is the issue?

Previously, GPU offloading didnt work on non AVX/AVX2 processors.

I thought that following #8539 this should now work and offload to GPU.

But running ollama run tinylama causes a crash

Relevant log output

[GIN] 2025/02/09 - 14:21:18 | 200 |      41.084µs |       127.0.0.1 | HEAD     "/"
[GIN] 2025/02/09 - 14:21:18 | 200 |   29.578114ms |       127.0.0.1 | POST     "/api/show"
time=2025-02-09T14:21:19.258Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 parallel=4 available=4179099648 required="1.7 GiB"
time=2025-02-09T14:21:19.465Z level=INFO source=server.go:100 msg="system memory" total="23.5 GiB" free="15.8 GiB" free_swap="0 B"
time=2025-02-09T14:21:19.466Z level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=23 layers.offload=23 layers.split="" memory.available="[3.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="1.7 GiB" memory.required.partial="1.7 GiB" memory.required.kv="176.0 MiB" memory.required.allocations="[1.7 GiB]" memory.weights.total="696.1 MiB" memory.weights.repeating="644.8 MiB" memory.weights.nonrepeating="51.3 MiB" memory.graph.full="544.0 MiB" memory.graph.partial="546.3 MiB"
time=2025-02-09T14:21:19.466Z level=INFO source=server.go:381 msg="starting llama server" cmd="/usr/bin/ollama runner --model /models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 --ctx-size 8192 --batch-size 512 --n-gpu-layers 23 --threads 12 --parallel 4 --port 35123"
time=2025-02-09T14:21:19.466Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-02-09T14:21:19.466Z level=INFO source=server.go:558 msg="waiting for llama runner to start responding"
time=2025-02-09T14:21:19.467Z level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error"
time=2025-02-09T14:21:19.489Z level=INFO source=runner.go:936 msg="starting go runner"
time=2025-02-09T14:21:19.489Z level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=12
time=2025-02-09T14:21:19.490Z level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:35123"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1050 Ti, compute capability 6.1, VMM: yes
load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so
SIGILL: illegal instruction
PC=0x150e1017de80 m=8 sigcode=2
signal arrived during cgo execution
instruction bytes: 0xc5 0xf9 0x6f 0x5 0x38 0x15 0x6 0x0 0xc5 0xf9 0x7f 0x5 0xe0 0xdb 0xa 0x0

goroutine 11 gp=0xc000504c40 m=8 mp=0xc000580008 [syscall]:
runtime.cgocall(0x558be5189020, 0xc0000a3a48)
        runtime/cgocall.go:167 +0x4b fp=0xc0000a3a20 sp=0xc0000a39e8 pc=0x558be449caab
github.com/ollama/ollama/ml/backend/ggml/ggml/src._Cfunc_ggml_backend_load_all_from_path(0x558be5ef5a20)
        _cgo_gotypes.go:108 +0x45 fp=0xc0000a3a48 sp=0xc0000a3a20 pc=0x558be48506a5
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.func1.1({0xc000040070, 0xf})
        github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:87 +0xf5 fp=0xc0000a3ae0 sp=0xc0000a3a48 pc=0x558be48503f5
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.func1()
        github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:88 +0x35a fp=0xc0000a3cc0 sp=0xc0000a3ae0 pc=0x558be48502ba
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func2()
        sync/oncefunc.go:27 +0x62 fp=0xc0000a3d08 sp=0xc0000a3cc0 pc=0x558be484fe82
sync.(*Once).doSlow(0x0?, 0x0?)
        sync/once.go:76 +0xb4 fp=0xc0000a3d68 sp=0xc0000a3d08 pc=0x558be44b8954
sync.(*Once).Do(0x0?, 0x0?)
        sync/once.go:67 +0x19 fp=0xc0000a3d88 sp=0xc0000a3d68 pc=0x558be44b8879
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func3()
        sync/oncefunc.go:32 +0x2d fp=0xc0000a3db8 sp=0xc0000a3d88 pc=0x558be484fded
github.com/ollama/ollama/llama.BackendInit()
        github.com/ollama/ollama/llama/llama.go:57 +0x16 fp=0xc0000a3dc8 sp=0xc0000a3db8 pc=0x558be4853af6
github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc00019f560, {0x17, 0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0xc0004093c0, 0x0}, ...)
        github.com/ollama/ollama/llama/runner/runner.go:847 +0x6a fp=0xc0000a3f10 sp=0xc0000a3dc8 pc=0x558be48627ca
github.com/ollama/ollama/llama/runner.Execute.gowrap1()
        github.com/ollama/ollama/llama/runner/runner.go:970 +0xda fp=0xc0000a3fe0 sp=0xc0000a3f10 pc=0x558be486417a
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a3fe8 sp=0xc0000a3fe0 pc=0x558be44ab581
created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
        github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d

goroutine 1 gp=0xc0000061c0 m=nil [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc0004955e8 sp=0xc0004955c8 pc=0x558be44a31ae
runtime.netpollblock(0x10?, 0xe4439fc6?, 0x8b?)
        runtime/netpoll.go:575 +0xf7 fp=0xc000495620 sp=0xc0004955e8 pc=0x558be4466e17
internal/poll.runtime_pollWait(0x150e133e0df0, 0x72)
        runtime/netpoll.go:351 +0x85 fp=0xc000495640 sp=0xc000495620 pc=0x558be44a24a5
internal/poll.(*pollDesc).wait(0xc0004aaf80?, 0x10?, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000495668 sp=0xc000495640 pc=0x558be4529c67
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0004aaf80)
        internal/poll/fd_unix.go:620 +0x295 fp=0xc000495710 sp=0xc000495668 pc=0x558be452f035
net.(*netFD).accept(0xc0004aaf80)
        net/fd_unix.go:172 +0x29 fp=0xc0004957c8 sp=0xc000495710 pc=0x558be4598129
net.(*TCPListener).accept(0xc0004af140)
        net/tcpsock_posix.go:159 +0x1e fp=0xc000495818 sp=0xc0004957c8 pc=0x558be45add9e
net.(*TCPListener).Accept(0xc0004af140)
        net/tcpsock.go:372 +0x30 fp=0xc000495848 sp=0xc000495818 pc=0x558be45acc50
net/http.(*onceCloseListener).Accept(0x558be566ff30?)
        <autogenerated>:1 +0x24 fp=0xc000495860 sp=0xc000495848 pc=0x558be47f6ec4
net/http.(*Server).Serve(0xc00052ee10, {0x558be566dc50, 0xc0004af140})
        net/http/server.go:3330 +0x30c fp=0xc000495990 sp=0xc000495860 pc=0x558be47cee4c
github.com/ollama/ollama/llama/runner.Execute({0xc000136010?, 0x0?, 0x0?})
        github.com/ollama/ollama/llama/runner/runner.go:996 +0x11a9 fp=0xc000495d30 sp=0xc000495990 pc=0x558be4863d49
github.com/ollama/ollama/cmd.NewCLI.func2(0xc000036f00?, {0x558be523001a?, 0x4?, 0x558be523001e?})
        github.com/ollama/ollama/cmd/cmd.go:1277 +0x45 fp=0xc000495d58 sp=0xc000495d30 pc=0x558be5043685
github.com/spf13/cobra.(*Command).execute(0xc0000fdb08, {0xc0001a6d20, 0xe, 0xe})
        github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 fp=0xc000495e78 sp=0xc000495d58 pc=0x558be4610e62
github.com/spf13/cobra.(*Command).ExecuteC(0xc0005bd508)
        github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc000495f30 sp=0xc000495e78 pc=0x558be46116a5
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
        github.com/ollama/ollama/main.go:12 +0x4d fp=0xc000495f50 sp=0xc000495f30 pc=0x558be5043a0d
runtime.main()
        runtime/proc.go:272 +0x29d fp=0xc000495fe0 sp=0xc000495f50 pc=0x558be446e4bd
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000495fe8 sp=0xc000495fe0 pc=0x558be44ab581

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008efa8 sp=0xc00008ef88 pc=0x558be44a31ae
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.forcegchelper()
        runtime/proc.go:337 +0xb8 fp=0xc00008efe0 sp=0xc00008efa8 pc=0x558be446e7f8
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008efe8 sp=0xc00008efe0 pc=0x558be44ab581
created by runtime.init.7 in goroutine 1
        runtime/proc.go:325 +0x1a

goroutine 18 gp=0xc000104380 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008a780 sp=0xc00008a760 pc=0x558be44a31ae
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.bgsweep(0xc000112000)
        runtime/mgcsweep.go:317 +0xdf fp=0xc00008a7c8 sp=0xc00008a780 pc=0x558be4458e9f
runtime.gcenable.gowrap1()
        runtime/mgc.go:204 +0x25 fp=0xc00008a7e0 sp=0xc00008a7c8 pc=0x558be444d4e5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008a7e8 sp=0xc00008a7e0 pc=0x558be44ab581
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0x66

goroutine 19 gp=0xc000104540 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x558be53d6b90?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008af78 sp=0xc00008af58 pc=0x558be44a31ae
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.(*scavengerState).park(0x558be5e1c820)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc00008afa8 sp=0xc00008af78 pc=0x558be4456869
runtime.bgscavenge(0xc000112000)
        runtime/mgcscavenge.go:658 +0x59 fp=0xc00008afc8 sp=0xc00008afa8 pc=0x558be4456df9
runtime.gcenable.gowrap2()
        runtime/mgc.go:205 +0x25 fp=0xc00008afe0 sp=0xc00008afc8 pc=0x558be444d485
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x558be44ab581
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:205 +0xa5

goroutine 20 gp=0xc000104a80 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008b620 sp=0xc00008b600 pc=0x558be44a31ae
runtime.runfinq()
        runtime/mfinal.go:193 +0x107 fp=0xc00008b7e0 sp=0xc00008b620 pc=0x558be444c567
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008b7e8 sp=0xc00008b7e0 pc=0x558be44ab581
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:163 +0x3d

goroutine 21 gp=0xc000222380 m=nil [chan receive]:
runtime.gopark(0xc00008bf60?, 0x558be457f7a5?, 0x40?, 0x8?, 0x558be56818e0?)
        runtime/proc.go:424 +0xce fp=0xc00008bf18 sp=0xc00008bef8 pc=0x558be44a31ae
runtime.chanrecv(0xc00011a310, 0x0, 0x1)
        runtime/chan.go:639 +0x41c fp=0xc00008bf90 sp=0xc00008bf18 pc=0x558be443cbdc
runtime.chanrecv1(0x0?, 0x0?)
        runtime/chan.go:489 +0x12 fp=0xc00008bfb8 sp=0xc00008bf90 pc=0x558be443c792
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        runtime/mgc.go:1781
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        runtime/mgc.go:1784 +0x2f fp=0xc00008bfe0 sp=0xc00008bfb8 pc=0x558be445054f
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x558be44ab581
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        runtime/mgc.go:1779 +0x96

goroutine 22 gp=0xc000222700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008c738 sp=0xc00008c718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008c7c8 sp=0xc00008c738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008c7e0 sp=0xc00008c7c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008c7e8 sp=0xc00008c7e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 23 gp=0xc0002228c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008cf38 sp=0xc00008cf18 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008cfc8 sp=0xc00008cf38 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008cfe0 sp=0xc00008cfc8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008cfe8 sp=0xc00008cfe0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 24 gp=0xc000222a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008d738 sp=0xc00008d718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008d7c8 sp=0xc00008d738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008d7e0 sp=0xc00008d7c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008d7e8 sp=0xc00008d7e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 25 gp=0xc000222c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008df38 sp=0xc00008df18 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008dfc8 sp=0xc00008df38 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008dfe0 sp=0xc00008dfc8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008dfe8 sp=0xc00008dfe0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 26 gp=0xc000222e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000472738 sp=0xc000472718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc0004727c8 sp=0xc000472738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0004727e0 sp=0xc0004727c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0004727e8 sp=0xc0004727e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 27 gp=0xc000222fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000472f38 sp=0xc000472f18 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc000472fc8 sp=0xc000472f38 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc000472fe0 sp=0xc000472fc8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000472fe8 sp=0xc000472fe0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 28 gp=0xc000223180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000473738 sp=0xc000473718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc0004737c8 sp=0xc000473738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0004737e0 sp=0xc0004737c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0004737e8 sp=0xc0004737e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 34 gp=0xc000484000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00046e738 sp=0xc00046e718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00046e7c8 sp=0xc00046e738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00046e7e0 sp=0xc00046e7c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00046e7e8 sp=0xc00046e7e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 35 gp=0xc0004841c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00046ef38 sp=0xc00046ef18 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00046efc8 sp=0xc00046ef38 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00046efe0 sp=0xc00046efc8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00046efe8 sp=0xc00046efe0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 3 gp=0xc000007500 m=nil [GC worker (idle)]:
runtime.gopark(0x558be5e1ef20?, 0xc00008e7d0?, 0x77?, 0xe5?, 0x558be5dbee70?)
        runtime/proc.go:424 +0xce fp=0xc00008e738 sp=0xc00008e718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008e7c8 sp=0xc00008e738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008e7e0 sp=0xc00008e7c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008e7e8 sp=0xc00008e7e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 4 gp=0xc0000076c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008f738 sp=0xc00008f718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008f7c8 sp=0xc00008f738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008f7e0 sp=0xc00008f7c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008f7e8 sp=0xc00008f7e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 36 gp=0xc000484380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00046f738 sp=0xc00046f718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00046f7c8 sp=0xc00046f738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00046f7e0 sp=0xc00046f7c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00046f7e8 sp=0xc00046f7e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 29 gp=0xc000223340 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000473f38 sp=0xc000473f18 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc000473fc8 sp=0xc000473f38 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc000473fe0 sp=0xc000473fc8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000473fe8 sp=0xc000473fe0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 5 gp=0xc000007880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008ff38 sp=0xc00008ff18 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008ffc8 sp=0xc00008ff38 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008ffe0 sp=0xc00008ffc8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 6 gp=0xc000007a40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000090738 sp=0xc000090718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc0000907c8 sp=0xc000090738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0000907e0 sp=0xc0000907c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000907e8 sp=0xc0000907e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 7 gp=0xc000007c00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000090f38 sp=0xc000090f18 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc000090fc8 sp=0xc000090f38 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc000090fe0 sp=0xc000090fc8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 8 gp=0xc000007dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000091738 sp=0xc000091718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc0000917c8 sp=0xc000091738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0000917e0 sp=0xc0000917c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000917e8 sp=0xc0000917e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 37 gp=0xc000484540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00046ff38 sp=0xc00046ff18 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc00046ffc8 sp=0xc00046ff38 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00046ffe0 sp=0xc00046ffc8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00046ffe8 sp=0xc00046ffe0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 30 gp=0xc000223500 m=nil [GC worker (idle)]:
runtime.gopark(0x116e8826eef?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000474738 sp=0xc000474718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc0004747c8 sp=0xc000474738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0004747e0 sp=0xc0004747c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0004747e8 sp=0xc0004747e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 9 gp=0xc0000ca000 m=nil [GC worker (idle)]:
runtime.gopark(0x558be5ecae80?, 0x1?, 0x17?, 0xbc?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000091f38 sp=0xc000091f18 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc000091fc8 sp=0xc000091f38 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc000091fe0 sp=0xc000091fc8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 10 gp=0xc0000ca1c0 m=nil [GC worker (idle)]:
runtime.gopark(0x558be5ecae80?, 0x1?, 0x30?, 0x16?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc0000d0738 sp=0xc0000d0718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc0000d07c8 sp=0xc0000d0738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0000d07e0 sp=0xc0000d07c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d07e8 sp=0xc0000d07e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 38 gp=0xc000484700 m=nil [GC worker (idle)]:
runtime.gopark(0x558be5ecae80?, 0x1?, 0xe?, 0xab?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000470738 sp=0xc000470718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc0004707c8 sp=0xc000470738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0004707e0 sp=0xc0004707c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0004707e8 sp=0xc0004707e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 39 gp=0xc0004848c0 m=nil [GC worker (idle)]:
runtime.gopark(0x116e882a817?, 0x1?, 0x30?, 0x7e?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000470f38 sp=0xc000470f18 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc000470fc8 sp=0xc000470f38 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc000470fe0 sp=0xc000470fc8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000470fe8 sp=0xc000470fe0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 40 gp=0xc000484a80 m=nil [GC worker (idle)]:
runtime.gopark(0x116e8832f6f?, 0x1?, 0xf9?, 0x7c?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000471738 sp=0xc000471718 pc=0x558be44a31ae
runtime.gcBgMarkWorker(0xc00011b730)
        runtime/mgc.go:1412 +0xe9 fp=0xc0004717c8 sp=0xc000471738 pc=0x558be444f849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0004717e0 sp=0xc0004717c8 pc=0x558be444f725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0004717e8 sp=0xc0004717e0 pc=0x558be44ab581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 12 gp=0xc000504e00 m=nil [semacquire]:
runtime.gopark(0x0?, 0x0?, 0xe0?, 0x81?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc0000d2e18 sp=0xc0000d2df8 pc=0x558be44a31ae
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.semacquire1(0xc00019f568, 0x0, 0x1, 0x0, 0x12)
        runtime/sema.go:178 +0x22c fp=0xc0000d2e80 sp=0xc0000d2e18 pc=0x558be448156c
sync.runtime_Semacquire(0x0?)
        runtime/sema.go:71 +0x25 fp=0xc0000d2eb8 sp=0xc0000d2e80 pc=0x558be44a49c5
sync.(*WaitGroup).Wait(0x0?)
        sync/waitgroup.go:118 +0x48 fp=0xc0000d2ee0 sp=0xc0000d2eb8 pc=0x558be44b9da8
github.com/ollama/ollama/llama/runner.(*Server).run(0xc00019f560, {0x558be566ffa0, 0xc0000d69b0})
        github.com/ollama/ollama/llama/runner/runner.go:315 +0x47 fp=0xc0000d2fb8 sp=0xc0000d2ee0 pc=0x558be485efe7
github.com/ollama/ollama/llama/runner.Execute.gowrap2()
        github.com/ollama/ollama/llama/runner/runner.go:975 +0x28 fp=0xc0000d2fe0 sp=0xc0000d2fb8 pc=0x558be4864068
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d2fe8 sp=0xc0000d2fe0 pc=0x558be44ab581
created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
        github.com/ollama/ollama/llama/runner/runner.go:975 +0xde5

rax    0x0
rbx    0x150e101e6590
rcx    0x2c0
rdx    0x7ffc2a824e50
rdi    0x10
rsi    0x7ffc2a824dc8
rbp    0x10
rsp    0x150e12d436d8
r8     0x0
r9     0x1
r10    0x150e101e6e98
r11    0x206
r12    0x7ffc2a824dc8
r13    0x7ffc2a824e50
r14    0x150e101e6598
r15    0x0
rip    0x150e1017de80
rflags 0x10202
cs     0x33
fs     0x0
gs     0x0
time=2025-02-09T14:21:19.717Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: exit status 2"
[GIN] 2025/02/09 - 14:21:19 | 500 |  792.734941ms |       127.0.0.1 | POST     "/api/generate"
time=2025-02-09T14:21:24.926Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.208828438 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
time=2025-02-09T14:21:25.176Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.458397688 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
time=2025-02-09T14:21:25.426Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.708953432 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

ollama version is 0.5.8-rc12

Originally created by @tris203 on GitHub (Feb 9, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8972 ### What is the issue? Previously, GPU offloading didnt work on non AVX/AVX2 processors. I thought that following #8539 this should now work and offload to GPU. But running `ollama run tinylama` causes a crash ### Relevant log output ```shell [GIN] 2025/02/09 - 14:21:18 | 200 | 41.084µs | 127.0.0.1 | HEAD "/" [GIN] 2025/02/09 - 14:21:18 | 200 | 29.578114ms | 127.0.0.1 | POST "/api/show" time=2025-02-09T14:21:19.258Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 parallel=4 available=4179099648 required="1.7 GiB" time=2025-02-09T14:21:19.465Z level=INFO source=server.go:100 msg="system memory" total="23.5 GiB" free="15.8 GiB" free_swap="0 B" time=2025-02-09T14:21:19.466Z level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=23 layers.offload=23 layers.split="" memory.available="[3.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="1.7 GiB" memory.required.partial="1.7 GiB" memory.required.kv="176.0 MiB" memory.required.allocations="[1.7 GiB]" memory.weights.total="696.1 MiB" memory.weights.repeating="644.8 MiB" memory.weights.nonrepeating="51.3 MiB" memory.graph.full="544.0 MiB" memory.graph.partial="546.3 MiB" time=2025-02-09T14:21:19.466Z level=INFO source=server.go:381 msg="starting llama server" cmd="/usr/bin/ollama runner --model /models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 --ctx-size 8192 --batch-size 512 --n-gpu-layers 23 --threads 12 --parallel 4 --port 35123" time=2025-02-09T14:21:19.466Z level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2025-02-09T14:21:19.466Z level=INFO source=server.go:558 msg="waiting for llama runner to start responding" time=2025-02-09T14:21:19.467Z level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error" time=2025-02-09T14:21:19.489Z level=INFO source=runner.go:936 msg="starting go runner" time=2025-02-09T14:21:19.489Z level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=12 time=2025-02-09T14:21:19.490Z level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:35123" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce GTX 1050 Ti, compute capability 6.1, VMM: yes load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so SIGILL: illegal instruction PC=0x150e1017de80 m=8 sigcode=2 signal arrived during cgo execution instruction bytes: 0xc5 0xf9 0x6f 0x5 0x38 0x15 0x6 0x0 0xc5 0xf9 0x7f 0x5 0xe0 0xdb 0xa 0x0 goroutine 11 gp=0xc000504c40 m=8 mp=0xc000580008 [syscall]: runtime.cgocall(0x558be5189020, 0xc0000a3a48) runtime/cgocall.go:167 +0x4b fp=0xc0000a3a20 sp=0xc0000a39e8 pc=0x558be449caab github.com/ollama/ollama/ml/backend/ggml/ggml/src._Cfunc_ggml_backend_load_all_from_path(0x558be5ef5a20) _cgo_gotypes.go:108 +0x45 fp=0xc0000a3a48 sp=0xc0000a3a20 pc=0x558be48506a5 github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.func1.1({0xc000040070, 0xf}) github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:87 +0xf5 fp=0xc0000a3ae0 sp=0xc0000a3a48 pc=0x558be48503f5 github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.func1() github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:88 +0x35a fp=0xc0000a3cc0 sp=0xc0000a3ae0 pc=0x558be48502ba github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func2() sync/oncefunc.go:27 +0x62 fp=0xc0000a3d08 sp=0xc0000a3cc0 pc=0x558be484fe82 sync.(*Once).doSlow(0x0?, 0x0?) sync/once.go:76 +0xb4 fp=0xc0000a3d68 sp=0xc0000a3d08 pc=0x558be44b8954 sync.(*Once).Do(0x0?, 0x0?) sync/once.go:67 +0x19 fp=0xc0000a3d88 sp=0xc0000a3d68 pc=0x558be44b8879 github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func3() sync/oncefunc.go:32 +0x2d fp=0xc0000a3db8 sp=0xc0000a3d88 pc=0x558be484fded github.com/ollama/ollama/llama.BackendInit() github.com/ollama/ollama/llama/llama.go:57 +0x16 fp=0xc0000a3dc8 sp=0xc0000a3db8 pc=0x558be4853af6 github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc00019f560, {0x17, 0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0xc0004093c0, 0x0}, ...) github.com/ollama/ollama/llama/runner/runner.go:847 +0x6a fp=0xc0000a3f10 sp=0xc0000a3dc8 pc=0x558be48627ca github.com/ollama/ollama/llama/runner.Execute.gowrap1() github.com/ollama/ollama/llama/runner/runner.go:970 +0xda fp=0xc0000a3fe0 sp=0xc0000a3f10 pc=0x558be486417a runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000a3fe8 sp=0xc0000a3fe0 pc=0x558be44ab581 created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d goroutine 1 gp=0xc0000061c0 m=nil [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc0004955e8 sp=0xc0004955c8 pc=0x558be44a31ae runtime.netpollblock(0x10?, 0xe4439fc6?, 0x8b?) runtime/netpoll.go:575 +0xf7 fp=0xc000495620 sp=0xc0004955e8 pc=0x558be4466e17 internal/poll.runtime_pollWait(0x150e133e0df0, 0x72) runtime/netpoll.go:351 +0x85 fp=0xc000495640 sp=0xc000495620 pc=0x558be44a24a5 internal/poll.(*pollDesc).wait(0xc0004aaf80?, 0x10?, 0x0) internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000495668 sp=0xc000495640 pc=0x558be4529c67 internal/poll.(*pollDesc).waitRead(...) internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc0004aaf80) internal/poll/fd_unix.go:620 +0x295 fp=0xc000495710 sp=0xc000495668 pc=0x558be452f035 net.(*netFD).accept(0xc0004aaf80) net/fd_unix.go:172 +0x29 fp=0xc0004957c8 sp=0xc000495710 pc=0x558be4598129 net.(*TCPListener).accept(0xc0004af140) net/tcpsock_posix.go:159 +0x1e fp=0xc000495818 sp=0xc0004957c8 pc=0x558be45add9e net.(*TCPListener).Accept(0xc0004af140) net/tcpsock.go:372 +0x30 fp=0xc000495848 sp=0xc000495818 pc=0x558be45acc50 net/http.(*onceCloseListener).Accept(0x558be566ff30?) <autogenerated>:1 +0x24 fp=0xc000495860 sp=0xc000495848 pc=0x558be47f6ec4 net/http.(*Server).Serve(0xc00052ee10, {0x558be566dc50, 0xc0004af140}) net/http/server.go:3330 +0x30c fp=0xc000495990 sp=0xc000495860 pc=0x558be47cee4c github.com/ollama/ollama/llama/runner.Execute({0xc000136010?, 0x0?, 0x0?}) github.com/ollama/ollama/llama/runner/runner.go:996 +0x11a9 fp=0xc000495d30 sp=0xc000495990 pc=0x558be4863d49 github.com/ollama/ollama/cmd.NewCLI.func2(0xc000036f00?, {0x558be523001a?, 0x4?, 0x558be523001e?}) github.com/ollama/ollama/cmd/cmd.go:1277 +0x45 fp=0xc000495d58 sp=0xc000495d30 pc=0x558be5043685 github.com/spf13/cobra.(*Command).execute(0xc0000fdb08, {0xc0001a6d20, 0xe, 0xe}) github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 fp=0xc000495e78 sp=0xc000495d58 pc=0x558be4610e62 github.com/spf13/cobra.(*Command).ExecuteC(0xc0005bd508) github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc000495f30 sp=0xc000495e78 pc=0x558be46116a5 github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) github.com/spf13/cobra@v1.7.0/command.go:985 main.main() github.com/ollama/ollama/main.go:12 +0x4d fp=0xc000495f50 sp=0xc000495f30 pc=0x558be5043a0d runtime.main() runtime/proc.go:272 +0x29d fp=0xc000495fe0 sp=0xc000495f50 pc=0x558be446e4bd runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000495fe8 sp=0xc000495fe0 pc=0x558be44ab581 goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008efa8 sp=0xc00008ef88 pc=0x558be44a31ae runtime.goparkunlock(...) runtime/proc.go:430 runtime.forcegchelper() runtime/proc.go:337 +0xb8 fp=0xc00008efe0 sp=0xc00008efa8 pc=0x558be446e7f8 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008efe8 sp=0xc00008efe0 pc=0x558be44ab581 created by runtime.init.7 in goroutine 1 runtime/proc.go:325 +0x1a goroutine 18 gp=0xc000104380 m=nil [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008a780 sp=0xc00008a760 pc=0x558be44a31ae runtime.goparkunlock(...) runtime/proc.go:430 runtime.bgsweep(0xc000112000) runtime/mgcsweep.go:317 +0xdf fp=0xc00008a7c8 sp=0xc00008a780 pc=0x558be4458e9f runtime.gcenable.gowrap1() runtime/mgc.go:204 +0x25 fp=0xc00008a7e0 sp=0xc00008a7c8 pc=0x558be444d4e5 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008a7e8 sp=0xc00008a7e0 pc=0x558be44ab581 created by runtime.gcenable in goroutine 1 runtime/mgc.go:204 +0x66 goroutine 19 gp=0xc000104540 m=nil [GC scavenge wait]: runtime.gopark(0x10000?, 0x558be53d6b90?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008af78 sp=0xc00008af58 pc=0x558be44a31ae runtime.goparkunlock(...) runtime/proc.go:430 runtime.(*scavengerState).park(0x558be5e1c820) runtime/mgcscavenge.go:425 +0x49 fp=0xc00008afa8 sp=0xc00008af78 pc=0x558be4456869 runtime.bgscavenge(0xc000112000) runtime/mgcscavenge.go:658 +0x59 fp=0xc00008afc8 sp=0xc00008afa8 pc=0x558be4456df9 runtime.gcenable.gowrap2() runtime/mgc.go:205 +0x25 fp=0xc00008afe0 sp=0xc00008afc8 pc=0x558be444d485 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x558be44ab581 created by runtime.gcenable in goroutine 1 runtime/mgc.go:205 +0xa5 goroutine 20 gp=0xc000104a80 m=nil [finalizer wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008b620 sp=0xc00008b600 pc=0x558be44a31ae runtime.runfinq() runtime/mfinal.go:193 +0x107 fp=0xc00008b7e0 sp=0xc00008b620 pc=0x558be444c567 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008b7e8 sp=0xc00008b7e0 pc=0x558be44ab581 created by runtime.createfing in goroutine 1 runtime/mfinal.go:163 +0x3d goroutine 21 gp=0xc000222380 m=nil [chan receive]: runtime.gopark(0xc00008bf60?, 0x558be457f7a5?, 0x40?, 0x8?, 0x558be56818e0?) runtime/proc.go:424 +0xce fp=0xc00008bf18 sp=0xc00008bef8 pc=0x558be44a31ae runtime.chanrecv(0xc00011a310, 0x0, 0x1) runtime/chan.go:639 +0x41c fp=0xc00008bf90 sp=0xc00008bf18 pc=0x558be443cbdc runtime.chanrecv1(0x0?, 0x0?) runtime/chan.go:489 +0x12 fp=0xc00008bfb8 sp=0xc00008bf90 pc=0x558be443c792 runtime.unique_runtime_registerUniqueMapCleanup.func1(...) runtime/mgc.go:1781 runtime.unique_runtime_registerUniqueMapCleanup.gowrap1() runtime/mgc.go:1784 +0x2f fp=0xc00008bfe0 sp=0xc00008bfb8 pc=0x558be445054f runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x558be44ab581 created by unique.runtime_registerUniqueMapCleanup in goroutine 1 runtime/mgc.go:1779 +0x96 goroutine 22 gp=0xc000222700 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008c738 sp=0xc00008c718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00008c7c8 sp=0xc00008c738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008c7e0 sp=0xc00008c7c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008c7e8 sp=0xc00008c7e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 23 gp=0xc0002228c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008cf38 sp=0xc00008cf18 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00008cfc8 sp=0xc00008cf38 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008cfe0 sp=0xc00008cfc8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008cfe8 sp=0xc00008cfe0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 24 gp=0xc000222a80 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008d738 sp=0xc00008d718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00008d7c8 sp=0xc00008d738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008d7e0 sp=0xc00008d7c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008d7e8 sp=0xc00008d7e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 25 gp=0xc000222c40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008df38 sp=0xc00008df18 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00008dfc8 sp=0xc00008df38 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008dfe0 sp=0xc00008dfc8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008dfe8 sp=0xc00008dfe0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 26 gp=0xc000222e00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000472738 sp=0xc000472718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc0004727c8 sp=0xc000472738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0004727e0 sp=0xc0004727c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004727e8 sp=0xc0004727e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 27 gp=0xc000222fc0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000472f38 sp=0xc000472f18 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc000472fc8 sp=0xc000472f38 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc000472fe0 sp=0xc000472fc8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000472fe8 sp=0xc000472fe0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 28 gp=0xc000223180 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000473738 sp=0xc000473718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc0004737c8 sp=0xc000473738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0004737e0 sp=0xc0004737c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004737e8 sp=0xc0004737e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 34 gp=0xc000484000 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00046e738 sp=0xc00046e718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00046e7c8 sp=0xc00046e738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00046e7e0 sp=0xc00046e7c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00046e7e8 sp=0xc00046e7e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 35 gp=0xc0004841c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00046ef38 sp=0xc00046ef18 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00046efc8 sp=0xc00046ef38 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00046efe0 sp=0xc00046efc8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00046efe8 sp=0xc00046efe0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 3 gp=0xc000007500 m=nil [GC worker (idle)]: runtime.gopark(0x558be5e1ef20?, 0xc00008e7d0?, 0x77?, 0xe5?, 0x558be5dbee70?) runtime/proc.go:424 +0xce fp=0xc00008e738 sp=0xc00008e718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00008e7c8 sp=0xc00008e738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008e7e0 sp=0xc00008e7c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008e7e8 sp=0xc00008e7e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 4 gp=0xc0000076c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008f738 sp=0xc00008f718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00008f7c8 sp=0xc00008f738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008f7e0 sp=0xc00008f7c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008f7e8 sp=0xc00008f7e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 36 gp=0xc000484380 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00046f738 sp=0xc00046f718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00046f7c8 sp=0xc00046f738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00046f7e0 sp=0xc00046f7c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00046f7e8 sp=0xc00046f7e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 29 gp=0xc000223340 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000473f38 sp=0xc000473f18 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc000473fc8 sp=0xc000473f38 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc000473fe0 sp=0xc000473fc8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000473fe8 sp=0xc000473fe0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 5 gp=0xc000007880 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008ff38 sp=0xc00008ff18 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00008ffc8 sp=0xc00008ff38 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008ffe0 sp=0xc00008ffc8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 6 gp=0xc000007a40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000090738 sp=0xc000090718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc0000907c8 sp=0xc000090738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0000907e0 sp=0xc0000907c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000907e8 sp=0xc0000907e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 7 gp=0xc000007c00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000090f38 sp=0xc000090f18 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc000090fc8 sp=0xc000090f38 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc000090fe0 sp=0xc000090fc8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 8 gp=0xc000007dc0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000091738 sp=0xc000091718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc0000917c8 sp=0xc000091738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0000917e0 sp=0xc0000917c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000917e8 sp=0xc0000917e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 37 gp=0xc000484540 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00046ff38 sp=0xc00046ff18 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc00046ffc8 sp=0xc00046ff38 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00046ffe0 sp=0xc00046ffc8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00046ffe8 sp=0xc00046ffe0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 30 gp=0xc000223500 m=nil [GC worker (idle)]: runtime.gopark(0x116e8826eef?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000474738 sp=0xc000474718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc0004747c8 sp=0xc000474738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0004747e0 sp=0xc0004747c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004747e8 sp=0xc0004747e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 9 gp=0xc0000ca000 m=nil [GC worker (idle)]: runtime.gopark(0x558be5ecae80?, 0x1?, 0x17?, 0xbc?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000091f38 sp=0xc000091f18 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc000091fc8 sp=0xc000091f38 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc000091fe0 sp=0xc000091fc8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 10 gp=0xc0000ca1c0 m=nil [GC worker (idle)]: runtime.gopark(0x558be5ecae80?, 0x1?, 0x30?, 0x16?, 0x0?) runtime/proc.go:424 +0xce fp=0xc0000d0738 sp=0xc0000d0718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc0000d07c8 sp=0xc0000d0738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0000d07e0 sp=0xc0000d07c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d07e8 sp=0xc0000d07e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 38 gp=0xc000484700 m=nil [GC worker (idle)]: runtime.gopark(0x558be5ecae80?, 0x1?, 0xe?, 0xab?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000470738 sp=0xc000470718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc0004707c8 sp=0xc000470738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0004707e0 sp=0xc0004707c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004707e8 sp=0xc0004707e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 39 gp=0xc0004848c0 m=nil [GC worker (idle)]: runtime.gopark(0x116e882a817?, 0x1?, 0x30?, 0x7e?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000470f38 sp=0xc000470f18 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc000470fc8 sp=0xc000470f38 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc000470fe0 sp=0xc000470fc8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000470fe8 sp=0xc000470fe0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 40 gp=0xc000484a80 m=nil [GC worker (idle)]: runtime.gopark(0x116e8832f6f?, 0x1?, 0xf9?, 0x7c?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000471738 sp=0xc000471718 pc=0x558be44a31ae runtime.gcBgMarkWorker(0xc00011b730) runtime/mgc.go:1412 +0xe9 fp=0xc0004717c8 sp=0xc000471738 pc=0x558be444f849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0004717e0 sp=0xc0004717c8 pc=0x558be444f725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004717e8 sp=0xc0004717e0 pc=0x558be44ab581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 12 gp=0xc000504e00 m=nil [semacquire]: runtime.gopark(0x0?, 0x0?, 0xe0?, 0x81?, 0x0?) runtime/proc.go:424 +0xce fp=0xc0000d2e18 sp=0xc0000d2df8 pc=0x558be44a31ae runtime.goparkunlock(...) runtime/proc.go:430 runtime.semacquire1(0xc00019f568, 0x0, 0x1, 0x0, 0x12) runtime/sema.go:178 +0x22c fp=0xc0000d2e80 sp=0xc0000d2e18 pc=0x558be448156c sync.runtime_Semacquire(0x0?) runtime/sema.go:71 +0x25 fp=0xc0000d2eb8 sp=0xc0000d2e80 pc=0x558be44a49c5 sync.(*WaitGroup).Wait(0x0?) sync/waitgroup.go:118 +0x48 fp=0xc0000d2ee0 sp=0xc0000d2eb8 pc=0x558be44b9da8 github.com/ollama/ollama/llama/runner.(*Server).run(0xc00019f560, {0x558be566ffa0, 0xc0000d69b0}) github.com/ollama/ollama/llama/runner/runner.go:315 +0x47 fp=0xc0000d2fb8 sp=0xc0000d2ee0 pc=0x558be485efe7 github.com/ollama/ollama/llama/runner.Execute.gowrap2() github.com/ollama/ollama/llama/runner/runner.go:975 +0x28 fp=0xc0000d2fe0 sp=0xc0000d2fb8 pc=0x558be4864068 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000d2fe8 sp=0xc0000d2fe0 pc=0x558be44ab581 created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 github.com/ollama/ollama/llama/runner/runner.go:975 +0xde5 rax 0x0 rbx 0x150e101e6590 rcx 0x2c0 rdx 0x7ffc2a824e50 rdi 0x10 rsi 0x7ffc2a824dc8 rbp 0x10 rsp 0x150e12d436d8 r8 0x0 r9 0x1 r10 0x150e101e6e98 r11 0x206 r12 0x7ffc2a824dc8 r13 0x7ffc2a824e50 r14 0x150e101e6598 r15 0x0 rip 0x150e1017de80 rflags 0x10202 cs 0x33 fs 0x0 gs 0x0 time=2025-02-09T14:21:19.717Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: exit status 2" [GIN] 2025/02/09 - 14:21:19 | 500 | 792.734941ms | 127.0.0.1 | POST "/api/generate" time=2025-02-09T14:21:24.926Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.208828438 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 time=2025-02-09T14:21:25.176Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.458397688 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 time=2025-02-09T14:21:25.426Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.708953432 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version ollama version is 0.5.8-rc12
GiteaMirror added the bug label 2026-04-28 23:04:03 -05:00
GiteaMirror commented 2026-04-28 23:04:18 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 9, 2025):

Failed in ggml_backend_load_all_from_path, perhaps the same problem as https://github.com/ollama/ollama/issues/8926

<!-- gh-comment-id:2646344965 --> @rick-github commented on GitHub (Feb 9, 2025): Failed in `ggml_backend_load_all_from_path`, perhaps the same problem as https://github.com/ollama/ollama/issues/8926
GiteaMirror commented 2026-04-28 23:04:19 -05:00
Author
Owner
Copy Link

@tris203 commented on GitHub (Feb 9, 2025):

I should add, that 0.57 works fine albeit CPU only

<!-- gh-comment-id:2646566692 --> @tris203 commented on GitHub (Feb 9, 2025): I should add, that 0.57 works fine albeit CPU only
GiteaMirror commented 2026-04-28 23:04:19 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Feb 9, 2025):

@tris203 would it be possible to run ollama serve with the OLLAMA_DEBUG=1 environment variable set? This will help us debug why – working on a fix!

<!-- gh-comment-id:2646616919 --> @jmorganca commented on GitHub (Feb 9, 2025): @tris203 would it be possible to run `ollama serve` with the `OLLAMA_DEBUG=1` environment variable set? This will help us debug why – working on a fix!
GiteaMirror commented 2026-04-28 23:04:20 -05:00
Author
Owner
Copy Link

@tris203 commented on GitHub (Feb 9, 2025):

I think this is what you are looking for, i think my stdout scrollback got cutoff slightly, but let me know if you need more

debug log 
time=2025-02-09T21:59:37.882Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
time=2025-02-09T21:59:37.882Z level=INFO source=types.go:130 msg="inference compute" id=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 library=cuda variant=v12 compute=6.1 driver=12.4 name="NVIDIA GeForce GTX 1050 Ti" total="3.9 GiB" available="3.9 GiB"
[GIN] 2025/02/09 - 21:59:55 | 200 |     546.252µs |       127.0.0.1 | HEAD     "/"
[GIN] 2025/02/09 - 21:59:55 | 200 |    31.24696ms |       127.0.0.1 | POST     "/api/show"
time=2025-02-09T21:59:55.448Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.4 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:55.703Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:55.703Z level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x55a55def69c0 gpu_count=1
time=2025-02-09T21:59:55.743Z level=DEBUG source=sched.go:224 msg="loading first model" model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
time=2025-02-09T21:59:55.743Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[3.9 GiB]"
time=2025-02-09T21:59:55.743Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 parallel=4 available=4179099648 required="1.7 GiB"
time=2025-02-09T21:59:55.743Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.2 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:55.977Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:55.977Z level=INFO source=server.go:100 msg="system memory" total="23.5 GiB" free="15.2 GiB" free_swap="0 B"
time=2025-02-09T21:59:55.977Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[3.9 GiB]"
time=2025-02-09T21:59:55.978Z level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=23 layers.offload=23 layers.split="" memory.available="[3.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="1.7 GiB" memory.required.partial="1.7 GiB" memory.required.kv="176.0 MiB" memory.required.allocations="[1.7 GiB]" memory.weights.total="696.1 MiB" memory.weights.repeating="644.8 MiB" memory.weights.nonrepeating="51.3 MiB" memory.graph.full="544.0 MiB" memory.graph.partial="546.3 MiB"
time=2025-02-09T21:59:55.978Z level=DEBUG source=server.go:262 msg="compatible gpu libraries" compatible="[cuda_v12 cuda_v11]"
time=2025-02-09T21:59:55.978Z level=DEBUG source=server.go:302 msg="adding gpu library" path=/usr/lib/ollama/cuda_v12
time=2025-02-09T21:59:55.978Z level=DEBUG source=server.go:310 msg="adding gpu dependency paths" paths=[/usr/lib/ollama/cuda_v12]
time=2025-02-09T21:59:55.978Z level=INFO source=server.go:381 msg="starting llama server" cmd="/usr/bin/ollama runner --model /models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 --ctx-size 8192 --batch-size 512 --n-gpu-layers 23 --verbose --threads 12 --parallel 4 --port 38567"
time=2025-02-09T21:59:55.978Z level=DEBUG source=server.go:399 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/usr/lib/ollama/cuda_v12:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/lib/ollama/cuda_v12:/usr/lib/ollama CUDA_VISIBLE_DEVICES=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47]"
time=2025-02-09T21:59:55.979Z level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-02-09T21:59:55.979Z level=INFO source=server.go:558 msg="waiting for llama runner to start responding"
time=2025-02-09T21:59:55.979Z level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error"
time=2025-02-09T21:59:56.003Z level=INFO source=runner.go:936 msg="starting go runner"
time=2025-02-09T21:59:56.003Z level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=12
time=2025-02-09T21:59:56.003Z level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/usr/lib/ollama/cuda_v12
time=2025-02-09T21:59:56.003Z level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:38567"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1050 Ti, compute capability 6.1, VMM: yes
load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so
time=2025-02-09T21:59:56.150Z level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/usr/local/nvidia/lib
time=2025-02-09T21:59:56.150Z level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/usr/local/nvidia/lib64
time=2025-02-09T21:59:56.150Z level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/usr/lib/ollama
SIGILL: illegal instruction
PC=0x14bac2d05e80 m=4 sigcode=2
signal arrived during cgo execution
instruction bytes: 0xc5 0xf9 0x6f 0x5 0x38 0x15 0x6 0x0 0xc5 0xf9 0x7f 0x5 0xe0 0xdb 0xa 0x0

goroutine 29 gp=0xc000504700 m=4 mp=0xc000095508 [syscall]:
runtime.cgocall(0x55bd44c9d020, 0xc0000cda48)
        runtime/cgocall.go:167 +0x4b fp=0xc0000cda20 sp=0xc0000cd9e8 pc=0x55bd43fb0aab
github.com/ollama/ollama/ml/backend/ggml/ggml/src._Cfunc_ggml_backend_load_all_from_path(0x55bd45a09a20)
        _cgo_gotypes.go:108 +0x45 fp=0xc0000cda48 sp=0xc0000cda20 pc=0x55bd443646a5
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.func1.1({0xc000040070, 0xf})
        github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:87 +0xf5 fp=0xc0000cdae0 sp=0xc0000cda48 pc=0x55bd443643f5
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.func1()
        github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:88 +0x35a fp=0xc0000cdcc0 sp=0xc0000cdae0 pc=0x55bd443642ba
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func2()
        sync/oncefunc.go:27 +0x62 fp=0xc0000cdd08 sp=0xc0000cdcc0 pc=0x55bd44363e82
sync.(*Once).doSlow(0x0?, 0x0?)
        sync/once.go:76 +0xb4 fp=0xc0000cdd68 sp=0xc0000cdd08 pc=0x55bd43fcc954
sync.(*Once).Do(0x0?, 0x0?)
        sync/once.go:67 +0x19 fp=0xc0000cdd88 sp=0xc0000cdd68 pc=0x55bd43fcc879
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func3()
        sync/oncefunc.go:32 +0x2d fp=0xc0000cddb8 sp=0xc0000cdd88 pc=0x55bd44363ded
github.com/ollama/ollama/llama.BackendInit()
        github.com/ollama/ollama/llama/llama.go:57 +0x16 fp=0xc0000cddc8 sp=0xc0000cddb8 pc=0x55bd44367af6
github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc00019f560, {0x17, 0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0xc0004837c0, 0x0}, ...)
        github.com/ollama/ollama/llama/runner/runner.go:847 +0x6a fp=0xc0000cdf10 sp=0xc0000cddc8 pc=0x55bd443767ca
github.com/ollama/ollama/llama/runner.Execute.gowrap1()
        github.com/ollama/ollama/llama/runner/runner.go:970 +0xda fp=0xc0000cdfe0 sp=0xc0000cdf10 pc=0x55bd4437817a
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000cdfe8 sp=0xc0000cdfe0 pc=0x55bd43fbf581
created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
        github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d

goroutine 1 gp=0xc0000061c0 m=nil [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc0004a75e8 sp=0xc0004a75c8 pc=0x55bd43fb71ae
runtime.netpollblock(0x10?, 0x43f4dfc6?, 0xbd?)
        runtime/netpoll.go:575 +0xf7 fp=0xc0004a7620 sp=0xc0004a75e8 pc=0x55bd43f7ae17
internal/poll.runtime_pollWait(0x14bac3c6bdf0, 0x72)
        runtime/netpoll.go:351 +0x85 fp=0xc0004a7640 sp=0xc0004a7620 pc=0x55bd43fb64a5
internal/poll.(*pollDesc).wait(0xc0005a1300?, 0x10?, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0004a7668 sp=0xc0004a7640 pc=0x55bd4403dc67
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0005a1300)
        internal/poll/fd_unix.go:620 +0x295 fp=0xc0004a7710 sp=0xc0004a7668 pc=0x55bd44043035
net.(*netFD).accept(0xc0005a1300)
        net/fd_unix.go:172 +0x29 fp=0xc0004a77c8 sp=0xc0004a7710 pc=0x55bd440ac129
net.(*TCPListener).accept(0xc00051d300)
        net/tcpsock_posix.go:159 +0x1e fp=0xc0004a7818 sp=0xc0004a77c8 pc=0x55bd440c1d9e
net.(*TCPListener).Accept(0xc00051d300)
        net/tcpsock.go:372 +0x30 fp=0xc0004a7848 sp=0xc0004a7818 pc=0x55bd440c0c50
net/http.(*onceCloseListener).Accept(0x55bd45183f30?)
        <autogenerated>:1 +0x24 fp=0xc0004a7860 sp=0xc0004a7848 pc=0x55bd4430aec4
net/http.(*Server).Serve(0xc000545590, {0x55bd45181c50, 0xc00051d300})
        net/http/server.go:3330 +0x30c fp=0xc0004a7990 sp=0xc0004a7860 pc=0x55bd442e2e4c
github.com/ollama/ollama/llama/runner.Execute({0xc000136010?, 0x0?, 0x0?})
        github.com/ollama/ollama/llama/runner/runner.go:996 +0x11a9 fp=0xc0004a7d30 sp=0xc0004a7990 pc=0x55bd44377d49
github.com/ollama/ollama/cmd.NewCLI.func2(0xc000038c00?, {0x55bd44d4401a?, 0x4?, 0x55bd44d4401e?})
        github.com/ollama/ollama/cmd/cmd.go:1277 +0x45 fp=0xc0004a7d58 sp=0xc0004a7d30 pc=0x55bd44b57685
github.com/spf13/cobra.(*Command).execute(0xc000163b08, {0xc000544f00, 0xf, 0xf})
        github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 fp=0xc0004a7e78 sp=0xc0004a7d58 pc=0x55bd44124e62
github.com/spf13/cobra.(*Command).ExecuteC(0xc0005b5b08)
        github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc0004a7f30 sp=0xc0004a7e78 pc=0x55bd441256a5
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
        github.com/ollama/ollama/main.go:12 +0x4d fp=0xc0004a7f50 sp=0xc0004a7f30 pc=0x55bd44b57a0d
runtime.main()
        runtime/proc.go:272 +0x29d fp=0xc0004a7fe0 sp=0xc0004a7f50 pc=0x55bd43f824bd
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a7fe8 sp=0xc0004a7fe0 pc=0x55bd43fbf581

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008efa8 sp=0xc00008ef88 pc=0x55bd43fb71ae
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.forcegchelper()
        runtime/proc.go:337 +0xb8 fp=0xc00008efe0 sp=0xc00008efa8 pc=0x55bd43f827f8
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008efe8 sp=0xc00008efe0 pc=0x55bd43fbf581
created by runtime.init.7 in goroutine 1
        runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008f780 sp=0xc00008f760 pc=0x55bd43fb71ae
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.bgsweep(0xc000040080)
        runtime/mgcsweep.go:317 +0xdf fp=0xc00008f7c8 sp=0xc00008f780 pc=0x55bd43f6ce9f
runtime.gcenable.gowrap1()
        runtime/mgc.go:204 +0x25 fp=0xc00008f7e0 sp=0xc00008f7c8 pc=0x55bd43f614e5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008f7e8 sp=0xc00008f7e0 pc=0x55bd43fbf581
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0x66

goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x55bd44eeab90?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008ff78 sp=0xc00008ff58 pc=0x55bd43fb71ae
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.(*scavengerState).park(0x55bd45930820)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc00008ffa8 sp=0xc00008ff78 pc=0x55bd43f6a869
runtime.bgscavenge(0xc000040080)
        runtime/mgcscavenge.go:658 +0x59 fp=0xc00008ffc8 sp=0xc00008ffa8 pc=0x55bd43f6adf9
runtime.gcenable.gowrap2()
        runtime/mgc.go:205 +0x25 fp=0xc00008ffe0 sp=0xc00008ffc8 pc=0x55bd43f61485
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0x55bd43fbf581
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:205 +0xa5

goroutine 18 gp=0xc000104700 m=nil [finalizer wait]:
runtime.gopark(0xc00008e648?, 0x55bd43f579e5?, 0xb0?, 0x1?, 0xc0000061c0?)
        runtime/proc.go:424 +0xce fp=0xc00008e620 sp=0xc00008e600 pc=0x55bd43fb71ae
runtime.runfinq()
        runtime/mfinal.go:193 +0x107 fp=0xc00008e7e0 sp=0xc00008e620 pc=0x55bd43f60567
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008e7e8 sp=0xc00008e7e0 pc=0x55bd43fbf581
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:163 +0x3d

goroutine 19 gp=0xc000260000 m=nil [chan receive]:
runtime.gopark(0xc00008a760?, 0x55bd440937a5?, 0x70?, 0xa8?, 0x55bd451958e0?)
        runtime/proc.go:424 +0xce fp=0xc00008a718 sp=0xc00008a6f8 pc=0x55bd43fb71ae
runtime.chanrecv(0xc000112310, 0x0, 0x1)
        runtime/chan.go:639 +0x41c fp=0xc00008a790 sp=0xc00008a718 pc=0x55bd43f50bdc
runtime.chanrecv1(0x0?, 0x0?)
        runtime/chan.go:489 +0x12 fp=0xc00008a7b8 sp=0xc00008a790 pc=0x55bd43f50792
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        runtime/mgc.go:1781
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        runtime/mgc.go:1784 +0x2f fp=0xc00008a7e0 sp=0xc00008a7b8 pc=0x55bd43f6454f
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008a7e8 sp=0xc00008a7e0 pc=0x55bd43fbf581
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        runtime/mgc.go:1779 +0x96

goroutine 20 gp=0xc000260380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008af38 sp=0xc00008af18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008afc8 sp=0xc00008af38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008afe0 sp=0xc00008afc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 21 gp=0xc000260540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008b738 sp=0xc00008b718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008b7c8 sp=0xc00008b738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008b7e0 sp=0xc00008b7c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008b7e8 sp=0xc00008b7e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 34 gp=0xc000484000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00048a738 sp=0xc00048a718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00048a7c8 sp=0xc00048a738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00048a7e0 sp=0xc00048a7c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00048a7e8 sp=0xc00048a7e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 5 gp=0xc000007880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000090738 sp=0xc000090718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc0000907c8 sp=0xc000090738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0000907e0 sp=0xc0000907c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000907e8 sp=0xc0000907e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 6 gp=0xc000007a40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000090f38 sp=0xc000090f18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc000090fc8 sp=0xc000090f38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc000090fe0 sp=0xc000090fc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 22 gp=0xc000260700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008bf38 sp=0xc00008bf18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008bfc8 sp=0xc00008bf38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008bfe0 sp=0xc00008bfc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 35 gp=0xc0004841c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00048af38 sp=0xc00048af18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00048afc8 sp=0xc00048af38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00048afe0 sp=0xc00048afc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00048afe8 sp=0xc00048afe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 36 gp=0xc000484380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00048b738 sp=0xc00048b718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00048b7c8 sp=0xc00048b738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00048b7e0 sp=0xc00048b7c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00048b7e8 sp=0xc00048b7e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 37 gp=0xc000484540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00048bf38 sp=0xc00048bf18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00048bfc8 sp=0xc00048bf38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00048bfe0 sp=0xc00048bfc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00048bfe8 sp=0xc00048bfe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 38 gp=0xc000484700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00048c738 sp=0xc00048c718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00048c7c8 sp=0xc00048c738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00048c7e0 sp=0xc00048c7c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00048c7e8 sp=0xc00048c7e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 39 gp=0xc0004848c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00048cf38 sp=0xc00048cf18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00048cfc8 sp=0xc00048cf38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00048cfe0 sp=0xc00048cfc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00048cfe8 sp=0xc00048cfe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 23 gp=0xc0002608c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008c738 sp=0xc00008c718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008c7c8 sp=0xc00008c738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008c7e0 sp=0xc00008c7c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008c7e8 sp=0xc00008c7e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 24 gp=0xc000260a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008cf38 sp=0xc00008cf18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008cfc8 sp=0xc00008cf38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008cfe0 sp=0xc00008cfc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008cfe8 sp=0xc00008cfe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 25 gp=0xc000260c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008d738 sp=0xc00008d718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008d7c8 sp=0xc00008d738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008d7e0 sp=0xc00008d7c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008d7e8 sp=0xc00008d7e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 26 gp=0xc000260e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008df38 sp=0xc00008df18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00008dfc8 sp=0xc00008df38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00008dfe0 sp=0xc00008dfc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008dfe8 sp=0xc00008dfe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 40 gp=0xc000484a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00048d738 sp=0xc00048d718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00048d7c8 sp=0xc00048d738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00048d7e0 sp=0xc00048d7c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00048d7e8 sp=0xc00048d7e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 7 gp=0xc000007c00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000091738 sp=0xc000091718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc0000917c8 sp=0xc000091738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0000917e0 sp=0xc0000917c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000917e8 sp=0xc0000917e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 27 gp=0xc000260fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000486738 sp=0xc000486718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc0004867c8 sp=0xc000486738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0004867e0 sp=0xc0004867c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0004867e8 sp=0xc0004867e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 28 gp=0xc000261180 m=nil [GC worker (idle)]:
runtime.gopark(0x1a1d987ea5d4?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000486f38 sp=0xc000486f18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc000486fc8 sp=0xc000486f38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc000486fe0 sp=0xc000486fc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000486fe8 sp=0xc000486fe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 41 gp=0xc000484c40 m=nil [GC worker (idle)]:
runtime.gopark(0x1a1d987e6ebc?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00048df38 sp=0xc00048df18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc00048dfc8 sp=0xc00048df38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc00048dfe0 sp=0xc00048dfc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00048dfe8 sp=0xc00048dfe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 42 gp=0xc000484e00 m=nil [GC worker (idle)]:
runtime.gopark(0x55bd459dee80?, 0x1?, 0x3e?, 0xc0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000494738 sp=0xc000494718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc0004947c8 sp=0xc000494738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0004947e0 sp=0xc0004947c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0004947e8 sp=0xc0004947e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 43 gp=0xc000484fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x55bd459dee80?, 0x1?, 0x6?, 0xfe?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000494f38 sp=0xc000494f18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc000494fc8 sp=0xc000494f38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc000494fe0 sp=0xc000494fc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000494fe8 sp=0xc000494fe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 44 gp=0xc000485180 m=nil [GC worker (idle)]:
runtime.gopark(0x1a1d987e80df?, 0x1?, 0x27?, 0xbb?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000495738 sp=0xc000495718 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc0004957c8 sp=0xc000495738 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc0004957e0 sp=0xc0004957c8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0004957e8 sp=0xc0004957e0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 45 gp=0xc000485340 m=nil [GC worker (idle)]:
runtime.gopark(0x1a1d987ea238?, 0xc000480180?, 0x1a?, 0xa?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000495f38 sp=0xc000495f18 pc=0x55bd43fb71ae
runtime.gcBgMarkWorker(0xc0001138f0)
        runtime/mgc.go:1412 +0xe9 fp=0xc000495fc8 sp=0xc000495f38 pc=0x55bd43f63849
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1328 +0x25 fp=0xc000495fe0 sp=0xc000495fc8 pc=0x55bd43f63725
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000495fe8 sp=0xc000495fe0 pc=0x55bd43fbf581
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1328 +0x105

goroutine 30 gp=0xc0005048c0 m=nil [semacquire]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0xe0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000091e18 sp=0xc000091df8 pc=0x55bd43fb71ae
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.semacquire1(0xc00019f568, 0x0, 0x1, 0x0, 0x12)
        runtime/sema.go:178 +0x22c fp=0xc000091e80 sp=0xc000091e18 pc=0x55bd43f9556c
sync.runtime_Semacquire(0x0?)
        runtime/sema.go:71 +0x25 fp=0xc000091eb8 sp=0xc000091e80 pc=0x55bd43fb89c5
sync.(*WaitGroup).Wait(0x0?)
        sync/waitgroup.go:118 +0x48 fp=0xc000091ee0 sp=0xc000091eb8 pc=0x55bd43fcdda8
github.com/ollama/ollama/llama/runner.(*Server).run(0xc00019f560, {0x55bd45183fa0, 0xc000400dc0})
        github.com/ollama/ollama/llama/runner/runner.go:315 +0x47 fp=0xc000091fb8 sp=0xc000091ee0 pc=0x55bd44372fe7
github.com/ollama/ollama/llama/runner.Execute.gowrap2()
        github.com/ollama/ollama/llama/runner/runner.go:975 +0x28 fp=0xc000091fe0 sp=0xc000091fb8 pc=0x55bd44378068
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0x55bd43fbf581
created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
        github.com/ollama/ollama/llama/runner/runner.go:975 +0xde5

rax    0x0
rbx    0x14bac2d6e590
rcx    0x2c0
rdx    0x7fffc9fb0138
rdi    0x11
rsi    0x7fffc9fb00a8
rbp    0x11
rsp    0x14bac3ea26d8
r8     0x0
r9     0x1
r10    0x14bac2d6ee98
r11    0x206
r12    0x7fffc9fb00a8
r13    0x7fffc9fb0138
r14    0x14bac2d6e598
r15    0x0
rip    0x14bac2d05e80
rflags 0x10202
cs     0x33
fs     0x0
gs     0x0
time=2025-02-09T21:59:56.230Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: exit status 2"
time=2025-02-09T21:59:56.230Z level=DEBUG source=sched.go:458 msg="triggering expiration for failed load" model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
time=2025-02-09T21:59:56.230Z level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
time=2025-02-09T21:59:56.230Z level=DEBUG source=sched.go:375 msg="got lock to unload" modelPath=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
time=2025-02-09T21:59:56.230Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.2 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.2 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
[GIN] 2025/02/09 - 21:59:56 | 500 |  820.363963ms |       127.0.0.1 | POST     "/api/generate"
time=2025-02-09T21:59:56.469Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:56.469Z level=DEBUG source=server.go:1082 msg="stopping llama server"
time=2025-02-09T21:59:56.469Z level=DEBUG source=sched.go:380 msg="runner released" modelPath=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
time=2025-02-09T21:59:56.720Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.2 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.2 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:56.951Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:56.970Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.2 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.2 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:57.227Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:57.227Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.2 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.2 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:57.456Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:57.469Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.2 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:57.731Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:57.731Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:57.979Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:57.979Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:58.216Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:58.220Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:58.456Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:58.469Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:58.714Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:58.720Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:58.954Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:58.969Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:59.202Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:59.220Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:59.455Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:59.470Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:59.695Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:59.720Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T21:59:59.948Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T21:59:59.970Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T22:00:00.206Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T22:00:00.220Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T22:00:00.452Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T22:00:00.470Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T22:00:00.705Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T22:00:00.720Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T22:00:00.944Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T22:00:00.970Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T22:00:01.207Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T22:00:01.220Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T22:00:01.448Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T22:00:01.469Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.238721931 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
time=2025-02-09T22:00:01.469Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T22:00:01.469Z level=DEBUG source=sched.go:384 msg="sending an unloaded event" modelPath=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
time=2025-02-09T22:00:01.469Z level=DEBUG source=sched.go:308 msg="ignoring unload event with no pending requests"
time=2025-02-09T22:00:01.704Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T22:00:01.720Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.489264324 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
time=2025-02-09T22:00:01.720Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
dlsym: cuInit - 0x14e2a8813170
dlsym: cuDriverGetVersion - 0x14e2a8813190
dlsym: cuDeviceGetCount - 0x14e2a88131d0
dlsym: cuDeviceGet - 0x14e2a88131b0
dlsym: cuDeviceGetAttribute - 0x14e2a88132b0
dlsym: cuDeviceGetUuid - 0x14e2a8813210
dlsym: cuDeviceGetName - 0x14e2a88131f0
dlsym: cuCtxCreate_v3 - 0x14e2a8813490
dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0
dlsym: cuCtxDestroy - 0x14e2a88782e0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f08
CUDA driver version: 12.4
calling cuDeviceGetCount
device count 1
time=2025-02-09T22:00:01.946Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB"
releasing cuda driver library
time=2025-02-09T22:00:01.969Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.738913206 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816
<!-- gh-comment-id:2646621253 --> @tris203 commented on GitHub (Feb 9, 2025): I think this is what you are looking for, i think my stdout scrollback got cutoff slightly, but let me know if you need more <details> <summary>debug log</summary> ``` time=2025-02-09T21:59:37.882Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2025-02-09T21:59:37.882Z level=INFO source=types.go:130 msg="inference compute" id=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 library=cuda variant=v12 compute=6.1 driver=12.4 name="NVIDIA GeForce GTX 1050 Ti" total="3.9 GiB" available="3.9 GiB" [GIN] 2025/02/09 - 21:59:55 | 200 | 546.252µs | 127.0.0.1 | HEAD "/" [GIN] 2025/02/09 - 21:59:55 | 200 | 31.24696ms | 127.0.0.1 | POST "/api/show" time=2025-02-09T21:59:55.448Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.4 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:55.703Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:55.703Z level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x55a55def69c0 gpu_count=1 time=2025-02-09T21:59:55.743Z level=DEBUG source=sched.go:224 msg="loading first model" model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 time=2025-02-09T21:59:55.743Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[3.9 GiB]" time=2025-02-09T21:59:55.743Z level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 parallel=4 available=4179099648 required="1.7 GiB" time=2025-02-09T21:59:55.743Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.2 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:55.977Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:55.977Z level=INFO source=server.go:100 msg="system memory" total="23.5 GiB" free="15.2 GiB" free_swap="0 B" time=2025-02-09T21:59:55.977Z level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[3.9 GiB]" time=2025-02-09T21:59:55.978Z level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=23 layers.offload=23 layers.split="" memory.available="[3.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="1.7 GiB" memory.required.partial="1.7 GiB" memory.required.kv="176.0 MiB" memory.required.allocations="[1.7 GiB]" memory.weights.total="696.1 MiB" memory.weights.repeating="644.8 MiB" memory.weights.nonrepeating="51.3 MiB" memory.graph.full="544.0 MiB" memory.graph.partial="546.3 MiB" time=2025-02-09T21:59:55.978Z level=DEBUG source=server.go:262 msg="compatible gpu libraries" compatible="[cuda_v12 cuda_v11]" time=2025-02-09T21:59:55.978Z level=DEBUG source=server.go:302 msg="adding gpu library" path=/usr/lib/ollama/cuda_v12 time=2025-02-09T21:59:55.978Z level=DEBUG source=server.go:310 msg="adding gpu dependency paths" paths=[/usr/lib/ollama/cuda_v12] time=2025-02-09T21:59:55.978Z level=INFO source=server.go:381 msg="starting llama server" cmd="/usr/bin/ollama runner --model /models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 --ctx-size 8192 --batch-size 512 --n-gpu-layers 23 --verbose --threads 12 --parallel 4 --port 38567" time=2025-02-09T21:59:55.978Z level=DEBUG source=server.go:399 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/usr/lib/ollama/cuda_v12:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/lib/ollama/cuda_v12:/usr/lib/ollama CUDA_VISIBLE_DEVICES=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47]" time=2025-02-09T21:59:55.979Z level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2025-02-09T21:59:55.979Z level=INFO source=server.go:558 msg="waiting for llama runner to start responding" time=2025-02-09T21:59:55.979Z level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error" time=2025-02-09T21:59:56.003Z level=INFO source=runner.go:936 msg="starting go runner" time=2025-02-09T21:59:56.003Z level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=12 time=2025-02-09T21:59:56.003Z level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/usr/lib/ollama/cuda_v12 time=2025-02-09T21:59:56.003Z level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:38567" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce GTX 1050 Ti, compute capability 6.1, VMM: yes load_backend: loaded CUDA backend from /usr/lib/ollama/cuda_v12/libggml-cuda.so time=2025-02-09T21:59:56.150Z level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/usr/local/nvidia/lib time=2025-02-09T21:59:56.150Z level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/usr/local/nvidia/lib64 time=2025-02-09T21:59:56.150Z level=DEBUG source=ggml.go:84 msg="ggml backend load all from path" path=/usr/lib/ollama SIGILL: illegal instruction PC=0x14bac2d05e80 m=4 sigcode=2 signal arrived during cgo execution instruction bytes: 0xc5 0xf9 0x6f 0x5 0x38 0x15 0x6 0x0 0xc5 0xf9 0x7f 0x5 0xe0 0xdb 0xa 0x0 goroutine 29 gp=0xc000504700 m=4 mp=0xc000095508 [syscall]: runtime.cgocall(0x55bd44c9d020, 0xc0000cda48) runtime/cgocall.go:167 +0x4b fp=0xc0000cda20 sp=0xc0000cd9e8 pc=0x55bd43fb0aab github.com/ollama/ollama/ml/backend/ggml/ggml/src._Cfunc_ggml_backend_load_all_from_path(0x55bd45a09a20) _cgo_gotypes.go:108 +0x45 fp=0xc0000cda48 sp=0xc0000cda20 pc=0x55bd443646a5 github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.func1.1({0xc000040070, 0xf}) github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:87 +0xf5 fp=0xc0000cdae0 sp=0xc0000cda48 pc=0x55bd443643f5 github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.func1() github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:88 +0x35a fp=0xc0000cdcc0 sp=0xc0000cdae0 pc=0x55bd443642ba github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func2() sync/oncefunc.go:27 +0x62 fp=0xc0000cdd08 sp=0xc0000cdcc0 pc=0x55bd44363e82 sync.(*Once).doSlow(0x0?, 0x0?) sync/once.go:76 +0xb4 fp=0xc0000cdd68 sp=0xc0000cdd08 pc=0x55bd43fcc954 sync.(*Once).Do(0x0?, 0x0?) sync/once.go:67 +0x19 fp=0xc0000cdd88 sp=0xc0000cdd68 pc=0x55bd43fcc879 github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func3() sync/oncefunc.go:32 +0x2d fp=0xc0000cddb8 sp=0xc0000cdd88 pc=0x55bd44363ded github.com/ollama/ollama/llama.BackendInit() github.com/ollama/ollama/llama/llama.go:57 +0x16 fp=0xc0000cddc8 sp=0xc0000cddb8 pc=0x55bd44367af6 github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc00019f560, {0x17, 0x0, 0x1, 0x0, {0x0, 0x0, 0x0}, 0xc0004837c0, 0x0}, ...) github.com/ollama/ollama/llama/runner/runner.go:847 +0x6a fp=0xc0000cdf10 sp=0xc0000cddc8 pc=0x55bd443767ca github.com/ollama/ollama/llama/runner.Execute.gowrap1() github.com/ollama/ollama/llama/runner/runner.go:970 +0xda fp=0xc0000cdfe0 sp=0xc0000cdf10 pc=0x55bd4437817a runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000cdfe8 sp=0xc0000cdfe0 pc=0x55bd43fbf581 created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d goroutine 1 gp=0xc0000061c0 m=nil [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc0004a75e8 sp=0xc0004a75c8 pc=0x55bd43fb71ae runtime.netpollblock(0x10?, 0x43f4dfc6?, 0xbd?) runtime/netpoll.go:575 +0xf7 fp=0xc0004a7620 sp=0xc0004a75e8 pc=0x55bd43f7ae17 internal/poll.runtime_pollWait(0x14bac3c6bdf0, 0x72) runtime/netpoll.go:351 +0x85 fp=0xc0004a7640 sp=0xc0004a7620 pc=0x55bd43fb64a5 internal/poll.(*pollDesc).wait(0xc0005a1300?, 0x10?, 0x0) internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0004a7668 sp=0xc0004a7640 pc=0x55bd4403dc67 internal/poll.(*pollDesc).waitRead(...) internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc0005a1300) internal/poll/fd_unix.go:620 +0x295 fp=0xc0004a7710 sp=0xc0004a7668 pc=0x55bd44043035 net.(*netFD).accept(0xc0005a1300) net/fd_unix.go:172 +0x29 fp=0xc0004a77c8 sp=0xc0004a7710 pc=0x55bd440ac129 net.(*TCPListener).accept(0xc00051d300) net/tcpsock_posix.go:159 +0x1e fp=0xc0004a7818 sp=0xc0004a77c8 pc=0x55bd440c1d9e net.(*TCPListener).Accept(0xc00051d300) net/tcpsock.go:372 +0x30 fp=0xc0004a7848 sp=0xc0004a7818 pc=0x55bd440c0c50 net/http.(*onceCloseListener).Accept(0x55bd45183f30?) <autogenerated>:1 +0x24 fp=0xc0004a7860 sp=0xc0004a7848 pc=0x55bd4430aec4 net/http.(*Server).Serve(0xc000545590, {0x55bd45181c50, 0xc00051d300}) net/http/server.go:3330 +0x30c fp=0xc0004a7990 sp=0xc0004a7860 pc=0x55bd442e2e4c github.com/ollama/ollama/llama/runner.Execute({0xc000136010?, 0x0?, 0x0?}) github.com/ollama/ollama/llama/runner/runner.go:996 +0x11a9 fp=0xc0004a7d30 sp=0xc0004a7990 pc=0x55bd44377d49 github.com/ollama/ollama/cmd.NewCLI.func2(0xc000038c00?, {0x55bd44d4401a?, 0x4?, 0x55bd44d4401e?}) github.com/ollama/ollama/cmd/cmd.go:1277 +0x45 fp=0xc0004a7d58 sp=0xc0004a7d30 pc=0x55bd44b57685 github.com/spf13/cobra.(*Command).execute(0xc000163b08, {0xc000544f00, 0xf, 0xf}) github.com/spf13/cobra@v1.7.0/command.go:940 +0x862 fp=0xc0004a7e78 sp=0xc0004a7d58 pc=0x55bd44124e62 github.com/spf13/cobra.(*Command).ExecuteC(0xc0005b5b08) github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc0004a7f30 sp=0xc0004a7e78 pc=0x55bd441256a5 github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) github.com/spf13/cobra@v1.7.0/command.go:985 main.main() github.com/ollama/ollama/main.go:12 +0x4d fp=0xc0004a7f50 sp=0xc0004a7f30 pc=0x55bd44b57a0d runtime.main() runtime/proc.go:272 +0x29d fp=0xc0004a7fe0 sp=0xc0004a7f50 pc=0x55bd43f824bd runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a7fe8 sp=0xc0004a7fe0 pc=0x55bd43fbf581 goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008efa8 sp=0xc00008ef88 pc=0x55bd43fb71ae runtime.goparkunlock(...) runtime/proc.go:430 runtime.forcegchelper() runtime/proc.go:337 +0xb8 fp=0xc00008efe0 sp=0xc00008efa8 pc=0x55bd43f827f8 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008efe8 sp=0xc00008efe0 pc=0x55bd43fbf581 created by runtime.init.7 in goroutine 1 runtime/proc.go:325 +0x1a goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008f780 sp=0xc00008f760 pc=0x55bd43fb71ae runtime.goparkunlock(...) runtime/proc.go:430 runtime.bgsweep(0xc000040080) runtime/mgcsweep.go:317 +0xdf fp=0xc00008f7c8 sp=0xc00008f780 pc=0x55bd43f6ce9f runtime.gcenable.gowrap1() runtime/mgc.go:204 +0x25 fp=0xc00008f7e0 sp=0xc00008f7c8 pc=0x55bd43f614e5 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008f7e8 sp=0xc00008f7e0 pc=0x55bd43fbf581 created by runtime.gcenable in goroutine 1 runtime/mgc.go:204 +0x66 goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]: runtime.gopark(0x10000?, 0x55bd44eeab90?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008ff78 sp=0xc00008ff58 pc=0x55bd43fb71ae runtime.goparkunlock(...) runtime/proc.go:430 runtime.(*scavengerState).park(0x55bd45930820) runtime/mgcscavenge.go:425 +0x49 fp=0xc00008ffa8 sp=0xc00008ff78 pc=0x55bd43f6a869 runtime.bgscavenge(0xc000040080) runtime/mgcscavenge.go:658 +0x59 fp=0xc00008ffc8 sp=0xc00008ffa8 pc=0x55bd43f6adf9 runtime.gcenable.gowrap2() runtime/mgc.go:205 +0x25 fp=0xc00008ffe0 sp=0xc00008ffc8 pc=0x55bd43f61485 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0x55bd43fbf581 created by runtime.gcenable in goroutine 1 runtime/mgc.go:205 +0xa5 goroutine 18 gp=0xc000104700 m=nil [finalizer wait]: runtime.gopark(0xc00008e648?, 0x55bd43f579e5?, 0xb0?, 0x1?, 0xc0000061c0?) runtime/proc.go:424 +0xce fp=0xc00008e620 sp=0xc00008e600 pc=0x55bd43fb71ae runtime.runfinq() runtime/mfinal.go:193 +0x107 fp=0xc00008e7e0 sp=0xc00008e620 pc=0x55bd43f60567 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008e7e8 sp=0xc00008e7e0 pc=0x55bd43fbf581 created by runtime.createfing in goroutine 1 runtime/mfinal.go:163 +0x3d goroutine 19 gp=0xc000260000 m=nil [chan receive]: runtime.gopark(0xc00008a760?, 0x55bd440937a5?, 0x70?, 0xa8?, 0x55bd451958e0?) runtime/proc.go:424 +0xce fp=0xc00008a718 sp=0xc00008a6f8 pc=0x55bd43fb71ae runtime.chanrecv(0xc000112310, 0x0, 0x1) runtime/chan.go:639 +0x41c fp=0xc00008a790 sp=0xc00008a718 pc=0x55bd43f50bdc runtime.chanrecv1(0x0?, 0x0?) runtime/chan.go:489 +0x12 fp=0xc00008a7b8 sp=0xc00008a790 pc=0x55bd43f50792 runtime.unique_runtime_registerUniqueMapCleanup.func1(...) runtime/mgc.go:1781 runtime.unique_runtime_registerUniqueMapCleanup.gowrap1() runtime/mgc.go:1784 +0x2f fp=0xc00008a7e0 sp=0xc00008a7b8 pc=0x55bd43f6454f runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008a7e8 sp=0xc00008a7e0 pc=0x55bd43fbf581 created by unique.runtime_registerUniqueMapCleanup in goroutine 1 runtime/mgc.go:1779 +0x96 goroutine 20 gp=0xc000260380 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008af38 sp=0xc00008af18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00008afc8 sp=0xc00008af38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008afe0 sp=0xc00008afc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 21 gp=0xc000260540 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008b738 sp=0xc00008b718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00008b7c8 sp=0xc00008b738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008b7e0 sp=0xc00008b7c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008b7e8 sp=0xc00008b7e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 34 gp=0xc000484000 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00048a738 sp=0xc00048a718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00048a7c8 sp=0xc00048a738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00048a7e0 sp=0xc00048a7c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00048a7e8 sp=0xc00048a7e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 5 gp=0xc000007880 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000090738 sp=0xc000090718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc0000907c8 sp=0xc000090738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0000907e0 sp=0xc0000907c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000907e8 sp=0xc0000907e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 6 gp=0xc000007a40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000090f38 sp=0xc000090f18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc000090fc8 sp=0xc000090f38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc000090fe0 sp=0xc000090fc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 22 gp=0xc000260700 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008bf38 sp=0xc00008bf18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00008bfc8 sp=0xc00008bf38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008bfe0 sp=0xc00008bfc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 35 gp=0xc0004841c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00048af38 sp=0xc00048af18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00048afc8 sp=0xc00048af38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00048afe0 sp=0xc00048afc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00048afe8 sp=0xc00048afe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 36 gp=0xc000484380 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00048b738 sp=0xc00048b718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00048b7c8 sp=0xc00048b738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00048b7e0 sp=0xc00048b7c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00048b7e8 sp=0xc00048b7e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 37 gp=0xc000484540 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00048bf38 sp=0xc00048bf18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00048bfc8 sp=0xc00048bf38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00048bfe0 sp=0xc00048bfc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00048bfe8 sp=0xc00048bfe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 38 gp=0xc000484700 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00048c738 sp=0xc00048c718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00048c7c8 sp=0xc00048c738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00048c7e0 sp=0xc00048c7c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00048c7e8 sp=0xc00048c7e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 39 gp=0xc0004848c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00048cf38 sp=0xc00048cf18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00048cfc8 sp=0xc00048cf38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00048cfe0 sp=0xc00048cfc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00048cfe8 sp=0xc00048cfe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 23 gp=0xc0002608c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008c738 sp=0xc00008c718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00008c7c8 sp=0xc00008c738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008c7e0 sp=0xc00008c7c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008c7e8 sp=0xc00008c7e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 24 gp=0xc000260a80 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008cf38 sp=0xc00008cf18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00008cfc8 sp=0xc00008cf38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008cfe0 sp=0xc00008cfc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008cfe8 sp=0xc00008cfe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 25 gp=0xc000260c40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008d738 sp=0xc00008d718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00008d7c8 sp=0xc00008d738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008d7e0 sp=0xc00008d7c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008d7e8 sp=0xc00008d7e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 26 gp=0xc000260e00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00008df38 sp=0xc00008df18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00008dfc8 sp=0xc00008df38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00008dfe0 sp=0xc00008dfc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00008dfe8 sp=0xc00008dfe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 40 gp=0xc000484a80 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00048d738 sp=0xc00048d718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00048d7c8 sp=0xc00048d738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00048d7e0 sp=0xc00048d7c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00048d7e8 sp=0xc00048d7e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 7 gp=0xc000007c00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000091738 sp=0xc000091718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc0000917c8 sp=0xc000091738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0000917e0 sp=0xc0000917c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000917e8 sp=0xc0000917e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 27 gp=0xc000260fc0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000486738 sp=0xc000486718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc0004867c8 sp=0xc000486738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0004867e0 sp=0xc0004867c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004867e8 sp=0xc0004867e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 28 gp=0xc000261180 m=nil [GC worker (idle)]: runtime.gopark(0x1a1d987ea5d4?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000486f38 sp=0xc000486f18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc000486fc8 sp=0xc000486f38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc000486fe0 sp=0xc000486fc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000486fe8 sp=0xc000486fe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 41 gp=0xc000484c40 m=nil [GC worker (idle)]: runtime.gopark(0x1a1d987e6ebc?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc00048df38 sp=0xc00048df18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc00048dfc8 sp=0xc00048df38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc00048dfe0 sp=0xc00048dfc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00048dfe8 sp=0xc00048dfe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 42 gp=0xc000484e00 m=nil [GC worker (idle)]: runtime.gopark(0x55bd459dee80?, 0x1?, 0x3e?, 0xc0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000494738 sp=0xc000494718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc0004947c8 sp=0xc000494738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0004947e0 sp=0xc0004947c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004947e8 sp=0xc0004947e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 43 gp=0xc000484fc0 m=nil [GC worker (idle)]: runtime.gopark(0x55bd459dee80?, 0x1?, 0x6?, 0xfe?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000494f38 sp=0xc000494f18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc000494fc8 sp=0xc000494f38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc000494fe0 sp=0xc000494fc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000494fe8 sp=0xc000494fe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 44 gp=0xc000485180 m=nil [GC worker (idle)]: runtime.gopark(0x1a1d987e80df?, 0x1?, 0x27?, 0xbb?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000495738 sp=0xc000495718 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc0004957c8 sp=0xc000495738 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc0004957e0 sp=0xc0004957c8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0004957e8 sp=0xc0004957e0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 45 gp=0xc000485340 m=nil [GC worker (idle)]: runtime.gopark(0x1a1d987ea238?, 0xc000480180?, 0x1a?, 0xa?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000495f38 sp=0xc000495f18 pc=0x55bd43fb71ae runtime.gcBgMarkWorker(0xc0001138f0) runtime/mgc.go:1412 +0xe9 fp=0xc000495fc8 sp=0xc000495f38 pc=0x55bd43f63849 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1328 +0x25 fp=0xc000495fe0 sp=0xc000495fc8 pc=0x55bd43f63725 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000495fe8 sp=0xc000495fe0 pc=0x55bd43fbf581 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1328 +0x105 goroutine 30 gp=0xc0005048c0 m=nil [semacquire]: runtime.gopark(0x0?, 0x0?, 0x0?, 0xe0?, 0x0?) runtime/proc.go:424 +0xce fp=0xc000091e18 sp=0xc000091df8 pc=0x55bd43fb71ae runtime.goparkunlock(...) runtime/proc.go:430 runtime.semacquire1(0xc00019f568, 0x0, 0x1, 0x0, 0x12) runtime/sema.go:178 +0x22c fp=0xc000091e80 sp=0xc000091e18 pc=0x55bd43f9556c sync.runtime_Semacquire(0x0?) runtime/sema.go:71 +0x25 fp=0xc000091eb8 sp=0xc000091e80 pc=0x55bd43fb89c5 sync.(*WaitGroup).Wait(0x0?) sync/waitgroup.go:118 +0x48 fp=0xc000091ee0 sp=0xc000091eb8 pc=0x55bd43fcdda8 github.com/ollama/ollama/llama/runner.(*Server).run(0xc00019f560, {0x55bd45183fa0, 0xc000400dc0}) github.com/ollama/ollama/llama/runner/runner.go:315 +0x47 fp=0xc000091fb8 sp=0xc000091ee0 pc=0x55bd44372fe7 github.com/ollama/ollama/llama/runner.Execute.gowrap2() github.com/ollama/ollama/llama/runner/runner.go:975 +0x28 fp=0xc000091fe0 sp=0xc000091fb8 pc=0x55bd44378068 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0x55bd43fbf581 created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 github.com/ollama/ollama/llama/runner/runner.go:975 +0xde5 rax 0x0 rbx 0x14bac2d6e590 rcx 0x2c0 rdx 0x7fffc9fb0138 rdi 0x11 rsi 0x7fffc9fb00a8 rbp 0x11 rsp 0x14bac3ea26d8 r8 0x0 r9 0x1 r10 0x14bac2d6ee98 r11 0x206 r12 0x7fffc9fb00a8 r13 0x7fffc9fb0138 r14 0x14bac2d6e598 r15 0x0 rip 0x14bac2d05e80 rflags 0x10202 cs 0x33 fs 0x0 gs 0x0 time=2025-02-09T21:59:56.230Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: exit status 2" time=2025-02-09T21:59:56.230Z level=DEBUG source=sched.go:458 msg="triggering expiration for failed load" model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 time=2025-02-09T21:59:56.230Z level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 time=2025-02-09T21:59:56.230Z level=DEBUG source=sched.go:375 msg="got lock to unload" modelPath=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 time=2025-02-09T21:59:56.230Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.2 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.2 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 [GIN] 2025/02/09 - 21:59:56 | 500 | 820.363963ms | 127.0.0.1 | POST "/api/generate" time=2025-02-09T21:59:56.469Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:56.469Z level=DEBUG source=server.go:1082 msg="stopping llama server" time=2025-02-09T21:59:56.469Z level=DEBUG source=sched.go:380 msg="runner released" modelPath=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 time=2025-02-09T21:59:56.720Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.2 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.2 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:56.951Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:56.970Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.2 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.2 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:57.227Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:57.227Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.2 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.2 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:57.456Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:57.469Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.2 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:57.731Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:57.731Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:57.979Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:57.979Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:58.216Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:58.220Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:58.456Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:58.469Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:58.714Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:58.720Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:58.954Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:58.969Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:59.202Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:59.220Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:59.455Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:59.470Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:59.695Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:59.720Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T21:59:59.948Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T21:59:59.970Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T22:00:00.206Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T22:00:00.220Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T22:00:00.452Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T22:00:00.470Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T22:00:00.705Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T22:00:00.720Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T22:00:00.944Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T22:00:00.970Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T22:00:01.207Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T22:00:01.220Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T22:00:01.448Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T22:00:01.469Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.238721931 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 time=2025-02-09T22:00:01.469Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T22:00:01.469Z level=DEBUG source=sched.go:384 msg="sending an unloaded event" modelPath=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 time=2025-02-09T22:00:01.469Z level=DEBUG source=sched.go:308 msg="ignoring unload event with no pending requests" time=2025-02-09T22:00:01.704Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T22:00:01.720Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.489264324 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 time=2025-02-09T22:00:01.720Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="23.5 GiB" before.free="15.3 GiB" before.free_swap="0 B" now.total="23.5 GiB" now.free="15.3 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 dlsym: cuInit - 0x14e2a8813170 dlsym: cuDriverGetVersion - 0x14e2a8813190 dlsym: cuDeviceGetCount - 0x14e2a88131d0 dlsym: cuDeviceGet - 0x14e2a88131b0 dlsym: cuDeviceGetAttribute - 0x14e2a88132b0 dlsym: cuDeviceGetUuid - 0x14e2a8813210 dlsym: cuDeviceGetName - 0x14e2a88131f0 dlsym: cuCtxCreate_v3 - 0x14e2a8813490 dlsym: cuMemGetInfo_v2 - 0x14e2a881d3f0 dlsym: cuCtxDestroy - 0x14e2a88782e0 calling cuInit calling cuDriverGetVersion raw version 0x2f08 CUDA driver version: 12.4 calling cuDeviceGetCount device count 1 time=2025-02-09T22:00:01.946Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-6c0b6f95-4b1d-5897-6fba-93f9d27eaa47 name="NVIDIA GeForce GTX 1050 Ti" overhead="0 B" before.total="3.9 GiB" before.free="3.9 GiB" now.total="3.9 GiB" now.free="3.9 GiB" now.used="46.1 MiB" releasing cuda driver library time=2025-02-09T22:00:01.969Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.738913206 model=/models/blobs/sha256-2af3b81862c6be03c769683af18efdadb2c33f60ff32ab6f83e42c043d6c7816 ``` </details>
GiteaMirror commented 2026-04-28 23:04:20 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Feb 10, 2025):

Fix is here. A new pre-release version should be ready in the morning

<!-- gh-comment-id:2646966970 --> @jmorganca commented on GitHub (Feb 10, 2025): Fix is [here](https://github.com/ollama/ollama/pull/8976). A new pre-release version should be ready in the morning
GiteaMirror commented 2026-04-28 23:04:21 -05:00
Author
Owner
Copy Link

@tris203 commented on GitHub (Feb 10, 2025):

Just tested @jmorganca. Works perfectly, thank you!

<!-- gh-comment-id:2649370772 --> @tris203 commented on GitHub (Feb 10, 2025): Just tested @jmorganca. Works perfectly, thank you!
GiteaMirror commented 2026-04-28 23:04:22 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Feb 10, 2025):

@tris203 thanks!

<!-- gh-comment-id:2649515988 --> @jmorganca commented on GitHub (Feb 10, 2025): @tris203 thanks!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#52335
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?