[GH-ISSUE #9707] Crash on Gemma:12b model #32100

New Issue

GiteaMirror · 2026-04-22T13:02:58-05:00

GiteaMirror commented

2026-04-22 13:02:58 -05:00

Originally created by @tandr on GitHub (Mar 13, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9707

What is the issue?

Seems like a crash of ollama service. Would be nice to know how to

same sequence of 2 questions gives reproducible crash on my machine.

$ ollama ls gemma3
NAME          ID              SIZE      MODIFIED       
gemma3:12b    6fd036cefda5    8.1 GB    12 minutes ago    

$ ollama run gemma3:12b
>>> who are u
Hello! I am Gemma, an open-weights AI assistant. Essentially, I'm a large language model created by the Gemma team at Google DeepMind. 
                                                                                                                                                                        
Here's a breakdown of what that means:                                                                                                                                  
                                                                                                                                                                        
*   **Large Language Model:** I'm a computer program trained on a massive amount of text data. This allows me to understand and generate human-like text.               
*   **Open-weights:** This means my underlying model weights are publicly available. This allows researchers, developers, and anyone interested to use, study, and 
build upon my capabilities.                                                                                                                                             
*   **Google DeepMind:** I was created by the Gemma team at Google DeepMind, a leading AI research lab.                                                                 
                                                                                                                                                                        
I'm here to help you with a variety of tasks, like answering questions, generating creative content, and more! Just let me know what you'd like me to do.

>>> please be concise in your future answers
Error: POST predict: Post "http://127.0.0.1:33533/completion": EOF

Relevant log output

Mar 12 20:08:33 ubuntu ollama[24113]: time=2025-03-12T20:08:33.811-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.233588939 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.060-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.483348353 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.350-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.772661791 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.524-06:00 level=INFO source=server.go:105 msg="system memory" total="30.6 GiB" free="25.1 GiB" free_swap="18.0 GiB"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.525-06:00 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=49 layers.offload=39 layers.split="" memory.available="[7.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="9.5 GiB" memory.required.partial="7.3 GiB" memory.required.kv="768.0 MiB" memory.required.allocations="[7.3 GiB]" memory.weights.total="6.8 GiB" memory.weights.repeating="6.0 GiB" memory.weights.nonrepeating="787.5 MiB" memory.graph.full="519.5 MiB" memory.graph.partial="1.3 GiB"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.608-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.613-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.616-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.623-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3 --ctx-size 2048 --batch-size 512 --n-gpu-layers 39 --threads 8 --parallel 1 --port 33533"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=INFO source=server.go:585 msg="waiting for llama runner to start responding"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.625-06:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server error"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.636-06:00 level=INFO source=runner.go:882 msg="starting ollama engine"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.636-06:00 level=INFO source=runner.go:938 msg="Server listening on 127.0.0.1:33533"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.721-06:00 level=WARN source=ggml.go:149 msg="key not found" key=general.name default=""
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.721-06:00 level=WARN source=ggml.go:149 msg="key not found" key=general.description default=""
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.722-06:00 level=INFO source=ggml.go:67 msg="" architecture=gemma3 file_type=Q4_K_M name="" description="" num_tensors=1065 num_key_values=36
Mar 12 20:08:34 ubuntu ollama[24113]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Mar 12 20:08:34 ubuntu ollama[24113]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Mar 12 20:08:34 ubuntu ollama[24113]: ggml_cuda_init: found 1 CUDA devices:
Mar 12 20:08:34 ubuntu ollama[24113]:   Device 0: NVIDIA GeForce RTX 3070 Ti Laptop GPU, compute capability 8.6, VMM: yes
Mar 12 20:08:34 ubuntu ollama[24113]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so
Mar 12 20:08:34 ubuntu ollama[24113]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.794-06:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.875-06:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server loading model"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.908-06:00 level=INFO source=ggml.go:289 msg="model weights" buffer=CPU size="3.5 GiB"
Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.908-06:00 level=INFO source=ggml.go:289 msg="model weights" buffer=CUDA0 size="4.8 GiB"
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.636-06:00 level=INFO source=ggml.go:356 msg="compute graph" backend=CUDA0 buffer_type=CUDA0
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.636-06:00 level=INFO source=ggml.go:356 msg="compute graph" backend=CPU buffer_type=CUDA_Host
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.636-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.639-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.642-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.652-06:00 level=INFO source=server.go:624 msg="llama runner started in 2.03 seconds"
Mar 12 20:08:36 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:08:36 | 200 |  8.109182517s |       127.0.0.1 | POST     "/api/generate"
Mar 12 20:08:39 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:08:39 | 200 |  803.272765ms |       127.0.0.1 | POST     "/api/chat"
Mar 12 20:08:43 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:08:43 | 200 |      30.948��s |       127.0.0.1 | HEAD     "/"
Mar 12 20:08:43 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:08:43 | 200 |   36.182274ms |       127.0.0.1 | POST     "/api/show"
Mar 12 20:08:43 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:08:43 | 200 |    37.66236ms |       127.0.0.1 | POST     "/api/generate"
Mar 12 20:09:01 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:09:01 | 200 | 15.335197463s |       127.0.0.1 | POST     "/api/chat"
Mar 12 20:09:04 ubuntu ollama[24113]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 5157.92 MiB on device 0: cudaMalloc failed: out of memory
Mar 12 20:09:04 ubuntu ollama[24113]: ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 5408466944
Mar 12 20:09:04 ubuntu ollama[24113]: SIGSEGV: segmentation violation
Mar 12 20:09:04 ubuntu ollama[24113]: PC=0x6073b5d6dfd0 m=200 sigcode=1 addr=0x58
Mar 12 20:09:04 ubuntu ollama[24113]: signal arrived during cgo execution
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 52 gp=0xc000103500 m=200 mp=0xc004827008 [syscall]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.cgocall(0x6073b5dc1fe0, 0xc00011bb00)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/cgocall.go:167 +0x4b fp=0xc00011bad8 sp=0xc00011baa0 pc=0x6073b4f9060b
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/ml/backend/ggml._Cfunc_ggml_backend_sched_graph_compute_async(0x78f360011370, 0x78f348327f60)
Mar 12 20:09:04 ubuntu ollama[24113]:         _cgo_gotypes.go:485 +0x4a fp=0xc00011bb00 sp=0xc00011bad8 pc=0x6073b537b1aa
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/ml/backend/ggml.Context.Compute.func1(...)
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/ml/backend/ggml/ggml.go:497
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/ml/backend/ggml.Context.Compute({0xc000438400, 0x78f348005310, 0x78f348327f60, 0x0, 0x2000}, {0xc18fed41e0, 0x1, 0x6073b627d210?})
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/ml/backend/ggml/ggml.go:497 +0xbd fp=0xc00011bb90 sp=0xc00011bb00 pc=0x6073b5383a7d
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/ml/backend/ggml.(*Context).Compute(0xc114d45a10?, {0xc18fed41e0?, 0x100?, 0x0?})
Mar 12 20:09:04 ubuntu ollama[24113]:         <autogenerated>:1 +0x72 fp=0xc00011bc08 sp=0xc00011bb90 pc=0x6073b53894f2
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/model.Forward({0x6073b6274c00, 0xc114d45a10}, {0x6073b626c2b0, 0xc000352070}, {{0xc0050ad800, 0x98, 0x100}, {0x0, 0x0, 0x0}, ...})
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/model/model.go:303 +0x218 fp=0xc00011bcf0 sp=0xc00011bc08 pc=0x6073b53b0718
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.(*Server).processBatch(0xc0005458c0)
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:395 +0x3bb fp=0xc00011bf98 sp=0xc00011bcf0 pc=0x6073b541cdfb
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.(*Server).run(0xc0005458c0, {0x6073b626d5e0, 0xc000390910})
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:321 +0x4e fp=0xc00011bfb8 sp=0xc00011bf98 pc=0x6073b541c9ee
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap2()
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:919 +0x28 fp=0xc00011bfe0 sp=0xc00011bfb8 pc=0x6073b5421668
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:919 +0xa9c
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 1 gp=0xc000002380 m=nil [IO wait]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc00011d648 sp=0xc00011d628 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.netpollblock(0xc00011d698?, 0xb4f2d226?, 0x73?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/netpoll.go:575 +0xf7 fp=0xc00011d680 sp=0xc00011d648 pc=0x6073b4f586f7
Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.runtime_pollWait(0x78f787cb7eb0, 0x72)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/netpoll.go:351 +0x85 fp=0xc00011d6a0 sp=0xc00011d680 pc=0x6073b4f92b05
Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*pollDesc).wait(0xc000612b00?, 0x900f36cac?, 0x0)
Mar 12 20:09:04 ubuntu ollama[24113]:         internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00011d6c8 sp=0xc00011d6a0 pc=0x6073b5019f87
Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*pollDesc).waitRead(...)
Mar 12 20:09:04 ubuntu ollama[24113]:         internal/poll/fd_poll_runtime.go:89
Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*FD).Accept(0xc000612b00)
Mar 12 20:09:04 ubuntu ollama[24113]:         internal/poll/fd_unix.go:620 +0x295 fp=0xc00011d770 sp=0xc00011d6c8 pc=0x6073b501f355
Mar 12 20:09:04 ubuntu ollama[24113]: net.(*netFD).accept(0xc000612b00)
Mar 12 20:09:04 ubuntu ollama[24113]:         net/fd_unix.go:172 +0x29 fp=0xc00011d828 sp=0xc00011d770 pc=0x6073b5092169
Mar 12 20:09:04 ubuntu ollama[24113]: net.(*TCPListener).accept(0xc00061c080)
Mar 12 20:09:04 ubuntu ollama[24113]:         net/tcpsock_posix.go:159 +0x1b fp=0xc00011d878 sp=0xc00011d828 pc=0x6073b50a7b1b
Mar 12 20:09:04 ubuntu ollama[24113]: net.(*TCPListener).Accept(0xc00061c080)
Mar 12 20:09:04 ubuntu ollama[24113]:         net/tcpsock.go:380 +0x30 fp=0xc00011d8a8 sp=0xc00011d878 pc=0x6073b50a69d0
Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*onceCloseListener).Accept(0xc0004ca120?)
Mar 12 20:09:04 ubuntu ollama[24113]:         <autogenerated>:1 +0x24 fp=0xc00011d8c0 sp=0xc00011d8a8 pc=0x6073b52bdb44
Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*Server).Serve(0xc0001fc600, {0x6073b626b318, 0xc00061c080})
Mar 12 20:09:04 ubuntu ollama[24113]:         net/http/server.go:3424 +0x30c fp=0xc00011d9f0 sp=0xc00011d8c0 pc=0x6073b529540c
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.Execute({0xc000034150, 0xe, 0xf})
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:939 +0xe6a fp=0xc00011dd08 sp=0xc00011d9f0 pc=0x6073b54213aa
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner.Execute({0xc000034130?, 0x0?, 0x0?})
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/runner/runner.go:20 +0xc9 fp=0xc00011dd30 sp=0xc00011dd08 pc=0x6073b5421f09
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/cmd.NewCLI.func2(0xc000116f00?, {0x6073b5ddd054?, 0x4?, 0x6073b5ddd058?})
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/cmd/cmd.go:1285 +0x45 fp=0xc00011dd58 sp=0xc00011dd30 pc=0x6073b5b926a5
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra.(*Command).execute(0xc0004cef08, {0xc000540d20, 0xf, 0xf})
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/spf13/cobra@v1.7.0/command.go:940 +0x85c fp=0xc00011de78 sp=0xc00011dd58 pc=0x6073b510b2fc
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000542f08)
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc00011df30 sp=0xc00011de78 pc=0x6073b510bb45
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra.(*Command).Execute(...)
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/spf13/cobra@v1.7.0/command.go:992
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/spf13/cobra@v1.7.0/command.go:985
Mar 12 20:09:04 ubuntu ollama[24113]: main.main()
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/main.go:12 +0x4d fp=0xc00011df50 sp=0xc00011df30 pc=0x6073b5b92a0d
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.main()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:283 +0x29d fp=0xc00011dfe0 sp=0xc00011df50 pc=0x6073b4f5fcfd
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 2 gp=0xc000002e00 m=nil [force gc (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000084fa8 sp=0xc000084f88 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goparkunlock(...)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:441
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.forcegchelper()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:348 +0xb8 fp=0xc000084fe0 sp=0xc000084fa8 pc=0x6073b4f60038
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000084fe8 sp=0xc000084fe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.init.7 in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:336 +0x1a
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 3 gp=0xc000003340 m=nil [GC sweep wait]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6acff01?, 0x0?, 0x0?, 0x0?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000085780 sp=0xc000085760 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goparkunlock(...)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:441
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.bgsweep(0xc0000ae000)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgcsweep.go:316 +0xdf fp=0xc0000857c8 sp=0xc000085780 pc=0x6073b4f4a85f
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcenable.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:204 +0x25 fp=0xc0000857e0 sp=0xc0000857c8 pc=0x6073b4f3ec45
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0000857e8 sp=0xc0000857e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcenable in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:204 +0x66
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 4 gp=0xc000003500 m=nil [GC scavenge wait]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x10000?, 0x6073b5f93a00?, 0x0?, 0x0?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goparkunlock(...)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:441
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.(*scavengerState).park(0x6073b6ad1b40)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0x6073b4f482a9
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.bgscavenge(0xc0000ae000)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgcscavenge.go:658 +0x59 fp=0xc000085fc8 sp=0xc000085fa8 pc=0x6073b4f48839
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcenable.gowrap2()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:205 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0x6073b4f3ebe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcenable in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:205 +0xa5
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 5 gp=0xc000003dc0 m=nil [finalizer wait]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x1b8?, 0x6073b4f622a9?, 0x1?, 0x23?, 0xc000084688?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000084630 sp=0xc000084610 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.runfinq()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mfinal.go:196 +0x107 fp=0xc0000847e0 sp=0xc000084630 pc=0x6073b4f3dc07
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0000847e8 sp=0xc0000847e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.createfing in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mfinal.go:166 +0x3d
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 6 gp=0xc0001e08c0 m=nil [chan receive]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xc0001dd900?, 0xc00055e3c0?, 0x60?, 0x67?, 0x6073b5078ea8?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000086718 sp=0xc0000866f8 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.chanrecv(0xc0000443f0, 0x0, 0x1)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/chan.go:664 +0x445 fp=0xc000086790 sp=0xc000086718 pc=0x6073b4f2fe05
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.chanrecv1(0x0?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/chan.go:506 +0x12 fp=0xc0000867b8 sp=0xc000086790 pc=0x6073b4f2f992
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.unique_runtime_registerUniqueMapCleanup.func2(...)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1796
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1799 +0x2f fp=0xc0000867e0 sp=0xc0000867b8 pc=0x6073b4f41def
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by unique.runtime_registerUniqueMapCleanup in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1794 +0x85
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 7 gp=0xc0001e0c40 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94dc27?, 0x3?, 0xd6?, 0x8d?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000086f38 sp=0xc000086f18 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc000086fc8 sp=0xc000086f38 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc000086fe0 sp=0xc000086fc8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000086fe8 sp=0xc000086fe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 8 gp=0xc0001e0e00 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94dbe1?, 0x1?, 0xcb?, 0x17?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000087738 sp=0xc000087718 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc0000877c8 sp=0xc000087738 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc0000877e0 sp=0xc0000877c8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 9 gp=0xc0001e0fc0 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x1?, 0x12?, 0x2e?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000087f38 sp=0xc000087f18 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc000087fc8 sp=0xc000087f38 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc000087fe0 sp=0xc000087fc8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 10 gp=0xc0001e1180 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49fafb32b5?, 0x1?, 0x4a?, 0x23?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000080738 sp=0xc000080718 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc0000807c8 sp=0xc000080738 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc0000807e0 sp=0xc0000807c8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0000807e8 sp=0xc0000807e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 11 gp=0xc0001e1340 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x3?, 0x68?, 0xf2?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000080f38 sp=0xc000080f18 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc000080fc8 sp=0xc000080f38 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc000080fe0 sp=0xc000080fc8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000080fe8 sp=0xc000080fe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 12 gp=0xc0001e1500 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94ddcc?, 0x3?, 0x9b?, 0x9c?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000081738 sp=0xc000081718 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc0000817c8 sp=0xc000081738 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc0000817e0 sp=0xc0000817c8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0000817e8 sp=0xc0000817e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 13 gp=0xc0001e16c0 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49fb26387e?, 0x1?, 0xc9?, 0x8f?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000081f38 sp=0xc000081f18 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc000081fc8 sp=0xc000081f38 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc000081fe0 sp=0xc000081fc8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000081fe8 sp=0xc000081fe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 18 gp=0xc000504000 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x1?, 0x6f?, 0xe8?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc00050a738 sp=0xc00050a718 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc00050a7c8 sp=0xc00050a738 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc00050a7e0 sp=0xc00050a7c8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc00050a7e8 sp=0xc00050a7e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 19 gp=0xc0005041c0 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94dd18?, 0x3?, 0x23?, 0x88?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc00050af38 sp=0xc00050af18 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc00050afc8 sp=0xc00050af38 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc00050afe0 sp=0xc00050afc8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc00050afe8 sp=0xc00050afe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 34 gp=0xc000102380 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49c80fb600?, 0x3?, 0xe6?, 0x40?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000506738 sp=0xc000506718 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc0005067c8 sp=0xc000506738 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc0005067e0 sp=0xc0005067c8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0005067e8 sp=0xc0005067e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 14 gp=0xc0001e1880 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49fab60a73?, 0x3?, 0xdc?, 0xb5?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000082738 sp=0xc000082718 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc0000827c8 sp=0xc000082738 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc0000827e0 sp=0xc0000827c8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0000827e8 sp=0xc0000827e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 20 gp=0xc000504380 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x3?, 0xdf?, 0x26?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc00050b738 sp=0xc00050b718 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc00050b7c8 sp=0xc00050b738 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc00050b7e0 sp=0xc00050b7c8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc00050b7e8 sp=0xc00050b7e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 35 gp=0xc000102540 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94dbeb?, 0x3?, 0xf4?, 0x4f?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000506f38 sp=0xc000506f18 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc000506fc8 sp=0xc000506f38 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc000506fe0 sp=0xc000506fc8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000506fe8 sp=0xc000506fe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 15 gp=0xc0001e1a40 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x1?, 0x50?, 0x70?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000082f38 sp=0xc000082f18 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc000082fc8 sp=0xc000082f38 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc000082fe0 sp=0xc000082fc8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000082fe8 sp=0xc000082fe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 16 gp=0xc0001e1c00 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x3?, 0x71?, 0x35?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000083738 sp=0xc000083718 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc0000837c8 sp=0xc000083738 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc0000837e0 sp=0xc0000837c8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc0000837e8 sp=0xc0000837e0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 50 gp=0xc0001e1dc0 m=nil [GC worker (idle)]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94dc45?, 0x3?, 0x67?, 0xbe?, 0x0?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc000083f38 sp=0xc000083f18 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1423 +0xe9 fp=0xc000083fc8 sp=0xc000083f38 pc=0x6073b4f41109
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1()
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x25 fp=0xc000083fe0 sp=0xc000083fc8 pc=0x6073b4f40fe5
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc000083fe8 sp=0xc000083fe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/mgc.go:1339 +0x105
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 53 gp=0xc0004a6700 m=nil [select]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xc004a8da68?, 0x2?, 0x50?, 0xe7?, 0xc004a8d80c?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc004a8d620 sp=0xc004a8d600 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.selectgo(0xc004a8da68, 0xc004a8d808, 0xd2?, 0x0, 0x1?, 0x1)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/select.go:351 +0x837 fp=0xc004a8d758 sp=0xc004a8d620 pc=0x6073b4f721f7
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.(*Server).completion(0xc0005458c0, {0x6073b626b4f8, 0xc000001ce0}, 0xc114d32000)
Mar 12 20:09:04 ubuntu ollama[24113]:         github.com/ollama/ollama/runner/ollamarunner/runner.go:649 +0xad0 fp=0xc004a8dac0 sp=0xc004a8d758 pc=0x6073b541ee70
Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.(*Server).completion-fm({0x6073b626b4f8?, 0xc000001ce0?}, 0xc004a8db40?)
Mar 12 20:09:04 ubuntu ollama[24113]:         <autogenerated>:1 +0x36 fp=0xc004a8daf0 sp=0xc004a8dac0 pc=0x6073b5421a36
Mar 12 20:09:04 ubuntu ollama[24113]: net/http.HandlerFunc.ServeHTTP(0xc0000f0000?, {0x6073b626b4f8?, 0xc000001ce0?}, 0xc004a8db60?)
Mar 12 20:09:04 ubuntu ollama[24113]:         net/http/server.go:2294 +0x29 fp=0xc004a8db18 sp=0xc004a8daf0 pc=0x6073b5291a49
Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*ServeMux).ServeHTTP(0x6073b4f38125?, {0x6073b626b4f8, 0xc000001ce0}, 0xc114d32000)
Mar 12 20:09:04 ubuntu ollama[24113]:         net/http/server.go:2822 +0x1c4 fp=0xc004a8db68 sp=0xc004a8db18 pc=0x6073b5293944
Mar 12 20:09:04 ubuntu ollama[24113]: net/http.serverHandler.ServeHTTP({0x6073b6267b10?}, {0x6073b626b4f8?, 0xc000001ce0?}, 0x1?)
Mar 12 20:09:04 ubuntu ollama[24113]:         net/http/server.go:3301 +0x8e fp=0xc004a8db98 sp=0xc004a8db68 pc=0x6073b52b13ce
Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*conn).serve(0xc0004ca120, {0x6073b626d5a8, 0xc000114000})
Mar 12 20:09:04 ubuntu ollama[24113]:         net/http/server.go:2102 +0x625 fp=0xc004a8dfb8 sp=0xc004a8db98 pc=0x6073b528ff45
Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*Server).Serve.gowrap3()
Mar 12 20:09:04 ubuntu ollama[24113]:         net/http/server.go:3454 +0x28 fp=0xc004a8dfe0 sp=0xc004a8dfb8 pc=0x6073b5295808
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc004a8dfe8 sp=0xc004a8dfe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by net/http.(*Server).Serve in goroutine 1
Mar 12 20:09:04 ubuntu ollama[24113]:         net/http/server.go:3454 +0x485
Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 1192 gp=0xc07ce75880 m=nil [IO wait]:
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xc07ce01df8?, 0x6073b4f9ac7c?, 0x20?, 0x1e?, 0xb?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/proc.go:435 +0xce fp=0xc07ce01dd8 sp=0xc07ce01db8 pc=0x6073b4f938ee
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.netpollblock(0x6073b4fb6d78?, 0xb4f2d226?, 0x73?)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/netpoll.go:575 +0xf7 fp=0xc07ce01e10 sp=0xc07ce01dd8 pc=0x6073b4f586f7
Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.runtime_pollWait(0x78f787cb7d98, 0x72)
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/netpoll.go:351 +0x85 fp=0xc07ce01e30 sp=0xc07ce01e10 pc=0x6073b4f92b05
Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*pollDesc).wait(0xc000612200?, 0xc00022e071?, 0x0)
Mar 12 20:09:04 ubuntu ollama[24113]:         internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc07ce01e58 sp=0xc07ce01e30 pc=0x6073b5019f87
Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*pollDesc).waitRead(...)
Mar 12 20:09:04 ubuntu ollama[24113]:         internal/poll/fd_poll_runtime.go:89
Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*FD).Read(0xc000612200, {0xc00022e071, 0x1, 0x1})
Mar 12 20:09:04 ubuntu ollama[24113]:         internal/poll/fd_unix.go:165 +0x27a fp=0xc07ce01ef0 sp=0xc07ce01e58 pc=0x6073b501b27a
Mar 12 20:09:04 ubuntu ollama[24113]: net.(*netFD).Read(0xc000612200, {0xc00022e071?, 0x6073b5379e89?, 0x0?})
Mar 12 20:09:04 ubuntu ollama[24113]:         net/fd_posix.go:55 +0x25 fp=0xc07ce01f38 sp=0xc07ce01ef0 pc=0x6073b50901c5
Mar 12 20:09:04 ubuntu ollama[24113]: net.(*conn).Read(0xc00007e1f8, {0xc00022e071?, 0xc003059840?, 0x6073b5379e40?})
Mar 12 20:09:04 ubuntu ollama[24113]:         net/net.go:194 +0x45 fp=0xc07ce01f80 sp=0xc07ce01f38 pc=0x6073b509e585
Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*connReader).backgroundRead(0xc00022e060)
Mar 12 20:09:04 ubuntu ollama[24113]:         net/http/server.go:690 +0x37 fp=0xc07ce01fc8 sp=0xc07ce01f80 pc=0x6073b5289e17
Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*connReader).startBackgroundRead.gowrap2()
Mar 12 20:09:04 ubuntu ollama[24113]:         net/http/server.go:686 +0x25 fp=0xc07ce01fe0 sp=0xc07ce01fc8 pc=0x6073b5289d45
Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({})
Mar 12 20:09:04 ubuntu ollama[24113]:         runtime/asm_amd64.s:1700 +0x1 fp=0xc07ce01fe8 sp=0xc07ce01fe0 pc=0x6073b4f9b021
Mar 12 20:09:04 ubuntu ollama[24113]: created by net/http.(*connReader).startBackgroundRead in goroutine 53
Mar 12 20:09:04 ubuntu ollama[24113]:         net/http/server.go:686 +0xb6
Mar 12 20:09:04 ubuntu ollama[24113]: rax    0x78f36034b540
Mar 12 20:09:04 ubuntu ollama[24113]: rbx    0x78f36034b4b0
Mar 12 20:09:04 ubuntu ollama[24113]: rcx    0x2
Mar 12 20:09:04 ubuntu ollama[24113]: rdx    0x78f3486e56e0
Mar 12 20:09:04 ubuntu ollama[24113]: rdi    0x0
Mar 12 20:09:04 ubuntu ollama[24113]: rsi    0x78f310b3f030
Mar 12 20:09:04 ubuntu ollama[24113]: rbp    0x78f3486e56d0
Mar 12 20:09:04 ubuntu ollama[24113]: rsp    0x78f32affcc48
Mar 12 20:09:04 ubuntu ollama[24113]: r8     0x4
Mar 12 20:09:04 ubuntu ollama[24113]: r9     0xc00007e048
Mar 12 20:09:04 ubuntu ollama[24113]: r10    0x1
Mar 12 20:09:04 ubuntu ollama[24113]: r11    0x206
Mar 12 20:09:04 ubuntu ollama[24113]: r12    0x0
Mar 12 20:09:04 ubuntu ollama[24113]: r13    0x78f3600114c8
Mar 12 20:09:04 ubuntu ollama[24113]: r14    0x988
Mar 12 20:09:04 ubuntu ollama[24113]: r15    0x78f36034b4b0
Mar 12 20:09:04 ubuntu ollama[24113]: rip    0x6073b5d6dfd0
Mar 12 20:09:04 ubuntu ollama[24113]: rflags 0x10206
Mar 12 20:09:04 ubuntu ollama[24113]: cs     0x33
Mar 12 20:09:04 ubuntu ollama[24113]: fs     0x0
Mar 12 20:09:04 ubuntu ollama[24113]: gs     0x0
Mar 12 20:09:04 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:09:04 | 200 |  697.481441ms |       127.0.0.1 | POST     "/api/chat"
Mar 12 20:09:05 ubuntu ollama[24113]: time=2025-03-12T20:09:05.095-06:00 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 2"
Mar 12 20:12:06 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:12:06 | 200 |      19.817��s |       127.0.0.1 | HEAD     "/"
Mar 12 20:12:06 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:12:06 | 200 |    3.325454ms |       127.0.0.1 | GET      "/api/tags"
Mar 12 20:12:29 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:12:29 | 200 |      21.561��s |       127.0.0.1 | HEAD     "/"
Mar 12 20:12:29 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:12:29 | 200 |     2.07612ms |       127.0.0.1 | GET      "/api/tags"
Mar 12 20:14:10 ubuntu ollama[24113]: time=2025-03-12T20:14:10.224-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.234496648 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3
Mar 12 20:14:10 ubuntu ollama[24113]: time=2025-03-12T20:14:10.473-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.483451432 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3
Mar 12 20:14:10 ubuntu ollama[24113]: time=2025-03-12T20:14:10.724-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.734500292 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.6.0

Originally created by @tandr on GitHub (Mar 13, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9707 ### What is the issue? Seems like a crash of ollama service. Would be nice to know how to same sequence of 2 questions gives reproducible crash on my machine. ``` $ ollama ls gemma3 NAME ID SIZE MODIFIED gemma3:12b 6fd036cefda5 8.1 GB 12 minutes ago $ ollama run gemma3:12b >>> who are u Hello! I am Gemma, an open-weights AI assistant. Essentially, I'm a large language model created by the Gemma team at Google DeepMind. Here's a breakdown of what that means: * **Large Language Model:** I'm a computer program trained on a massive amount of text data. This allows me to understand and generate human-like text. * **Open-weights:** This means my underlying model weights are publicly available. This allows researchers, developers, and anyone interested to use, study, and build upon my capabilities. * **Google DeepMind:** I was created by the Gemma team at Google DeepMind, a leading AI research lab. I'm here to help you with a variety of tasks, like answering questions, generating creative content, and more! Just let me know what you'd like me to do. >>> please be concise in your future answers Error: POST predict: Post "http://127.0.0.1:33533/completion": EOF ``` ### Relevant log output ```shell Mar 12 20:08:33 ubuntu ollama[24113]: time=2025-03-12T20:08:33.811-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.233588939 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3 Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.060-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.483348353 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3 Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.350-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.772661791 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3 Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.524-06:00 level=INFO source=server.go:105 msg="system memory" total="30.6 GiB" free="25.1 GiB" free_swap="18.0 GiB" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.525-06:00 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=49 layers.offload=39 layers.split="" memory.available="[7.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="9.5 GiB" memory.required.partial="7.3 GiB" memory.required.kv="768.0 MiB" memory.required.allocations="[7.3 GiB]" memory.weights.total="6.8 GiB" memory.weights.repeating="6.0 GiB" memory.weights.nonrepeating="787.5 MiB" memory.graph.full="519.5 MiB" memory.graph.partial="1.3 GiB" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.608-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.613-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.616-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.623-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07 Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30 Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3 --ctx-size 2048 --batch-size 512 --n-gpu-layers 39 --threads 8 --parallel 1 --port 33533" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.624-06:00 level=INFO source=server.go:585 msg="waiting for llama runner to start responding" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.625-06:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server error" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.636-06:00 level=INFO source=runner.go:882 msg="starting ollama engine" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.636-06:00 level=INFO source=runner.go:938 msg="Server listening on 127.0.0.1:33533" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.721-06:00 level=WARN source=ggml.go:149 msg="key not found" key=general.name default="" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.721-06:00 level=WARN source=ggml.go:149 msg="key not found" key=general.description default="" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.722-06:00 level=INFO source=ggml.go:67 msg="" architecture=gemma3 file_type=Q4_K_M name="" description="" num_tensors=1065 num_key_values=36 Mar 12 20:08:34 ubuntu ollama[24113]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Mar 12 20:08:34 ubuntu ollama[24113]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Mar 12 20:08:34 ubuntu ollama[24113]: ggml_cuda_init: found 1 CUDA devices: Mar 12 20:08:34 ubuntu ollama[24113]: Device 0: NVIDIA GeForce RTX 3070 Ti Laptop GPU, compute capability 8.6, VMM: yes Mar 12 20:08:34 ubuntu ollama[24113]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so Mar 12 20:08:34 ubuntu ollama[24113]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.794-06:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.875-06:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server loading model" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.908-06:00 level=INFO source=ggml.go:289 msg="model weights" buffer=CPU size="3.5 GiB" Mar 12 20:08:34 ubuntu ollama[24113]: time=2025-03-12T20:08:34.908-06:00 level=INFO source=ggml.go:289 msg="model weights" buffer=CUDA0 size="4.8 GiB" Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.636-06:00 level=INFO source=ggml.go:356 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.636-06:00 level=INFO source=ggml.go:356 msg="compute graph" backend=CPU buffer_type=CUDA_Host Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.636-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.639-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.642-06:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07 Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30 Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.648-06:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 Mar 12 20:08:36 ubuntu ollama[24113]: time=2025-03-12T20:08:36.652-06:00 level=INFO source=server.go:624 msg="llama runner started in 2.03 seconds" Mar 12 20:08:36 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:08:36 | 200 | 8.109182517s | 127.0.0.1 | POST "/api/generate" Mar 12 20:08:39 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:08:39 | 200 | 803.272765ms | 127.0.0.1 | POST "/api/chat" Mar 12 20:08:43 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:08:43 | 200 | 30.948��s | 127.0.0.1 | HEAD "/" Mar 12 20:08:43 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:08:43 | 200 | 36.182274ms | 127.0.0.1 | POST "/api/show" Mar 12 20:08:43 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:08:43 | 200 | 37.66236ms | 127.0.0.1 | POST "/api/generate" Mar 12 20:09:01 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:09:01 | 200 | 15.335197463s | 127.0.0.1 | POST "/api/chat" Mar 12 20:09:04 ubuntu ollama[24113]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 5157.92 MiB on device 0: cudaMalloc failed: out of memory Mar 12 20:09:04 ubuntu ollama[24113]: ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 5408466944 Mar 12 20:09:04 ubuntu ollama[24113]: SIGSEGV: segmentation violation Mar 12 20:09:04 ubuntu ollama[24113]: PC=0x6073b5d6dfd0 m=200 sigcode=1 addr=0x58 Mar 12 20:09:04 ubuntu ollama[24113]: signal arrived during cgo execution Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 52 gp=0xc000103500 m=200 mp=0xc004827008 [syscall]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.cgocall(0x6073b5dc1fe0, 0xc00011bb00) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/cgocall.go:167 +0x4b fp=0xc00011bad8 sp=0xc00011baa0 pc=0x6073b4f9060b Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/ml/backend/ggml._Cfunc_ggml_backend_sched_graph_compute_async(0x78f360011370, 0x78f348327f60) Mar 12 20:09:04 ubuntu ollama[24113]: _cgo_gotypes.go:485 +0x4a fp=0xc00011bb00 sp=0xc00011bad8 pc=0x6073b537b1aa Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/ml/backend/ggml.Context.Compute.func1(...) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/ml/backend/ggml/ggml.go:497 Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/ml/backend/ggml.Context.Compute({0xc000438400, 0x78f348005310, 0x78f348327f60, 0x0, 0x2000}, {0xc18fed41e0, 0x1, 0x6073b627d210?}) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/ml/backend/ggml/ggml.go:497 +0xbd fp=0xc00011bb90 sp=0xc00011bb00 pc=0x6073b5383a7d Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/ml/backend/ggml.(*Context).Compute(0xc114d45a10?, {0xc18fed41e0?, 0x100?, 0x0?}) Mar 12 20:09:04 ubuntu ollama[24113]: <autogenerated>:1 +0x72 fp=0xc00011bc08 sp=0xc00011bb90 pc=0x6073b53894f2 Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/model.Forward({0x6073b6274c00, 0xc114d45a10}, {0x6073b626c2b0, 0xc000352070}, {{0xc0050ad800, 0x98, 0x100}, {0x0, 0x0, 0x0}, ...}) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/model/model.go:303 +0x218 fp=0xc00011bcf0 sp=0xc00011bc08 pc=0x6073b53b0718 Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.(*Server).processBatch(0xc0005458c0) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner/runner.go:395 +0x3bb fp=0xc00011bf98 sp=0xc00011bcf0 pc=0x6073b541cdfb Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.(*Server).run(0xc0005458c0, {0x6073b626d5e0, 0xc000390910}) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner/runner.go:321 +0x4e fp=0xc00011bfb8 sp=0xc00011bf98 pc=0x6073b541c9ee Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap2() Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner/runner.go:919 +0x28 fp=0xc00011bfe0 sp=0xc00011bfb8 pc=0x6073b5421668 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner/runner.go:919 +0xa9c Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 1 gp=0xc000002380 m=nil [IO wait]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc00011d648 sp=0xc00011d628 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.netpollblock(0xc00011d698?, 0xb4f2d226?, 0x73?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/netpoll.go:575 +0xf7 fp=0xc00011d680 sp=0xc00011d648 pc=0x6073b4f586f7 Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.runtime_pollWait(0x78f787cb7eb0, 0x72) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/netpoll.go:351 +0x85 fp=0xc00011d6a0 sp=0xc00011d680 pc=0x6073b4f92b05 Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*pollDesc).wait(0xc000612b00?, 0x900f36cac?, 0x0) Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00011d6c8 sp=0xc00011d6a0 pc=0x6073b5019f87 Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*pollDesc).waitRead(...) Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll/fd_poll_runtime.go:89 Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*FD).Accept(0xc000612b00) Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll/fd_unix.go:620 +0x295 fp=0xc00011d770 sp=0xc00011d6c8 pc=0x6073b501f355 Mar 12 20:09:04 ubuntu ollama[24113]: net.(*netFD).accept(0xc000612b00) Mar 12 20:09:04 ubuntu ollama[24113]: net/fd_unix.go:172 +0x29 fp=0xc00011d828 sp=0xc00011d770 pc=0x6073b5092169 Mar 12 20:09:04 ubuntu ollama[24113]: net.(*TCPListener).accept(0xc00061c080) Mar 12 20:09:04 ubuntu ollama[24113]: net/tcpsock_posix.go:159 +0x1b fp=0xc00011d878 sp=0xc00011d828 pc=0x6073b50a7b1b Mar 12 20:09:04 ubuntu ollama[24113]: net.(*TCPListener).Accept(0xc00061c080) Mar 12 20:09:04 ubuntu ollama[24113]: net/tcpsock.go:380 +0x30 fp=0xc00011d8a8 sp=0xc00011d878 pc=0x6073b50a69d0 Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*onceCloseListener).Accept(0xc0004ca120?) Mar 12 20:09:04 ubuntu ollama[24113]: <autogenerated>:1 +0x24 fp=0xc00011d8c0 sp=0xc00011d8a8 pc=0x6073b52bdb44 Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*Server).Serve(0xc0001fc600, {0x6073b626b318, 0xc00061c080}) Mar 12 20:09:04 ubuntu ollama[24113]: net/http/server.go:3424 +0x30c fp=0xc00011d9f0 sp=0xc00011d8c0 pc=0x6073b529540c Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.Execute({0xc000034150, 0xe, 0xf}) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner/runner.go:939 +0xe6a fp=0xc00011dd08 sp=0xc00011d9f0 pc=0x6073b54213aa Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner.Execute({0xc000034130?, 0x0?, 0x0?}) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/runner.go:20 +0xc9 fp=0xc00011dd30 sp=0xc00011dd08 pc=0x6073b5421f09 Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/cmd.NewCLI.func2(0xc000116f00?, {0x6073b5ddd054?, 0x4?, 0x6073b5ddd058?}) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/cmd/cmd.go:1285 +0x45 fp=0xc00011dd58 sp=0xc00011dd30 pc=0x6073b5b926a5 Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra.(*Command).execute(0xc0004cef08, {0xc000540d20, 0xf, 0xf}) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra@v1.7.0/command.go:940 +0x85c fp=0xc00011de78 sp=0xc00011dd58 pc=0x6073b510b2fc Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000542f08) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc00011df30 sp=0xc00011de78 pc=0x6073b510bb45 Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra.(*Command).Execute(...) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra@v1.7.0/command.go:992 Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra.(*Command).ExecuteContext(...) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/spf13/cobra@v1.7.0/command.go:985 Mar 12 20:09:04 ubuntu ollama[24113]: main.main() Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/main.go:12 +0x4d fp=0xc00011df50 sp=0xc00011df30 pc=0x6073b5b92a0d Mar 12 20:09:04 ubuntu ollama[24113]: runtime.main() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:283 +0x29d fp=0xc00011dfe0 sp=0xc00011df50 pc=0x6073b4f5fcfd Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 2 gp=0xc000002e00 m=nil [force gc (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000084fa8 sp=0xc000084f88 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goparkunlock(...) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:441 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.forcegchelper() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:348 +0xb8 fp=0xc000084fe0 sp=0xc000084fa8 pc=0x6073b4f60038 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000084fe8 sp=0xc000084fe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.init.7 in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:336 +0x1a Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 3 gp=0xc000003340 m=nil [GC sweep wait]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6acff01?, 0x0?, 0x0?, 0x0?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000085780 sp=0xc000085760 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goparkunlock(...) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:441 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.bgsweep(0xc0000ae000) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgcsweep.go:316 +0xdf fp=0xc0000857c8 sp=0xc000085780 pc=0x6073b4f4a85f Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcenable.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:204 +0x25 fp=0xc0000857e0 sp=0xc0000857c8 pc=0x6073b4f3ec45 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0000857e8 sp=0xc0000857e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcenable in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:204 +0x66 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 4 gp=0xc000003500 m=nil [GC scavenge wait]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x10000?, 0x6073b5f93a00?, 0x0?, 0x0?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goparkunlock(...) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:441 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.(*scavengerState).park(0x6073b6ad1b40) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0x6073b4f482a9 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.bgscavenge(0xc0000ae000) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgcscavenge.go:658 +0x59 fp=0xc000085fc8 sp=0xc000085fa8 pc=0x6073b4f48839 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcenable.gowrap2() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:205 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0x6073b4f3ebe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcenable in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:205 +0xa5 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 5 gp=0xc000003dc0 m=nil [finalizer wait]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x1b8?, 0x6073b4f622a9?, 0x1?, 0x23?, 0xc000084688?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000084630 sp=0xc000084610 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.runfinq() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mfinal.go:196 +0x107 fp=0xc0000847e0 sp=0xc000084630 pc=0x6073b4f3dc07 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0000847e8 sp=0xc0000847e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.createfing in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mfinal.go:166 +0x3d Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 6 gp=0xc0001e08c0 m=nil [chan receive]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xc0001dd900?, 0xc00055e3c0?, 0x60?, 0x67?, 0x6073b5078ea8?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000086718 sp=0xc0000866f8 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.chanrecv(0xc0000443f0, 0x0, 0x1) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/chan.go:664 +0x445 fp=0xc000086790 sp=0xc000086718 pc=0x6073b4f2fe05 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.chanrecv1(0x0?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/chan.go:506 +0x12 fp=0xc0000867b8 sp=0xc000086790 pc=0x6073b4f2f992 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.unique_runtime_registerUniqueMapCleanup.func2(...) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1796 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.unique_runtime_registerUniqueMapCleanup.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1799 +0x2f fp=0xc0000867e0 sp=0xc0000867b8 pc=0x6073b4f41def Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by unique.runtime_registerUniqueMapCleanup in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1794 +0x85 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 7 gp=0xc0001e0c40 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94dc27?, 0x3?, 0xd6?, 0x8d?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000086f38 sp=0xc000086f18 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc000086fc8 sp=0xc000086f38 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc000086fe0 sp=0xc000086fc8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000086fe8 sp=0xc000086fe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 8 gp=0xc0001e0e00 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94dbe1?, 0x1?, 0xcb?, 0x17?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000087738 sp=0xc000087718 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc0000877c8 sp=0xc000087738 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc0000877e0 sp=0xc0000877c8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 9 gp=0xc0001e0fc0 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x1?, 0x12?, 0x2e?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000087f38 sp=0xc000087f18 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc000087fc8 sp=0xc000087f38 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc000087fe0 sp=0xc000087fc8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 10 gp=0xc0001e1180 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49fafb32b5?, 0x1?, 0x4a?, 0x23?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000080738 sp=0xc000080718 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc0000807c8 sp=0xc000080738 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc0000807e0 sp=0xc0000807c8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0000807e8 sp=0xc0000807e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 11 gp=0xc0001e1340 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x3?, 0x68?, 0xf2?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000080f38 sp=0xc000080f18 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc000080fc8 sp=0xc000080f38 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc000080fe0 sp=0xc000080fc8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000080fe8 sp=0xc000080fe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 12 gp=0xc0001e1500 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94ddcc?, 0x3?, 0x9b?, 0x9c?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000081738 sp=0xc000081718 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc0000817c8 sp=0xc000081738 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc0000817e0 sp=0xc0000817c8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0000817e8 sp=0xc0000817e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 13 gp=0xc0001e16c0 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49fb26387e?, 0x1?, 0xc9?, 0x8f?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000081f38 sp=0xc000081f18 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc000081fc8 sp=0xc000081f38 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc000081fe0 sp=0xc000081fc8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000081fe8 sp=0xc000081fe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 18 gp=0xc000504000 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x1?, 0x6f?, 0xe8?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc00050a738 sp=0xc00050a718 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc00050a7c8 sp=0xc00050a738 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc00050a7e0 sp=0xc00050a7c8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc00050a7e8 sp=0xc00050a7e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 19 gp=0xc0005041c0 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94dd18?, 0x3?, 0x23?, 0x88?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc00050af38 sp=0xc00050af18 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc00050afc8 sp=0xc00050af38 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc00050afe0 sp=0xc00050afc8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc00050afe8 sp=0xc00050afe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 34 gp=0xc000102380 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49c80fb600?, 0x3?, 0xe6?, 0x40?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000506738 sp=0xc000506718 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc0005067c8 sp=0xc000506738 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc0005067e0 sp=0xc0005067c8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0005067e8 sp=0xc0005067e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 14 gp=0xc0001e1880 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49fab60a73?, 0x3?, 0xdc?, 0xb5?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000082738 sp=0xc000082718 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc0000827c8 sp=0xc000082738 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc0000827e0 sp=0xc0000827c8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0000827e8 sp=0xc0000827e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 20 gp=0xc000504380 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x3?, 0xdf?, 0x26?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc00050b738 sp=0xc00050b718 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc00050b7c8 sp=0xc00050b738 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc00050b7e0 sp=0xc00050b7c8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc00050b7e8 sp=0xc00050b7e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 35 gp=0xc000102540 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94dbeb?, 0x3?, 0xf4?, 0x4f?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000506f38 sp=0xc000506f18 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc000506fc8 sp=0xc000506f38 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc000506fe0 sp=0xc000506fc8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000506fe8 sp=0xc000506fe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 15 gp=0xc0001e1a40 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x1?, 0x50?, 0x70?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000082f38 sp=0xc000082f18 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc000082fc8 sp=0xc000082f38 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc000082fe0 sp=0xc000082fc8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000082fe8 sp=0xc000082fe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 16 gp=0xc0001e1c00 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0x6073b6b802c0?, 0x3?, 0x71?, 0x35?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000083738 sp=0xc000083718 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc0000837c8 sp=0xc000083738 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc0000837e0 sp=0xc0000837c8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc0000837e8 sp=0xc0000837e0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 50 gp=0xc0001e1dc0 m=nil [GC worker (idle)]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xa49bb94dc45?, 0x3?, 0x67?, 0xbe?, 0x0?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc000083f38 sp=0xc000083f18 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkWorker(0xc0000459d0) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1423 +0xe9 fp=0xc000083fc8 sp=0xc000083f38 pc=0x6073b4f41109 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gcBgMarkStartWorkers.gowrap1() Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x25 fp=0xc000083fe0 sp=0xc000083fc8 pc=0x6073b4f40fe5 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc000083fe8 sp=0xc000083fe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by runtime.gcBgMarkStartWorkers in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: runtime/mgc.go:1339 +0x105 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 53 gp=0xc0004a6700 m=nil [select]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xc004a8da68?, 0x2?, 0x50?, 0xe7?, 0xc004a8d80c?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc004a8d620 sp=0xc004a8d600 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.selectgo(0xc004a8da68, 0xc004a8d808, 0xd2?, 0x0, 0x1?, 0x1) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/select.go:351 +0x837 fp=0xc004a8d758 sp=0xc004a8d620 pc=0x6073b4f721f7 Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.(*Server).completion(0xc0005458c0, {0x6073b626b4f8, 0xc000001ce0}, 0xc114d32000) Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner/runner.go:649 +0xad0 fp=0xc004a8dac0 sp=0xc004a8d758 pc=0x6073b541ee70 Mar 12 20:09:04 ubuntu ollama[24113]: github.com/ollama/ollama/runner/ollamarunner.(*Server).completion-fm({0x6073b626b4f8?, 0xc000001ce0?}, 0xc004a8db40?) Mar 12 20:09:04 ubuntu ollama[24113]: <autogenerated>:1 +0x36 fp=0xc004a8daf0 sp=0xc004a8dac0 pc=0x6073b5421a36 Mar 12 20:09:04 ubuntu ollama[24113]: net/http.HandlerFunc.ServeHTTP(0xc0000f0000?, {0x6073b626b4f8?, 0xc000001ce0?}, 0xc004a8db60?) Mar 12 20:09:04 ubuntu ollama[24113]: net/http/server.go:2294 +0x29 fp=0xc004a8db18 sp=0xc004a8daf0 pc=0x6073b5291a49 Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*ServeMux).ServeHTTP(0x6073b4f38125?, {0x6073b626b4f8, 0xc000001ce0}, 0xc114d32000) Mar 12 20:09:04 ubuntu ollama[24113]: net/http/server.go:2822 +0x1c4 fp=0xc004a8db68 sp=0xc004a8db18 pc=0x6073b5293944 Mar 12 20:09:04 ubuntu ollama[24113]: net/http.serverHandler.ServeHTTP({0x6073b6267b10?}, {0x6073b626b4f8?, 0xc000001ce0?}, 0x1?) Mar 12 20:09:04 ubuntu ollama[24113]: net/http/server.go:3301 +0x8e fp=0xc004a8db98 sp=0xc004a8db68 pc=0x6073b52b13ce Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*conn).serve(0xc0004ca120, {0x6073b626d5a8, 0xc000114000}) Mar 12 20:09:04 ubuntu ollama[24113]: net/http/server.go:2102 +0x625 fp=0xc004a8dfb8 sp=0xc004a8db98 pc=0x6073b528ff45 Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*Server).Serve.gowrap3() Mar 12 20:09:04 ubuntu ollama[24113]: net/http/server.go:3454 +0x28 fp=0xc004a8dfe0 sp=0xc004a8dfb8 pc=0x6073b5295808 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc004a8dfe8 sp=0xc004a8dfe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by net/http.(*Server).Serve in goroutine 1 Mar 12 20:09:04 ubuntu ollama[24113]: net/http/server.go:3454 +0x485 Mar 12 20:09:04 ubuntu ollama[24113]: goroutine 1192 gp=0xc07ce75880 m=nil [IO wait]: Mar 12 20:09:04 ubuntu ollama[24113]: runtime.gopark(0xc07ce01df8?, 0x6073b4f9ac7c?, 0x20?, 0x1e?, 0xb?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/proc.go:435 +0xce fp=0xc07ce01dd8 sp=0xc07ce01db8 pc=0x6073b4f938ee Mar 12 20:09:04 ubuntu ollama[24113]: runtime.netpollblock(0x6073b4fb6d78?, 0xb4f2d226?, 0x73?) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/netpoll.go:575 +0xf7 fp=0xc07ce01e10 sp=0xc07ce01dd8 pc=0x6073b4f586f7 Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.runtime_pollWait(0x78f787cb7d98, 0x72) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/netpoll.go:351 +0x85 fp=0xc07ce01e30 sp=0xc07ce01e10 pc=0x6073b4f92b05 Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*pollDesc).wait(0xc000612200?, 0xc00022e071?, 0x0) Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc07ce01e58 sp=0xc07ce01e30 pc=0x6073b5019f87 Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*pollDesc).waitRead(...) Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll/fd_poll_runtime.go:89 Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll.(*FD).Read(0xc000612200, {0xc00022e071, 0x1, 0x1}) Mar 12 20:09:04 ubuntu ollama[24113]: internal/poll/fd_unix.go:165 +0x27a fp=0xc07ce01ef0 sp=0xc07ce01e58 pc=0x6073b501b27a Mar 12 20:09:04 ubuntu ollama[24113]: net.(*netFD).Read(0xc000612200, {0xc00022e071?, 0x6073b5379e89?, 0x0?}) Mar 12 20:09:04 ubuntu ollama[24113]: net/fd_posix.go:55 +0x25 fp=0xc07ce01f38 sp=0xc07ce01ef0 pc=0x6073b50901c5 Mar 12 20:09:04 ubuntu ollama[24113]: net.(*conn).Read(0xc00007e1f8, {0xc00022e071?, 0xc003059840?, 0x6073b5379e40?}) Mar 12 20:09:04 ubuntu ollama[24113]: net/net.go:194 +0x45 fp=0xc07ce01f80 sp=0xc07ce01f38 pc=0x6073b509e585 Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*connReader).backgroundRead(0xc00022e060) Mar 12 20:09:04 ubuntu ollama[24113]: net/http/server.go:690 +0x37 fp=0xc07ce01fc8 sp=0xc07ce01f80 pc=0x6073b5289e17 Mar 12 20:09:04 ubuntu ollama[24113]: net/http.(*connReader).startBackgroundRead.gowrap2() Mar 12 20:09:04 ubuntu ollama[24113]: net/http/server.go:686 +0x25 fp=0xc07ce01fe0 sp=0xc07ce01fc8 pc=0x6073b5289d45 Mar 12 20:09:04 ubuntu ollama[24113]: runtime.goexit({}) Mar 12 20:09:04 ubuntu ollama[24113]: runtime/asm_amd64.s:1700 +0x1 fp=0xc07ce01fe8 sp=0xc07ce01fe0 pc=0x6073b4f9b021 Mar 12 20:09:04 ubuntu ollama[24113]: created by net/http.(*connReader).startBackgroundRead in goroutine 53 Mar 12 20:09:04 ubuntu ollama[24113]: net/http/server.go:686 +0xb6 Mar 12 20:09:04 ubuntu ollama[24113]: rax 0x78f36034b540 Mar 12 20:09:04 ubuntu ollama[24113]: rbx 0x78f36034b4b0 Mar 12 20:09:04 ubuntu ollama[24113]: rcx 0x2 Mar 12 20:09:04 ubuntu ollama[24113]: rdx 0x78f3486e56e0 Mar 12 20:09:04 ubuntu ollama[24113]: rdi 0x0 Mar 12 20:09:04 ubuntu ollama[24113]: rsi 0x78f310b3f030 Mar 12 20:09:04 ubuntu ollama[24113]: rbp 0x78f3486e56d0 Mar 12 20:09:04 ubuntu ollama[24113]: rsp 0x78f32affcc48 Mar 12 20:09:04 ubuntu ollama[24113]: r8 0x4 Mar 12 20:09:04 ubuntu ollama[24113]: r9 0xc00007e048 Mar 12 20:09:04 ubuntu ollama[24113]: r10 0x1 Mar 12 20:09:04 ubuntu ollama[24113]: r11 0x206 Mar 12 20:09:04 ubuntu ollama[24113]: r12 0x0 Mar 12 20:09:04 ubuntu ollama[24113]: r13 0x78f3600114c8 Mar 12 20:09:04 ubuntu ollama[24113]: r14 0x988 Mar 12 20:09:04 ubuntu ollama[24113]: r15 0x78f36034b4b0 Mar 12 20:09:04 ubuntu ollama[24113]: rip 0x6073b5d6dfd0 Mar 12 20:09:04 ubuntu ollama[24113]: rflags 0x10206 Mar 12 20:09:04 ubuntu ollama[24113]: cs 0x33 Mar 12 20:09:04 ubuntu ollama[24113]: fs 0x0 Mar 12 20:09:04 ubuntu ollama[24113]: gs 0x0 Mar 12 20:09:04 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:09:04 | 200 | 697.481441ms | 127.0.0.1 | POST "/api/chat" Mar 12 20:09:05 ubuntu ollama[24113]: time=2025-03-12T20:09:05.095-06:00 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 2" Mar 12 20:12:06 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:12:06 | 200 | 19.817��s | 127.0.0.1 | HEAD "/" Mar 12 20:12:06 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:12:06 | 200 | 3.325454ms | 127.0.0.1 | GET "/api/tags" Mar 12 20:12:29 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:12:29 | 200 | 21.561��s | 127.0.0.1 | HEAD "/" Mar 12 20:12:29 ubuntu ollama[24113]: [GIN] 2025/03/12 - 20:12:29 | 200 | 2.07612ms | 127.0.0.1 | GET "/api/tags" Mar 12 20:14:10 ubuntu ollama[24113]: time=2025-03-12T20:14:10.224-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.234496648 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3 Mar 12 20:14:10 ubuntu ollama[24113]: time=2025-03-12T20:14:10.473-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.483451432 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3 Mar 12 20:14:10 ubuntu ollama[24113]: time=2025-03-12T20:14:10.724-06:00 level=WARN source=sched.go:647 msg="gpu VRAM usage didn't recover within timeout" seconds=5.734500292 model=/usr/share/ollama/.ollama/models/blobs/sha256-adca500fad9b54c565ae672184e0c9eb690eb6014ba63f8ec13849d4f73a32d3 ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.6.0

GiteaMirror added the bug label 2026-04-22 13:02:58 -05:00

GiteaMirror closed this issue

2026-04-22 13:02:58 -05:00

GiteaMirror commented

2026-04-22 13:02:59 -05:00

@rick-github commented on GitHub (Mar 13, 2025):

Mar 12 20:09:04 ubuntu ollama[24113]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 5157.92 MiB on device 0: cudaMalloc failed: out of memory
Mar 12 20:09:04 ubuntu ollama[24113]: ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 5408466944

#9699

@rick-github commented on GitHub (Mar 13, 2025): ``` Mar 12 20:09:04 ubuntu ollama[24113]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 5157.92 MiB on device 0: cudaMalloc failed: out of memory Mar 12 20:09:04 ubuntu ollama[24113]: ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 5408466944 ``` #9699

GiteaMirror commented

2026-04-22 13:03:00 -05:00

@tandr commented on GitHub (Mar 13, 2025):

adding Environment="GGML_CUDA_ENABLE_UNIFIED_MEMORY=1" to /etc/systemd/system/ollama.service helped.

@tandr commented on GitHub (Mar 13, 2025): adding `Environment="GGML_CUDA_ENABLE_UNIFIED_MEMORY=1"` to `/etc/systemd/system/ollama.service` helped.

GiteaMirror commented

2026-04-22 13:03:01 -05:00

@tandr commented on GitHub (Mar 14, 2025):

Before I close this - with systemd you need to use systemctl edit ollama.service to add this line as "override" in a special file, otheriwse each update of ollama wipes these changes out.

@tandr commented on GitHub (Mar 14, 2025): Before I close this - with systemd you need to use `systemctl edit ollama.service` to add this line as "override" in a special file, otheriwse each update of ollama wipes these changes out.

GiteaMirror commented

2026-04-22 13:03:01 -05:00

@last-partizan commented on GitHub (Mar 14, 2025):

Is this fix supposed to work for AMD?

I have RX 6800 XT with 16GB VRAM, and 32 GB RAM, and I tried running gemma3:27b (with GGML_CUDA_ENABLE_UNIFIED_MEMORY=1).

$ ollama run gemma3:27b
>>> Hello.
Hello there! 👋

How can I help you today? Just let me know what you're thinking, whether it's a question, a request, or just a
casual chat.

I'm ready when you are! 😊


>>> I just wanted to know it this works with ollama. Seems to be working for small requests, but larger requests are cra
... shing...
Error: POST predict: Post "http://127.0.0.1:40453/completion": EOF

journalctl -xfn --unit ollama -o cat

ollama.service: Consumed 39min 37.352s CPU time, 24.1G memory peak, 1.9G memory swap peak.
Started Ollama Service.
2025/03/14 12:01:38 routes.go:1225: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/var/lib/ollama OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-03-14T12:01:38.628+02:00 level=INFO source=images.go:432 msg="total blobs: 40"
time=2025-03-14T12:01:38.629+02:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-03-14T12:01:38.630+02:00 level=INFO source=routes.go:1292 msg="Listening on 127.0.0.1:11434 (version 0.6.0)"
time=2025-03-14T12:01:38.630+02:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-03-14T12:01:38.664+02:00 level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2025-03-14T12:01:38.673+02:00 level=INFO source=amd_linux.go:386 msg="amdgpu is supported" gpu=GPU-276bf8a36519a813 gpu_type=gfx1030
time=2025-03-14T12:01:38.673+02:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-276bf8a36519a813 library=rocm variant="" compute=gfx1030 driver=0.0 name=1002:73bf total="16.0 GiB" available="14.5 GiB"
[GIN] 2025/03/14 - 12:01:54 | 200 |      40.191µs |       127.0.0.1 | HEAD     "/"
[GIN] 2025/03/14 - 12:01:55 | 200 |   38.177328ms |       127.0.0.1 | POST     "/api/show"
time=2025-03-14T12:01:55.130+02:00 level=INFO source=server.go:105 msg="system memory" total="31.3 GiB" free="26.5 GiB" free_swap="4.5 GiB"
time=2025-03-14T12:01:55.131+02:00 level=INFO source=server.go:138 msg=offload library=rocm layers.requested=-1 layers.model=63 layers.offload=50 layers.split="" memory.available="[14.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="18.8 GiB" memory.required.partial="14.5 GiB" memory.required.kv="992.0 MiB" memory.required.allocations="[14.5 GiB]" memory.weights.total="15.3 GiB" memory.weights.repeating="14.2 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="522.5 MiB" memory.graph.partial="1.6 GiB"
time=2025-03-14T12:01:55.210+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
time=2025-03-14T12:01:55.213+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
time=2025-03-14T12:01:55.220+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07
time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30
time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
time=2025-03-14T12:01:55.226+02:00 level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/bin/ollama runner --ollama-engine --model /var/lib/ollama/blobs/sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541 --ctx-size 2048 --batch-size 512 --n-gpu-layers 50 --threads 8 --parallel 1 --port 40453"
time=2025-03-14T12:01:55.226+02:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2025-03-14T12:01:55.226+02:00 level=INFO source=server.go:585 msg="waiting for llama runner to start responding"
time=2025-03-14T12:01:55.226+02:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server error"
time=2025-03-14T12:01:55.234+02:00 level=INFO source=runner.go:882 msg="starting ollama engine"
time=2025-03-14T12:01:55.234+02:00 level=INFO source=runner.go:938 msg="Server listening on 127.0.0.1:40453"
time=2025-03-14T12:01:55.310+02:00 level=WARN source=ggml.go:149 msg="key not found" key=general.name default=""
time=2025-03-14T12:01:55.310+02:00 level=WARN source=ggml.go:149 msg="key not found" key=general.description default=""
time=2025-03-14T12:01:55.310+02:00 level=INFO source=ggml.go:67 msg="" architecture=gemma3 file_type=Q4_K_M name="" description="" num_tensors=1247 num_key_values=36
time=2025-03-14T12:01:55.477+02:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server loading model"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 6800 XT, gfx1030 (0x1030), VMM: no, Wave Size: 32
load_backend: loaded ROCm backend from /usr/lib/ollama/rocm/libggml-hip.so
load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-haswell.so
time=2025-03-14T12:01:58.356+02:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 ROCm.0.NO_VMM=1 ROCm.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
time=2025-03-14T12:01:58.808+02:00 level=INFO source=ggml.go:289 msg="model weights" buffer=CPU size="5.8 GiB"
time=2025-03-14T12:01:58.808+02:00 level=INFO source=ggml.go:289 msg="model weights" buffer=ROCm0 size="11.5 GiB"
time=2025-03-14T12:02:09.993+02:00 level=INFO source=ggml.go:356 msg="compute graph" backend=ROCm0 buffer_type=ROCm0
time=2025-03-14T12:02:10.016+02:00 level=INFO source=ggml.go:356 msg="compute graph" backend=CPU buffer_type=ROCm_Host
time=2025-03-14T12:02:10.019+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
time=2025-03-14T12:02:10.022+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
time=2025-03-14T12:02:10.038+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
time=2025-03-14T12:02:10.042+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07
time=2025-03-14T12:02:10.043+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
time=2025-03-14T12:02:10.043+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
time=2025-03-14T12:02:10.043+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
time=2025-03-14T12:02:10.043+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30
time=2025-03-14T12:02:10.043+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
time=2025-03-14T12:02:10.181+02:00 level=INFO source=server.go:624 msg="llama runner started in 14.95 seconds"
[GIN] 2025/03/14 - 12:02:10 | 200 | 15.169570486s |       127.0.0.1 | POST     "/api/generate"
[GIN] 2025/03/14 - 12:02:26 | 200 | 10.623129145s |       127.0.0.1 | POST     "/api/chat"
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 12083.39 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 12670354944
SIGSEGV: segmentation violation
PC=0x5b72a4c9a2a4 m=22 sigcode=1 addr=0x58
signal arrived during cgo execution
goroutine 9 gp=0xc000603180 m=22 mp=0xc000581008 [syscall]:
runtime.cgocall(0x5b72a4cf3ae0, 0xc000619b00)
	/usr/lib/go/src/runtime/cgocall.go:167 +0x4b fp=0xc000619ad8 sp=0xc000619aa0 pc=0x5b72a3e8d42b
github.com/ollama/ollama/ml/backend/ggml._Cfunc_ggml_backend_sched_graph_compute_async(0x74bc2c01f600, 0x74bc2c684420)
	_cgo_gotypes.go:489 +0x4a fp=0xc000619b00 sp=0xc000619ad8 pc=0x5b72a427904a
github.com/ollama/ollama/ml/backend/ggml.Context.Compute.func1(...)
	/build/ollama/src/ollama/ml/backend/ggml/ggml.go:497
github.com/ollama/ollama/ml/backend/ggml.Context.Compute({0xc0000aa6c0, 0x74bc2c3583d0, 0x74bc2c684420, 0x0, 0x2000}, {0xc08f3ecb20, 0x1, 0x5b72a5194630?})
	/build/ollama/src/ollama/ml/backend/ggml/ggml.go:497 +0xbd fp=0xc000619b90 sp=0xc000619b00 pc=0x5b72a428191d
github.com/ollama/ollama/ml/backend/ggml.(*Context).Compute(0xc08f401980?, {0xc08f3ecb20?, 0x40?, 0x0?})
	<autogenerated>:1 +0x72 fp=0xc000619c08 sp=0xc000619b90 pc=0x5b72a4287392
github.com/ollama/ollama/model.Forward({0x5b72a518c020, 0xc08f401980}, {0x5b72a51836d0, 0xc00043c000}, {{0xc2817abf00, 0x24, 0x40}, {0x0, 0x0, 0x0}, ...})
	/build/ollama/src/ollama/model/model.go:303 +0x218 fp=0xc000619cf0 sp=0xc000619c08 pc=0x5b72a42ae5b8
github.com/ollama/ollama/runner/ollamarunner.(*Server).processBatch(0xc00012c360)
	/build/ollama/src/ollama/runner/ollamarunner/runner.go:395 +0x3bb fp=0xc000619f98 sp=0xc000619cf0 pc=0x5b72a431ac9b
github.com/ollama/ollama/runner/ollamarunner.(*Server).run(0xc00012c360, {0x5b72a5184a00, 0xc0000c34a0})
	/build/ollama/src/ollama/runner/ollamarunner/runner.go:321 +0x4e fp=0xc000619fb8 sp=0xc000619f98 pc=0x5b72a431a88e
github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap2()
	/build/ollama/src/ollama/runner/ollamarunner/runner.go:919 +0x28 fp=0xc000619fe0 sp=0xc000619fb8 pc=0x5b72a431f508
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000619fe8 sp=0xc000619fe0 pc=0x5b72a3e98681
created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1
	/build/ollama/src/ollama/runner/ollamarunner/runner.go:919 +0xa9c
goroutine 1 gp=0xc000002380 m=nil [IO wait, 1 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000617648 sp=0xc000617628 pc=0x5b72a3e9070e
runtime.netpollblock(0xc000617698?, 0xa3e2a886?, 0x72?)
	/usr/lib/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc000617680 sp=0xc000617648 pc=0x5b72a3e55d57
internal/poll.runtime_pollWait(0x74c2d6256de0, 0x72)
	/usr/lib/go/src/runtime/netpoll.go:351 +0x85 fp=0xc0006176a0 sp=0xc000617680 pc=0x5b72a3e8f925
internal/poll.(*pollDesc).wait(0xc000705b00?, 0x900000036?, 0x0)
	/usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0006176c8 sp=0xc0006176a0 pc=0x5b72a3f17da7
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000705b00)
	/usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295 fp=0xc000617770 sp=0xc0006176c8 pc=0x5b72a3f1d175
net.(*netFD).accept(0xc000705b00)
	/usr/lib/go/src/net/fd_unix.go:172 +0x29 fp=0xc000617828 sp=0xc000617770 pc=0x5b72a3f8ff89
net.(*TCPListener).accept(0xc0000ab780)
	/usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b fp=0xc000617878 sp=0xc000617828 pc=0x5b72a3fa593b
net.(*TCPListener).Accept(0xc0000ab780)
	/usr/lib/go/src/net/tcpsock.go:380 +0x30 fp=0xc0006178a8 sp=0xc000617878 pc=0x5b72a3fa47f0
net/http.(*onceCloseListener).Accept(0xc000234120?)
	<autogenerated>:1 +0x24 fp=0xc0006178c0 sp=0xc0006178a8 pc=0x5b72a41bb9e4
net/http.(*Server).Serve(0xc0001f7600, {0x5b72a5182738, 0xc0000ab780})
	/usr/lib/go/src/net/http/server.go:3424 +0x30c fp=0xc0006179f0 sp=0xc0006178c0 pc=0x5b72a41932ac
github.com/ollama/ollama/runner/ollamarunner.Execute({0xc00012c150, 0xe, 0xf})
	/build/ollama/src/ollama/runner/ollamarunner/runner.go:939 +0xe6a fp=0xc000617d08 sp=0xc0006179f0 pc=0x5b72a431f24a
github.com/ollama/ollama/runner.Execute({0xc00012c130?, 0x0?, 0x0?})
	/build/ollama/src/ollama/runner/runner.go:20 +0xc9 fp=0xc000617d30 sp=0xc000617d08 pc=0x5b72a431fda9
github.com/ollama/ollama/cmd.NewCLI.func2(0xc0001f7400?, {0x5b72a4cf6194?, 0x4?, 0x5b72a4cf6198?})
	/build/ollama/src/ollama/cmd/cmd.go:1285 +0x45 fp=0xc000617d58 sp=0xc000617d30 pc=0x5b72a4a90545
github.com/spf13/cobra.(*Command).execute(0xc000236f08, {0xc0000e6ff0, 0xf, 0xf})
	/build/ollama/src/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x85c fp=0xc000617e78 sp=0xc000617d58 pc=0x5b72a400911c
github.com/spf13/cobra.(*Command).ExecuteC(0xc0000ea908)
	/build/ollama/src/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc000617f30 sp=0xc000617e78 pc=0x5b72a4009965
github.com/spf13/cobra.(*Command).Execute(...)
	/build/ollama/src/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/build/ollama/src/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
	/build/ollama/src/ollama/main.go:12 +0x4d fp=0xc000617f50 sp=0xc000617f30 pc=0x5b72a4a908ad
runtime.main()
	/usr/lib/go/src/runtime/proc.go:283 +0x29d fp=0xc000617fe0 sp=0xc000617f50 pc=0x5b72a3e5d35d
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000617fe8 sp=0xc000617fe0 pc=0x5b72a3e98681
goroutine 2 gp=0xc000002e00 m=nil [force gc (idle), 1 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007cfa8 sp=0xc00007cf88 pc=0x5b72a3e9070e
runtime.goparkunlock(...)
	/usr/lib/go/src/runtime/proc.go:441
runtime.forcegchelper()
	/usr/lib/go/src/runtime/proc.go:348 +0xb8 fp=0xc00007cfe0 sp=0xc00007cfa8 pc=0x5b72a3e5d698
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007cfe8 sp=0xc00007cfe0 pc=0x5b72a3e98681
created by runtime.init.7 in goroutine 1
	/usr/lib/go/src/runtime/proc.go:336 +0x1a
goroutine 3 gp=0xc000003340 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007d780 sp=0xc00007d760 pc=0x5b72a3e9070e
runtime.goparkunlock(...)
	/usr/lib/go/src/runtime/proc.go:441
runtime.bgsweep(0xc0000a8000)
	/usr/lib/go/src/runtime/mgcsweep.go:316 +0xdf fp=0xc00007d7c8 sp=0xc00007d780 pc=0x5b72a3e47ebf
runtime.gcenable.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007d7e0 sp=0xc00007d7c8 pc=0x5b72a3e3c2a5
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007d7e8 sp=0xc00007d7e0 pc=0x5b72a3e98681
created by runtime.gcenable in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:204 +0x66
goroutine 4 gp=0xc000003500 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x5b72a4eacc88?, 0x0?, 0x0?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007df78 sp=0xc00007df58 pc=0x5b72a3e9070e
runtime.goparkunlock(...)
	/usr/lib/go/src/runtime/proc.go:441
runtime.(*scavengerState).park(0x5b72a59f2460)
	/usr/lib/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007dfa8 sp=0xc00007df78 pc=0x5b72a3e45909
runtime.bgscavenge(0xc0000a8000)
	/usr/lib/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00007dfc8 sp=0xc00007dfa8 pc=0x5b72a3e45e99
runtime.gcenable.gowrap2()
	/usr/lib/go/src/runtime/mgc.go:205 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 pc=0x5b72a3e3c245
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x5b72a3e98681
created by runtime.gcenable in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:205 +0xa5
goroutine 18 gp=0xc000102700 m=nil [finalizer wait, 1 minutes]:
runtime.gopark(0x1b8?, 0xc000002380?, 0x1?, 0x23?, 0xc00007c688?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007c630 sp=0xc00007c610 pc=0x5b72a3e9070e
runtime.runfinq()
	/usr/lib/go/src/runtime/mfinal.go:196 +0x107 fp=0xc00007c7e0 sp=0xc00007c630 pc=0x5b72a3e3b267
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007c7e8 sp=0xc00007c7e0 pc=0x5b72a3e98681
created by runtime.createfing in goroutine 1
	/usr/lib/go/src/runtime/mfinal.go:166 +0x3d
goroutine 19 gp=0xc000103180 m=nil [chan receive]:
runtime.gopark(0xc000229860?, 0xc00016a180?, 0x60?, 0x87?, 0x5b72a3f76cc8?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000078718 sp=0xc0000786f8 pc=0x5b72a3e9070e
runtime.chanrecv(0xc000110380, 0x0, 0x1)
	/usr/lib/go/src/runtime/chan.go:664 +0x445 fp=0xc000078790 sp=0xc000078718 pc=0x5b72a3e2d465
runtime.chanrecv1(0x0?, 0x0?)
	/usr/lib/go/src/runtime/chan.go:506 +0x12 fp=0xc0000787b8 sp=0xc000078790 pc=0x5b72a3e2cff2
runtime.unique_runtime_registerUniqueMapCleanup.func2(...)
	/usr/lib/go/src/runtime/mgc.go:1796
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1799 +0x2f fp=0xc0000787e0 sp=0xc0000787b8 pc=0x5b72a3e3f44f
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000787e8 sp=0xc0000787e0 pc=0x5b72a3e98681
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1794 +0x85
goroutine 20 gp=0xc000103500 m=nil [GC worker (idle), 1 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000078f38 sp=0xc000078f18 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc000078fc8 sp=0xc000078f38 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc000078fe0 sp=0xc000078fc8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000078fe8 sp=0xc000078fe0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 21 gp=0xc0001036c0 m=nil [GC worker (idle)]:
runtime.gopark(0xede5e73158e?, 0x1?, 0xa?, 0x4b?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000079738 sp=0xc000079718 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc0000797c8 sp=0xc000079738 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc0000797e0 sp=0xc0000797c8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000797e8 sp=0xc0000797e0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 34 gp=0xc000504000 m=nil [GC worker (idle)]:
runtime.gopark(0xede67800fe3?, 0x1?, 0x1?, 0x3c?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00050a738 sp=0xc00050a718 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00050a7c8 sp=0xc00050a738 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00050a7e0 sp=0xc00050a7c8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050a7e8 sp=0xc00050a7e0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 5 gp=0xc000003a40 m=nil [GC worker (idle)]:
runtime.gopark(0xede5e85e70d?, 0x3?, 0x1a?, 0x18?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007e738 sp=0xc00007e718 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007e7c8 sp=0xc00007e738 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007e7e0 sp=0xc00007e7c8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007e7e8 sp=0xc00007e7e0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 22 gp=0xc000103880 m=nil [GC worker (idle)]:
runtime.gopark(0xede5e7119d0?, 0x1?, 0xa3?, 0x61?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000079f38 sp=0xc000079f18 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc000079fc8 sp=0xc000079f38 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc000079fe0 sp=0xc000079fc8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 35 gp=0xc0005041c0 m=nil [GC worker (idle)]:
runtime.gopark(0xede5e9063e6?, 0x3?, 0x66?, 0xd6?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00050af38 sp=0xc00050af18 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00050afc8 sp=0xc00050af38 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00050afe0 sp=0xc00050afc8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050afe8 sp=0xc00050afe0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 6 gp=0xc000003c00 m=nil [GC worker (idle)]:
runtime.gopark(0xede5e82d894?, 0x1?, 0xb0?, 0x27?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007ef38 sp=0xc00007ef18 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007efc8 sp=0xc00007ef38 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007efe0 sp=0xc00007efc8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007efe8 sp=0xc00007efe0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 23 gp=0xc000103a40 m=nil [GC worker (idle)]:
runtime.gopark(0xede681a8c6d?, 0x1?, 0xa7?, 0x59?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007a738 sp=0xc00007a718 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007a7c8 sp=0xc00007a738 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007a7e0 sp=0xc00007a7c8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007a7e8 sp=0xc00007a7e0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 24 gp=0xc000103c00 m=nil [GC worker (idle)]:
runtime.gopark(0x5b72a5aa0be0?, 0x1?, 0x33?, 0x36?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007af38 sp=0xc00007af18 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007afc8 sp=0xc00007af38 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007afe0 sp=0xc00007afc8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007afe8 sp=0xc00007afe0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 25 gp=0xc000103dc0 m=nil [GC worker (idle)]:
runtime.gopark(0xede5e75699e?, 0x1?, 0xff?, 0xce?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007b738 sp=0xc00007b718 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007b7c8 sp=0xc00007b738 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007b7e0 sp=0xc00007b7c8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007b7e8 sp=0xc00007b7e0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 26 gp=0xc000482000 m=nil [GC worker (idle)]:
runtime.gopark(0x5b72a5aa0be0?, 0x1?, 0xe4?, 0xda?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007bf38 sp=0xc00007bf18 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007bfc8 sp=0xc00007bf38 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 27 gp=0xc0004821c0 m=nil [GC worker (idle), 1 minutes]:
runtime.gopark(0xedd898ba6db?, 0x3?, 0xa0?, 0x7d?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000506738 sp=0xc000506718 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc0005067c8 sp=0xc000506738 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc0005067e0 sp=0xc0005067c8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005067e8 sp=0xc0005067e0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 28 gp=0xc000482380 m=nil [GC worker (idle)]:
runtime.gopark(0x5b72a5aa0be0?, 0x1?, 0x43?, 0xcc?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000506f38 sp=0xc000506f18 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc000506fc8 sp=0xc000506f38 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc000506fe0 sp=0xc000506fc8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000506fe8 sp=0xc000506fe0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 29 gp=0xc000482540 m=nil [GC worker (idle)]:
runtime.gopark(0x5b72a5aa0be0?, 0x1?, 0xc8?, 0xfb?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000507738 sp=0xc000507718 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc0005077c8 sp=0xc000507738 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc0005077e0 sp=0xc0005077c8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005077e8 sp=0xc0005077e0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 36 gp=0xc000504380 m=nil [GC worker (idle)]:
runtime.gopark(0xede67f939d3?, 0x1?, 0xdf?, 0xfe?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00050b738 sp=0xc00050b718 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00050b7c8 sp=0xc00050b738 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00050b7e0 sp=0xc00050b7c8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050b7e8 sp=0xc00050b7e0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 7 gp=0xc000003dc0 m=nil [GC worker (idle)]:
runtime.gopark(0xede6804b387?, 0x1?, 0x49?, 0xbc?, 0x0?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007f738 sp=0xc00007f718 pc=0x5b72a3e9070e
runtime.gcBgMarkWorker(0xc0001117a0)
	/usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007f7c8 sp=0xc00007f738 pc=0x5b72a3e3e769
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007f7e0 sp=0xc00007f7c8 pc=0x5b72a3e3e645
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007f7e8 sp=0xc00007f7e0 pc=0x5b72a3e98681
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/usr/lib/go/src/runtime/mgc.go:1339 +0x105
goroutine 1403 gp=0xc000602fc0 m=nil [IO wait]:
runtime.gopark(0x5b72a3e356a6?, 0xc09fe3f340?, 0x0?, 0x0?, 0xb?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc09fe46dd8 sp=0xc09fe46db8 pc=0x5b72a3e9070e
runtime.netpollblock(0x5b72a3eb43d8?, 0xa3e2a886?, 0x72?)
	/usr/lib/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc09fe46e10 sp=0xc09fe46dd8 pc=0x5b72a3e55d57
internal/poll.runtime_pollWait(0x74c2d6256cc8, 0x72)
	/usr/lib/go/src/runtime/netpoll.go:351 +0x85 fp=0xc09fe46e30 sp=0xc09fe46e10 pc=0x5b72a3e8f925
internal/poll.(*pollDesc).wait(0xc000704100?, 0xc000624251?, 0x0)
	/usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc09fe46e58 sp=0xc09fe46e30 pc=0x5b72a3f17da7
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000704100, {0xc000624251, 0x1, 0x1})
	/usr/lib/go/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc09fe46ef0 sp=0xc09fe46e58 pc=0x5b72a3f1909a
net.(*netFD).Read(0xc000704100, {0xc000624251?, 0xc0000aa0d8?, 0xc09fe46f70?})
	/usr/lib/go/src/net/fd_posix.go:55 +0x25 fp=0xc09fe46f38 sp=0xc09fe46ef0 pc=0x5b72a3f8dfe5
net.(*conn).Read(0xc000708020, {0xc000624251?, 0xc0000aa1c0?, 0x5b72a4277ce0?})
	/usr/lib/go/src/net/net.go:194 +0x45 fp=0xc09fe46f80 sp=0xc09fe46f38 pc=0x5b72a3f9c3a5
net/http.(*connReader).backgroundRead(0xc000624240)
	/usr/lib/go/src/net/http/server.go:690 +0x37 fp=0xc09fe46fc8 sp=0xc09fe46f80 pc=0x5b72a4187cb7
net/http.(*connReader).startBackgroundRead.gowrap2()
	/usr/lib/go/src/net/http/server.go:686 +0x25 fp=0xc09fe46fe0 sp=0xc09fe46fc8 pc=0x5b72a4187be5
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc09fe46fe8 sp=0xc09fe46fe0 pc=0x5b72a3e98681
created by net/http.(*connReader).startBackgroundRead in goroutine 10
	/usr/lib/go/src/net/http/server.go:686 +0xb6
goroutine 10 gp=0xc000603500 m=nil [select]:
runtime.gopark(0xc08fe87a68?, 0x2?, 0x70?, 0x87?, 0xc08fe8780c?)
	/usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc08fe87620 sp=0xc08fe87600 pc=0x5b72a3e9070e
runtime.selectgo(0xc08fe87a68, 0xc08fe87808, 0x62?, 0x0, 0x1?, 0x1)
	/usr/lib/go/src/runtime/select.go:351 +0x837 fp=0xc08fe87758 sp=0xc08fe87620 pc=0x5b72a3e6f857
github.com/ollama/ollama/runner/ollamarunner.(*Server).completion(0xc00012c360, {0x5b72a5182918, 0xc04903f260}, 0xc04956a3c0)
	/build/ollama/src/ollama/runner/ollamarunner/runner.go:649 +0xad0 fp=0xc08fe87ac0 sp=0xc08fe87758 pc=0x5b72a431cd10
github.com/ollama/ollama/runner/ollamarunner.(*Server).completion-fm({0x5b72a5182918?, 0xc04903f260?}, 0xc08fe87b40?)
	<autogenerated>:1 +0x36 fp=0xc08fe87af0 sp=0xc08fe87ac0 pc=0x5b72a431f8d6
net/http.HandlerFunc.ServeHTTP(0xc000020780?, {0x5b72a5182918?, 0xc04903f260?}, 0xc08fe87b60?)
	/usr/lib/go/src/net/http/server.go:2294 +0x29 fp=0xc08fe87b18 sp=0xc08fe87af0 pc=0x5b72a418f8e9
net/http.(*ServeMux).ServeHTTP(0x5b72a3e35785?, {0x5b72a5182918, 0xc04903f260}, 0xc04956a3c0)
	/usr/lib/go/src/net/http/server.go:2822 +0x1c4 fp=0xc08fe87b68 sp=0xc08fe87b18 pc=0x5b72a41917e4
net/http.serverHandler.ServeHTTP({0x5b72a517ef30?}, {0x5b72a5182918?, 0xc04903f260?}, 0x1?)
	/usr/lib/go/src/net/http/server.go:3301 +0x8e fp=0xc08fe87b98 sp=0xc08fe87b68 pc=0x5b72a41af26e
net/http.(*conn).serve(0xc000234120, {0x5b72a51849c8, 0xc000484000})
	/usr/lib/go/src/net/http/server.go:2102 +0x625 fp=0xc08fe87fb8 sp=0xc08fe87b98 pc=0x5b72a418dde5
net/http.(*Server).Serve.gowrap3()
	/usr/lib/go/src/net/http/server.go:3454 +0x28 fp=0xc08fe87fe0 sp=0xc08fe87fb8 pc=0x5b72a41936a8
runtime.goexit({})
	/usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc08fe87fe8 sp=0xc08fe87fe0 pc=0x5b72a3e98681
created by net/http.(*Server).Serve in goroutine 1
	/usr/lib/go/src/net/http/server.go:3454 +0x485
rax    0x74bc2c359b50
rbx    0x1
rcx    0x2
rdx    0x74bc2ca7add0
rdi    0x0
rsi    0x74b8e553f030
rbp    0x74c0a0ffec40
rsp    0x74c0a0ffec18
r8     0x74c0a0ffee10
r9     0xc00011c040
r10    0x1
r11    0x206
r12    0x74bc2c359ac0
r13    0x0
r14    0x74bc2ca7adc0
r15    0x74bc2c359ac0
rip    0x5b72a4c9a2a4
rflags 0x10202
cs     0x33
fs     0x0
gs     0x0
time=2025-03-14T12:03:04.277+02:00 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 2"
[GIN] 2025/03/14 - 12:03:04 | 200 |  1.479187146s |       127.0.0.1 | POST     "/api/chat"

@last-partizan commented on GitHub (Mar 14, 2025): Is this fix supposed to work for AMD? I have RX 6800 XT with 16GB VRAM, and 32 GB RAM, and I tried running gemma3:27b (with GGML_CUDA_ENABLE_UNIFIED_MEMORY=1). ```sh $ ollama run gemma3:27b >>> Hello. Hello there! 👋 How can I help you today? Just let me know what you're thinking, whether it's a question, a request, or just a casual chat. I'm ready when you are! 😊 >>> I just wanted to know it this works with ollama. Seems to be working for small requests, but larger requests are cra ... shing... Error: POST predict: Post "http://127.0.0.1:40453/completion": EOF ``` <details> <summary>journalctl -xfn --unit ollama -o cat</summary> ``` ollama.service: Consumed 39min 37.352s CPU time, 24.1G memory peak, 1.9G memory swap peak. Started Ollama Service. 2025/03/14 12:01:38 routes.go:1225: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/var/lib/ollama OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-03-14T12:01:38.628+02:00 level=INFO source=images.go:432 msg="total blobs: 40" time=2025-03-14T12:01:38.629+02:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" time=2025-03-14T12:01:38.630+02:00 level=INFO source=routes.go:1292 msg="Listening on 127.0.0.1:11434 (version 0.6.0)" time=2025-03-14T12:01:38.630+02:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-03-14T12:01:38.664+02:00 level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2025-03-14T12:01:38.673+02:00 level=INFO source=amd_linux.go:386 msg="amdgpu is supported" gpu=GPU-276bf8a36519a813 gpu_type=gfx1030 time=2025-03-14T12:01:38.673+02:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-276bf8a36519a813 library=rocm variant="" compute=gfx1030 driver=0.0 name=1002:73bf total="16.0 GiB" available="14.5 GiB" [GIN] 2025/03/14 - 12:01:54 | 200 | 40.191µs | 127.0.0.1 | HEAD "/" [GIN] 2025/03/14 - 12:01:55 | 200 | 38.177328ms | 127.0.0.1 | POST "/api/show" time=2025-03-14T12:01:55.130+02:00 level=INFO source=server.go:105 msg="system memory" total="31.3 GiB" free="26.5 GiB" free_swap="4.5 GiB" time=2025-03-14T12:01:55.131+02:00 level=INFO source=server.go:138 msg=offload library=rocm layers.requested=-1 layers.model=63 layers.offload=50 layers.split="" memory.available="[14.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="18.8 GiB" memory.required.partial="14.5 GiB" memory.required.kv="992.0 MiB" memory.required.allocations="[14.5 GiB]" memory.weights.total="15.3 GiB" memory.weights.repeating="14.2 GiB" memory.weights.nonrepeating="1.1 GiB" memory.graph.full="522.5 MiB" memory.graph.partial="1.6 GiB" time=2025-03-14T12:01:55.210+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-03-14T12:01:55.213+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false time=2025-03-14T12:01:55.220+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07 time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30 time=2025-03-14T12:01:55.225+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 time=2025-03-14T12:01:55.226+02:00 level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/bin/ollama runner --ollama-engine --model /var/lib/ollama/blobs/sha256-afa0ea2ef463c87a1eebb9af070e76a353107493b5d9a62e5e66f65a65409541 --ctx-size 2048 --batch-size 512 --n-gpu-layers 50 --threads 8 --parallel 1 --port 40453" time=2025-03-14T12:01:55.226+02:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 time=2025-03-14T12:01:55.226+02:00 level=INFO source=server.go:585 msg="waiting for llama runner to start responding" time=2025-03-14T12:01:55.226+02:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server error" time=2025-03-14T12:01:55.234+02:00 level=INFO source=runner.go:882 msg="starting ollama engine" time=2025-03-14T12:01:55.234+02:00 level=INFO source=runner.go:938 msg="Server listening on 127.0.0.1:40453" time=2025-03-14T12:01:55.310+02:00 level=WARN source=ggml.go:149 msg="key not found" key=general.name default="" time=2025-03-14T12:01:55.310+02:00 level=WARN source=ggml.go:149 msg="key not found" key=general.description default="" time=2025-03-14T12:01:55.310+02:00 level=INFO source=ggml.go:67 msg="" architecture=gemma3 file_type=Q4_K_M name="" description="" num_tensors=1247 num_key_values=36 time=2025-03-14T12:01:55.477+02:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server loading model" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon RX 6800 XT, gfx1030 (0x1030), VMM: no, Wave Size: 32 load_backend: loaded ROCm backend from /usr/lib/ollama/rocm/libggml-hip.so load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-haswell.so time=2025-03-14T12:01:58.356+02:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 ROCm.0.NO_VMM=1 ROCm.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-03-14T12:01:58.808+02:00 level=INFO source=ggml.go:289 msg="model weights" buffer=CPU size="5.8 GiB" time=2025-03-14T12:01:58.808+02:00 level=INFO source=ggml.go:289 msg="model weights" buffer=ROCm0 size="11.5 GiB" time=2025-03-14T12:02:09.993+02:00 level=INFO source=ggml.go:356 msg="compute graph" backend=ROCm0 buffer_type=ROCm0 time=2025-03-14T12:02:10.016+02:00 level=INFO source=ggml.go:356 msg="compute graph" backend=CPU buffer_type=ROCm_Host time=2025-03-14T12:02:10.019+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-03-14T12:02:10.022+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false time=2025-03-14T12:02:10.038+02:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.pretokenizer default="(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" time=2025-03-14T12:02:10.042+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07 time=2025-03-14T12:02:10.043+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 time=2025-03-14T12:02:10.043+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 time=2025-03-14T12:02:10.043+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 time=2025-03-14T12:02:10.043+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.final_logit_softcapping default=30 time=2025-03-14T12:02:10.043+02:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 time=2025-03-14T12:02:10.181+02:00 level=INFO source=server.go:624 msg="llama runner started in 14.95 seconds" [GIN] 2025/03/14 - 12:02:10 | 200 | 15.169570486s | 127.0.0.1 | POST "/api/generate" [GIN] 2025/03/14 - 12:02:26 | 200 | 10.623129145s | 127.0.0.1 | POST "/api/chat" ggml_backend_cuda_buffer_type_alloc_buffer: allocating 12083.39 MiB on device 0: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 12670354944 SIGSEGV: segmentation violation PC=0x5b72a4c9a2a4 m=22 sigcode=1 addr=0x58 signal arrived during cgo execution goroutine 9 gp=0xc000603180 m=22 mp=0xc000581008 [syscall]: runtime.cgocall(0x5b72a4cf3ae0, 0xc000619b00) /usr/lib/go/src/runtime/cgocall.go:167 +0x4b fp=0xc000619ad8 sp=0xc000619aa0 pc=0x5b72a3e8d42b github.com/ollama/ollama/ml/backend/ggml._Cfunc_ggml_backend_sched_graph_compute_async(0x74bc2c01f600, 0x74bc2c684420) _cgo_gotypes.go:489 +0x4a fp=0xc000619b00 sp=0xc000619ad8 pc=0x5b72a427904a github.com/ollama/ollama/ml/backend/ggml.Context.Compute.func1(...) /build/ollama/src/ollama/ml/backend/ggml/ggml.go:497 github.com/ollama/ollama/ml/backend/ggml.Context.Compute({0xc0000aa6c0, 0x74bc2c3583d0, 0x74bc2c684420, 0x0, 0x2000}, {0xc08f3ecb20, 0x1, 0x5b72a5194630?}) /build/ollama/src/ollama/ml/backend/ggml/ggml.go:497 +0xbd fp=0xc000619b90 sp=0xc000619b00 pc=0x5b72a428191d github.com/ollama/ollama/ml/backend/ggml.(*Context).Compute(0xc08f401980?, {0xc08f3ecb20?, 0x40?, 0x0?}) <autogenerated>:1 +0x72 fp=0xc000619c08 sp=0xc000619b90 pc=0x5b72a4287392 github.com/ollama/ollama/model.Forward({0x5b72a518c020, 0xc08f401980}, {0x5b72a51836d0, 0xc00043c000}, {{0xc2817abf00, 0x24, 0x40}, {0x0, 0x0, 0x0}, ...}) /build/ollama/src/ollama/model/model.go:303 +0x218 fp=0xc000619cf0 sp=0xc000619c08 pc=0x5b72a42ae5b8 github.com/ollama/ollama/runner/ollamarunner.(*Server).processBatch(0xc00012c360) /build/ollama/src/ollama/runner/ollamarunner/runner.go:395 +0x3bb fp=0xc000619f98 sp=0xc000619cf0 pc=0x5b72a431ac9b github.com/ollama/ollama/runner/ollamarunner.(*Server).run(0xc00012c360, {0x5b72a5184a00, 0xc0000c34a0}) /build/ollama/src/ollama/runner/ollamarunner/runner.go:321 +0x4e fp=0xc000619fb8 sp=0xc000619f98 pc=0x5b72a431a88e github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap2() /build/ollama/src/ollama/runner/ollamarunner/runner.go:919 +0x28 fp=0xc000619fe0 sp=0xc000619fb8 pc=0x5b72a431f508 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000619fe8 sp=0xc000619fe0 pc=0x5b72a3e98681 created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1 /build/ollama/src/ollama/runner/ollamarunner/runner.go:919 +0xa9c goroutine 1 gp=0xc000002380 m=nil [IO wait, 1 minutes]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000617648 sp=0xc000617628 pc=0x5b72a3e9070e runtime.netpollblock(0xc000617698?, 0xa3e2a886?, 0x72?) /usr/lib/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc000617680 sp=0xc000617648 pc=0x5b72a3e55d57 internal/poll.runtime_pollWait(0x74c2d6256de0, 0x72) /usr/lib/go/src/runtime/netpoll.go:351 +0x85 fp=0xc0006176a0 sp=0xc000617680 pc=0x5b72a3e8f925 internal/poll.(*pollDesc).wait(0xc000705b00?, 0x900000036?, 0x0) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0006176c8 sp=0xc0006176a0 pc=0x5b72a3f17da7 internal/poll.(*pollDesc).waitRead(...) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc000705b00) /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295 fp=0xc000617770 sp=0xc0006176c8 pc=0x5b72a3f1d175 net.(*netFD).accept(0xc000705b00) /usr/lib/go/src/net/fd_unix.go:172 +0x29 fp=0xc000617828 sp=0xc000617770 pc=0x5b72a3f8ff89 net.(*TCPListener).accept(0xc0000ab780) /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b fp=0xc000617878 sp=0xc000617828 pc=0x5b72a3fa593b net.(*TCPListener).Accept(0xc0000ab780) /usr/lib/go/src/net/tcpsock.go:380 +0x30 fp=0xc0006178a8 sp=0xc000617878 pc=0x5b72a3fa47f0 net/http.(*onceCloseListener).Accept(0xc000234120?) <autogenerated>:1 +0x24 fp=0xc0006178c0 sp=0xc0006178a8 pc=0x5b72a41bb9e4 net/http.(*Server).Serve(0xc0001f7600, {0x5b72a5182738, 0xc0000ab780}) /usr/lib/go/src/net/http/server.go:3424 +0x30c fp=0xc0006179f0 sp=0xc0006178c0 pc=0x5b72a41932ac github.com/ollama/ollama/runner/ollamarunner.Execute({0xc00012c150, 0xe, 0xf}) /build/ollama/src/ollama/runner/ollamarunner/runner.go:939 +0xe6a fp=0xc000617d08 sp=0xc0006179f0 pc=0x5b72a431f24a github.com/ollama/ollama/runner.Execute({0xc00012c130?, 0x0?, 0x0?}) /build/ollama/src/ollama/runner/runner.go:20 +0xc9 fp=0xc000617d30 sp=0xc000617d08 pc=0x5b72a431fda9 github.com/ollama/ollama/cmd.NewCLI.func2(0xc0001f7400?, {0x5b72a4cf6194?, 0x4?, 0x5b72a4cf6198?}) /build/ollama/src/ollama/cmd/cmd.go:1285 +0x45 fp=0xc000617d58 sp=0xc000617d30 pc=0x5b72a4a90545 github.com/spf13/cobra.(*Command).execute(0xc000236f08, {0xc0000e6ff0, 0xf, 0xf}) /build/ollama/src/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x85c fp=0xc000617e78 sp=0xc000617d58 pc=0x5b72a400911c github.com/spf13/cobra.(*Command).ExecuteC(0xc0000ea908) /build/ollama/src/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc000617f30 sp=0xc000617e78 pc=0x5b72a4009965 github.com/spf13/cobra.(*Command).Execute(...) /build/ollama/src/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) /build/ollama/src/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985 main.main() /build/ollama/src/ollama/main.go:12 +0x4d fp=0xc000617f50 sp=0xc000617f30 pc=0x5b72a4a908ad runtime.main() /usr/lib/go/src/runtime/proc.go:283 +0x29d fp=0xc000617fe0 sp=0xc000617f50 pc=0x5b72a3e5d35d runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000617fe8 sp=0xc000617fe0 pc=0x5b72a3e98681 goroutine 2 gp=0xc000002e00 m=nil [force gc (idle), 1 minutes]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007cfa8 sp=0xc00007cf88 pc=0x5b72a3e9070e runtime.goparkunlock(...) /usr/lib/go/src/runtime/proc.go:441 runtime.forcegchelper() /usr/lib/go/src/runtime/proc.go:348 +0xb8 fp=0xc00007cfe0 sp=0xc00007cfa8 pc=0x5b72a3e5d698 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007cfe8 sp=0xc00007cfe0 pc=0x5b72a3e98681 created by runtime.init.7 in goroutine 1 /usr/lib/go/src/runtime/proc.go:336 +0x1a goroutine 3 gp=0xc000003340 m=nil [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007d780 sp=0xc00007d760 pc=0x5b72a3e9070e runtime.goparkunlock(...) /usr/lib/go/src/runtime/proc.go:441 runtime.bgsweep(0xc0000a8000) /usr/lib/go/src/runtime/mgcsweep.go:316 +0xdf fp=0xc00007d7c8 sp=0xc00007d780 pc=0x5b72a3e47ebf runtime.gcenable.gowrap1() /usr/lib/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007d7e0 sp=0xc00007d7c8 pc=0x5b72a3e3c2a5 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007d7e8 sp=0xc00007d7e0 pc=0x5b72a3e98681 created by runtime.gcenable in goroutine 1 /usr/lib/go/src/runtime/mgc.go:204 +0x66 goroutine 4 gp=0xc000003500 m=nil [GC scavenge wait]: runtime.gopark(0x10000?, 0x5b72a4eacc88?, 0x0?, 0x0?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007df78 sp=0xc00007df58 pc=0x5b72a3e9070e runtime.goparkunlock(...) /usr/lib/go/src/runtime/proc.go:441 runtime.(*scavengerState).park(0x5b72a59f2460) /usr/lib/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007dfa8 sp=0xc00007df78 pc=0x5b72a3e45909 runtime.bgscavenge(0xc0000a8000) /usr/lib/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00007dfc8 sp=0xc00007dfa8 pc=0x5b72a3e45e99 runtime.gcenable.gowrap2() /usr/lib/go/src/runtime/mgc.go:205 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 pc=0x5b72a3e3c245 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x5b72a3e98681 created by runtime.gcenable in goroutine 1 /usr/lib/go/src/runtime/mgc.go:205 +0xa5 goroutine 18 gp=0xc000102700 m=nil [finalizer wait, 1 minutes]: runtime.gopark(0x1b8?, 0xc000002380?, 0x1?, 0x23?, 0xc00007c688?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007c630 sp=0xc00007c610 pc=0x5b72a3e9070e runtime.runfinq() /usr/lib/go/src/runtime/mfinal.go:196 +0x107 fp=0xc00007c7e0 sp=0xc00007c630 pc=0x5b72a3e3b267 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007c7e8 sp=0xc00007c7e0 pc=0x5b72a3e98681 created by runtime.createfing in goroutine 1 /usr/lib/go/src/runtime/mfinal.go:166 +0x3d goroutine 19 gp=0xc000103180 m=nil [chan receive]: runtime.gopark(0xc000229860?, 0xc00016a180?, 0x60?, 0x87?, 0x5b72a3f76cc8?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000078718 sp=0xc0000786f8 pc=0x5b72a3e9070e runtime.chanrecv(0xc000110380, 0x0, 0x1) /usr/lib/go/src/runtime/chan.go:664 +0x445 fp=0xc000078790 sp=0xc000078718 pc=0x5b72a3e2d465 runtime.chanrecv1(0x0?, 0x0?) /usr/lib/go/src/runtime/chan.go:506 +0x12 fp=0xc0000787b8 sp=0xc000078790 pc=0x5b72a3e2cff2 runtime.unique_runtime_registerUniqueMapCleanup.func2(...) /usr/lib/go/src/runtime/mgc.go:1796 runtime.unique_runtime_registerUniqueMapCleanup.gowrap1() /usr/lib/go/src/runtime/mgc.go:1799 +0x2f fp=0xc0000787e0 sp=0xc0000787b8 pc=0x5b72a3e3f44f runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000787e8 sp=0xc0000787e0 pc=0x5b72a3e98681 created by unique.runtime_registerUniqueMapCleanup in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1794 +0x85 goroutine 20 gp=0xc000103500 m=nil [GC worker (idle), 1 minutes]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000078f38 sp=0xc000078f18 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc000078fc8 sp=0xc000078f38 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc000078fe0 sp=0xc000078fc8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000078fe8 sp=0xc000078fe0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 21 gp=0xc0001036c0 m=nil [GC worker (idle)]: runtime.gopark(0xede5e73158e?, 0x1?, 0xa?, 0x4b?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000079738 sp=0xc000079718 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc0000797c8 sp=0xc000079738 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc0000797e0 sp=0xc0000797c8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000797e8 sp=0xc0000797e0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 34 gp=0xc000504000 m=nil [GC worker (idle)]: runtime.gopark(0xede67800fe3?, 0x1?, 0x1?, 0x3c?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00050a738 sp=0xc00050a718 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00050a7c8 sp=0xc00050a738 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00050a7e0 sp=0xc00050a7c8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050a7e8 sp=0xc00050a7e0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 5 gp=0xc000003a40 m=nil [GC worker (idle)]: runtime.gopark(0xede5e85e70d?, 0x3?, 0x1a?, 0x18?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007e738 sp=0xc00007e718 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007e7c8 sp=0xc00007e738 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007e7e0 sp=0xc00007e7c8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007e7e8 sp=0xc00007e7e0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 22 gp=0xc000103880 m=nil [GC worker (idle)]: runtime.gopark(0xede5e7119d0?, 0x1?, 0xa3?, 0x61?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000079f38 sp=0xc000079f18 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc000079fc8 sp=0xc000079f38 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc000079fe0 sp=0xc000079fc8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 35 gp=0xc0005041c0 m=nil [GC worker (idle)]: runtime.gopark(0xede5e9063e6?, 0x3?, 0x66?, 0xd6?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00050af38 sp=0xc00050af18 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00050afc8 sp=0xc00050af38 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00050afe0 sp=0xc00050afc8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050afe8 sp=0xc00050afe0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 6 gp=0xc000003c00 m=nil [GC worker (idle)]: runtime.gopark(0xede5e82d894?, 0x1?, 0xb0?, 0x27?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007ef38 sp=0xc00007ef18 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007efc8 sp=0xc00007ef38 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007efe0 sp=0xc00007efc8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007efe8 sp=0xc00007efe0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 23 gp=0xc000103a40 m=nil [GC worker (idle)]: runtime.gopark(0xede681a8c6d?, 0x1?, 0xa7?, 0x59?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007a738 sp=0xc00007a718 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007a7c8 sp=0xc00007a738 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007a7e0 sp=0xc00007a7c8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007a7e8 sp=0xc00007a7e0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 24 gp=0xc000103c00 m=nil [GC worker (idle)]: runtime.gopark(0x5b72a5aa0be0?, 0x1?, 0x33?, 0x36?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007af38 sp=0xc00007af18 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007afc8 sp=0xc00007af38 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007afe0 sp=0xc00007afc8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007afe8 sp=0xc00007afe0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 25 gp=0xc000103dc0 m=nil [GC worker (idle)]: runtime.gopark(0xede5e75699e?, 0x1?, 0xff?, 0xce?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007b738 sp=0xc00007b718 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007b7c8 sp=0xc00007b738 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007b7e0 sp=0xc00007b7c8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007b7e8 sp=0xc00007b7e0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 26 gp=0xc000482000 m=nil [GC worker (idle)]: runtime.gopark(0x5b72a5aa0be0?, 0x1?, 0xe4?, 0xda?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007bf38 sp=0xc00007bf18 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007bfc8 sp=0xc00007bf38 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 27 gp=0xc0004821c0 m=nil [GC worker (idle), 1 minutes]: runtime.gopark(0xedd898ba6db?, 0x3?, 0xa0?, 0x7d?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000506738 sp=0xc000506718 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc0005067c8 sp=0xc000506738 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc0005067e0 sp=0xc0005067c8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005067e8 sp=0xc0005067e0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 28 gp=0xc000482380 m=nil [GC worker (idle)]: runtime.gopark(0x5b72a5aa0be0?, 0x1?, 0x43?, 0xcc?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000506f38 sp=0xc000506f18 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc000506fc8 sp=0xc000506f38 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc000506fe0 sp=0xc000506fc8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000506fe8 sp=0xc000506fe0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 29 gp=0xc000482540 m=nil [GC worker (idle)]: runtime.gopark(0x5b72a5aa0be0?, 0x1?, 0xc8?, 0xfb?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc000507738 sp=0xc000507718 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc0005077c8 sp=0xc000507738 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc0005077e0 sp=0xc0005077c8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0005077e8 sp=0xc0005077e0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 36 gp=0xc000504380 m=nil [GC worker (idle)]: runtime.gopark(0xede67f939d3?, 0x1?, 0xdf?, 0xfe?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00050b738 sp=0xc00050b718 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00050b7c8 sp=0xc00050b738 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00050b7e0 sp=0xc00050b7c8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050b7e8 sp=0xc00050b7e0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 7 gp=0xc000003dc0 m=nil [GC worker (idle)]: runtime.gopark(0xede6804b387?, 0x1?, 0x49?, 0xbc?, 0x0?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc00007f738 sp=0xc00007f718 pc=0x5b72a3e9070e runtime.gcBgMarkWorker(0xc0001117a0) /usr/lib/go/src/runtime/mgc.go:1423 +0xe9 fp=0xc00007f7c8 sp=0xc00007f738 pc=0x5b72a3e3e769 runtime.gcBgMarkStartWorkers.gowrap1() /usr/lib/go/src/runtime/mgc.go:1339 +0x25 fp=0xc00007f7e0 sp=0xc00007f7c8 pc=0x5b72a3e3e645 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00007f7e8 sp=0xc00007f7e0 pc=0x5b72a3e98681 created by runtime.gcBgMarkStartWorkers in goroutine 1 /usr/lib/go/src/runtime/mgc.go:1339 +0x105 goroutine 1403 gp=0xc000602fc0 m=nil [IO wait]: runtime.gopark(0x5b72a3e356a6?, 0xc09fe3f340?, 0x0?, 0x0?, 0xb?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc09fe46dd8 sp=0xc09fe46db8 pc=0x5b72a3e9070e runtime.netpollblock(0x5b72a3eb43d8?, 0xa3e2a886?, 0x72?) /usr/lib/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc09fe46e10 sp=0xc09fe46dd8 pc=0x5b72a3e55d57 internal/poll.runtime_pollWait(0x74c2d6256cc8, 0x72) /usr/lib/go/src/runtime/netpoll.go:351 +0x85 fp=0xc09fe46e30 sp=0xc09fe46e10 pc=0x5b72a3e8f925 internal/poll.(*pollDesc).wait(0xc000704100?, 0xc000624251?, 0x0) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc09fe46e58 sp=0xc09fe46e30 pc=0x5b72a3f17da7 internal/poll.(*pollDesc).waitRead(...) /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc000704100, {0xc000624251, 0x1, 0x1}) /usr/lib/go/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc09fe46ef0 sp=0xc09fe46e58 pc=0x5b72a3f1909a net.(*netFD).Read(0xc000704100, {0xc000624251?, 0xc0000aa0d8?, 0xc09fe46f70?}) /usr/lib/go/src/net/fd_posix.go:55 +0x25 fp=0xc09fe46f38 sp=0xc09fe46ef0 pc=0x5b72a3f8dfe5 net.(*conn).Read(0xc000708020, {0xc000624251?, 0xc0000aa1c0?, 0x5b72a4277ce0?}) /usr/lib/go/src/net/net.go:194 +0x45 fp=0xc09fe46f80 sp=0xc09fe46f38 pc=0x5b72a3f9c3a5 net/http.(*connReader).backgroundRead(0xc000624240) /usr/lib/go/src/net/http/server.go:690 +0x37 fp=0xc09fe46fc8 sp=0xc09fe46f80 pc=0x5b72a4187cb7 net/http.(*connReader).startBackgroundRead.gowrap2() /usr/lib/go/src/net/http/server.go:686 +0x25 fp=0xc09fe46fe0 sp=0xc09fe46fc8 pc=0x5b72a4187be5 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc09fe46fe8 sp=0xc09fe46fe0 pc=0x5b72a3e98681 created by net/http.(*connReader).startBackgroundRead in goroutine 10 /usr/lib/go/src/net/http/server.go:686 +0xb6 goroutine 10 gp=0xc000603500 m=nil [select]: runtime.gopark(0xc08fe87a68?, 0x2?, 0x70?, 0x87?, 0xc08fe8780c?) /usr/lib/go/src/runtime/proc.go:435 +0xce fp=0xc08fe87620 sp=0xc08fe87600 pc=0x5b72a3e9070e runtime.selectgo(0xc08fe87a68, 0xc08fe87808, 0x62?, 0x0, 0x1?, 0x1) /usr/lib/go/src/runtime/select.go:351 +0x837 fp=0xc08fe87758 sp=0xc08fe87620 pc=0x5b72a3e6f857 github.com/ollama/ollama/runner/ollamarunner.(*Server).completion(0xc00012c360, {0x5b72a5182918, 0xc04903f260}, 0xc04956a3c0) /build/ollama/src/ollama/runner/ollamarunner/runner.go:649 +0xad0 fp=0xc08fe87ac0 sp=0xc08fe87758 pc=0x5b72a431cd10 github.com/ollama/ollama/runner/ollamarunner.(*Server).completion-fm({0x5b72a5182918?, 0xc04903f260?}, 0xc08fe87b40?) <autogenerated>:1 +0x36 fp=0xc08fe87af0 sp=0xc08fe87ac0 pc=0x5b72a431f8d6 net/http.HandlerFunc.ServeHTTP(0xc000020780?, {0x5b72a5182918?, 0xc04903f260?}, 0xc08fe87b60?) /usr/lib/go/src/net/http/server.go:2294 +0x29 fp=0xc08fe87b18 sp=0xc08fe87af0 pc=0x5b72a418f8e9 net/http.(*ServeMux).ServeHTTP(0x5b72a3e35785?, {0x5b72a5182918, 0xc04903f260}, 0xc04956a3c0) /usr/lib/go/src/net/http/server.go:2822 +0x1c4 fp=0xc08fe87b68 sp=0xc08fe87b18 pc=0x5b72a41917e4 net/http.serverHandler.ServeHTTP({0x5b72a517ef30?}, {0x5b72a5182918?, 0xc04903f260?}, 0x1?) /usr/lib/go/src/net/http/server.go:3301 +0x8e fp=0xc08fe87b98 sp=0xc08fe87b68 pc=0x5b72a41af26e net/http.(*conn).serve(0xc000234120, {0x5b72a51849c8, 0xc000484000}) /usr/lib/go/src/net/http/server.go:2102 +0x625 fp=0xc08fe87fb8 sp=0xc08fe87b98 pc=0x5b72a418dde5 net/http.(*Server).Serve.gowrap3() /usr/lib/go/src/net/http/server.go:3454 +0x28 fp=0xc08fe87fe0 sp=0xc08fe87fb8 pc=0x5b72a41936a8 runtime.goexit({}) /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc08fe87fe8 sp=0xc08fe87fe0 pc=0x5b72a3e98681 created by net/http.(*Server).Serve in goroutine 1 /usr/lib/go/src/net/http/server.go:3454 +0x485 rax 0x74bc2c359b50 rbx 0x1 rcx 0x2 rdx 0x74bc2ca7add0 rdi 0x0 rsi 0x74b8e553f030 rbp 0x74c0a0ffec40 rsp 0x74c0a0ffec18 r8 0x74c0a0ffee10 r9 0xc00011c040 r10 0x1 r11 0x206 r12 0x74bc2c359ac0 r13 0x0 r14 0x74bc2ca7adc0 r15 0x74bc2c359ac0 rip 0x5b72a4c9a2a4 rflags 0x10202 cs 0x33 fs 0x0 gs 0x0 time=2025-03-14T12:03:04.277+02:00 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 2" [GIN] 2025/03/14 - 12:03:04 | 200 | 1.479187146s | 127.0.0.1 | POST "/api/chat" ``` </details>

GiteaMirror commented

2026-04-22 13:03:02 -05:00

@rick-github commented on GitHub (Mar 14, 2025):

Unified memory is an Nvidia driver feature, I don't know if ROCm has anything similar. Other mitigation techniques can be found here.

@rick-github commented on GitHub (Mar 14, 2025): Unified memory is an Nvidia driver feature, I don't know if ROCm has anything similar. Other mitigation techniques can be found [here](https://github.com/ollama/ollama/issues/8597#issuecomment-2614533288).

Sign in to join this conversation.

Branches Tags

main

parth-mlx-decode-checkpoints

dhiltgen/ci

hoyyeva/editor-config-repair

parth-launch-codex-app

hoyyeva/fix-codex-model-metadata-warning

hoyyeva/qwen

hoyyeva/launch-backup-ux

parth/hide-claude-desktop-till-release

hoyyeva/opencode-image-modality

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

hoyyeva/opencode-thinking

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

brucemacd/download-before-remove

parth/update-claude-docs

parth-anthropic-reference-images-path

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#32100