github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-05 23:53:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #10113] CUDA out of memory: Mixed VRAM Cards #6634

New Issue
Open
opened 2026-04-12 18:18:51 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-12 18:18:51 -05:00
Owner
Copy Link

Originally created by @Billnye29 on GitHub (Apr 3, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10113

What is the issue?

Ollama version 0.6.4
GPUs: 2x RTX 3060 12GB, RTX 3090 24GB
CPU: AMD Epyc 7C13
Ram: 512GB

I have two 3060 12GBs and one 3090 24GB.

The OOM error occurs even if I load the model partially into the CPU instead of just the GPUs.
The OOM error seems to occur when Ollama is reading prompt tokens.
When Ollama is reading prompt tokens it keeps trying to use the VRAM on my RTX 3060s instead of the RTX 3090.

The model I have been using is hf.co/mlabonne/gemma-3-27b-it-abliterated-GGUF:Q8_0.
Lets say I have two chats with identical settings.
The chat that has a long message history always errors out.
The chat that doesn't have a message history doesn't error out.

Here are the environment variables I use.
OLLAMA_NUM_PARALLEL = 1
OLLAMA_LOAD_TIMEOUT = 15m
OLLAMA_KEEP_ALIVE = -1m

Reordering the GPUs with the CUDA_VISIBLE_DEVICE parameter had no effect on the behavior of the error.
The error occurs when using any combination of RTX 3090 and RTX 3060s.
However, the error does not occur if I use either just the RTX 3090 or the pair of RTX 3060s.

Thanks

Relevant log output

2025/04/03 07:48:53 routes.go:1231: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:15m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\colin\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-04-03T07:48:53.468-07:00 level=INFO source=images.go:458 msg="total blobs: 56"
time=2025-04-03T07:48:53.472-07:00 level=INFO source=images.go:465 msg="total unused blobs removed: 0"
time=2025-04-03T07:48:53.476-07:00 level=INFO source=routes.go:1298 msg="Listening on 127.0.0.1:11434 (version 0.6.4)"
time=2025-04-03T07:48:53.476-07:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-04-03T07:48:53.476-07:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2025-04-03T07:48:53.476-07:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=64 efficiency=0 threads=128
time=2025-04-03T07:48:53.671-07:00 level=INFO source=gpu.go:319 msg="detected OS VRAM overhead" id=GPU-a34a2513-8707-531b-4b98-b2ac4622dccd library=cuda compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" overhead="328.5 MiB"
time=2025-04-03T07:48:53.770-07:00 level=INFO source=gpu.go:319 msg="detected OS VRAM overhead" id=GPU-34cc13ab-9559-ca5a-84ea-a321fc083eb4 library=cuda compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3060" overhead="868.0 MiB"
time=2025-04-03T07:48:53.874-07:00 level=INFO source=gpu.go:319 msg="detected OS VRAM overhead" id=GPU-48fd61d0-1c11-a994-8910-e9640c258a94 library=cuda compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3060" overhead="868.0 MiB"
time=2025-04-03T07:48:53.876-07:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-a34a2513-8707-531b-4b98-b2ac4622dccd library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" total="24.0 GiB" available="22.8 GiB"
time=2025-04-03T07:48:53.877-07:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-34cc13ab-9559-ca5a-84ea-a321fc083eb4 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3060" total="12.0 GiB" available="11.0 GiB"
time=2025-04-03T07:48:53.877-07:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-48fd61d0-1c11-a994-8910-e9640c258a94 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3060" total="12.0 GiB" available="11.0 GiB"
time=2025-04-03T07:49:14.601-07:00 level=INFO source=sched.go:732 msg="new model will fit in available VRAM, loading" model=C:\Users\colin\.ollama\models\blobs\sha256-157910b4292f3bc69735dd1d0ef93f02937d84d6921debaef661cac5696baad4 library=cuda parallel=1 required="39.8 GiB"
time=2025-04-03T07:49:14.652-07:00 level=INFO source=server.go:105 msg="system memory" total="511.8 GiB" free="498.1 GiB" free_swap="536.8 GiB"
time=2025-04-03T07:49:14.654-07:00 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=63 layers.offload=63 layers.split=31,18,14 memory.available="[22.7 GiB 11.0 GiB 11.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="39.8 GiB" memory.required.partial="39.8 GiB" memory.required.kv="3.1 GiB" memory.required.allocations="[18.4 GiB 10.9 GiB 10.6 GiB]" memory.weights.total="26.7 GiB" memory.weights.repeating="25.3 GiB" memory.weights.nonrepeating="1.4 GiB" memory.graph.full="2.2 GiB" memory.graph.partial="2.2 GiB" projector.weights="818.0 MiB" projector.graph="0 B"
time=2025-04-03T07:49:14.716-07:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
time=2025-04-03T07:49:14.720-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
time=2025-04-03T07:49:14.720-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
time=2025-04-03T07:49:14.720-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0
time=2025-04-03T07:49:14.721-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0
time=2025-04-03T07:49:14.722-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0
time=2025-04-03T07:49:14.723-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0
time=2025-04-03T07:49:14.723-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
time=2025-04-03T07:49:14.724-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
time=2025-04-03T07:49:14.726-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0
time=2025-04-03T07:49:14.736-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
time=2025-04-03T07:49:14.736-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
time=2025-04-03T07:49:14.736-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
time=2025-04-03T07:49:14.738-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
time=2025-04-03T07:49:14.745-07:00 level=INFO source=server.go:405 msg="starting llama server" cmd="C:\\Users\\colin\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\colin\\.ollama\\models\\blobs\\sha256-157910b4292f3bc69735dd1d0ef93f02937d84d6921debaef661cac5696baad4 --ctx-size 32384 --batch-size 512 --n-gpu-layers 63 --threads 64 --no-mmap --mlock --parallel 1 --tensor-split 31,18,14 --port 50311"
time=2025-04-03T07:49:14.748-07:00 level=INFO source=sched.go:451 msg="loaded runners" count=1
time=2025-04-03T07:49:14.748-07:00 level=INFO source=server.go:580 msg="waiting for llama runner to start responding"
time=2025-04-03T07:49:14.749-07:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error"
time=2025-04-03T07:49:14.775-07:00 level=INFO source=runner.go:821 msg="starting ollama engine"
time=2025-04-03T07:49:14.778-07:00 level=INFO source=runner.go:884 msg="Server listening on 127.0.0.1:50311"
time=2025-04-03T07:49:14.837-07:00 level=WARN source=ggml.go:149 msg="key not found" key=general.description default=""
time=2025-04-03T07:49:14.837-07:00 level=INFO source=ggml.go:66 msg="" architecture=gemma3 file_type=Q8_0 name="Gemma 3 27b It Abliterated" description="" num_tensors=808 num_key_values=41
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 3 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
  Device 2: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
time=2025-04-03T07:49:14.999-07:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model"
load_backend: loaded CUDA backend from C:\Users\colin\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\colin\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll
time=2025-04-03T07:49:15.175-07:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-04-03T07:49:15.516-07:00 level=INFO source=ggml.go:288 msg="model weights" buffer=CUDA0 size="12.7 GiB"
time=2025-04-03T07:49:15.516-07:00 level=INFO source=ggml.go:288 msg="model weights" buffer=CUDA1 size="7.4 GiB"
time=2025-04-03T07:49:15.516-07:00 level=INFO source=ggml.go:288 msg="model weights" buffer=CUDA2 size="6.7 GiB"
time=2025-04-03T07:49:15.516-07:00 level=INFO source=ggml.go:288 msg="model weights" buffer=CPU size="1.4 GiB"
time=2025-04-03T07:49:23.326-07:00 level=INFO source=ggml.go:380 msg="compute graph" backend=CUDA0 buffer_type=CUDA0
time=2025-04-03T07:49:23.326-07:00 level=INFO source=ggml.go:380 msg="compute graph" backend=CUDA1 buffer_type=CUDA1
time=2025-04-03T07:49:23.326-07:00 level=INFO source=ggml.go:380 msg="compute graph" backend=CUDA2 buffer_type=CUDA2
time=2025-04-03T07:49:23.326-07:00 level=INFO source=ggml.go:380 msg="compute graph" backend=CPU buffer_type=CUDA_Host
time=2025-04-03T07:49:23.331-07:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false
time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0
time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0
time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0
time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0
time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0
time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0
time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0
time=2025-04-03T07:49:23.346-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000
time=2025-04-03T07:49:23.346-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06
time=2025-04-03T07:49:23.346-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1
time=2025-04-03T07:49:23.346-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256
time=2025-04-03T07:49:23.526-07:00 level=INFO source=server.go:619 msg="llama runner started in 8.78 seconds"
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2202.51 MiB on device 1: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA1 buffer of size 2309496832
Exception 0xc0000005 0x0 0x58 0x7ff756bea554
PC=0x7ff756bea554
signal arrived during external code execution

runtime.cgocall(0x7ff756c53540, 0xc000047a88)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/cgocall.go:167 +0x3e fp=0xc000047a60 sp=0xc0000479f8 pc=0x7ff755dc259e
github.com/ollama/ollama/ml/backend/ggml._Cfunc_ggml_backend_sched_graph_compute_async(0x1855c3be370, 0x1856d541320)
        _cgo_gotypes.go:481 +0x50 fp=0xc000047a88 sp=0xc000047a60 pc=0x7ff7561d4230
github.com/ollama/ollama/ml/backend/ggml.Context.Compute.func1(...)
        C:/a/ollama/ollama/ml/backend/ggml/ggml.go:507
github.com/ollama/ollama/ml/backend/ggml.Context.Compute({0xc001572040, 0x18d9bac7cd0, 0x1856d541320, 0x0, 0x2000}, {0xc0011bc920, 0x1, 0xc001492048?})
        C:/a/ollama/ollama/ml/backend/ggml/ggml.go:507 +0xbd fp=0xc000047b18 sp=0xc000047a88 pc=0x7ff7561dd3bd
github.com/ollama/ollama/ml/backend/ggml.(*Context).Compute(0xc000570db0?, {0xc0011bc920?, 0x0?, 0x0?})
        <autogenerated>:1 +0x72 fp=0xc000047b90 sp=0xc000047b18 pc=0x7ff7561e2e92
github.com/ollama/ollama/model.Forward({0x7ff7570e82f8, 0xc000570db0}, {0x7ff7570dfa50, 0xc002be53e0}, {0xc002f73000, 0x200, 0x200}, {{0x7ff7570f2b38, 0xc0013ff3b0}, {0x0, ...}, ...})
        C:/a/ollama/ollama/model/model.go:312 +0x2b8 fp=0xc000047c70 sp=0xc000047b90 pc=0x7ff75620a3d8
github.com/ollama/ollama/runner/ollamarunner.(*Server).processBatch(0xc0001e6000)
        C:/a/ollama/ollama/runner/ollamarunner/runner.go:478 +0x476 fp=0xc000047f98 sp=0xc000047c70 pc=0x7ff756286a36
github.com/ollama/ollama/runner/ollamarunner.(*Server).run(0xc0001e6000, {0x7ff7570e0d80, 0xc0004bf680})
        C:/a/ollama/ollama/runner/ollamarunner/runner.go:364 +0x4e fp=0xc000047fb8 sp=0xc000047f98 pc=0x7ff75628656e
github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap2()
        C:/a/ollama/ollama/runner/ollamarunner/runner.go:861 +0x28 fp=0xc000047fe0 sp=0xc000047fb8 pc=0x7ff75628aba8
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x7ff755dcd161
created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1
        C:/a/ollama/ollama/runner/ollamarunner/runner.go:861 +0xb37

goroutine 1 gp=0xc0000021c0 m=nil [IO wait, 2 minutes]:
runtime.gopark(0x7ff755dce960?, 0x7ff7579f3ec0?, 0x20?, 0xe0?, 0xc00030e0cc?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc0014254e0 sp=0xc0014254c0 pc=0x7ff755dc596e
runtime.netpollblock(0x244?, 0x55d603e6?, 0xf7?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/netpoll.go:575 +0xf7 fp=0xc001425518 sp=0xc0014254e0 pc=0x7ff755d8b817
internal/poll.runtime_pollWait(0x18558964e00, 0x72)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/netpoll.go:351 +0x85 fp=0xc001425538 sp=0xc001425518 pc=0x7ff755dc4b05
internal/poll.(*pollDesc).wait(0x7ff755e59933?, 0x7ff755d71ef6?, 0x0)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc001425560 sp=0xc001425538 pc=0x7ff755e5af27
internal/poll.execIO(0xc00030e020, 0xc001425608)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_windows.go:177 +0x105 fp=0xc0014255d8 sp=0xc001425560 pc=0x7ff755e5c385
internal/poll.(*FD).acceptOne(0xc00030e008, 0xf64, {0xc000596000?, 0xc001425668?, 0x7ff755e64045?}, 0xc00142569c?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_windows.go:946 +0x65 fp=0xc001425638 sp=0xc0014255d8 pc=0x7ff755e60905
internal/poll.(*FD).Accept(0xc00030e008, 0xc0014257e8)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_windows.go:980 +0x1b6 fp=0xc0014256f0 sp=0xc001425638 pc=0x7ff755e60c36
net.(*netFD).accept(0xc00030e008)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/fd_windows.go:182 +0x4b fp=0xc001425808 sp=0xc0014256f0 pc=0x7ff755ed204b
net.(*TCPListener).accept(0xc0000fc000)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/tcpsock_posix.go:159 +0x1b fp=0xc001425858 sp=0xc001425808 pc=0x7ff755ee809b
net.(*TCPListener).Accept(0xc0000fc000)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/tcpsock.go:380 +0x30 fp=0xc001425888 sp=0xc001425858 pc=0x7ff755ee6e50
net/http.(*onceCloseListener).Accept(0xc0005741b0?)
        <autogenerated>:1 +0x24 fp=0xc0014258a0 sp=0xc001425888 pc=0x7ff756100124
net/http.(*Server).Serve(0xc00019c000, {0x7ff7570deae0, 0xc0000fc000})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/http/server.go:3424 +0x30c fp=0xc0014259d0 sp=0xc0014258a0 pc=0x7ff7560d79ec
github.com/ollama/ollama/runner/ollamarunner.Execute({0xc00004e230, 0x12, 0x1d})
        C:/a/ollama/ollama/runner/ollamarunner/runner.go:885 +0xec9 fp=0xc001425d08 sp=0xc0014259d0 pc=0x7ff75628a909
github.com/ollama/ollama/runner.Execute({0xc00004e210?, 0x0?, 0x0?})
        C:/a/ollama/ollama/runner/runner.go:20 +0xc9 fp=0xc001425d30 sp=0xc001425d08 pc=0x7ff75628b549
github.com/ollama/ollama/cmd.NewCLI.func2(0xc00017af00?, {0x7ff756f0e4a4?, 0x4?, 0x7ff756f0e4a8?})
        C:/a/ollama/ollama/cmd/cmd.go:1344 +0x45 fp=0xc001425d58 sp=0xc001425d30 pc=0x7ff7569d85a5
github.com/spf13/cobra.(*Command).execute(0xc000576f08, {0xc000562500, 0x13, 0x14})
        C:/Users/runneradmin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x85c fp=0xc001425e78 sp=0xc001425d58 pc=0x7ff755f4cb1c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000540908)
        C:/Users/runneradmin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc001425f30 sp=0xc001425e78 pc=0x7ff755f4d365
github.com/spf13/cobra.(*Command).Execute(...)
        C:/Users/runneradmin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        C:/Users/runneradmin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
        C:/a/ollama/ollama/main.go:12 +0x4d fp=0xc001425f50 sp=0xc001425f30 pc=0x7ff7569d890d
runtime.main()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:283 +0x27d fp=0xc001425fe0 sp=0xc001425f50 pc=0x7ff755d947fd
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc001425fe8 sp=0xc001425fe0 pc=0x7ff755dcd161

goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle), 5 minutes]:
runtime.gopark(0x42fc412ed1c?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc0000edfa8 sp=0xc0000edf88 pc=0x7ff755dc596e
runtime.goparkunlock(...)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:441
runtime.forcegchelper()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:348 +0xb8 fp=0xc0000edfe0 sp=0xc0000edfa8 pc=0x7ff755d94b18
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000edfe8 sp=0xc0000edfe0 pc=0x7ff755dcd161
created by runtime.init.7 in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000002c40 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc0000eff80 sp=0xc0000eff60 pc=0x7ff755dc596e
runtime.goparkunlock(...)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:441
runtime.bgsweep(0xc0000ea080)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgcsweep.go:316 +0xdf fp=0xc0000effc8 sp=0xc0000eff80 pc=0x7ff755d7d77f
runtime.gcenable.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:204 +0x25 fp=0xc0000effe0 sp=0xc0000effc8 pc=0x7ff755d71b45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000effe8 sp=0xc0000effe0 pc=0x7ff755dcd161
created by runtime.gcenable in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:204 +0x66

goroutine 4 gp=0xc000002e00 m=nil [GC scavenge wait]:
runtime.gopark(0x83aa4?, 0x79596?, 0x0?, 0x0?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000105f78 sp=0xc000105f58 pc=0x7ff755dc596e
runtime.goparkunlock(...)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:441
runtime.(*scavengerState).park(0x7ff757a1a500)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000105fa8 sp=0xc000105f78 pc=0x7ff755d7b1c9
runtime.bgscavenge(0xc0000ea080)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000105fc8 sp=0xc000105fa8 pc=0x7ff755d7b759
runtime.gcenable.gowrap2()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:205 +0x25 fp=0xc000105fe0 sp=0xc000105fc8 pc=0x7ff755d71ae5
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000105fe8 sp=0xc000105fe0 pc=0x7ff755dcd161
created by runtime.gcenable in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:205 +0xa5

goroutine 5 gp=0xc000003340 m=nil [finalizer wait, 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000107e30 sp=0xc000107e10 pc=0x7ff755dc596e
runtime.runfinq()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mfinal.go:196 +0x107 fp=0xc000107fe0 sp=0xc000107e30 pc=0x7ff755d70ac7
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000107fe8 sp=0xc000107fe0 pc=0x7ff755dcd161
created by runtime.createfing in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mfinal.go:166 +0x3d

goroutine 6 gp=0xc000003dc0 m=nil [chan receive]:
runtime.gopark(0xc000215860?, 0xc0014d0018?, 0x60?, 0x1f?, 0x7ff755ebb088?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc0000f1f18 sp=0xc0000f1ef8 pc=0x7ff755dc596e
runtime.chanrecv(0xc0001803f0, 0x0, 0x1)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/chan.go:664 +0x445 fp=0xc0000f1f90 sp=0xc0000f1f18 pc=0x7ff755d62d25
runtime.chanrecv1(0x7ff755d94960?, 0xc0000f1f76?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/chan.go:506 +0x12 fp=0xc0000f1fb8 sp=0xc0000f1f90 pc=0x7ff755d628b2
runtime.unique_runtime_registerUniqueMapCleanup.func2(...)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1796
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1799 +0x2f fp=0xc0000f1fe0 sp=0xc0000f1fb8 pc=0x7ff755d74d6f
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000f1fe8 sp=0xc0000f1fe0 pc=0x7ff755dcd161
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1794 +0x85

goroutine 7 gp=0xc00045e1c0 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x1?, 0x30?, 0x67?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000101f38 sp=0xc000101f18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000101fc8 sp=0xc000101f38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000101fe0 sp=0xc000101fc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000101fe8 sp=0xc000101fe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 18 gp=0xc00010e1c0 m=nil [GC worker (idle)]:
runtime.gopark(0x7ff757a68fe0?, 0x1?, 0x80?, 0xe?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00011bf38 sp=0xc00011bf18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00011bfc8 sp=0xc00011bf38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00011bfe0 sp=0xc00011bfc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 34 gp=0xc000484000 m=nil [GC worker (idle)]:
runtime.gopark(0x4534be175b0?, 0x3?, 0x88?, 0x72?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000117f38 sp=0xc000117f18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000117fc8 sp=0xc000117f38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000117fe0 sp=0xc000117fc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 8 gp=0xc00045e380 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x1?, 0xec?, 0x77?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000103f38 sp=0xc000103f18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000103fc8 sp=0xc000103f38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000103fe0 sp=0xc000103fc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000103fe8 sp=0xc000103fe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 9 gp=0xc00045e540 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x1?, 0x0?, 0x0?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00046df38 sp=0xc00046df18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00046dfc8 sp=0xc00046df38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00046dfe0 sp=0xc00046dfc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00046dfe8 sp=0xc00046dfe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 19 gp=0xc00010e380 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x3?, 0x0?, 0x0?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00011df38 sp=0xc00011df18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00011dfc8 sp=0xc00011df38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00011dfe0 sp=0xc00011dfc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 20 gp=0xc00010e540 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x3?, 0x8?, 0x64?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000469f38 sp=0xc000469f18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000469fc8 sp=0xc000469f38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000469fe0 sp=0xc000469fc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000469fe8 sp=0xc000469fe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 35 gp=0xc0004841c0 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x1?, 0xec?, 0x77?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000119f38 sp=0xc000119f18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000119fc8 sp=0xc000119f38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000119fe0 sp=0xc000119fc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 10 gp=0xc00045e700 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x3?, 0xf0?, 0x90?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00046ff38 sp=0xc00046ff18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00046ffc8 sp=0xc00046ff38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00046ffe0 sp=0xc00046ffc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00046ffe8 sp=0xc00046ffe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 11 gp=0xc00045e8c0 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bec3b80?, 0x3?, 0x48?, 0xdb?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000475f38 sp=0xc000475f18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000475fc8 sp=0xc000475f38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000475fe0 sp=0xc000475fc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000475fe8 sp=0xc000475fe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 12 gp=0xc00045ea80 m=nil [GC worker (idle)]:
runtime.gopark(0x7ff757a68fe0?, 0x3?, 0x50?, 0xd4?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000477f38 sp=0xc000477f18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000477fc8 sp=0xc000477f38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000477fe0 sp=0xc000477fc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000477fe8 sp=0xc000477fe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 13 gp=0xc00045ec40 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bec3b80?, 0x3?, 0xdc?, 0xf3?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000471f38 sp=0xc000471f18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000471fc8 sp=0xc000471f38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000471fe0 sp=0xc000471fc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000471fe8 sp=0xc000471fe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 14 gp=0xc00045ee00 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x3?, 0x0?, 0x0?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000473f38 sp=0xc000473f18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000473fc8 sp=0xc000473f38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000473fe0 sp=0xc000473fc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000473fe8 sp=0xc000473fe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 15 gp=0xc00045efc0 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x1?, 0x58?, 0x5c?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00047df38 sp=0xc00047df18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00047dfc8 sp=0xc00047df38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00047dfe0 sp=0xc00047dfc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00047dfe8 sp=0xc00047dfe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 16 gp=0xc00045f180 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bec3b80?, 0x3?, 0x58?, 0x38?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00047ff38 sp=0xc00047ff18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00047ffc8 sp=0xc00047ff38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00047ffe0 sp=0xc00047ffc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00047ffe8 sp=0xc00047ffe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 21 gp=0xc00010e700 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x1?, 0x4?, 0x1d?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00046bf38 sp=0xc00046bf18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00046bfc8 sp=0xc00046bf38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00046bfe0 sp=0xc00046bfc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00046bfe8 sp=0xc00046bfe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 22 gp=0xc00010e8c0 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x3?, 0x8?, 0x64?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000479f38 sp=0xc000479f18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000479fc8 sp=0xc000479f38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000479fe0 sp=0xc000479fc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000479fe8 sp=0xc000479fe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 23 gp=0xc00010ea80 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x1?, 0x8?, 0x64?, 0x0?)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00047bf38 sp=0xc00047bf18 pc=0x7ff755dc596e
runtime.gcBgMarkWorker(0xc000181650)
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00047bfc8 sp=0xc00047bf38 pc=0x7ff755d74069
runtime.gcBgMarkStartWorkers.gowrap1()
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00047bfe0 sp=0xc00047bfc8 pc=0x7ff755d73f45
runtime.goexit({})
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00047bfe8 sp=0xc00047bfe0 pc=0x7ff755dcd161
created by runtime.gcBgMarkStartWorkers in goroutine 1
        C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105

goroutine 24 gp=0xc00010ec40 m=nil [GC worker (idle)]:
runtime.gopark(0x4534bd96730?, 0x3?, 0x0
time=2025-04-03T07:54:27.296-07:00 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 2"

OS

Windows

GPU

Nvidia

CPU

AMD

Ollama version

0.6.4

Originally created by @Billnye29 on GitHub (Apr 3, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10113 ### What is the issue? Ollama version 0.6.4 GPUs: 2x RTX 3060 12GB, RTX 3090 24GB CPU: AMD Epyc 7C13 Ram: 512GB I have two 3060 12GBs and one 3090 24GB. The OOM error occurs even if I load the model partially into the CPU instead of just the GPUs. The OOM error seems to occur when Ollama is reading prompt tokens. When Ollama is reading prompt tokens it keeps trying to use the VRAM on my RTX 3060s instead of the RTX 3090. The model I have been using is hf.co/mlabonne/gemma-3-27b-it-abliterated-GGUF:Q8_0. Lets say I have two chats with identical settings. The chat that has a long message history always errors out. The chat that doesn't have a message history doesn't error out. Here are the environment variables I use. OLLAMA_NUM_PARALLEL = 1 OLLAMA_LOAD_TIMEOUT = 15m OLLAMA_KEEP_ALIVE = -1m Reordering the GPUs with the CUDA_VISIBLE_DEVICE parameter had no effect on the behavior of the error. The error occurs when using any combination of RTX 3090 and RTX 3060s. However, the error does not occur if I use either just the RTX 3090 or the pair of RTX 3060s. Thanks ### Relevant log output ```shell 2025/04/03 07:48:53 routes.go:1231: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:15m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\colin\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]" time=2025-04-03T07:48:53.468-07:00 level=INFO source=images.go:458 msg="total blobs: 56" time=2025-04-03T07:48:53.472-07:00 level=INFO source=images.go:465 msg="total unused blobs removed: 0" time=2025-04-03T07:48:53.476-07:00 level=INFO source=routes.go:1298 msg="Listening on 127.0.0.1:11434 (version 0.6.4)" time=2025-04-03T07:48:53.476-07:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-04-03T07:48:53.476-07:00 level=INFO source=gpu_windows.go:167 msg=packages count=1 time=2025-04-03T07:48:53.476-07:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=64 efficiency=0 threads=128 time=2025-04-03T07:48:53.671-07:00 level=INFO source=gpu.go:319 msg="detected OS VRAM overhead" id=GPU-a34a2513-8707-531b-4b98-b2ac4622dccd library=cuda compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" overhead="328.5 MiB" time=2025-04-03T07:48:53.770-07:00 level=INFO source=gpu.go:319 msg="detected OS VRAM overhead" id=GPU-34cc13ab-9559-ca5a-84ea-a321fc083eb4 library=cuda compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3060" overhead="868.0 MiB" time=2025-04-03T07:48:53.874-07:00 level=INFO source=gpu.go:319 msg="detected OS VRAM overhead" id=GPU-48fd61d0-1c11-a994-8910-e9640c258a94 library=cuda compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3060" overhead="868.0 MiB" time=2025-04-03T07:48:53.876-07:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-a34a2513-8707-531b-4b98-b2ac4622dccd library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3090" total="24.0 GiB" available="22.8 GiB" time=2025-04-03T07:48:53.877-07:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-34cc13ab-9559-ca5a-84ea-a321fc083eb4 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3060" total="12.0 GiB" available="11.0 GiB" time=2025-04-03T07:48:53.877-07:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-48fd61d0-1c11-a994-8910-e9640c258a94 library=cuda variant=v12 compute=8.6 driver=12.8 name="NVIDIA GeForce RTX 3060" total="12.0 GiB" available="11.0 GiB" time=2025-04-03T07:49:14.601-07:00 level=INFO source=sched.go:732 msg="new model will fit in available VRAM, loading" model=C:\Users\colin\.ollama\models\blobs\sha256-157910b4292f3bc69735dd1d0ef93f02937d84d6921debaef661cac5696baad4 library=cuda parallel=1 required="39.8 GiB" time=2025-04-03T07:49:14.652-07:00 level=INFO source=server.go:105 msg="system memory" total="511.8 GiB" free="498.1 GiB" free_swap="536.8 GiB" time=2025-04-03T07:49:14.654-07:00 level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=63 layers.offload=63 layers.split=31,18,14 memory.available="[22.7 GiB 11.0 GiB 11.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="39.8 GiB" memory.required.partial="39.8 GiB" memory.required.kv="3.1 GiB" memory.required.allocations="[18.4 GiB 10.9 GiB 10.6 GiB]" memory.weights.total="26.7 GiB" memory.weights.repeating="25.3 GiB" memory.weights.nonrepeating="1.4 GiB" memory.graph.full="2.2 GiB" memory.graph.partial="2.2 GiB" projector.weights="818.0 MiB" projector.graph="0 B" time=2025-04-03T07:49:14.716-07:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false time=2025-04-03T07:49:14.720-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 time=2025-04-03T07:49:14.720-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 time=2025-04-03T07:49:14.720-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0 time=2025-04-03T07:49:14.721-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0 time=2025-04-03T07:49:14.722-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0 time=2025-04-03T07:49:14.723-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0 time=2025-04-03T07:49:14.723-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 time=2025-04-03T07:49:14.724-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 time=2025-04-03T07:49:14.726-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0 time=2025-04-03T07:49:14.736-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 time=2025-04-03T07:49:14.736-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 time=2025-04-03T07:49:14.736-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 time=2025-04-03T07:49:14.738-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 time=2025-04-03T07:49:14.745-07:00 level=INFO source=server.go:405 msg="starting llama server" cmd="C:\\Users\\colin\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\colin\\.ollama\\models\\blobs\\sha256-157910b4292f3bc69735dd1d0ef93f02937d84d6921debaef661cac5696baad4 --ctx-size 32384 --batch-size 512 --n-gpu-layers 63 --threads 64 --no-mmap --mlock --parallel 1 --tensor-split 31,18,14 --port 50311" time=2025-04-03T07:49:14.748-07:00 level=INFO source=sched.go:451 msg="loaded runners" count=1 time=2025-04-03T07:49:14.748-07:00 level=INFO source=server.go:580 msg="waiting for llama runner to start responding" time=2025-04-03T07:49:14.749-07:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error" time=2025-04-03T07:49:14.775-07:00 level=INFO source=runner.go:821 msg="starting ollama engine" time=2025-04-03T07:49:14.778-07:00 level=INFO source=runner.go:884 msg="Server listening on 127.0.0.1:50311" time=2025-04-03T07:49:14.837-07:00 level=WARN source=ggml.go:149 msg="key not found" key=general.description default="" time=2025-04-03T07:49:14.837-07:00 level=INFO source=ggml.go:66 msg="" architecture=gemma3 file_type=Q8_0 name="Gemma 3 27b It Abliterated" description="" num_tensors=808 num_key_values=41 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 3 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 1: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes Device 2: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes time=2025-04-03T07:49:14.999-07:00 level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model" load_backend: loaded CUDA backend from C:\Users\colin\AppData\Local\Programs\Ollama\lib\ollama\cuda_v12\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\colin\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-haswell.dll time=2025-04-03T07:49:15.175-07:00 level=INFO source=ggml.go:109 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-04-03T07:49:15.516-07:00 level=INFO source=ggml.go:288 msg="model weights" buffer=CUDA0 size="12.7 GiB" time=2025-04-03T07:49:15.516-07:00 level=INFO source=ggml.go:288 msg="model weights" buffer=CUDA1 size="7.4 GiB" time=2025-04-03T07:49:15.516-07:00 level=INFO source=ggml.go:288 msg="model weights" buffer=CUDA2 size="6.7 GiB" time=2025-04-03T07:49:15.516-07:00 level=INFO source=ggml.go:288 msg="model weights" buffer=CPU size="1.4 GiB" time=2025-04-03T07:49:23.326-07:00 level=INFO source=ggml.go:380 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 time=2025-04-03T07:49:23.326-07:00 level=INFO source=ggml.go:380 msg="compute graph" backend=CUDA1 buffer_type=CUDA1 time=2025-04-03T07:49:23.326-07:00 level=INFO source=ggml.go:380 msg="compute graph" backend=CUDA2 buffer_type=CUDA2 time=2025-04-03T07:49:23.326-07:00 level=INFO source=ggml.go:380 msg="compute graph" backend=CPU buffer_type=CUDA_Host time=2025-04-03T07:49:23.331-07:00 level=WARN source=ggml.go:149 msg="key not found" key=tokenizer.ggml.add_eot_token default=false time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.num_channels default=0 time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.block_count default=0 time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.embedding_length default=0 time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.head_count default=0 time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.image_size default=0 time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.patch_size default=0 time=2025-04-03T07:49:23.335-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.vision.attention.layer_norm_epsilon default=0 time=2025-04-03T07:49:23.346-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.local.freq_base default=10000 time=2025-04-03T07:49:23.346-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.global.freq_base default=1e+06 time=2025-04-03T07:49:23.346-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.rope.freq_scale default=1 time=2025-04-03T07:49:23.346-07:00 level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 time=2025-04-03T07:49:23.526-07:00 level=INFO source=server.go:619 msg="llama runner started in 8.78 seconds" ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2202.51 MiB on device 1: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate CUDA1 buffer of size 2309496832 Exception 0xc0000005 0x0 0x58 0x7ff756bea554 PC=0x7ff756bea554 signal arrived during external code execution runtime.cgocall(0x7ff756c53540, 0xc000047a88) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/cgocall.go:167 +0x3e fp=0xc000047a60 sp=0xc0000479f8 pc=0x7ff755dc259e github.com/ollama/ollama/ml/backend/ggml._Cfunc_ggml_backend_sched_graph_compute_async(0x1855c3be370, 0x1856d541320) _cgo_gotypes.go:481 +0x50 fp=0xc000047a88 sp=0xc000047a60 pc=0x7ff7561d4230 github.com/ollama/ollama/ml/backend/ggml.Context.Compute.func1(...) C:/a/ollama/ollama/ml/backend/ggml/ggml.go:507 github.com/ollama/ollama/ml/backend/ggml.Context.Compute({0xc001572040, 0x18d9bac7cd0, 0x1856d541320, 0x0, 0x2000}, {0xc0011bc920, 0x1, 0xc001492048?}) C:/a/ollama/ollama/ml/backend/ggml/ggml.go:507 +0xbd fp=0xc000047b18 sp=0xc000047a88 pc=0x7ff7561dd3bd github.com/ollama/ollama/ml/backend/ggml.(*Context).Compute(0xc000570db0?, {0xc0011bc920?, 0x0?, 0x0?}) <autogenerated>:1 +0x72 fp=0xc000047b90 sp=0xc000047b18 pc=0x7ff7561e2e92 github.com/ollama/ollama/model.Forward({0x7ff7570e82f8, 0xc000570db0}, {0x7ff7570dfa50, 0xc002be53e0}, {0xc002f73000, 0x200, 0x200}, {{0x7ff7570f2b38, 0xc0013ff3b0}, {0x0, ...}, ...}) C:/a/ollama/ollama/model/model.go:312 +0x2b8 fp=0xc000047c70 sp=0xc000047b90 pc=0x7ff75620a3d8 github.com/ollama/ollama/runner/ollamarunner.(*Server).processBatch(0xc0001e6000) C:/a/ollama/ollama/runner/ollamarunner/runner.go:478 +0x476 fp=0xc000047f98 sp=0xc000047c70 pc=0x7ff756286a36 github.com/ollama/ollama/runner/ollamarunner.(*Server).run(0xc0001e6000, {0x7ff7570e0d80, 0xc0004bf680}) C:/a/ollama/ollama/runner/ollamarunner/runner.go:364 +0x4e fp=0xc000047fb8 sp=0xc000047f98 pc=0x7ff75628656e github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap2() C:/a/ollama/ollama/runner/ollamarunner/runner.go:861 +0x28 fp=0xc000047fe0 sp=0xc000047fb8 pc=0x7ff75628aba8 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x7ff755dcd161 created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1 C:/a/ollama/ollama/runner/ollamarunner/runner.go:861 +0xb37 goroutine 1 gp=0xc0000021c0 m=nil [IO wait, 2 minutes]: runtime.gopark(0x7ff755dce960?, 0x7ff7579f3ec0?, 0x20?, 0xe0?, 0xc00030e0cc?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc0014254e0 sp=0xc0014254c0 pc=0x7ff755dc596e runtime.netpollblock(0x244?, 0x55d603e6?, 0xf7?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/netpoll.go:575 +0xf7 fp=0xc001425518 sp=0xc0014254e0 pc=0x7ff755d8b817 internal/poll.runtime_pollWait(0x18558964e00, 0x72) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/netpoll.go:351 +0x85 fp=0xc001425538 sp=0xc001425518 pc=0x7ff755dc4b05 internal/poll.(*pollDesc).wait(0x7ff755e59933?, 0x7ff755d71ef6?, 0x0) C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc001425560 sp=0xc001425538 pc=0x7ff755e5af27 internal/poll.execIO(0xc00030e020, 0xc001425608) C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_windows.go:177 +0x105 fp=0xc0014255d8 sp=0xc001425560 pc=0x7ff755e5c385 internal/poll.(*FD).acceptOne(0xc00030e008, 0xf64, {0xc000596000?, 0xc001425668?, 0x7ff755e64045?}, 0xc00142569c?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_windows.go:946 +0x65 fp=0xc001425638 sp=0xc0014255d8 pc=0x7ff755e60905 internal/poll.(*FD).Accept(0xc00030e008, 0xc0014257e8) C:/hostedtoolcache/windows/go/1.24.0/x64/src/internal/poll/fd_windows.go:980 +0x1b6 fp=0xc0014256f0 sp=0xc001425638 pc=0x7ff755e60c36 net.(*netFD).accept(0xc00030e008) C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/fd_windows.go:182 +0x4b fp=0xc001425808 sp=0xc0014256f0 pc=0x7ff755ed204b net.(*TCPListener).accept(0xc0000fc000) C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/tcpsock_posix.go:159 +0x1b fp=0xc001425858 sp=0xc001425808 pc=0x7ff755ee809b net.(*TCPListener).Accept(0xc0000fc000) C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/tcpsock.go:380 +0x30 fp=0xc001425888 sp=0xc001425858 pc=0x7ff755ee6e50 net/http.(*onceCloseListener).Accept(0xc0005741b0?) <autogenerated>:1 +0x24 fp=0xc0014258a0 sp=0xc001425888 pc=0x7ff756100124 net/http.(*Server).Serve(0xc00019c000, {0x7ff7570deae0, 0xc0000fc000}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/net/http/server.go:3424 +0x30c fp=0xc0014259d0 sp=0xc0014258a0 pc=0x7ff7560d79ec github.com/ollama/ollama/runner/ollamarunner.Execute({0xc00004e230, 0x12, 0x1d}) C:/a/ollama/ollama/runner/ollamarunner/runner.go:885 +0xec9 fp=0xc001425d08 sp=0xc0014259d0 pc=0x7ff75628a909 github.com/ollama/ollama/runner.Execute({0xc00004e210?, 0x0?, 0x0?}) C:/a/ollama/ollama/runner/runner.go:20 +0xc9 fp=0xc001425d30 sp=0xc001425d08 pc=0x7ff75628b549 github.com/ollama/ollama/cmd.NewCLI.func2(0xc00017af00?, {0x7ff756f0e4a4?, 0x4?, 0x7ff756f0e4a8?}) C:/a/ollama/ollama/cmd/cmd.go:1344 +0x45 fp=0xc001425d58 sp=0xc001425d30 pc=0x7ff7569d85a5 github.com/spf13/cobra.(*Command).execute(0xc000576f08, {0xc000562500, 0x13, 0x14}) C:/Users/runneradmin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x85c fp=0xc001425e78 sp=0xc001425d58 pc=0x7ff755f4cb1c github.com/spf13/cobra.(*Command).ExecuteC(0xc000540908) C:/Users/runneradmin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc001425f30 sp=0xc001425e78 pc=0x7ff755f4d365 github.com/spf13/cobra.(*Command).Execute(...) C:/Users/runneradmin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) C:/Users/runneradmin/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985 main.main() C:/a/ollama/ollama/main.go:12 +0x4d fp=0xc001425f50 sp=0xc001425f30 pc=0x7ff7569d890d runtime.main() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:283 +0x27d fp=0xc001425fe0 sp=0xc001425f50 pc=0x7ff755d947fd runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc001425fe8 sp=0xc001425fe0 pc=0x7ff755dcd161 goroutine 2 gp=0xc0000028c0 m=nil [force gc (idle), 5 minutes]: runtime.gopark(0x42fc412ed1c?, 0x0?, 0x0?, 0x0?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc0000edfa8 sp=0xc0000edf88 pc=0x7ff755dc596e runtime.goparkunlock(...) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:441 runtime.forcegchelper() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:348 +0xb8 fp=0xc0000edfe0 sp=0xc0000edfa8 pc=0x7ff755d94b18 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000edfe8 sp=0xc0000edfe0 pc=0x7ff755dcd161 created by runtime.init.7 in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:336 +0x1a goroutine 3 gp=0xc000002c40 m=nil [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc0000eff80 sp=0xc0000eff60 pc=0x7ff755dc596e runtime.goparkunlock(...) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:441 runtime.bgsweep(0xc0000ea080) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgcsweep.go:316 +0xdf fp=0xc0000effc8 sp=0xc0000eff80 pc=0x7ff755d7d77f runtime.gcenable.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:204 +0x25 fp=0xc0000effe0 sp=0xc0000effc8 pc=0x7ff755d71b45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000effe8 sp=0xc0000effe0 pc=0x7ff755dcd161 created by runtime.gcenable in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:204 +0x66 goroutine 4 gp=0xc000002e00 m=nil [GC scavenge wait]: runtime.gopark(0x83aa4?, 0x79596?, 0x0?, 0x0?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000105f78 sp=0xc000105f58 pc=0x7ff755dc596e runtime.goparkunlock(...) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:441 runtime.(*scavengerState).park(0x7ff757a1a500) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000105fa8 sp=0xc000105f78 pc=0x7ff755d7b1c9 runtime.bgscavenge(0xc0000ea080) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000105fc8 sp=0xc000105fa8 pc=0x7ff755d7b759 runtime.gcenable.gowrap2() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:205 +0x25 fp=0xc000105fe0 sp=0xc000105fc8 pc=0x7ff755d71ae5 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000105fe8 sp=0xc000105fe0 pc=0x7ff755dcd161 created by runtime.gcenable in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:205 +0xa5 goroutine 5 gp=0xc000003340 m=nil [finalizer wait, 5 minutes]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000107e30 sp=0xc000107e10 pc=0x7ff755dc596e runtime.runfinq() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mfinal.go:196 +0x107 fp=0xc000107fe0 sp=0xc000107e30 pc=0x7ff755d70ac7 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000107fe8 sp=0xc000107fe0 pc=0x7ff755dcd161 created by runtime.createfing in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mfinal.go:166 +0x3d goroutine 6 gp=0xc000003dc0 m=nil [chan receive]: runtime.gopark(0xc000215860?, 0xc0014d0018?, 0x60?, 0x1f?, 0x7ff755ebb088?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc0000f1f18 sp=0xc0000f1ef8 pc=0x7ff755dc596e runtime.chanrecv(0xc0001803f0, 0x0, 0x1) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/chan.go:664 +0x445 fp=0xc0000f1f90 sp=0xc0000f1f18 pc=0x7ff755d62d25 runtime.chanrecv1(0x7ff755d94960?, 0xc0000f1f76?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/chan.go:506 +0x12 fp=0xc0000f1fb8 sp=0xc0000f1f90 pc=0x7ff755d628b2 runtime.unique_runtime_registerUniqueMapCleanup.func2(...) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1796 runtime.unique_runtime_registerUniqueMapCleanup.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1799 +0x2f fp=0xc0000f1fe0 sp=0xc0000f1fb8 pc=0x7ff755d74d6f runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000f1fe8 sp=0xc0000f1fe0 pc=0x7ff755dcd161 created by unique.runtime_registerUniqueMapCleanup in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1794 +0x85 goroutine 7 gp=0xc00045e1c0 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x1?, 0x30?, 0x67?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000101f38 sp=0xc000101f18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000101fc8 sp=0xc000101f38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000101fe0 sp=0xc000101fc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000101fe8 sp=0xc000101fe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 18 gp=0xc00010e1c0 m=nil [GC worker (idle)]: runtime.gopark(0x7ff757a68fe0?, 0x1?, 0x80?, 0xe?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00011bf38 sp=0xc00011bf18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00011bfc8 sp=0xc00011bf38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00011bfe0 sp=0xc00011bfc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 34 gp=0xc000484000 m=nil [GC worker (idle)]: runtime.gopark(0x4534be175b0?, 0x3?, 0x88?, 0x72?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000117f38 sp=0xc000117f18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000117fc8 sp=0xc000117f38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000117fe0 sp=0xc000117fc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 8 gp=0xc00045e380 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x1?, 0xec?, 0x77?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000103f38 sp=0xc000103f18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000103fc8 sp=0xc000103f38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000103fe0 sp=0xc000103fc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000103fe8 sp=0xc000103fe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 9 gp=0xc00045e540 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x1?, 0x0?, 0x0?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00046df38 sp=0xc00046df18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00046dfc8 sp=0xc00046df38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00046dfe0 sp=0xc00046dfc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00046dfe8 sp=0xc00046dfe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 19 gp=0xc00010e380 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x3?, 0x0?, 0x0?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00011df38 sp=0xc00011df18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00011dfc8 sp=0xc00011df38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00011dfe0 sp=0xc00011dfc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 20 gp=0xc00010e540 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x3?, 0x8?, 0x64?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000469f38 sp=0xc000469f18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000469fc8 sp=0xc000469f38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000469fe0 sp=0xc000469fc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000469fe8 sp=0xc000469fe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 35 gp=0xc0004841c0 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x1?, 0xec?, 0x77?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000119f38 sp=0xc000119f18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000119fc8 sp=0xc000119f38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000119fe0 sp=0xc000119fc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 10 gp=0xc00045e700 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x3?, 0xf0?, 0x90?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00046ff38 sp=0xc00046ff18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00046ffc8 sp=0xc00046ff38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00046ffe0 sp=0xc00046ffc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00046ffe8 sp=0xc00046ffe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 11 gp=0xc00045e8c0 m=nil [GC worker (idle)]: runtime.gopark(0x4534bec3b80?, 0x3?, 0x48?, 0xdb?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000475f38 sp=0xc000475f18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000475fc8 sp=0xc000475f38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000475fe0 sp=0xc000475fc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000475fe8 sp=0xc000475fe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 12 gp=0xc00045ea80 m=nil [GC worker (idle)]: runtime.gopark(0x7ff757a68fe0?, 0x3?, 0x50?, 0xd4?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000477f38 sp=0xc000477f18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000477fc8 sp=0xc000477f38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000477fe0 sp=0xc000477fc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000477fe8 sp=0xc000477fe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 13 gp=0xc00045ec40 m=nil [GC worker (idle)]: runtime.gopark(0x4534bec3b80?, 0x3?, 0xdc?, 0xf3?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000471f38 sp=0xc000471f18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000471fc8 sp=0xc000471f38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000471fe0 sp=0xc000471fc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000471fe8 sp=0xc000471fe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 14 gp=0xc00045ee00 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x3?, 0x0?, 0x0?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000473f38 sp=0xc000473f18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000473fc8 sp=0xc000473f38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000473fe0 sp=0xc000473fc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000473fe8 sp=0xc000473fe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 15 gp=0xc00045efc0 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x1?, 0x58?, 0x5c?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00047df38 sp=0xc00047df18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00047dfc8 sp=0xc00047df38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00047dfe0 sp=0xc00047dfc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00047dfe8 sp=0xc00047dfe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 16 gp=0xc00045f180 m=nil [GC worker (idle)]: runtime.gopark(0x4534bec3b80?, 0x3?, 0x58?, 0x38?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00047ff38 sp=0xc00047ff18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00047ffc8 sp=0xc00047ff38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00047ffe0 sp=0xc00047ffc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00047ffe8 sp=0xc00047ffe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 21 gp=0xc00010e700 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x1?, 0x4?, 0x1d?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00046bf38 sp=0xc00046bf18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00046bfc8 sp=0xc00046bf38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00046bfe0 sp=0xc00046bfc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00046bfe8 sp=0xc00046bfe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 22 gp=0xc00010e8c0 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x3?, 0x8?, 0x64?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc000479f38 sp=0xc000479f18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc000479fc8 sp=0xc000479f38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc000479fe0 sp=0xc000479fc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000479fe8 sp=0xc000479fe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 23 gp=0xc00010ea80 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x1?, 0x8?, 0x64?, 0x0?) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/proc.go:435 +0xce fp=0xc00047bf38 sp=0xc00047bf18 pc=0x7ff755dc596e runtime.gcBgMarkWorker(0xc000181650) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1423 +0xe9 fp=0xc00047bfc8 sp=0xc00047bf38 pc=0x7ff755d74069 runtime.gcBgMarkStartWorkers.gowrap1() C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x25 fp=0xc00047bfe0 sp=0xc00047bfc8 pc=0x7ff755d73f45 runtime.goexit({}) C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00047bfe8 sp=0xc00047bfe0 pc=0x7ff755dcd161 created by runtime.gcBgMarkStartWorkers in goroutine 1 C:/hostedtoolcache/windows/go/1.24.0/x64/src/runtime/mgc.go:1339 +0x105 goroutine 24 gp=0xc00010ec40 m=nil [GC worker (idle)]: runtime.gopark(0x4534bd96730?, 0x3?, 0x0 time=2025-04-03T07:54:27.296-07:00 level=ERROR source=server.go:449 msg="llama runner terminated" error="exit status 2" ``` ### OS Windows ### GPU Nvidia ### CPU AMD ### Ollama version 0.6.4
GiteaMirror added the bug label 2026-04-12 18:18:51 -05:00
GiteaMirror commented 2026-04-12 18:18:53 -05:00
Author
Owner
Copy Link

@Billnye29 commented on GitHub (Apr 3, 2025):

Just to add some additional clarification.
Here is my Ollama ps from before the error occurs.

hf.co/mlabonne/gemma-3-27b-it-abliterated-GGUF:Q8_0 2be432ad8c2e 42 GB 100% GPU Forever

Also, the OOM error occurs on all models, not just this one.

<!-- gh-comment-id:2776181227 --> @Billnye29 commented on GitHub (Apr 3, 2025): Just to add some additional clarification. Here is my Ollama ps from before the error occurs. hf.co/mlabonne/gemma-3-27b-it-abliterated-GGUF:Q8_0 2be432ad8c2e 42 GB 100% GPU Forever Also, the OOM error occurs on all models, not just this one.
GiteaMirror commented 2026-04-12 18:18:57 -05:00
Author
Owner
Copy Link

@Billnye29 commented on GitHub (Apr 3, 2025):

I've found a temporary fix.
I set the parameter
OLLAMA_GPU_OVERHEAD = 2GB
This seems to solve the issue for now.

<!-- gh-comment-id:2776391320 --> @Billnye29 commented on GitHub (Apr 3, 2025): I've found a temporary fix. I set the parameter OLLAMA_GPU_OVERHEAD = 2GB This seems to solve the issue for now.
GiteaMirror commented 2026-04-12 18:18:58 -05:00
Author
Owner
Copy Link

@dasoran commented on GitHub (Jun 26, 2025):

I've met same issue.

<!-- gh-comment-id:3008818775 --> @dasoran commented on GitHub (Jun 26, 2025): I've met same issue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6634
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?