github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #12419] 0.12.2 run qwen3 model error: Error: 500 Internal Server Error: do load #34008

New Issue
Closed
opened 2026-04-22 17:14:04 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-22 17:14:04 -05:00
Owner
Copy Link

Originally created by @RaidenAZ on GitHub (Sep 26, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/12419

What is the issue?

I upgraded from version 0.11.5 to 0.12.2 by running tar -C /usr -xzf ollama-linux-amd64.tgz. When I tried to launch the qwen3 model with ollama run, I got the error: Error: 500 Internal Server Error: do load request: Post "http://127.0.0.1:43393/load": EOF
After reviewing the logs and trying the fixes suggested in other issues—such as deleting the cuda_v* directories—the fault still occurs. The log shows:

load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so

load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so

SIGSEGV: segmentation violation

PC=0x55e63fa050d9 m=12 sigcode=128 addr=0x0

signal arrived during cgo execution

The libraries are not being loaded from cuda_v12 or cuda_v13. Rolling back to 0.11.5 restores normal operation—the model loads without any problem. I then completely removed /usr/bin/ollama and performed a fresh install, but the same fault still occurs.

Relevant log output

[GIN] 2025/09/26 - 09:41:37 | 200 |       45.07µs |       127.0.0.1 | HEAD     "/"
time=2025-09-26T09:41:38.973+08:00 level=INFO source=download.go:177 msg="downloading 7f4030143c1c in 6 100 MB part(s)"
time=2025-09-26T09:41:47.693+08:00 level=INFO source=download.go:177 msg="downloading ae370d884f10 in 1 1.7 KB part(s)"
time=2025-09-26T09:41:49.366+08:00 level=INFO source=download.go:177 msg="downloading d18a5cc71b84 in 1 11 KB part(s)"
time=2025-09-26T09:41:51.046+08:00 level=INFO source=download.go:177 msg="downloading cff3f395ef37 in 1 120 B part(s)"
time=2025-09-26T09:41:52.716+08:00 level=INFO source=download.go:177 msg="downloading b0830f4ff6a0 in 1 490 B part(s)"
[GIN] 2025/09/26 - 09:41:54 | 200 | 16.648641496s |       127.0.0.1 | POST     "/api/pull"
[GIN] 2025/09/26 - 09:42:00 | 200 |      32.828µs |       127.0.0.1 | HEAD     "/"
time=2025-09-26T09:42:00.546+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
[GIN] 2025/09/26 - 09:42:00 | 200 |   61.427513ms |       127.0.0.1 | POST     "/api/show"
time=2025-09-26T09:42:00.628+08:00 level=DEBUG source=gpu.go:410 msg="updating system memory data" before.total="62.7 GiB" before.free="58.6 GiB" before.free_swap="2.0 GiB" now.total="62.7 GiB" now.free="58.5 GiB" now.free_swap="2.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08
dlsym: cuInit - 0x7f7d03e47ab0
dlsym: cuDriverGetVersion - 0x7f7d03e47ad0
dlsym: cuDeviceGetCount - 0x7f7d03e47b10
dlsym: cuDeviceGet - 0x7f7d03e47af0
dlsym: cuDeviceGetAttribute - 0x7f7d03e47bf0
dlsym: cuDeviceGetUuid - 0x7f7d03e47b50
dlsym: cuDeviceGetName - 0x7f7d03e47b30
dlsym: cuCtxCreate_v3 - 0x7f7d03e47dd0
dlsym: cuMemGetInfo_v2 - 0x7f7d03e68190
dlsym: cuCtxDestroy - 0x7f7d03ea6ae0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 4
time=2025-09-26T09:42:00.799+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
time=2025-09-26T09:42:00.960+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-cb7d892b-02b5-256f-997d-c93a0f014ea1 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
time=2025-09-26T09:42:01.121+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-fc8226bf-f6bf-8c07-157a-003fa3217d8c name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
time=2025-09-26T09:42:01.281+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-7e1e6171-9d02-0e32-5065-c0d4c5dd3b88 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
releasing cuda driver library
time=2025-09-26T09:42:01.281+08:00 level=DEBUG source=sched.go:188 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=12 gpu_count=4
time=2025-09-26T09:42:01.298+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
time=2025-09-26T09:42:01.298+08:00 level=DEBUG source=sched.go:208 msg="loading first model" model=/root/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa
time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0
time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_eos_token default=false
time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}"
time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.rope.scaling.factor default=1
time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_count default=0
time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_used_count default=0
time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.norm_top_k_prob default=true
time=2025-09-26T09:42:01.370+08:00 level=DEBUG source=gpu.go:410 msg="updating system memory data" before.total="62.7 GiB" before.free="58.5 GiB" before.free_swap="2.0 GiB" now.total="62.7 GiB" now.free="58.4 GiB" now.free_swap="2.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08
dlsym: cuInit - 0x7f7d03e47ab0
dlsym: cuDriverGetVersion - 0x7f7d03e47ad0
dlsym: cuDeviceGetCount - 0x7f7d03e47b10
dlsym: cuDeviceGet - 0x7f7d03e47af0
dlsym: cuDeviceGetAttribute - 0x7f7d03e47bf0
dlsym: cuDeviceGetUuid - 0x7f7d03e47b50
dlsym: cuDeviceGetName - 0x7f7d03e47b30
dlsym: cuCtxCreate_v3 - 0x7f7d03e47dd0
dlsym: cuMemGetInfo_v2 - 0x7f7d03e68190
dlsym: cuCtxDestroy - 0x7f7d03ea6ae0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 4
time=2025-09-26T09:42:01.528+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
time=2025-09-26T09:42:01.681+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-cb7d892b-02b5-256f-997d-c93a0f014ea1 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
time=2025-09-26T09:42:01.837+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-fc8226bf-f6bf-8c07-157a-003fa3217d8c name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
time=2025-09-26T09:42:01.992+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-7e1e6171-9d02-0e32-5065-c0d4c5dd3b88 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
releasing cuda driver library
time=2025-09-26T09:42:01.992+08:00 level=INFO source=server.go:399 msg="starting runner" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa --port 43377"
time=2025-09-26T09:42:01.992+08:00 level=DEBUG source=server.go:400 msg=subprocess OLLAMA_KEEP_ALIVE=-1 OLLAMA_DEBUG=1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_MAX_LOADED_MODELS=12 OLLAMA_LIBRARY_PATH=/usr/lib/ollama LD_LIBRARY_PATH=/usr/lib/ollama:/usr/lib/ollama
time=2025-09-26T09:42:01.993+08:00 level=INFO source=server.go:672 msg="loading model" "model layers"=29 requested=-1
time=2025-09-26T09:42:01.993+08:00 level=DEBUG source=gpu.go:410 msg="updating system memory data" before.total="62.7 GiB" before.free="58.4 GiB" before.free_swap="2.0 GiB" now.total="62.7 GiB" now.free="58.4 GiB" now.free_swap="2.0 GiB"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08
dlsym: cuInit - 0x7f7d03e47ab0
dlsym: cuDriverGetVersion - 0x7f7d03e47ad0
dlsym: cuDeviceGetCount - 0x7f7d03e47b10
dlsym: cuDeviceGet - 0x7f7d03e47af0
dlsym: cuDeviceGetAttribute - 0x7f7d03e47bf0
dlsym: cuDeviceGetUuid - 0x7f7d03e47b50
dlsym: cuDeviceGetName - 0x7f7d03e47b30
dlsym: cuCtxCreate_v3 - 0x7f7d03e47dd0
dlsym: cuMemGetInfo_v2 - 0x7f7d03e68190
dlsym: cuCtxDestroy - 0x7f7d03ea6ae0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 4
time=2025-09-26T09:42:02.008+08:00 level=INFO source=runner.go:1252 msg="starting ollama engine"
time=2025-09-26T09:42:02.009+08:00 level=INFO source=runner.go:1287 msg="Server listening on 127.0.0.1:43377"
time=2025-09-26T09:42:02.148+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
time=2025-09-26T09:42:02.306+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-cb7d892b-02b5-256f-997d-c93a0f014ea1 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
time=2025-09-26T09:42:02.463+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-fc8226bf-f6bf-8c07-157a-003fa3217d8c name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
time=2025-09-26T09:42:02.617+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-7e1e6171-9d02-0e32-5065-c0d4c5dd3b88 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB"
releasing cuda driver library
time=2025-09-26T09:42:02.617+08:00 level=INFO source=server.go:678 msg="system memory" total="62.7 GiB" free="58.4 GiB" free_swap="2.0 GiB"
time=2025-09-26T09:42:02.617+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 available="46.7 GiB" free="47.1 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-09-26T09:42:02.617+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-cb7d892b-02b5-256f-997d-c93a0f014ea1 available="46.7 GiB" free="47.1 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-09-26T09:42:02.617+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-fc8226bf-f6bf-8c07-157a-003fa3217d8c available="46.7 GiB" free="47.1 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-09-26T09:42:02.617+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-7e1e6171-9d02-0e32-5065-c0d4c5dd3b88 available="46.7 GiB" free="47.1 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-09-26T09:42:02.618+08:00 level=INFO source=runner.go:1171 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:false KvSize:4096 KvCacheType: NumThreads:16 GPULayers:29[ID:GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 Layers:29(0..28)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-09-26T09:42:02.650+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
time=2025-09-26T09:42:02.650+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.description default=""
time=2025-09-26T09:42:02.650+08:00 level=INFO source=ggml.go:131 msg="" architecture=qwen3 file_type=Q4_K_M name="Qwen3 0.6B" description="" num_tensors=311 num_key_values=29
time=2025-09-26T09:42:02.650+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/lib/ollama
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA A40, compute capability 8.6, VMM: yes, ID: GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26
  Device 1: NVIDIA A40, compute capability 8.6, VMM: yes, ID: GPU-cb7d892b-02b5-256f-997d-c93a0f014ea1
  Device 2: NVIDIA A40, compute capability 8.6, VMM: yes, ID: GPU-fc8226bf-f6bf-8c07-157a-003fa3217d8c
  Device 3: NVIDIA A40, compute capability 8.6, VMM: yes, ID: GPU-7e1e6171-9d02-0e32-5065-c0d4c5dd3b88
load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so
load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so
SIGSEGV: segmentation violation
PC=0x55e63fa050d9 m=12 sigcode=128 addr=0x0
signal arrived during cgo execution

goroutine 14 gp=0xc000505180 m=12 mp=0xc000580808 [syscall]:
runtime.cgocall(0x55e63f9b1f40, 0xc0000479b8)
        runtime/cgocall.go:167 +0x4b fp=0xc000047990 sp=0xc000047958 pc=0x55e63eca93eb
github.com/ollama/ollama/ml/backend/ggml/ggml/src._Cfunc_ggml_backend_reg_get_proc_address(0x7f6a7cf74260, 0x7f6a7c000c30)
        _cgo_gotypes.go:208 +0x4b fp=0xc0000479b8 sp=0xc000047990 pc=0x55e63f05906b
github.com/ollama/ollama/ml/backend/ggml/ggml/src.system.LogValue.func2.2(...)
        github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:119
github.com/ollama/ollama/ml/backend/ggml/ggml/src.system.LogValue.func2(0x7f6a7cf74260, 0xc000047ba0, 0xc000047c88)
        github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:119 +0xd0 fp=0xc000047b10 sp=0xc0000479b8 pc=0x55e63f0599b0
github.com/ollama/ollama/ml/backend/ggml/ggml/src.system.LogValue({})
        github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:129 +0x145 fp=0xc000047cc8 sp=0xc000047b10 pc=0x55e63f059585
github.com/ollama/ollama/ml/backend/ggml/ggml/src.(*system).LogValue(0x55e64096d760?)
        <autogenerated>:1 +0x1d fp=0xc000047cd8 sp=0xc000047cc8 pc=0x55e63f059efd
log/slog.Value.Resolve({{}, 0xc000047dd0?, {0x55e6400312e0?, 0x55e640ad0ba0?}})
        log/slog/value.go:512 +0x9f fp=0xc000047d90 sp=0xc000047cd8 pc=0x55e63f02209f
log/slog.(*handleState).appendAttr(0xc000048208, {{0x0, 0x0}, {{}, 0x0, {0x55e6400312e0, 0x55e640ad0ba0}}})
        log/slog/handler.go:468 +0x5d fp=0xc000047e78 sp=0xc000047d90 pc=0x55e63f0178dd
log/slog.(*handleState).appendNonBuiltIns.func1(...)
        log/slog/handler.go:342
log/slog.Record.Attrs(...)
        log/slog/record.go:84
log/slog.(*handleState).appendNonBuiltIns(_, {{0xc22d99bac439b370, 0x3fd436cd, 0x55e640a22f80}, {0x55e63fc8b0c7, 0x6}, 0x0, 0x55e63f0588ef, {{{0x0, 0x0}, ...}, ...}, ...})
        log/slog/handler.go:341 +0x4e9 fp=0xc0000480a0 sp=0xc000047e78 pc=0x55e63f016f69
log/slog.(*commonHandler).handle(_, {{0xc22d99bac439b370, 0x3fd436cd, 0x55e640a22f80}, {0x55e63fc8b0c7, 0x6}, 0x0, 0x55e63f0588ef, {{{0x0, 0x0}, ...}, ...}, ...})
        log/slog/handler.go:308 +0x5c5 fp=0xc0000482e0 sp=0xc0000480a0 pc=0x55e63f016825
log/slog.(*TextHandler).Handle(_, {_, _}, {{0xc22d99bac439b370, 0x3fd436cd, 0x55e640a22f80}, {0x55e63fc8b0c7, 0x6}, 0x0, 0x55e63f0588ef, ...})
        log/slog/text_handler.go:95 +0x3f fp=0xc000048418 sp=0xc0000482e0 pc=0x55e63f01e45f
log/slog.(*Logger).log(0xc00047b200, {0x55e640175a40, 0x55e640ad0ba0}, 0x0, {0x55e63fc8b0c7, 0x6}, {0xc0000488d8, 0x2, 0x2})
        log/slog/logger.go:256 +0x1e4 fp=0xc0000487b0 sp=0xc000048418 pc=0x55e63f01be04
log/slog.Info(...)
        log/slog/logger.go:291
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.func1()
        github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:104 +0x5be fp=0xc000048a40 sp=0xc0000487b0 pc=0x55e63f05895e
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func2()
        sync/oncefunc.go:27 +0x62 fp=0xc000048a88 sp=0xc000048a40 pc=0x55e63f0582c2
sync.(*Once).doSlow(0x55e640175a40?, 0x55e640ad0ba0?)
        sync/once.go:78 +0xab fp=0xc000048ae0 sp=0xc000048a88 pc=0x55e63ecbe4ab
sync.(*Once).Do(0x0?, 0xc000048b88?)
        sync/once.go:69 +0x19 fp=0xc000048b00 sp=0xc000048ae0 pc=0x55e63ecbe3d9
github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func3()
        sync/oncefunc.go:32 +0x2d fp=0xc000048b30 sp=0xc000048b00 pc=0x55e63f05822d
github.com/ollama/ollama/ml/backend/ggml.init.func1()
        github.com/ollama/ollama/ml/backend/ggml/ggml.go:44 +0x23 fp=0xc000048bc0 sp=0xc000048b30 pc=0x55e63f0e9863
github.com/ollama/ollama/ml/backend/ggml.init.OnceFunc.func2()
        sync/oncefunc.go:27 +0x62 fp=0xc000048c08 sp=0xc000048bc0 pc=0x55e63f0e9762
sync.(*Once).doSlow(0x155e64016b718?, 0xc000594058?)
        sync/once.go:78 +0xab fp=0xc000048c60 sp=0xc000048c08 pc=0x55e63ecbe4ab
sync.(*Once).Do(0x55e63ecbe560?, 0x55e640ad11bc?)
        sync/once.go:69 +0x19 fp=0xc000048c80 sp=0xc000048c60 pc=0x55e63ecbe3d9
github.com/ollama/ollama/ml/backend/ggml.init.OnceFunc.func3()
        sync/oncefunc.go:32 +0x2d fp=0xc000048cb0 sp=0xc000048c80 pc=0x55e63f0e96cd
github.com/ollama/ollama/ml/backend/ggml.New({0x7ffddf4da665, 0x62}, {0x0, 0x10, {0xc000189dd0, 0x1, 0x1}, 0x0})
        github.com/ollama/ollama/ml/backend/ggml/ggml.go:142 +0x124 fp=0xc000049580 sp=0xc000048cb0 pc=0x55e63f0f1f64
github.com/ollama/ollama/ml.NewBackend({0x7ffddf4da665, 0x62}, {0x0, 0x10, {0xc000189dd0, 0x1, 0x1}, 0x0})
        github.com/ollama/ollama/ml/backend.go:358 +0x9c fp=0xc0000495d0 sp=0xc000049580 pc=0x55e63f09127c
github.com/ollama/ollama/model.New({0x7ffddf4da665?, 0x0?}, {0x0, 0x10, {0xc000189dd0, 0x1, 0x1}, 0x0})
        github.com/ollama/ollama/model/model.go:106 +0x7e fp=0xc0000496a0 sp=0xc0000495d0 pc=0x55e63f10641e
github.com/ollama/ollama/runner/ollamarunner.(*Server).allocModel(0xc0000374a0, {0x7ffddf4da665?, 0x55e63efa043a?}, {0x0, 0x10, {0xc000189dd0, 0x1, 0x1}, 0x0}, {0x0, ...}, ...)
        github.com/ollama/ollama/runner/ollamarunner/runner.go:1100 +0xcc fp=0xc000049730 sp=0xc0000496a0 pc=0x55e63f1bb1cc
github.com/ollama/ollama/runner/ollamarunner.(*Server).load(0xc0000374a0, {0x55e640173748, 0xc00014c000}, 0xc00046a000)
        github.com/ollama/ollama/runner/ollamarunner/runner.go:1198 +0x54d fp=0xc000049ac0 sp=0xc000049730 pc=0x55e63f1bbdad
github.com/ollama/ollama/runner/ollamarunner.(*Server).load-fm({0x55e640173748?, 0xc00014c000?}, 0xc000123b40?)
        <autogenerated>:1 +0x36 fp=0xc000049af0 sp=0xc000049ac0 pc=0x55e63f1bd5d6
net/http.HandlerFunc.ServeHTTP(0xc0004a3200?, {0x55e640173748?, 0xc00014c000?}, 0xc000123b60?)
        net/http/server.go:2294 +0x29 fp=0xc000049b18 sp=0xc000049af0 pc=0x55e63efab109
net/http.(*ServeMux).ServeHTTP(0x55e63ec50d85?, {0x55e640173748, 0xc00014c000}, 0xc00046a000)
        net/http/server.go:2822 +0x1c4 fp=0xc000049b68 sp=0xc000049b18 pc=0x55e63efad004
net/http.serverHandler.ServeHTTP({0x55e64016fd90?}, {0x55e640173748?, 0xc00014c000?}, 0x1?)
        net/http/server.go:3301 +0x8e fp=0xc000049b98 sp=0xc000049b68 pc=0x55e63efcaa8e
net/http.(*conn).serve(0xc0004cc3f0, {0x55e640175a78, 0xc000690d50})
        net/http/server.go:2102 +0x625 fp=0xc000049fb8 sp=0xc000049b98 pc=0x55e63efa9605
net/http.(*Server).Serve.gowrap3()
        net/http/server.go:3454 +0x28 fp=0xc000049fe0 sp=0xc000049fb8 pc=0x55e63efaeec8
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000049fe8 sp=0xc000049fe0 pc=0x55e63ecb3fa1
created by net/http.(*Server).Serve in goroutine 1
        net/http/server.go:3454 +0x485

goroutine 1 gp=0xc000002380 m=nil [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000bf9788 sp=0xc000bf9768 pc=0x55e63ecac86e
runtime.netpollblock(0xc00050f7d8?, 0x3ec45666?, 0xe6?)
        runtime/netpoll.go:575 +0xf7 fp=0xc000bf97c0 sp=0xc000bf9788 pc=0x55e63ec71357
internal/poll.runtime_pollWait(0x7f6ab2b5eeb0, 0x72)
        runtime/netpoll.go:351 +0x85 fp=0xc000bf97e0 sp=0xc000bf97c0 pc=0x55e63ecaba85
internal/poll.(*pollDesc).wait(0xc000686a80?, 0x900000036?, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000bf9808 sp=0xc000bf97e0 pc=0x55e63ed32ec7
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000686a80)
        internal/poll/fd_unix.go:620 +0x295 fp=0xc000bf98b0 sp=0xc000bf9808 pc=0x55e63ed38295
net.(*netFD).accept(0xc000686a80)
        net/fd_unix.go:172 +0x29 fp=0xc000bf9968 sp=0xc000bf98b0 pc=0x55e63edab249
net.(*TCPListener).accept(0xc0006a0f40)
        net/tcpsock_posix.go:159 +0x1b fp=0xc000bf99b8 sp=0xc000bf9968 pc=0x55e63edc0bfb
net.(*TCPListener).Accept(0xc0006a0f40)
        net/tcpsock.go:380 +0x30 fp=0xc000bf99e8 sp=0xc000bf99b8 pc=0x55e63edbfab0
net/http.(*onceCloseListener).Accept(0xc0004cc3f0?)
        <autogenerated>:1 +0x24 fp=0xc000bf9a00 sp=0xc000bf99e8 pc=0x55e63efd7204
net/http.(*Server).Serve(0xc000507100, {0x55e640173568, 0xc0006a0f40})
        net/http/server.go:3424 +0x30c fp=0xc000bf9b30 sp=0xc000bf9a00 pc=0x55e63efaeacc
github.com/ollama/ollama/runner/ollamarunner.Execute({0xc0000340a0, 0x4, 0x4})
        github.com/ollama/ollama/runner/ollamarunner/runner.go:1288 +0x935 fp=0xc000bf9d08 sp=0xc000bf9b30 pc=0x55e63f1bcff5
github.com/ollama/ollama/runner.Execute({0xc000034080?, 0x0?, 0x0?})
        github.com/ollama/ollama/runner/runner.go:20 +0xc9 fp=0xc000bf9d30 sp=0xc000bf9d08 pc=0x55e63f1bd869
github.com/ollama/ollama/cmd.NewCLI.func2(0xc000506f00?, {0x55e63fc850b2?, 0x4?, 0x55e63fc850b6?})
        github.com/ollama/ollama/cmd/cmd.go:1706 +0x45 fp=0xc000bf9d58 sp=0xc000bf9d30 pc=0x55e63f92f3a5
github.com/spf13/cobra.(*Command).execute(0xc0004cf508, {0xc000390690, 0x5, 0x5})
        github.com/spf13/cobra@v1.7.0/command.go:940 +0x85c fp=0xc000bf9e78 sp=0xc000bf9d58 pc=0x55e63ee2489c
github.com/spf13/cobra.(*Command).ExecuteC(0xc0004b0908)
        github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc000bf9f30 sp=0xc000bf9e78 pc=0x55e63ee250e5
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
        github.com/ollama/ollama/main.go:12 +0x4d fp=0xc000bf9f50 sp=0xc000bf9f30 pc=0x55e63f92fe6d
runtime.main()
        runtime/proc.go:283 +0x29d fp=0xc000bf9fe0 sp=0xc000bf9f50 pc=0x55e63ec789dd
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000bf9fe8 sp=0xc000bf9fe0 pc=0x55e63ecb3fa1

goroutine 2 gp=0xc000002e00 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000084fa8 sp=0xc000084f88 pc=0x55e63ecac86e
runtime.goparkunlock(...)
        runtime/proc.go:441
runtime.forcegchelper()
        runtime/proc.go:348 +0xb8 fp=0xc000084fe0 sp=0xc000084fa8 pc=0x55e63ec78d18
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000084fe8 sp=0xc000084fe0 pc=0x55e63ecb3fa1
created by runtime.init.7 in goroutine 1
        runtime/proc.go:336 +0x1a

goroutine 3 gp=0xc000003340 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000085780 sp=0xc000085760 pc=0x55e63ecac86e
runtime.goparkunlock(...)
        runtime/proc.go:441
runtime.bgsweep(0xc0000ac000)
        runtime/mgcsweep.go:316 +0xdf fp=0xc0000857c8 sp=0xc000085780 pc=0x55e63ec634bf
runtime.gcenable.gowrap1()
        runtime/mgc.go:204 +0x25 fp=0xc0000857e0 sp=0xc0000857c8 pc=0x55e63ec578a5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000857e8 sp=0xc0000857e0 pc=0x55e63ecb3fa1
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0x66

goroutine 4 gp=0xc000003500 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x55e63fe4d728?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0x55e63ecac86e
runtime.goparkunlock(...)
        runtime/proc.go:441
runtime.(*scavengerState).park(0x55e640a231a0)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0x55e63ec60f09
runtime.bgscavenge(0xc0000ac000)
        runtime/mgcscavenge.go:658 +0x59 fp=0xc000085fc8 sp=0xc000085fa8 pc=0x55e63ec61499
runtime.gcenable.gowrap2()
        runtime/mgc.go:205 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0x55e63ec57845
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x55e63ecb3fa1
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:205 +0xa5

goroutine 5 gp=0xc000003dc0 m=nil [finalizer wait]:
runtime.gopark(0x1b8?, 0xc000002380?, 0x1?, 0x23?, 0xc000084688?)
        runtime/proc.go:435 +0xce fp=0xc000084630 sp=0xc000084610 pc=0x55e63ecac86e
runtime.runfinq()
        runtime/mfinal.go:196 +0x107 fp=0xc0000847e0 sp=0xc000084630 pc=0x55e63ec56867
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000847e8 sp=0xc0000847e0 pc=0x55e63ecb3fa1
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:166 +0x3d

goroutine 6 gp=0xc0001e08c0 m=nil [chan receive]:
runtime.gopark(0xc000235ae0?, 0xc0000101b0?, 0x60?, 0x67?, 0x55e63ed91e88?)
        runtime/proc.go:435 +0xce fp=0xc000086718 sp=0xc0000866f8 pc=0x55e63ecac86e
runtime.chanrecv(0xc000042380, 0x0, 0x1)
        runtime/chan.go:664 +0x445 fp=0xc000086790 sp=0xc000086718 pc=0x55e63ec48245
runtime.chanrecv1(0x0?, 0x0?)
        runtime/chan.go:506 +0x12 fp=0xc0000867b8 sp=0xc000086790 pc=0x55e63ec47dd2
runtime.unique_runtime_registerUniqueMapCleanup.func2(...)
        runtime/mgc.go:1796
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        runtime/mgc.go:1799 +0x2f fp=0xc0000867e0 sp=0xc0000867b8 pc=0x55e63ec5aa4f
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x55e63ecb3fa1
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        runtime/mgc.go:1794 +0x85

goroutine 7 gp=0xc0001e0c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000086f38 sp=0xc000086f18 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc000086fc8 sp=0xc000086f38 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc000086fe0 sp=0xc000086fc8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000086fe8 sp=0xc000086fe0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 8 gp=0xc0001e0e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000087738 sp=0xc000087718 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc0000877c8 sp=0xc000087738 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0000877e0 sp=0xc0000877c8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 18 gp=0xc000504000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000080738 sp=0xc000080718 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc0000807c8 sp=0xc000080738 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0000807e0 sp=0xc0000807c8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000807e8 sp=0xc0000807e0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 19 gp=0xc0005041c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000080f38 sp=0xc000080f18 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc000080fc8 sp=0xc000080f38 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc000080fe0 sp=0xc000080fc8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000080fe8 sp=0xc000080fe0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 34 gp=0xc000102380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc00011a738 sp=0xc00011a718 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc00011a7c8 sp=0xc00011a738 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc00011a7e0 sp=0xc00011a7c8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00011a7e8 sp=0xc00011a7e0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 35 gp=0xc000102540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc00011af38 sp=0xc00011af18 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc00011afc8 sp=0xc00011af38 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc00011afe0 sp=0xc00011afc8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00011afe8 sp=0xc00011afe0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 9 gp=0xc0001e0fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000087f38 sp=0xc000087f18 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc000087fc8 sp=0xc000087f38 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc000087fe0 sp=0xc000087fc8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 10 gp=0xc0001e1180 m=nil [GC worker (idle)]:
runtime.gopark(0xb0b92c230e30c?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000116738 sp=0xc000116718 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc0001167c8 sp=0xc000116738 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0001167e0 sp=0xc0001167c8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0001167e8 sp=0xc0001167e0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 20 gp=0xc000504380 m=nil [GC worker (idle)]:
runtime.gopark(0xb0b92c230d7b4?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000081738 sp=0xc000081718 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc0000817c8 sp=0xc000081738 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0000817e0 sp=0xc0000817c8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000817e8 sp=0xc0000817e0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 21 gp=0xc000504540 m=nil [GC worker (idle)]:
runtime.gopark(0xb0b92c23119d0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000081f38 sp=0xc000081f18 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc000081fc8 sp=0xc000081f38 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc000081fe0 sp=0xc000081fc8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000081fe8 sp=0xc000081fe0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 22 gp=0xc000504700 m=nil [GC worker (idle)]:
runtime.gopark(0xb0b92c231cf0a?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000082738 sp=0xc000082718 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc0000827c8 sp=0xc000082738 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0000827e0 sp=0xc0000827c8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0000827e8 sp=0xc0000827e0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 36 gp=0xc000102700 m=nil [GC worker (idle)]:
runtime.gopark(0xb0b92c230e467?, 0x1?, 0x3?, 0xee?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc00011b738 sp=0xc00011b718 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc00011b7c8 sp=0xc00011b738 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc00011b7e0 sp=0xc00011b7c8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00011b7e8 sp=0xc00011b7e0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 11 gp=0xc0001e1340 m=nil [GC worker (idle)]:
runtime.gopark(0xb0b92c230de16?, 0x3?, 0xc7?, 0x62?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000116f38 sp=0xc000116f18 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc000116fc8 sp=0xc000116f38 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc000116fe0 sp=0xc000116fc8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000116fe8 sp=0xc000116fe0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 12 gp=0xc0001e1500 m=nil [GC worker (idle)]:
runtime.gopark(0xb0b92c248d14b?, 0x1?, 0xcd?, 0x8b?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000117738 sp=0xc000117718 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc0001177c8 sp=0xc000117738 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc0001177e0 sp=0xc0001177c8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc0001177e8 sp=0xc0001177e0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 37 gp=0xc0001028c0 m=nil [GC worker (idle)]:
runtime.gopark(0xb0b92c230f2b4?, 0x1?, 0xec?, 0xd0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc00011bf38 sp=0xc00011bf18 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc00011bfc8 sp=0xc00011bf38 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc00011bfe0 sp=0xc00011bfc8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 38 gp=0xc000102a80 m=nil [GC worker (idle)]:
runtime.gopark(0x55e640ad2ea0?, 0x1?, 0x70?, 0x4a?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc00011c738 sp=0xc00011c718 pc=0x55e63ecac86e
runtime.gcBgMarkWorker(0xc000043960)
        runtime/mgc.go:1423 +0xe9 fp=0xc00011c7c8 sp=0xc00011c738 pc=0x55e63ec59d69
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1339 +0x25 fp=0xc00011c7e0 sp=0xc00011c7c8 pc=0x55e63ec59c45
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00011c7e8 sp=0xc00011c7e0 pc=0x55e63ecb3fa1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1339 +0x105

goroutine 13 gp=0xc000504fc0 m=nil [sync.WaitGroup.Wait]:
runtime.gopark(0x0?, 0x0?, 0xc0?, 0xe0?, 0x0?)
        runtime/proc.go:435 +0xce fp=0xc000093b30 sp=0xc000093b10 pc=0x55e63ecac86e
runtime.goparkunlock(...)
        runtime/proc.go:441
runtime.semacquire1(0xc000037558, 0x0, 0x1, 0x0, 0x18)
        runtime/sema.go:188 +0x229 fp=0xc000093b98 sp=0xc000093b30 pc=0x55e63ec8bfa9
sync.runtime_SemacquireWaitGroup(0x0?)
        runtime/sema.go:110 +0x25 fp=0xc000093bd0 sp=0xc000093b98 pc=0x55e63ecae285
sync.(*WaitGroup).Wait(0x0?)
        sync/waitgroup.go:118 +0x48 fp=0xc000093bf8 sp=0xc000093bd0 pc=0x55e63ecbf8e8
github.com/ollama/ollama/runner/ollamarunner.(*Server).run(0xc0000374a0, {0x55e640175ab0, 0xc000390730})
        github.com/ollama/ollama/runner/ollamarunner/runner.go:406 +0x45 fp=0xc000093fb8 sp=0xc000093bf8 pc=0x55e63f1b4b85
github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap1()
        github.com/ollama/ollama/runner/ollamarunner/runner.go:1266 +0x28 fp=0xc000093fe0 sp=0xc000093fb8 pc=0x55e63f1bd268
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x55e63ecb3fa1
created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1
        github.com/ollama/ollama/runner/ollamarunner/runner.go:1266 +0x505

goroutine 50 gp=0xc000505340 m=nil [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0xb?)
        runtime/proc.go:435 +0xce fp=0xc00011d5d8 sp=0xc00011d5b8 pc=0x55e63ecac86e
runtime.netpollblock(0x55e63eccfbd8?, 0x3ec45666?, 0xe6?)
        runtime/netpoll.go:575 +0xf7 fp=0xc00011d610 sp=0xc00011d5d8 pc=0x55e63ec71357
internal/poll.runtime_pollWait(0x7f6ab2b5ed98, 0x72)
        runtime/netpoll.go:351 +0x85 fp=0xc00011d630 sp=0xc00011d610 pc=0x55e63ecaba85
internal/poll.(*pollDesc).wait(0xc000686b00?, 0xc000690e51?, 0x0)
        internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00011d658 sp=0xc00011d630 pc=0x55e63ed32ec7
internal/poll.(*pollDesc).waitRead(...)
        internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000686b00, {0xc000690e51, 0x1, 0x1})
        internal/poll/fd_unix.go:165 +0x27a fp=0xc00011d6f0 sp=0xc00011d658 pc=0x55e63ed341ba
net.(*netFD).Read(0xc000686b00, {0xc000690e51?, 0xc0006a1018?, 0xc00011d770?})
        net/fd_posix.go:55 +0x25 fp=0xc00011d738 sp=0xc00011d6f0 pc=0x55e63eda92a5
net.(*conn).Read(0xc00007c8b0, {0xc000690e51?, 0x0?, 0x0?})
        net/net.go:194 +0x45 fp=0xc00011d780 sp=0xc00011d738 pc=0x55e63edb7665
net/http.(*connReader).backgroundRead(0xc000690e40)
        net/http/server.go:690 +0x37 fp=0xc00011d7c8 sp=0xc00011d780 pc=0x55e63efa34d7
net/http.(*connReader).startBackgroundRead.gowrap2()
        net/http/server.go:686 +0x25 fp=0xc00011d7e0 sp=0xc00011d7c8 pc=0x55e63efa3405
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00011d7e8 sp=0xc00011d7e0 pc=0x55e63ecb3fa1
created by net/http.(*connReader).startBackgroundRead in goroutine 14
        net/http/server.go:686 +0xb6

rax    0x365f4c414e524554
rbx    0xc0000479b8
rcx    0xffffffffffffffd8
rdx    0xc000047948
rdi    0x7f6a7cf74260
rsi    0x7f6a7c000c30
rbp    0xc000047948
rsp    0x7f6a91ffadf8
r8     0xc000580808
r9     0x0
r10    0x7f6ab003e580
r11    0x50
r12    0xc00004a000
r13    0xffffffffffffffff
r14    0xc000505180
r15    0xc000ce5770
rip    0x55e63fa050d9
rflags 0x10202
cs     0x33
fs     0x0
gs     0x0
time=2025-09-26T09:42:03.150+08:00 level=INFO source=sched.go:438 msg="Load failed" model=/root/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa error="do load request: Post \"http://127.0.0.1:43377/load\": EOF"
time=2025-09-26T09:42:03.150+08:00 level=DEBUG source=server.go:1683 msg="stopping llama server" pid=253632
time=2025-09-26T09:42:03.150+08:00 level=DEBUG source=server.go:1689 msg="waiting for llama server to exit" pid=253632
time=2025-09-26T09:42:03.248+08:00 level=DEBUG source=server.go:1693 msg="llama server stopped" pid=253632
[GIN] 2025/09/26 - 09:42:03 | 500 |  2.698911092s |       127.0.0.1 | POST     "/api/generate"

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.12.2

Originally created by @RaidenAZ on GitHub (Sep 26, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/12419 ### What is the issue? I upgraded from version 0.11.5 to 0.12.2 by `running tar -C /usr -xzf ollama-linux-amd64.tgz`. When I tried to launch the qwen3 model with ollama run, I got the error: `Error: 500 Internal Server Error: do load request: Post "http://127.0.0.1:43393/load": EOF` After reviewing the logs and trying the fixes suggested in other issues—such as deleting the cuda_v* directories—the fault still occurs. The log shows: ```text load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so SIGSEGV: segmentation violation PC=0x55e63fa050d9 m=12 sigcode=128 addr=0x0 signal arrived during cgo execution ``` The libraries are not being loaded from cuda_v12 or cuda_v13. Rolling back to 0.11.5 restores normal operation—the model loads without any problem. I then completely removed /usr/bin/ollama and performed a fresh install, but the same fault still occurs. ### Relevant log output ```shell [GIN] 2025/09/26 - 09:41:37 | 200 | 45.07µs | 127.0.0.1 | HEAD "/" time=2025-09-26T09:41:38.973+08:00 level=INFO source=download.go:177 msg="downloading 7f4030143c1c in 6 100 MB part(s)" time=2025-09-26T09:41:47.693+08:00 level=INFO source=download.go:177 msg="downloading ae370d884f10 in 1 1.7 KB part(s)" time=2025-09-26T09:41:49.366+08:00 level=INFO source=download.go:177 msg="downloading d18a5cc71b84 in 1 11 KB part(s)" time=2025-09-26T09:41:51.046+08:00 level=INFO source=download.go:177 msg="downloading cff3f395ef37 in 1 120 B part(s)" time=2025-09-26T09:41:52.716+08:00 level=INFO source=download.go:177 msg="downloading b0830f4ff6a0 in 1 490 B part(s)" [GIN] 2025/09/26 - 09:41:54 | 200 | 16.648641496s | 127.0.0.1 | POST "/api/pull" [GIN] 2025/09/26 - 09:42:00 | 200 | 32.828µs | 127.0.0.1 | HEAD "/" time=2025-09-26T09:42:00.546+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 [GIN] 2025/09/26 - 09:42:00 | 200 | 61.427513ms | 127.0.0.1 | POST "/api/show" time=2025-09-26T09:42:00.628+08:00 level=DEBUG source=gpu.go:410 msg="updating system memory data" before.total="62.7 GiB" before.free="58.6 GiB" before.free_swap="2.0 GiB" now.total="62.7 GiB" now.free="58.5 GiB" now.free_swap="2.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08 dlsym: cuInit - 0x7f7d03e47ab0 dlsym: cuDriverGetVersion - 0x7f7d03e47ad0 dlsym: cuDeviceGetCount - 0x7f7d03e47b10 dlsym: cuDeviceGet - 0x7f7d03e47af0 dlsym: cuDeviceGetAttribute - 0x7f7d03e47bf0 dlsym: cuDeviceGetUuid - 0x7f7d03e47b50 dlsym: cuDeviceGetName - 0x7f7d03e47b30 dlsym: cuCtxCreate_v3 - 0x7f7d03e47dd0 dlsym: cuMemGetInfo_v2 - 0x7f7d03e68190 dlsym: cuCtxDestroy - 0x7f7d03ea6ae0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 4 time=2025-09-26T09:42:00.799+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" time=2025-09-26T09:42:00.960+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-cb7d892b-02b5-256f-997d-c93a0f014ea1 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" time=2025-09-26T09:42:01.121+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-fc8226bf-f6bf-8c07-157a-003fa3217d8c name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" time=2025-09-26T09:42:01.281+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-7e1e6171-9d02-0e32-5065-c0d4c5dd3b88 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" releasing cuda driver library time=2025-09-26T09:42:01.281+08:00 level=DEBUG source=sched.go:188 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=12 gpu_count=4 time=2025-09-26T09:42:01.298+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 time=2025-09-26T09:42:01.298+08:00 level=DEBUG source=sched.go:208 msg="loading first model" model=/root/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0 time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_eos_token default=false time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}" time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.rope.scaling.factor default=1 time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_count default=0 time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_used_count default=0 time=2025-09-26T09:42:01.369+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.norm_top_k_prob default=true time=2025-09-26T09:42:01.370+08:00 level=DEBUG source=gpu.go:410 msg="updating system memory data" before.total="62.7 GiB" before.free="58.5 GiB" before.free_swap="2.0 GiB" now.total="62.7 GiB" now.free="58.4 GiB" now.free_swap="2.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08 dlsym: cuInit - 0x7f7d03e47ab0 dlsym: cuDriverGetVersion - 0x7f7d03e47ad0 dlsym: cuDeviceGetCount - 0x7f7d03e47b10 dlsym: cuDeviceGet - 0x7f7d03e47af0 dlsym: cuDeviceGetAttribute - 0x7f7d03e47bf0 dlsym: cuDeviceGetUuid - 0x7f7d03e47b50 dlsym: cuDeviceGetName - 0x7f7d03e47b30 dlsym: cuCtxCreate_v3 - 0x7f7d03e47dd0 dlsym: cuMemGetInfo_v2 - 0x7f7d03e68190 dlsym: cuCtxDestroy - 0x7f7d03ea6ae0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 4 time=2025-09-26T09:42:01.528+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" time=2025-09-26T09:42:01.681+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-cb7d892b-02b5-256f-997d-c93a0f014ea1 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" time=2025-09-26T09:42:01.837+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-fc8226bf-f6bf-8c07-157a-003fa3217d8c name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" time=2025-09-26T09:42:01.992+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-7e1e6171-9d02-0e32-5065-c0d4c5dd3b88 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" releasing cuda driver library time=2025-09-26T09:42:01.992+08:00 level=INFO source=server.go:399 msg="starting runner" cmd="/usr/bin/ollama runner --ollama-engine --model /root/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa --port 43377" time=2025-09-26T09:42:01.992+08:00 level=DEBUG source=server.go:400 msg=subprocess OLLAMA_KEEP_ALIVE=-1 OLLAMA_DEBUG=1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_MAX_LOADED_MODELS=12 OLLAMA_LIBRARY_PATH=/usr/lib/ollama LD_LIBRARY_PATH=/usr/lib/ollama:/usr/lib/ollama time=2025-09-26T09:42:01.993+08:00 level=INFO source=server.go:672 msg="loading model" "model layers"=29 requested=-1 time=2025-09-26T09:42:01.993+08:00 level=DEBUG source=gpu.go:410 msg="updating system memory data" before.total="62.7 GiB" before.free="58.4 GiB" before.free_swap="2.0 GiB" now.total="62.7 GiB" now.free="58.4 GiB" now.free_swap="2.0 GiB" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08 dlsym: cuInit - 0x7f7d03e47ab0 dlsym: cuDriverGetVersion - 0x7f7d03e47ad0 dlsym: cuDeviceGetCount - 0x7f7d03e47b10 dlsym: cuDeviceGet - 0x7f7d03e47af0 dlsym: cuDeviceGetAttribute - 0x7f7d03e47bf0 dlsym: cuDeviceGetUuid - 0x7f7d03e47b50 dlsym: cuDeviceGetName - 0x7f7d03e47b30 dlsym: cuCtxCreate_v3 - 0x7f7d03e47dd0 dlsym: cuMemGetInfo_v2 - 0x7f7d03e68190 dlsym: cuCtxDestroy - 0x7f7d03ea6ae0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 4 time=2025-09-26T09:42:02.008+08:00 level=INFO source=runner.go:1252 msg="starting ollama engine" time=2025-09-26T09:42:02.009+08:00 level=INFO source=runner.go:1287 msg="Server listening on 127.0.0.1:43377" time=2025-09-26T09:42:02.148+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" time=2025-09-26T09:42:02.306+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-cb7d892b-02b5-256f-997d-c93a0f014ea1 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" time=2025-09-26T09:42:02.463+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-fc8226bf-f6bf-8c07-157a-003fa3217d8c name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" time=2025-09-26T09:42:02.617+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-7e1e6171-9d02-0e32-5065-c0d4c5dd3b88 name="NVIDIA A40" overhead="0 B" before.total="47.4 GiB" before.free="47.1 GiB" now.total="47.4 GiB" now.free="47.1 GiB" now.used="281.7 MiB" releasing cuda driver library time=2025-09-26T09:42:02.617+08:00 level=INFO source=server.go:678 msg="system memory" total="62.7 GiB" free="58.4 GiB" free_swap="2.0 GiB" time=2025-09-26T09:42:02.617+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 available="46.7 GiB" free="47.1 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-09-26T09:42:02.617+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-cb7d892b-02b5-256f-997d-c93a0f014ea1 available="46.7 GiB" free="47.1 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-09-26T09:42:02.617+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-fc8226bf-f6bf-8c07-157a-003fa3217d8c available="46.7 GiB" free="47.1 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-09-26T09:42:02.617+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-7e1e6171-9d02-0e32-5065-c0d4c5dd3b88 available="46.7 GiB" free="47.1 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-09-26T09:42:02.618+08:00 level=INFO source=runner.go:1171 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:false KvSize:4096 KvCacheType: NumThreads:16 GPULayers:29[ID:GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 Layers:29(0..28)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-09-26T09:42:02.650+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 time=2025-09-26T09:42:02.650+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.description default="" time=2025-09-26T09:42:02.650+08:00 level=INFO source=ggml.go:131 msg="" architecture=qwen3 file_type=Q4_K_M name="Qwen3 0.6B" description="" num_tensors=311 num_key_values=29 time=2025-09-26T09:42:02.650+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/lib/ollama ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 4 CUDA devices: Device 0: NVIDIA A40, compute capability 8.6, VMM: yes, ID: GPU-16f80610-8f19-4f3a-5b0f-5dad99ecbf26 Device 1: NVIDIA A40, compute capability 8.6, VMM: yes, ID: GPU-cb7d892b-02b5-256f-997d-c93a0f014ea1 Device 2: NVIDIA A40, compute capability 8.6, VMM: yes, ID: GPU-fc8226bf-f6bf-8c07-157a-003fa3217d8c Device 3: NVIDIA A40, compute capability 8.6, VMM: yes, ID: GPU-7e1e6171-9d02-0e32-5065-c0d4c5dd3b88 load_backend: loaded CUDA backend from /usr/lib/ollama/libggml-cuda.so load_backend: loaded CPU backend from /usr/lib/ollama/libggml-cpu-icelake.so SIGSEGV: segmentation violation PC=0x55e63fa050d9 m=12 sigcode=128 addr=0x0 signal arrived during cgo execution goroutine 14 gp=0xc000505180 m=12 mp=0xc000580808 [syscall]: runtime.cgocall(0x55e63f9b1f40, 0xc0000479b8) runtime/cgocall.go:167 +0x4b fp=0xc000047990 sp=0xc000047958 pc=0x55e63eca93eb github.com/ollama/ollama/ml/backend/ggml/ggml/src._Cfunc_ggml_backend_reg_get_proc_address(0x7f6a7cf74260, 0x7f6a7c000c30) _cgo_gotypes.go:208 +0x4b fp=0xc0000479b8 sp=0xc000047990 pc=0x55e63f05906b github.com/ollama/ollama/ml/backend/ggml/ggml/src.system.LogValue.func2.2(...) github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:119 github.com/ollama/ollama/ml/backend/ggml/ggml/src.system.LogValue.func2(0x7f6a7cf74260, 0xc000047ba0, 0xc000047c88) github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:119 +0xd0 fp=0xc000047b10 sp=0xc0000479b8 pc=0x55e63f0599b0 github.com/ollama/ollama/ml/backend/ggml/ggml/src.system.LogValue({}) github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:129 +0x145 fp=0xc000047cc8 sp=0xc000047b10 pc=0x55e63f059585 github.com/ollama/ollama/ml/backend/ggml/ggml/src.(*system).LogValue(0x55e64096d760?) <autogenerated>:1 +0x1d fp=0xc000047cd8 sp=0xc000047cc8 pc=0x55e63f059efd log/slog.Value.Resolve({{}, 0xc000047dd0?, {0x55e6400312e0?, 0x55e640ad0ba0?}}) log/slog/value.go:512 +0x9f fp=0xc000047d90 sp=0xc000047cd8 pc=0x55e63f02209f log/slog.(*handleState).appendAttr(0xc000048208, {{0x0, 0x0}, {{}, 0x0, {0x55e6400312e0, 0x55e640ad0ba0}}}) log/slog/handler.go:468 +0x5d fp=0xc000047e78 sp=0xc000047d90 pc=0x55e63f0178dd log/slog.(*handleState).appendNonBuiltIns.func1(...) log/slog/handler.go:342 log/slog.Record.Attrs(...) log/slog/record.go:84 log/slog.(*handleState).appendNonBuiltIns(_, {{0xc22d99bac439b370, 0x3fd436cd, 0x55e640a22f80}, {0x55e63fc8b0c7, 0x6}, 0x0, 0x55e63f0588ef, {{{0x0, 0x0}, ...}, ...}, ...}) log/slog/handler.go:341 +0x4e9 fp=0xc0000480a0 sp=0xc000047e78 pc=0x55e63f016f69 log/slog.(*commonHandler).handle(_, {{0xc22d99bac439b370, 0x3fd436cd, 0x55e640a22f80}, {0x55e63fc8b0c7, 0x6}, 0x0, 0x55e63f0588ef, {{{0x0, 0x0}, ...}, ...}, ...}) log/slog/handler.go:308 +0x5c5 fp=0xc0000482e0 sp=0xc0000480a0 pc=0x55e63f016825 log/slog.(*TextHandler).Handle(_, {_, _}, {{0xc22d99bac439b370, 0x3fd436cd, 0x55e640a22f80}, {0x55e63fc8b0c7, 0x6}, 0x0, 0x55e63f0588ef, ...}) log/slog/text_handler.go:95 +0x3f fp=0xc000048418 sp=0xc0000482e0 pc=0x55e63f01e45f log/slog.(*Logger).log(0xc00047b200, {0x55e640175a40, 0x55e640ad0ba0}, 0x0, {0x55e63fc8b0c7, 0x6}, {0xc0000488d8, 0x2, 0x2}) log/slog/logger.go:256 +0x1e4 fp=0xc0000487b0 sp=0xc000048418 pc=0x55e63f01be04 log/slog.Info(...) log/slog/logger.go:291 github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.func1() github.com/ollama/ollama/ml/backend/ggml/ggml/src/ggml.go:104 +0x5be fp=0xc000048a40 sp=0xc0000487b0 pc=0x55e63f05895e github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func2() sync/oncefunc.go:27 +0x62 fp=0xc000048a88 sp=0xc000048a40 pc=0x55e63f0582c2 sync.(*Once).doSlow(0x55e640175a40?, 0x55e640ad0ba0?) sync/once.go:78 +0xab fp=0xc000048ae0 sp=0xc000048a88 pc=0x55e63ecbe4ab sync.(*Once).Do(0x0?, 0xc000048b88?) sync/once.go:69 +0x19 fp=0xc000048b00 sp=0xc000048ae0 pc=0x55e63ecbe3d9 github.com/ollama/ollama/ml/backend/ggml/ggml/src.init.OnceFunc.func3() sync/oncefunc.go:32 +0x2d fp=0xc000048b30 sp=0xc000048b00 pc=0x55e63f05822d github.com/ollama/ollama/ml/backend/ggml.init.func1() github.com/ollama/ollama/ml/backend/ggml/ggml.go:44 +0x23 fp=0xc000048bc0 sp=0xc000048b30 pc=0x55e63f0e9863 github.com/ollama/ollama/ml/backend/ggml.init.OnceFunc.func2() sync/oncefunc.go:27 +0x62 fp=0xc000048c08 sp=0xc000048bc0 pc=0x55e63f0e9762 sync.(*Once).doSlow(0x155e64016b718?, 0xc000594058?) sync/once.go:78 +0xab fp=0xc000048c60 sp=0xc000048c08 pc=0x55e63ecbe4ab sync.(*Once).Do(0x55e63ecbe560?, 0x55e640ad11bc?) sync/once.go:69 +0x19 fp=0xc000048c80 sp=0xc000048c60 pc=0x55e63ecbe3d9 github.com/ollama/ollama/ml/backend/ggml.init.OnceFunc.func3() sync/oncefunc.go:32 +0x2d fp=0xc000048cb0 sp=0xc000048c80 pc=0x55e63f0e96cd github.com/ollama/ollama/ml/backend/ggml.New({0x7ffddf4da665, 0x62}, {0x0, 0x10, {0xc000189dd0, 0x1, 0x1}, 0x0}) github.com/ollama/ollama/ml/backend/ggml/ggml.go:142 +0x124 fp=0xc000049580 sp=0xc000048cb0 pc=0x55e63f0f1f64 github.com/ollama/ollama/ml.NewBackend({0x7ffddf4da665, 0x62}, {0x0, 0x10, {0xc000189dd0, 0x1, 0x1}, 0x0}) github.com/ollama/ollama/ml/backend.go:358 +0x9c fp=0xc0000495d0 sp=0xc000049580 pc=0x55e63f09127c github.com/ollama/ollama/model.New({0x7ffddf4da665?, 0x0?}, {0x0, 0x10, {0xc000189dd0, 0x1, 0x1}, 0x0}) github.com/ollama/ollama/model/model.go:106 +0x7e fp=0xc0000496a0 sp=0xc0000495d0 pc=0x55e63f10641e github.com/ollama/ollama/runner/ollamarunner.(*Server).allocModel(0xc0000374a0, {0x7ffddf4da665?, 0x55e63efa043a?}, {0x0, 0x10, {0xc000189dd0, 0x1, 0x1}, 0x0}, {0x0, ...}, ...) github.com/ollama/ollama/runner/ollamarunner/runner.go:1100 +0xcc fp=0xc000049730 sp=0xc0000496a0 pc=0x55e63f1bb1cc github.com/ollama/ollama/runner/ollamarunner.(*Server).load(0xc0000374a0, {0x55e640173748, 0xc00014c000}, 0xc00046a000) github.com/ollama/ollama/runner/ollamarunner/runner.go:1198 +0x54d fp=0xc000049ac0 sp=0xc000049730 pc=0x55e63f1bbdad github.com/ollama/ollama/runner/ollamarunner.(*Server).load-fm({0x55e640173748?, 0xc00014c000?}, 0xc000123b40?) <autogenerated>:1 +0x36 fp=0xc000049af0 sp=0xc000049ac0 pc=0x55e63f1bd5d6 net/http.HandlerFunc.ServeHTTP(0xc0004a3200?, {0x55e640173748?, 0xc00014c000?}, 0xc000123b60?) net/http/server.go:2294 +0x29 fp=0xc000049b18 sp=0xc000049af0 pc=0x55e63efab109 net/http.(*ServeMux).ServeHTTP(0x55e63ec50d85?, {0x55e640173748, 0xc00014c000}, 0xc00046a000) net/http/server.go:2822 +0x1c4 fp=0xc000049b68 sp=0xc000049b18 pc=0x55e63efad004 net/http.serverHandler.ServeHTTP({0x55e64016fd90?}, {0x55e640173748?, 0xc00014c000?}, 0x1?) net/http/server.go:3301 +0x8e fp=0xc000049b98 sp=0xc000049b68 pc=0x55e63efcaa8e net/http.(*conn).serve(0xc0004cc3f0, {0x55e640175a78, 0xc000690d50}) net/http/server.go:2102 +0x625 fp=0xc000049fb8 sp=0xc000049b98 pc=0x55e63efa9605 net/http.(*Server).Serve.gowrap3() net/http/server.go:3454 +0x28 fp=0xc000049fe0 sp=0xc000049fb8 pc=0x55e63efaeec8 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000049fe8 sp=0xc000049fe0 pc=0x55e63ecb3fa1 created by net/http.(*Server).Serve in goroutine 1 net/http/server.go:3454 +0x485 goroutine 1 gp=0xc000002380 m=nil [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000bf9788 sp=0xc000bf9768 pc=0x55e63ecac86e runtime.netpollblock(0xc00050f7d8?, 0x3ec45666?, 0xe6?) runtime/netpoll.go:575 +0xf7 fp=0xc000bf97c0 sp=0xc000bf9788 pc=0x55e63ec71357 internal/poll.runtime_pollWait(0x7f6ab2b5eeb0, 0x72) runtime/netpoll.go:351 +0x85 fp=0xc000bf97e0 sp=0xc000bf97c0 pc=0x55e63ecaba85 internal/poll.(*pollDesc).wait(0xc000686a80?, 0x900000036?, 0x0) internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000bf9808 sp=0xc000bf97e0 pc=0x55e63ed32ec7 internal/poll.(*pollDesc).waitRead(...) internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Accept(0xc000686a80) internal/poll/fd_unix.go:620 +0x295 fp=0xc000bf98b0 sp=0xc000bf9808 pc=0x55e63ed38295 net.(*netFD).accept(0xc000686a80) net/fd_unix.go:172 +0x29 fp=0xc000bf9968 sp=0xc000bf98b0 pc=0x55e63edab249 net.(*TCPListener).accept(0xc0006a0f40) net/tcpsock_posix.go:159 +0x1b fp=0xc000bf99b8 sp=0xc000bf9968 pc=0x55e63edc0bfb net.(*TCPListener).Accept(0xc0006a0f40) net/tcpsock.go:380 +0x30 fp=0xc000bf99e8 sp=0xc000bf99b8 pc=0x55e63edbfab0 net/http.(*onceCloseListener).Accept(0xc0004cc3f0?) <autogenerated>:1 +0x24 fp=0xc000bf9a00 sp=0xc000bf99e8 pc=0x55e63efd7204 net/http.(*Server).Serve(0xc000507100, {0x55e640173568, 0xc0006a0f40}) net/http/server.go:3424 +0x30c fp=0xc000bf9b30 sp=0xc000bf9a00 pc=0x55e63efaeacc github.com/ollama/ollama/runner/ollamarunner.Execute({0xc0000340a0, 0x4, 0x4}) github.com/ollama/ollama/runner/ollamarunner/runner.go:1288 +0x935 fp=0xc000bf9d08 sp=0xc000bf9b30 pc=0x55e63f1bcff5 github.com/ollama/ollama/runner.Execute({0xc000034080?, 0x0?, 0x0?}) github.com/ollama/ollama/runner/runner.go:20 +0xc9 fp=0xc000bf9d30 sp=0xc000bf9d08 pc=0x55e63f1bd869 github.com/ollama/ollama/cmd.NewCLI.func2(0xc000506f00?, {0x55e63fc850b2?, 0x4?, 0x55e63fc850b6?}) github.com/ollama/ollama/cmd/cmd.go:1706 +0x45 fp=0xc000bf9d58 sp=0xc000bf9d30 pc=0x55e63f92f3a5 github.com/spf13/cobra.(*Command).execute(0xc0004cf508, {0xc000390690, 0x5, 0x5}) github.com/spf13/cobra@v1.7.0/command.go:940 +0x85c fp=0xc000bf9e78 sp=0xc000bf9d58 pc=0x55e63ee2489c github.com/spf13/cobra.(*Command).ExecuteC(0xc0004b0908) github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc000bf9f30 sp=0xc000bf9e78 pc=0x55e63ee250e5 github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/cobra@v1.7.0/command.go:992 github.com/spf13/cobra.(*Command).ExecuteContext(...) github.com/spf13/cobra@v1.7.0/command.go:985 main.main() github.com/ollama/ollama/main.go:12 +0x4d fp=0xc000bf9f50 sp=0xc000bf9f30 pc=0x55e63f92fe6d runtime.main() runtime/proc.go:283 +0x29d fp=0xc000bf9fe0 sp=0xc000bf9f50 pc=0x55e63ec789dd runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000bf9fe8 sp=0xc000bf9fe0 pc=0x55e63ecb3fa1 goroutine 2 gp=0xc000002e00 m=nil [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000084fa8 sp=0xc000084f88 pc=0x55e63ecac86e runtime.goparkunlock(...) runtime/proc.go:441 runtime.forcegchelper() runtime/proc.go:348 +0xb8 fp=0xc000084fe0 sp=0xc000084fa8 pc=0x55e63ec78d18 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000084fe8 sp=0xc000084fe0 pc=0x55e63ecb3fa1 created by runtime.init.7 in goroutine 1 runtime/proc.go:336 +0x1a goroutine 3 gp=0xc000003340 m=nil [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000085780 sp=0xc000085760 pc=0x55e63ecac86e runtime.goparkunlock(...) runtime/proc.go:441 runtime.bgsweep(0xc0000ac000) runtime/mgcsweep.go:316 +0xdf fp=0xc0000857c8 sp=0xc000085780 pc=0x55e63ec634bf runtime.gcenable.gowrap1() runtime/mgc.go:204 +0x25 fp=0xc0000857e0 sp=0xc0000857c8 pc=0x55e63ec578a5 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000857e8 sp=0xc0000857e0 pc=0x55e63ecb3fa1 created by runtime.gcenable in goroutine 1 runtime/mgc.go:204 +0x66 goroutine 4 gp=0xc000003500 m=nil [GC scavenge wait]: runtime.gopark(0x10000?, 0x55e63fe4d728?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0x55e63ecac86e runtime.goparkunlock(...) runtime/proc.go:441 runtime.(*scavengerState).park(0x55e640a231a0) runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0x55e63ec60f09 runtime.bgscavenge(0xc0000ac000) runtime/mgcscavenge.go:658 +0x59 fp=0xc000085fc8 sp=0xc000085fa8 pc=0x55e63ec61499 runtime.gcenable.gowrap2() runtime/mgc.go:205 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0x55e63ec57845 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x55e63ecb3fa1 created by runtime.gcenable in goroutine 1 runtime/mgc.go:205 +0xa5 goroutine 5 gp=0xc000003dc0 m=nil [finalizer wait]: runtime.gopark(0x1b8?, 0xc000002380?, 0x1?, 0x23?, 0xc000084688?) runtime/proc.go:435 +0xce fp=0xc000084630 sp=0xc000084610 pc=0x55e63ecac86e runtime.runfinq() runtime/mfinal.go:196 +0x107 fp=0xc0000847e0 sp=0xc000084630 pc=0x55e63ec56867 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000847e8 sp=0xc0000847e0 pc=0x55e63ecb3fa1 created by runtime.createfing in goroutine 1 runtime/mfinal.go:166 +0x3d goroutine 6 gp=0xc0001e08c0 m=nil [chan receive]: runtime.gopark(0xc000235ae0?, 0xc0000101b0?, 0x60?, 0x67?, 0x55e63ed91e88?) runtime/proc.go:435 +0xce fp=0xc000086718 sp=0xc0000866f8 pc=0x55e63ecac86e runtime.chanrecv(0xc000042380, 0x0, 0x1) runtime/chan.go:664 +0x445 fp=0xc000086790 sp=0xc000086718 pc=0x55e63ec48245 runtime.chanrecv1(0x0?, 0x0?) runtime/chan.go:506 +0x12 fp=0xc0000867b8 sp=0xc000086790 pc=0x55e63ec47dd2 runtime.unique_runtime_registerUniqueMapCleanup.func2(...) runtime/mgc.go:1796 runtime.unique_runtime_registerUniqueMapCleanup.gowrap1() runtime/mgc.go:1799 +0x2f fp=0xc0000867e0 sp=0xc0000867b8 pc=0x55e63ec5aa4f runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x55e63ecb3fa1 created by unique.runtime_registerUniqueMapCleanup in goroutine 1 runtime/mgc.go:1794 +0x85 goroutine 7 gp=0xc0001e0c40 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000086f38 sp=0xc000086f18 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc000086fc8 sp=0xc000086f38 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000086fe0 sp=0xc000086fc8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000086fe8 sp=0xc000086fe0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 8 gp=0xc0001e0e00 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000087738 sp=0xc000087718 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc0000877c8 sp=0xc000087738 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0000877e0 sp=0xc0000877c8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 18 gp=0xc000504000 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000080738 sp=0xc000080718 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc0000807c8 sp=0xc000080738 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0000807e0 sp=0xc0000807c8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000807e8 sp=0xc0000807e0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 19 gp=0xc0005041c0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000080f38 sp=0xc000080f18 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc000080fc8 sp=0xc000080f38 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000080fe0 sp=0xc000080fc8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000080fe8 sp=0xc000080fe0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 34 gp=0xc000102380 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00011a738 sp=0xc00011a718 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc00011a7c8 sp=0xc00011a738 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00011a7e0 sp=0xc00011a7c8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00011a7e8 sp=0xc00011a7e0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 35 gp=0xc000102540 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00011af38 sp=0xc00011af18 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc00011afc8 sp=0xc00011af38 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00011afe0 sp=0xc00011afc8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00011afe8 sp=0xc00011afe0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 9 gp=0xc0001e0fc0 m=nil [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000087f38 sp=0xc000087f18 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc000087fc8 sp=0xc000087f38 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000087fe0 sp=0xc000087fc8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 10 gp=0xc0001e1180 m=nil [GC worker (idle)]: runtime.gopark(0xb0b92c230e30c?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000116738 sp=0xc000116718 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc0001167c8 sp=0xc000116738 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0001167e0 sp=0xc0001167c8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0001167e8 sp=0xc0001167e0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 20 gp=0xc000504380 m=nil [GC worker (idle)]: runtime.gopark(0xb0b92c230d7b4?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000081738 sp=0xc000081718 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc0000817c8 sp=0xc000081738 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0000817e0 sp=0xc0000817c8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000817e8 sp=0xc0000817e0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 21 gp=0xc000504540 m=nil [GC worker (idle)]: runtime.gopark(0xb0b92c23119d0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000081f38 sp=0xc000081f18 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc000081fc8 sp=0xc000081f38 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000081fe0 sp=0xc000081fc8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000081fe8 sp=0xc000081fe0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 22 gp=0xc000504700 m=nil [GC worker (idle)]: runtime.gopark(0xb0b92c231cf0a?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000082738 sp=0xc000082718 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc0000827c8 sp=0xc000082738 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0000827e0 sp=0xc0000827c8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0000827e8 sp=0xc0000827e0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 36 gp=0xc000102700 m=nil [GC worker (idle)]: runtime.gopark(0xb0b92c230e467?, 0x1?, 0x3?, 0xee?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00011b738 sp=0xc00011b718 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc00011b7c8 sp=0xc00011b738 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00011b7e0 sp=0xc00011b7c8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00011b7e8 sp=0xc00011b7e0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 11 gp=0xc0001e1340 m=nil [GC worker (idle)]: runtime.gopark(0xb0b92c230de16?, 0x3?, 0xc7?, 0x62?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000116f38 sp=0xc000116f18 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc000116fc8 sp=0xc000116f38 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc000116fe0 sp=0xc000116fc8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000116fe8 sp=0xc000116fe0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 12 gp=0xc0001e1500 m=nil [GC worker (idle)]: runtime.gopark(0xb0b92c248d14b?, 0x1?, 0xcd?, 0x8b?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000117738 sp=0xc000117718 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc0001177c8 sp=0xc000117738 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc0001177e0 sp=0xc0001177c8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc0001177e8 sp=0xc0001177e0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 37 gp=0xc0001028c0 m=nil [GC worker (idle)]: runtime.gopark(0xb0b92c230f2b4?, 0x1?, 0xec?, 0xd0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00011bf38 sp=0xc00011bf18 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc00011bfc8 sp=0xc00011bf38 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00011bfe0 sp=0xc00011bfc8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 38 gp=0xc000102a80 m=nil [GC worker (idle)]: runtime.gopark(0x55e640ad2ea0?, 0x1?, 0x70?, 0x4a?, 0x0?) runtime/proc.go:435 +0xce fp=0xc00011c738 sp=0xc00011c718 pc=0x55e63ecac86e runtime.gcBgMarkWorker(0xc000043960) runtime/mgc.go:1423 +0xe9 fp=0xc00011c7c8 sp=0xc00011c738 pc=0x55e63ec59d69 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1339 +0x25 fp=0xc00011c7e0 sp=0xc00011c7c8 pc=0x55e63ec59c45 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00011c7e8 sp=0xc00011c7e0 pc=0x55e63ecb3fa1 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1339 +0x105 goroutine 13 gp=0xc000504fc0 m=nil [sync.WaitGroup.Wait]: runtime.gopark(0x0?, 0x0?, 0xc0?, 0xe0?, 0x0?) runtime/proc.go:435 +0xce fp=0xc000093b30 sp=0xc000093b10 pc=0x55e63ecac86e runtime.goparkunlock(...) runtime/proc.go:441 runtime.semacquire1(0xc000037558, 0x0, 0x1, 0x0, 0x18) runtime/sema.go:188 +0x229 fp=0xc000093b98 sp=0xc000093b30 pc=0x55e63ec8bfa9 sync.runtime_SemacquireWaitGroup(0x0?) runtime/sema.go:110 +0x25 fp=0xc000093bd0 sp=0xc000093b98 pc=0x55e63ecae285 sync.(*WaitGroup).Wait(0x0?) sync/waitgroup.go:118 +0x48 fp=0xc000093bf8 sp=0xc000093bd0 pc=0x55e63ecbf8e8 github.com/ollama/ollama/runner/ollamarunner.(*Server).run(0xc0000374a0, {0x55e640175ab0, 0xc000390730}) github.com/ollama/ollama/runner/ollamarunner/runner.go:406 +0x45 fp=0xc000093fb8 sp=0xc000093bf8 pc=0x55e63f1b4b85 github.com/ollama/ollama/runner/ollamarunner.Execute.gowrap1() github.com/ollama/ollama/runner/ollamarunner/runner.go:1266 +0x28 fp=0xc000093fe0 sp=0xc000093fb8 pc=0x55e63f1bd268 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x55e63ecb3fa1 created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1 github.com/ollama/ollama/runner/ollamarunner/runner.go:1266 +0x505 goroutine 50 gp=0xc000505340 m=nil [IO wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0xb?) runtime/proc.go:435 +0xce fp=0xc00011d5d8 sp=0xc00011d5b8 pc=0x55e63ecac86e runtime.netpollblock(0x55e63eccfbd8?, 0x3ec45666?, 0xe6?) runtime/netpoll.go:575 +0xf7 fp=0xc00011d610 sp=0xc00011d5d8 pc=0x55e63ec71357 internal/poll.runtime_pollWait(0x7f6ab2b5ed98, 0x72) runtime/netpoll.go:351 +0x85 fp=0xc00011d630 sp=0xc00011d610 pc=0x55e63ecaba85 internal/poll.(*pollDesc).wait(0xc000686b00?, 0xc000690e51?, 0x0) internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc00011d658 sp=0xc00011d630 pc=0x55e63ed32ec7 internal/poll.(*pollDesc).waitRead(...) internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc000686b00, {0xc000690e51, 0x1, 0x1}) internal/poll/fd_unix.go:165 +0x27a fp=0xc00011d6f0 sp=0xc00011d658 pc=0x55e63ed341ba net.(*netFD).Read(0xc000686b00, {0xc000690e51?, 0xc0006a1018?, 0xc00011d770?}) net/fd_posix.go:55 +0x25 fp=0xc00011d738 sp=0xc00011d6f0 pc=0x55e63eda92a5 net.(*conn).Read(0xc00007c8b0, {0xc000690e51?, 0x0?, 0x0?}) net/net.go:194 +0x45 fp=0xc00011d780 sp=0xc00011d738 pc=0x55e63edb7665 net/http.(*connReader).backgroundRead(0xc000690e40) net/http/server.go:690 +0x37 fp=0xc00011d7c8 sp=0xc00011d780 pc=0x55e63efa34d7 net/http.(*connReader).startBackgroundRead.gowrap2() net/http/server.go:686 +0x25 fp=0xc00011d7e0 sp=0xc00011d7c8 pc=0x55e63efa3405 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00011d7e8 sp=0xc00011d7e0 pc=0x55e63ecb3fa1 created by net/http.(*connReader).startBackgroundRead in goroutine 14 net/http/server.go:686 +0xb6 rax 0x365f4c414e524554 rbx 0xc0000479b8 rcx 0xffffffffffffffd8 rdx 0xc000047948 rdi 0x7f6a7cf74260 rsi 0x7f6a7c000c30 rbp 0xc000047948 rsp 0x7f6a91ffadf8 r8 0xc000580808 r9 0x0 r10 0x7f6ab003e580 r11 0x50 r12 0xc00004a000 r13 0xffffffffffffffff r14 0xc000505180 r15 0xc000ce5770 rip 0x55e63fa050d9 rflags 0x10202 cs 0x33 fs 0x0 gs 0x0 time=2025-09-26T09:42:03.150+08:00 level=INFO source=sched.go:438 msg="Load failed" model=/root/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa error="do load request: Post \"http://127.0.0.1:43377/load\": EOF" time=2025-09-26T09:42:03.150+08:00 level=DEBUG source=server.go:1683 msg="stopping llama server" pid=253632 time=2025-09-26T09:42:03.150+08:00 level=DEBUG source=server.go:1689 msg="waiting for llama server to exit" pid=253632 time=2025-09-26T09:42:03.248+08:00 level=DEBUG source=server.go:1693 msg="llama server stopped" pid=253632 [GIN] 2025/09/26 - 09:42:03 | 500 | 2.698911092s | 127.0.0.1 | POST "/api/generate" ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.12.2
GiteaMirror added the bug label 2026-04-22 17:14:04 -05:00
GiteaMirror commented 2026-04-22 17:14:05 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Sep 26, 2025):

Please make sure you remove all traces of the prior version of ollama binaries before extracting the tar files with the new binaries. The crash you're seeing looks like the pattern where there are multiple versions lingering on the filesystem and they have different sizes of structures leading to a memory corruption.

<!-- gh-comment-id:3339128964 --> @dhiltgen commented on GitHub (Sep 26, 2025): Please make sure you remove all traces of the prior version of ollama binaries before extracting the tar files with the new binaries. The crash you're seeing looks like the pattern where there are multiple versions lingering on the filesystem and they have different sizes of structures leading to a memory corruption.
GiteaMirror commented 2026-04-22 17:14:06 -05:00
Author
Owner
Copy Link

@RaidenAZ commented on GitHub (Sep 28, 2025):

I searched everywhere but couldn’t figure out where the leftover old files were. After redeploying the system, everything is back to normal—thanks!

<!-- gh-comment-id:3342255770 --> @RaidenAZ commented on GitHub (Sep 28, 2025): I searched everywhere but couldn’t figure out where the leftover old files were. After redeploying the system, everything is back to normal—thanks!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#34008
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?