github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-22 21:51:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #11987] Error: 500 Internal Server Error: do load request: Post "http://127.0.0.1:38703/load": EOF #33720

New Issue
Closed
opened 2026-04-22 16:39:20 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-22 16:39:20 -05:00
Owner
Copy Link

Originally created by @ainub on GitHub (Aug 20, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11987

What is the issue?

Starting with version 0.11.4 it started throwing this error. Initially in 0.11.4 there were only a few models, I didn't pay attention as I thought it will be fixed. 0.11.5-rc2 was throwing it on every model, didn't report as I thought it will be fixed, 0.11.5 is now throwing it for every single model no matter how big/small.

Relevant log output

time=2025-08-20T11:53:39.671+03:00 level=DEBUG source=gpu.go:443 msg="updating cuda memory data" gpu=GPU-33e43125-1065-2e07-d3e4-acad4edb74a7 name="NVIDIA RTX 6000 Ada Generation" overhead="0 B" before.total="47.4 GiB" before.free="46.9 GiB" now.total="47.4 GiB" now.free="46.9 GiB" now.used="433.2 MiB"
time=2025-08-20T11:53:39.854+03:00 level=DEBUG source=gpu.go:443 msg="updating cuda memory data" gpu=GPU-44df9b8d-dd26-a5cd-49db-9c424b90a193 name="NVIDIA RTX 6000 Ada Generation" overhead="0 B" before.total="47.4 GiB" before.free="46.9 GiB" now.total="47.4 GiB" now.free="46.9 GiB" now.used="433.2 MiB"
releasing cuda driver library
time=2025-08-20T11:53:39.855+03:00 level=DEBUG source=server.go:318 msg="adding gpu library" path=/home/some-user/tools/ollama/lib/ollama/cuda_v12
time=2025-08-20T11:53:39.855+03:00 level=DEBUG source=server.go:318 msg="adding gpu library" path=/home/some-user/tools/ollama/lib/ollama/cuda_v12
time=2025-08-20T11:53:39.855+03:00 level=DEBUG source=server.go:326 msg="adding gpu dependency paths" paths=[/home/some-user/tools/ollama/lib/ollama/cuda_v12]
time=2025-08-20T11:53:39.855+03:00 level=DEBUG source=server.go:326 msg="adding gpu dependency paths" paths=[/home/some-user/tools/ollama/lib/ollama/cuda_v12]
time=2025-08-20T11:53:39.855+03:00 level=INFO source=server.go:383 msg="starting runner" cmd="/home/some-user/tools/ollama/bin/ollama runner --model /home/some-user/.ollama/models/blobs/sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 --port 38703"
time=2025-08-20T11:53:39.855+03:00 level=DEBUG source=server.go:384 msg=subprocess OLLAMA_DEBUG=1 PATH=/home/some-user/.conda/envs/some-user/bin:/usr/condabin:/home/some-user/.local/bin:/home/some-user/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/some-user/work/MONAILabel/monailabel/scripts OLLAMA_KEEP_ALIVE=60m OLLAMA_MAX_LOADED_MODELS=6 OLLAMA_LIBRARY_PATH=/home/some-user/tools/ollama/lib/ollama:/home/some-user/tools/ollama/lib/ollama/cuda_v12:/home/some-user/tools/ollama/lib/ollama/cuda_v12 LD_LIBRARY_PATH=/home/some-user/tools/ollama/lib/ollama/cuda_v12:/home/some-user/tools/ollama/lib/ollama/cuda_v12:/home/some-user/tools/ollama/lib/ollama/cuda_v12:/home/some-user/tools/ollama/lib/ollama/cuda_v12:/home/some-user/tools/ollama/lib/ollama:/home/some-user/tools/ollama/lib/ollama
time=2025-08-20T11:53:39.856+03:00 level=DEBUG source=gpu.go:393 msg="updating system memory data" before.total="62.8 GiB" before.free="59.5 GiB" before.free_swap="8.0 GiB" now.total="62.8 GiB" now.free="59.5 GiB" now.free_swap="8.0 GiB"
initializing /usr/lib64/libcuda.so.575.64.05
dlsym: cuInit - 0x7f382a974640
dlsym: cuDriverGetVersion - 0x7f382a974700
dlsym: cuDeviceGetCount - 0x7f382a974880
dlsym: cuDeviceGet - 0x7f382a9747c0
dlsym: cuDeviceGetAttribute - 0x7f382a974dc0
dlsym: cuDeviceGetUuid - 0x7f382a974a00
dlsym: cuDeviceGetName - 0x7f382a974940
dlsym: cuCtxCreate_v3 - 0x7f382a975900
dlsym: cuMemGetInfo_v2 - 0x7f382a9785a0
dlsym: cuCtxDestroy - 0x7f382a9da4a0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f3a
CUDA driver version: 12.9
calling cuDeviceGetCount
device count 2
time=2025-08-20T11:53:39.886+03:00 level=INFO source=runner.go:864 msg="starting go runner"
time=2025-08-20T11:53:39.886+03:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/some-user/tools/ollama/lib/ollama
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes, ID: GPU-33e43125-1065-2e07-d3e4-acad4edb74a7
  Device 1: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes, ID: GPU-44df9b8d-dd26-a5cd-49db-9c424b90a193
time=2025-08-20T11:53:40.102+03:00 level=DEBUG source=gpu.go:443 msg="updating cuda memory data" gpu=GPU-33e43125-1065-2e07-d3e4-acad4edb74a7 name="NVIDIA RTX 6000 Ada Generation" overhead="0 B" before.total="47.4 GiB" before.free="46.9 GiB" now.total="47.4 GiB" now.free="46.9 GiB" now.used="433.2 MiB"
load_backend: loaded CUDA backend from /home/some-user/tools/ollama/lib/ollama/libggml-cuda.so
load_backend: loaded CPU backend from /home/some-user/tools/ollama/lib/ollama/libggml-cpu-icelake.so
time=2025-08-20T11:53:40.263+03:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/some-user/tools/ollama/lib/ollama/cuda_v12
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes
  Device 1: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes
time=2025-08-20T11:53:40.392+03:00 level=DEBUG source=gpu.go:443 msg="updating cuda memory data" gpu=GPU-44df9b8d-dd26-a5cd-49db-9c424b90a193 name="NVIDIA RTX 6000 Ada Generation" overhead="0 B" before.total="47.4 GiB" before.free="46.9 GiB" now.total="47.4 GiB" now.free="46.5 GiB" now.used="862.6 MiB"
releasing cuda driver library
time=2025-08-20T11:53:40.392+03:00 level=INFO source=server.go:488 msg="system memory" total="62.8 GiB" free="59.5 GiB" free_swap="8.0 GiB"
time=2025-08-20T11:53:40.392+03:00 level=DEBUG source=memory.go:177 msg=evaluating library=cuda gpu_count=1 available="[46.9 GiB]"
time=2025-08-20T11:53:40.393+03:00 level=DEBUG source=ggml.go:208 msg="key with type not found" key=qwen3.vision.block_count default=0
time=2025-08-20T11:53:40.393+03:00 level=INFO source=memory.go:36 msg="new model will fit in available VRAM across minimum required GPUs, loading" model=/home/some-user/.ollama/models/blobs/sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 library=cuda parallel=1 required="35.2 GiB" gpus=1
time=2025-08-20T11:53:40.394+03:00 level=DEBUG source=memory.go:177 msg=evaluating library=cuda gpu_count=1 available="[46.9 GiB]"
time=2025-08-20T11:53:40.394+03:00 level=DEBUG source=ggml.go:208 msg="key with type not found" key=qwen3.vision.block_count default=0
time=2025-08-20T11:53:40.394+03:00 level=INFO source=server.go:531 msg=offload library=cuda layers.requested=-1 layers.model=65 layers.offload=65 layers.split=[65] memory.available="[46.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="35.2 GiB" memory.required.partial="35.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[35.2 GiB]" memory.weights.total="31.9 GiB" memory.weights.repeating="31.2 GiB" memory.weights.nonrepeating="788.3 MiB" memory.graph.full="1.3 GiB" memory.graph.partial="1.3 GiB"
load_backend: loaded CUDA backend from /home/some-user/tools/ollama/lib/ollama/cuda_v12/libggml-cuda.so
time=2025-08-20T11:53:40.452+03:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
time=2025-08-20T11:53:40.453+03:00 level=INFO source=runner.go:900 msg="Server listening on 127.0.0.1:38703"
time=2025-08-20T11:53:40.464+03:00 level=INFO source=runner.go:799 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:false KvSize:4096 KvCacheType: NumThreads:32 GPULayers:65[ID:GPU-33e43125-1065-2e07-d3e4-acad4edb74a7 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:true}"
unexpected fault address 0xbbcbb0000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xbbcbb0000 pc=0x557cbf7082a0]

goroutine 67 gp=0xc0001036c0 m=0 mp=0x557cc14c0d20 [running]:
runtime.throw({0x557cc073849a?, 0xc0001036c0?})
	runtime/panic.go:1096 +0x4a fp=0xc00004d678 sp=0xc00004d648 pc=0x557cbf77874a
runtime.sigpanic()
	runtime/signal_unix.go:939 +0x26c fp=0xc00004d6d8 sp=0xc00004d678 pc=0x557cbf77abcc
indexbytebody()
	internal/bytealg/indexbyte_amd64.s:131 +0xe0 fp=0xc00004d6e0 sp=0xc00004d6d8 pc=0x557cbf7082a0
runtime.findnull(0xc00004d760?)
	runtime/string.go:577 +0x79 fp=0xc00004d738 sp=0xc00004d6e0 pc=0x557cbf7603d9
runtime.gostring(0xbbcbb0000)
	runtime/string.go:363 +0x1c fp=0xc00004d770 sp=0xc00004d738 pc=0x557cbf77ba3c
github.com/ollama/ollama/llama._Cfunc_GoString(...)
	_cgo_gotypes.go:332
github.com/ollama/ollama/llama.EnumerateGPUs()
	github.com/ollama/ollama/llama/llama.go:74 +0x1a5 fp=0xc00004d820 sp=0xc00004d770 pc=0x557cbfb26da5
github.com/ollama/ollama/runner/llamarunner.(*Server).load(0xc0003828c0, {0x557cc0c1c2c8, 0xc000001260}, 0xc0003d4140)
	github.com/ollama/ollama/runner/llamarunner/runner.go:810 +0x40a fp=0xc00004dac0 sp=0xc00004d820 pc=0x557cbfbf124a
github.com/ollama/ollama/runner/llamarunner.(*Server).load-fm({0x557cc0c1c2c8?, 0xc000001260?}, 0xc00026fb40?)
	<autogenerated>:1 +0x36 fp=0xc00004daf0 sp=0xc00004dac0 pc=0x557cbfbf2b36
net/http.HandlerFunc.ServeHTTP(0xc00019e000?, {0x557cc0c1c2c8?, 0xc000001260?}, 0xc00026fb60?)
	net/http/server.go:2294 +0x29 fp=0xc00004db18 sp=0xc00004daf0 pc=0x557cbfa77109
net/http.(*ServeMux).ServeHTTP(0x557cbf71cd85?, {0x557cc0c1c2c8, 0xc000001260}, 0xc0003d4140)
	net/http/server.go:2822 +0x1c4 fp=0xc00004db68 sp=0xc00004db18 pc=0x557cbfa79004
net/http.serverHandler.ServeHTTP({0x557cc0c18950?}, {0x557cc0c1c2c8?, 0xc000001260?}, 0x1?)
	net/http/server.go:3301 +0x8e fp=0xc00004db98 sp=0xc00004db68 pc=0x557cbfa96a8e
net/http.(*conn).serve(0xc0005bc090, {0x557cc0c1e568, 0xc000680570})
	net/http/server.go:2102 +0x625 fp=0xc00004dfb8 sp=0xc00004db98 pc=0x557cbfa75605
net/http.(*Server).Serve.gowrap3()
	net/http/server.go:3454 +0x28 fp=0xc00004dfe0 sp=0xc00004dfb8 pc=0x557cbfa7aec8
runtime.goexit({})
	runtime/asm_amd64.s:1700 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0x557cbf77ffa1
created by net/http.(*Server).Serve in goroutine 1
	net/http/server.go:3454 +0x485

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.11.5, 0.11.4

Originally created by @ainub on GitHub (Aug 20, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11987 ### What is the issue? Starting with version 0.11.4 it started throwing this error. Initially in 0.11.4 there were only a few models, I didn't pay attention as I thought it will be fixed. 0.11.5-rc2 was throwing it on every model, didn't report as I thought it will be fixed, 0.11.5 is now throwing it for every single model no matter how big/small. ### Relevant log output ```shell time=2025-08-20T11:53:39.671+03:00 level=DEBUG source=gpu.go:443 msg="updating cuda memory data" gpu=GPU-33e43125-1065-2e07-d3e4-acad4edb74a7 name="NVIDIA RTX 6000 Ada Generation" overhead="0 B" before.total="47.4 GiB" before.free="46.9 GiB" now.total="47.4 GiB" now.free="46.9 GiB" now.used="433.2 MiB" time=2025-08-20T11:53:39.854+03:00 level=DEBUG source=gpu.go:443 msg="updating cuda memory data" gpu=GPU-44df9b8d-dd26-a5cd-49db-9c424b90a193 name="NVIDIA RTX 6000 Ada Generation" overhead="0 B" before.total="47.4 GiB" before.free="46.9 GiB" now.total="47.4 GiB" now.free="46.9 GiB" now.used="433.2 MiB" releasing cuda driver library time=2025-08-20T11:53:39.855+03:00 level=DEBUG source=server.go:318 msg="adding gpu library" path=/home/some-user/tools/ollama/lib/ollama/cuda_v12 time=2025-08-20T11:53:39.855+03:00 level=DEBUG source=server.go:318 msg="adding gpu library" path=/home/some-user/tools/ollama/lib/ollama/cuda_v12 time=2025-08-20T11:53:39.855+03:00 level=DEBUG source=server.go:326 msg="adding gpu dependency paths" paths=[/home/some-user/tools/ollama/lib/ollama/cuda_v12] time=2025-08-20T11:53:39.855+03:00 level=DEBUG source=server.go:326 msg="adding gpu dependency paths" paths=[/home/some-user/tools/ollama/lib/ollama/cuda_v12] time=2025-08-20T11:53:39.855+03:00 level=INFO source=server.go:383 msg="starting runner" cmd="/home/some-user/tools/ollama/bin/ollama runner --model /home/some-user/.ollama/models/blobs/sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 --port 38703" time=2025-08-20T11:53:39.855+03:00 level=DEBUG source=server.go:384 msg=subprocess OLLAMA_DEBUG=1 PATH=/home/some-user/.conda/envs/some-user/bin:/usr/condabin:/home/some-user/.local/bin:/home/some-user/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/some-user/work/MONAILabel/monailabel/scripts OLLAMA_KEEP_ALIVE=60m OLLAMA_MAX_LOADED_MODELS=6 OLLAMA_LIBRARY_PATH=/home/some-user/tools/ollama/lib/ollama:/home/some-user/tools/ollama/lib/ollama/cuda_v12:/home/some-user/tools/ollama/lib/ollama/cuda_v12 LD_LIBRARY_PATH=/home/some-user/tools/ollama/lib/ollama/cuda_v12:/home/some-user/tools/ollama/lib/ollama/cuda_v12:/home/some-user/tools/ollama/lib/ollama/cuda_v12:/home/some-user/tools/ollama/lib/ollama/cuda_v12:/home/some-user/tools/ollama/lib/ollama:/home/some-user/tools/ollama/lib/ollama time=2025-08-20T11:53:39.856+03:00 level=DEBUG source=gpu.go:393 msg="updating system memory data" before.total="62.8 GiB" before.free="59.5 GiB" before.free_swap="8.0 GiB" now.total="62.8 GiB" now.free="59.5 GiB" now.free_swap="8.0 GiB" initializing /usr/lib64/libcuda.so.575.64.05 dlsym: cuInit - 0x7f382a974640 dlsym: cuDriverGetVersion - 0x7f382a974700 dlsym: cuDeviceGetCount - 0x7f382a974880 dlsym: cuDeviceGet - 0x7f382a9747c0 dlsym: cuDeviceGetAttribute - 0x7f382a974dc0 dlsym: cuDeviceGetUuid - 0x7f382a974a00 dlsym: cuDeviceGetName - 0x7f382a974940 dlsym: cuCtxCreate_v3 - 0x7f382a975900 dlsym: cuMemGetInfo_v2 - 0x7f382a9785a0 dlsym: cuCtxDestroy - 0x7f382a9da4a0 calling cuInit calling cuDriverGetVersion raw version 0x2f3a CUDA driver version: 12.9 calling cuDeviceGetCount device count 2 time=2025-08-20T11:53:39.886+03:00 level=INFO source=runner.go:864 msg="starting go runner" time=2025-08-20T11:53:39.886+03:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/some-user/tools/ollama/lib/ollama ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 2 CUDA devices: Device 0: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes, ID: GPU-33e43125-1065-2e07-d3e4-acad4edb74a7 Device 1: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes, ID: GPU-44df9b8d-dd26-a5cd-49db-9c424b90a193 time=2025-08-20T11:53:40.102+03:00 level=DEBUG source=gpu.go:443 msg="updating cuda memory data" gpu=GPU-33e43125-1065-2e07-d3e4-acad4edb74a7 name="NVIDIA RTX 6000 Ada Generation" overhead="0 B" before.total="47.4 GiB" before.free="46.9 GiB" now.total="47.4 GiB" now.free="46.9 GiB" now.used="433.2 MiB" load_backend: loaded CUDA backend from /home/some-user/tools/ollama/lib/ollama/libggml-cuda.so load_backend: loaded CPU backend from /home/some-user/tools/ollama/lib/ollama/libggml-cpu-icelake.so time=2025-08-20T11:53:40.263+03:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/some-user/tools/ollama/lib/ollama/cuda_v12 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 2 CUDA devices: Device 0: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes Device 1: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes time=2025-08-20T11:53:40.392+03:00 level=DEBUG source=gpu.go:443 msg="updating cuda memory data" gpu=GPU-44df9b8d-dd26-a5cd-49db-9c424b90a193 name="NVIDIA RTX 6000 Ada Generation" overhead="0 B" before.total="47.4 GiB" before.free="46.9 GiB" now.total="47.4 GiB" now.free="46.5 GiB" now.used="862.6 MiB" releasing cuda driver library time=2025-08-20T11:53:40.392+03:00 level=INFO source=server.go:488 msg="system memory" total="62.8 GiB" free="59.5 GiB" free_swap="8.0 GiB" time=2025-08-20T11:53:40.392+03:00 level=DEBUG source=memory.go:177 msg=evaluating library=cuda gpu_count=1 available="[46.9 GiB]" time=2025-08-20T11:53:40.393+03:00 level=DEBUG source=ggml.go:208 msg="key with type not found" key=qwen3.vision.block_count default=0 time=2025-08-20T11:53:40.393+03:00 level=INFO source=memory.go:36 msg="new model will fit in available VRAM across minimum required GPUs, loading" model=/home/some-user/.ollama/models/blobs/sha256-de447d788da3df6b4ea340408b13fc2c3a2043a2dfc19178b12d501a4bd96484 library=cuda parallel=1 required="35.2 GiB" gpus=1 time=2025-08-20T11:53:40.394+03:00 level=DEBUG source=memory.go:177 msg=evaluating library=cuda gpu_count=1 available="[46.9 GiB]" time=2025-08-20T11:53:40.394+03:00 level=DEBUG source=ggml.go:208 msg="key with type not found" key=qwen3.vision.block_count default=0 time=2025-08-20T11:53:40.394+03:00 level=INFO source=server.go:531 msg=offload library=cuda layers.requested=-1 layers.model=65 layers.offload=65 layers.split=[65] memory.available="[46.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="35.2 GiB" memory.required.partial="35.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[35.2 GiB]" memory.weights.total="31.9 GiB" memory.weights.repeating="31.2 GiB" memory.weights.nonrepeating="788.3 MiB" memory.graph.full="1.3 GiB" memory.graph.partial="1.3 GiB" load_backend: loaded CUDA backend from /home/some-user/tools/ollama/lib/ollama/cuda_v12/libggml-cuda.so time=2025-08-20T11:53:40.452+03:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-08-20T11:53:40.453+03:00 level=INFO source=runner.go:900 msg="Server listening on 127.0.0.1:38703" time=2025-08-20T11:53:40.464+03:00 level=INFO source=runner.go:799 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:false KvSize:4096 KvCacheType: NumThreads:32 GPULayers:65[ID:GPU-33e43125-1065-2e07-d3e4-acad4edb74a7 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:true}" unexpected fault address 0xbbcbb0000 fatal error: fault [signal SIGSEGV: segmentation violation code=0x1 addr=0xbbcbb0000 pc=0x557cbf7082a0] goroutine 67 gp=0xc0001036c0 m=0 mp=0x557cc14c0d20 [running]: runtime.throw({0x557cc073849a?, 0xc0001036c0?}) runtime/panic.go:1096 +0x4a fp=0xc00004d678 sp=0xc00004d648 pc=0x557cbf77874a runtime.sigpanic() runtime/signal_unix.go:939 +0x26c fp=0xc00004d6d8 sp=0xc00004d678 pc=0x557cbf77abcc indexbytebody() internal/bytealg/indexbyte_amd64.s:131 +0xe0 fp=0xc00004d6e0 sp=0xc00004d6d8 pc=0x557cbf7082a0 runtime.findnull(0xc00004d760?) runtime/string.go:577 +0x79 fp=0xc00004d738 sp=0xc00004d6e0 pc=0x557cbf7603d9 runtime.gostring(0xbbcbb0000) runtime/string.go:363 +0x1c fp=0xc00004d770 sp=0xc00004d738 pc=0x557cbf77ba3c github.com/ollama/ollama/llama._Cfunc_GoString(...) _cgo_gotypes.go:332 github.com/ollama/ollama/llama.EnumerateGPUs() github.com/ollama/ollama/llama/llama.go:74 +0x1a5 fp=0xc00004d820 sp=0xc00004d770 pc=0x557cbfb26da5 github.com/ollama/ollama/runner/llamarunner.(*Server).load(0xc0003828c0, {0x557cc0c1c2c8, 0xc000001260}, 0xc0003d4140) github.com/ollama/ollama/runner/llamarunner/runner.go:810 +0x40a fp=0xc00004dac0 sp=0xc00004d820 pc=0x557cbfbf124a github.com/ollama/ollama/runner/llamarunner.(*Server).load-fm({0x557cc0c1c2c8?, 0xc000001260?}, 0xc00026fb40?) <autogenerated>:1 +0x36 fp=0xc00004daf0 sp=0xc00004dac0 pc=0x557cbfbf2b36 net/http.HandlerFunc.ServeHTTP(0xc00019e000?, {0x557cc0c1c2c8?, 0xc000001260?}, 0xc00026fb60?) net/http/server.go:2294 +0x29 fp=0xc00004db18 sp=0xc00004daf0 pc=0x557cbfa77109 net/http.(*ServeMux).ServeHTTP(0x557cbf71cd85?, {0x557cc0c1c2c8, 0xc000001260}, 0xc0003d4140) net/http/server.go:2822 +0x1c4 fp=0xc00004db68 sp=0xc00004db18 pc=0x557cbfa79004 net/http.serverHandler.ServeHTTP({0x557cc0c18950?}, {0x557cc0c1c2c8?, 0xc000001260?}, 0x1?) net/http/server.go:3301 +0x8e fp=0xc00004db98 sp=0xc00004db68 pc=0x557cbfa96a8e net/http.(*conn).serve(0xc0005bc090, {0x557cc0c1e568, 0xc000680570}) net/http/server.go:2102 +0x625 fp=0xc00004dfb8 sp=0xc00004db98 pc=0x557cbfa75605 net/http.(*Server).Serve.gowrap3() net/http/server.go:3454 +0x28 fp=0xc00004dfe0 sp=0xc00004dfb8 pc=0x557cbfa7aec8 runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0x557cbf77ffa1 created by net/http.(*Server).Serve in goroutine 1 net/http/server.go:3454 +0x485 ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.11.5, 0.11.4
GiteaMirror added the bug label 2026-04-22 16:39:20 -05:00
GiteaMirror commented 2026-04-22 16:39:21 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 20, 2025):

load_backend: loaded CUDA backend from /home/some-user/tools/ollama/lib/ollama/libggml-cuda.so
load_backend: loaded CUDA backend from /home/some-user/tools/ollama/lib/ollama/cuda_v12/libggml-cuda.so

#11211. Delete /home/some-user/tools/ollama/lib/ollama/cuda_v*.

<!-- gh-comment-id:3206184618 --> @rick-github commented on GitHub (Aug 20, 2025): ``` load_backend: loaded CUDA backend from /home/some-user/tools/ollama/lib/ollama/libggml-cuda.so load_backend: loaded CUDA backend from /home/some-user/tools/ollama/lib/ollama/cuda_v12/libggml-cuda.so ``` #11211. Delete `/home/some-user/tools/ollama/lib/ollama/cuda_v*`.
GiteaMirror commented 2026-04-22 16:39:22 -05:00
Author
Owner
Copy Link

@ainub commented on GitHub (Aug 20, 2025):

That actually worked. Thank you for the tip. It now works as intended.

<!-- gh-comment-id:3206214520 --> @ainub commented on GitHub (Aug 20, 2025): That actually worked. Thank you for the tip. It now works as intended.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#33720
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?