github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-22 13:42:25 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #11504] Cannot load model on the GPU (Error: llama runner process has terminated: exit status 2) #7600

New Issue
Closed
opened 2026-04-12 19:41:10 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-12 19:41:10 -05:00
Owner
Copy Link

Originally created by @Cope-233 on GitHub (Jul 23, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11504

What is the issue?

I have already set the environment path:
CUDA_VISIBLE_DEVICES=0
OLLAMA_GPU_LAYER=cuda
But ollama cannot load the model to my RTX5090
Driver Version: 577.00 CUDA Version: 12.9

Relevant log output

time=2025-07-23T23:30:49.433+08:00 level=INFO source=routes.go:1235 msg="server config" env="map[CUDA_VISIBLE_DEVICES:0 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:5 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\Sonic_Dong\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:500 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]"
time=2025-07-23T23:30:49.438+08:00 level=INFO source=images.go:476 msg="total blobs: 0"
time=2025-07-23T23:30:49.438+08:00 level=INFO source=images.go:483 msg="total unused blobs removed: 0"
time=2025-07-23T23:30:49.439+08:00 level=INFO source=routes.go:1288 msg="Listening on [::]:11434 (version 0.9.6)"
time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=sched.go:108 msg="starting llm scheduler"
time=2025-07-23T23:30:49.439+08:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-07-23T23:30:49.439+08:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2025-07-23T23:30:49.439+08:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=16 efficiency=0 threads=32
time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=nvml.dll
time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvml.dll D:\\VMware\\bin\\nvml.dll C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\bin\\nvml.dll C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\libnvvp\\nvml.dll C:\\WINDOWS\\system32\\nvml.dll C:\\WINDOWS\\nvml.dll C:\\WINDOWS\\System32\\Wbem\\nvml.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\nvml.dll C:\\WINDOWS\\System32\\OpenSSH\\nvml.dll C:\\Program Files\\Zero Install\\nvml.dll C:\\Program Files\\dotnet\\nvml.dll C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR\\nvml.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvml.dll D:\\Git\\Git\\cmd\\nvml.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\nvml.dll D:\\MATLAB\\bin\\nvml.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Microsoft\\WindowsApps\\nvml.dll C:\\Users\\Sonic_Dong\\.lmstudio\\bin\\nvml.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\nvml.dll C:\\ProgramData\\anaconda3\\nvml.dll C:\\ProgramData\\anaconda3\\Scripts\\nvml.dll C:\\ProgramData\\anaconda3\\Library\\mingw-w64\\bin\\nvml.dll C:\\ProgramData\\anaconda3\\Library\\usr\\bin\\nvml.dll C:\\ProgramData\\anaconda3\\Library\\bin\\nvml.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\nvml.dll c:\\Windows\\System32\\nvml.dll]"
time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=gpu.go:529 msg="skipping PhysX cuda library path" path="C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvml.dll"
time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[C:\\WINDOWS\\system32\\nvml.dll c:\\Windows\\System32\\nvml.dll]"
time=2025-07-23T23:30:49.449+08:00 level=DEBUG source=gpu.go:111 msg="nvidia-ml loaded" library=C:\WINDOWS\system32\nvml.dll
time=2025-07-23T23:30:49.449+08:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=nvcuda.dll
time=2025-07-23T23:30:49.449+08:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvcuda.dll D:\\VMware\\bin\\nvcuda.dll C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\bin\\nvcuda.dll C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\libnvvp\\nvcuda.dll C:\\WINDOWS\\system32\\nvcuda.dll C:\\WINDOWS\\nvcuda.dll C:\\WINDOWS\\System32\\Wbem\\nvcuda.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\nvcuda.dll C:\\WINDOWS\\System32\\OpenSSH\\nvcuda.dll C:\\Program Files\\Zero Install\\nvcuda.dll C:\\Program Files\\dotnet\\nvcuda.dll C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR\\nvcuda.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvcuda.dll D:\\Git\\Git\\cmd\\nvcuda.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\nvcuda.dll D:\\MATLAB\\bin\\nvcuda.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Microsoft\\WindowsApps\\nvcuda.dll C:\\Users\\Sonic_Dong\\.lmstudio\\bin\\nvcuda.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll C:\\ProgramData\\anaconda3\\nvcuda.dll C:\\ProgramData\\anaconda3\\Scripts\\nvcuda.dll C:\\ProgramData\\anaconda3\\Library\\mingw-w64\\bin\\nvcuda.dll C:\\ProgramData\\anaconda3\\Library\\usr\\bin\\nvcuda.dll C:\\ProgramData\\anaconda3\\Library\\bin\\nvcuda.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\nvcuda.dll c:\\windows\\system*\\nvcuda.dll]"
time=2025-07-23T23:30:49.449+08:00 level=DEBUG source=gpu.go:529 msg="skipping PhysX cuda library path" path="C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvcuda.dll"
time=2025-07-23T23:30:49.450+08:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[C:\WINDOWS\system32\nvcuda.dll]
initializing C:\WINDOWS\system32\nvcuda.dll
dlsym: cuInit - 00007FF985DA1F80
dlsym: cuDriverGetVersion - 00007FF985DA2020
dlsym: cuDeviceGetCount - 00007FF985DA2816
dlsym: cuDeviceGet - 00007FF985DA2810
dlsym: cuDeviceGetAttribute - 00007FF985DA2170
dlsym: cuDeviceGetUuid - 00007FF985DA2822
dlsym: cuDeviceGetName - 00007FF985DA281C
dlsym: cuCtxCreate_v3 - 00007FF985DA2894
dlsym: cuMemGetInfo_v2 - 00007FF985DA2996
dlsym: cuCtxDestroy - 00007FF985DA28A6
calling cuInit
calling cuDriverGetVersion
raw version 0x2f3a
CUDA driver version: 12.9
calling cuDeviceGetCount
device count 1
time=2025-07-23T23:30:49.459+08:00 level=DEBUG source=gpu.go:125 msg="detected GPUs" count=1 library=C:\WINDOWS\system32\nvcuda.dll
[GPU-cfb69b01-0098-7635-99b9-9e2c46877913] CUDA totalMem 32606mb
[GPU-cfb69b01-0098-7635-99b9-9e2c46877913] CUDA freeMem 30843mb
[GPU-cfb69b01-0098-7635-99b9-9e2c46877913] Compute Capability 12.0
time=2025-07-23T23:30:49.544+08:00 level=DEBUG source=amd_hip_windows.go:88 msg=hipDriverGetVersion version=60450101
time=2025-07-23T23:30:49.544+08:00 level=INFO source=amd_hip_windows.go:103 msg="AMD ROCm reports no devices found"
time=2025-07-23T23:30:49.544+08:00 level=INFO source=amd_windows.go:49 msg="no compatible amdgpu devices detected"
releasing cuda driver library
releasing nvml library
time=2025-07-23T23:30:49.545+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 library=cuda variant=v12 compute=12.0 driver=12.9 name="NVIDIA GeForce RTX 5090" total="31.8 GiB" available="30.1 GiB"
[GIN] 2025/07/23 - 23:31:07 | 200 |       509.9µs |       127.0.0.1 | HEAD     "/"
[GIN] 2025/07/23 - 23:31:07 | 404 |      2.1248ms |       127.0.0.1 | POST     "/api/show"
time=2025-07-23T23:31:09.548+08:00 level=INFO source=download.go:177 msg="downloading aeda25e63ebd in 16 208 MB part(s)"
time=2025-07-23T23:33:59.975+08:00 level=INFO source=download.go:295 msg="aeda25e63ebd part 1 attempt 0 failed: unexpected EOF, retrying in 1s"
time=2025-07-23T23:34:37.564+08:00 level=INFO source=download.go:177 msg="downloading e0a42594d802 in 1 358 B part(s)"
time=2025-07-23T23:34:39.186+08:00 level=INFO source=download.go:177 msg="downloading dd084c7d92a3 in 1 8.4 KB part(s)"
time=2025-07-23T23:34:40.807+08:00 level=INFO source=download.go:177 msg="downloading 3116c5225075 in 1 77 B part(s)"
time=2025-07-23T23:34:42.455+08:00 level=INFO source=download.go:177 msg="downloading b6ae5839783f in 1 489 B part(s)"
[GIN] 2025/07/23 - 23:34:46 | 200 |         3m38s |       127.0.0.1 | POST     "/api/pull"
time=2025-07-23T23:34:46.342+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.alignment default=32
[GIN] 2025/07/23 - 23:34:46 | 200 |     67.9182ms |       127.0.0.1 | POST     "/api/show"
time=2025-07-23T23:34:46.387+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="80.3 GiB" before.free_swap="76.2 GiB" now.total="95.6 GiB" now.free="83.0 GiB" now.free_swap="78.9 GiB"
time=2025-07-23T23:34:46.394+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="30.1 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB"
releasing nvml library
time=2025-07-23T23:34:46.412+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.alignment default=32
time=2025-07-23T23:34:46.448+08:00 level=DEBUG source=sched.go:228 msg="loading first model" model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25
time=2025-07-23T23:34:46.448+08:00 level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=1 available="[29.9 GiB]"
time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=memory.go:201 msg="gpu has too little memory to allocate any layers" id=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 library=cuda variant=v12 compute=12.0 driver=12.9 name="NVIDIA GeForce RTX 5090" total="31.8 GiB" available="29.9 GiB" minimum_memory=479199232 layer_size="2.0 GiB" gpu_zer_overhead="0 B" partial_offload="50.8 GiB" full_offload="35.2 GiB"
time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=memory.go:351 msg="insufficient VRAM to load any model layers"
time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=1 available="[29.9 GiB]"
time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=memory.go:201 msg="gpu has too little memory to allocate any layers" id=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 library=cuda variant=v12 compute=12.0 driver=12.9 name="NVIDIA GeForce RTX 5090" total="31.8 GiB" available="29.9 GiB" minimum_memory=479199232 layer_size="2.0 GiB" gpu_zer_overhead="0 B" partial_offload="50.8 GiB" full_offload="35.2 GiB"
time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=memory.go:351 msg="insufficient VRAM to load any model layers"
time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.0 GiB" before.free_swap="78.9 GiB" now.total="95.6 GiB" now.free="83.0 GiB" now.free_swap="78.9 GiB"
time=2025-07-23T23:34:46.457+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB"
releasing nvml library
time=2025-07-23T23:34:46.457+08:00 level=INFO source=server.go:135 msg="system memory" total="95.6 GiB" free="83.0 GiB" free_swap="78.9 GiB"
time=2025-07-23T23:34:46.457+08:00 level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=1 available="[29.9 GiB]"
time=2025-07-23T23:34:46.458+08:00 level=DEBUG source=memory.go:201 msg="gpu has too little memory to allocate any layers" id=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 library=cuda variant=v12 compute=12.0 driver=12.9 name="NVIDIA GeForce RTX 5090" total="31.8 GiB" available="29.9 GiB" minimum_memory=479199232 layer_size="2.0 GiB" gpu_zer_overhead="0 B" partial_offload="50.8 GiB" full_offload="35.2 GiB"
time=2025-07-23T23:34:46.458+08:00 level=DEBUG source=memory.go:351 msg="insufficient VRAM to load any model layers"
time=2025-07-23T23:34:46.458+08:00 level=INFO source=server.go:175 msg=offload library=cuda layers.requested=-1 layers.model=35 layers.offload=0 layers.split="" memory.available="[29.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="99.9 GiB" memory.required.partial="0 B" memory.required.kv="95.8 GiB" memory.required.allocations="[0 B]" memory.weights.total="2.3 GiB" memory.weights.repeating="1.8 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="35.2 GiB" memory.graph.partial="50.8 GiB" projector.weights="795.9 MiB" projector.graph="1.0 GiB"
time=2025-07-23T23:34:46.458+08:00 level=DEBUG source=server.go:291 msg="compatible gpu libraries" compatible=[]
time=2025-07-23T23:34:46.487+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.alignment default=32
time=2025-07-23T23:34:46.487+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=tokenizer.ggml.eot_token_id default=106
time=2025-07-23T23:34:46.487+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}"
time=2025-07-23T23:34:46.490+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07
time=2025-07-23T23:34:46.490+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.local.freq_base default=10000
time=2025-07-23T23:34:46.490+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.global.freq_base default=1e+06
time=2025-07-23T23:34:46.490+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.freq_scale default=1
time=2025-07-23T23:34:46.490+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.mm_tokens_per_image default=256
time=2025-07-23T23:34:46.494+08:00 level=DEBUG source=gpu.go:695 msg="no filter required for library cpu"
time=2025-07-23T23:34:46.494+08:00 level=INFO source=server.go:438 msg="starting llama server" cmd="C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\Sonic_Dong\\.ollama\\models\\blobs\\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 --ctx-size 2048000 --batch-size 512 --threads 16 --no-mmap --parallel 500 --port 8037"
time=2025-07-23T23:34:46.494+08:00 level=DEBUG source=server.go:439 msg=subprocess CUDA_VISIBLE_DEVICES=0 OLLAMA_GPU_LAYER=cuda OLLAMA_ORIGINS=* OLLAMA_NUM_PARALLEL=500 CUDA_PATH_V12_9="C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9" PATH="C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\lib\\ollama;D:\\VMware\\bin\\;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\bin;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\libnvvp;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Program Files\\Zero Install;C:\\Program Files\\dotnet\\;C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;D:\\Git\\Git\\cmd;C:\\Program Files\\Docker\\Docker\\resources\\bin;D:\\MATLAB\\bin;C:\\Users\\Sonic_Dong\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\Sonic_Dong\\.lmstudio\\bin;C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama;C:\\ProgramData\\anaconda3;C:\\ProgramData\\anaconda3\\Scripts;C:\\ProgramData\\anaconda3\\Library\\mingw-w64\\bin;C:\\ProgramData\\anaconda3\\Library\\usr\\bin;C:\\ProgramData\\anaconda3\\Library\\bin;C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Microsoft VS Code\\bin;C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\lib\\ollama" OLLAMA_MODELS=C:\Users\Sonic_Dong\.ollama\models CUDA_HOME="C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9" OLLAMA_HOST=0.0.0.0 OLLAMA_MAX_LOADED_MODELS=5 CUDA_PATH="C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9" OLLAMA_DEBUG=1 OLLAMA_LIBRARY_PATH=C:\Users\Sonic_Dong\AppData\Local\Programs\Ollama\lib\ollama
time=2025-07-23T23:34:46.499+08:00 level=INFO source=sched.go:483 msg="loaded runners" count=1
time=2025-07-23T23:34:46.499+08:00 level=INFO source=server.go:598 msg="waiting for llama runner to start responding"
time=2025-07-23T23:34:46.499+08:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server error"
time=2025-07-23T23:34:46.524+08:00 level=INFO source=runner.go:925 msg="starting ollama engine"
time=2025-07-23T23:34:46.529+08:00 level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:8037"
time=2025-07-23T23:34:46.561+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.alignment default=32
time=2025-07-23T23:34:46.562+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.name default=""
time=2025-07-23T23:34:46.562+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.description default=""
time=2025-07-23T23:34:46.562+08:00 level=INFO source=ggml.go:92 msg="" architecture=gemma3 file_type=Q4_K_M name="" description="" num_tensors=883 num_key_values=36
time=2025-07-23T23:34:46.562+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=C:\Users\Sonic_Dong\AppData\Local\Programs\Ollama\lib\ollama
time=2025-07-23T23:34:46.750+08:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from C:\Users\Sonic_Dong\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll
load_backend: loaded CPU backend from C:\Users\Sonic_Dong\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-icelake.dll
time=2025-07-23T23:34:53.848+08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2025-07-23T23:34:53.907+08:00 level=INFO source=ggml.go:359 msg="offloading 0 repeating layers to GPU"
time=2025-07-23T23:34:53.907+08:00 level=INFO source=ggml.go:363 msg="offloading output layer to CPU"
time=2025-07-23T23:34:53.907+08:00 level=INFO source=ggml.go:375 msg="offloaded 0/35 layers to GPU"
time=2025-07-23T23:34:53.907+08:00 level=INFO source=ggml.go:377 msg="model weights" buffer=CPU size="3.6 GiB"
time=2025-07-23T23:34:53.908+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=tokenizer.ggml.eot_token_id default=106
time=2025-07-23T23:34:53.908+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}"
time=2025-07-23T23:34:53.909+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07
time=2025-07-23T23:34:53.909+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.local.freq_base default=10000
time=2025-07-23T23:34:53.909+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.global.freq_base default=1e+06
time=2025-07-23T23:34:53.909+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.freq_scale default=1
time=2025-07-23T23:34:53.909+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.mm_tokens_per_image default=256
time=2025-07-23T23:34:54.007+08:00 level=DEBUG source=ggml.go:648 msg="compute graph" nodes=972 splits=1
time=2025-07-23T23:34:54.007+08:00 level=INFO source=ggml.go:666 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 size="0 B"
time=2025-07-23T23:34:54.008+08:00 level=INFO source=ggml.go:666 msg="compute graph" backend=CPU buffer_type=CPU size="1.1 GiB"
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 4194304000
panic: insufficient memory - required allocations: {InputWeights:550502400A CPU:{Name:CPU UUID: Weights:[60561408A 60561408A 60561408A 60561408A 53803008A 53803008A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 59885568A 60561408A 59885568A 59885568A 60561408A 1390946752A] Cache:[2099249152A 2099249152A 2099249152A 2099249152A 2099249152A 8388608000A 2099249152A 2099249152A 2099249152A 2099249152A 2099249152A 8388608000A 2099249152A 2099249152A 2099249152A 2099249152A 2099249152A 8388608000A 2099249152A 2099249152A 2099249152A 2099249152A 2099249152A 4194304000F 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U] Graph:1174011904A} GPUs:[{Name:CUDA0 UUID:GPU-cfb69b01-0098-7635-99b9-9e2c46877913 Weights:[0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U] Cache:[0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U] Graph:0A}]}

goroutine 51 [running]:
github.com/ollama/ollama/ml/backend/ggml.(*Context).newTensor(0xc000416a00, 0x1?, {0xc001743248, 0x3, 0x7ff7c397b8c5?})
	C:/a/ollama/ollama/ml/backend/ggml/ggml.go:742 +0x696
github.com/ollama/ollama/ml/backend/ggml.(*Context).Zeros(0x7ff7c4a8f780?, 0xc0017fd410?, {0xc001743248?, 0xc000606a80?, 0x300000002?})
	C:/a/ollama/ollama/ml/backend/ggml/ggml.go:755 +0x1c
github.com/ollama/ollama/kvcache.(*Causal).Put(0xc0017d4d20, {0x7ff7c4d9bc70, 0xc0000bc100}, {0x7ff7c4da8dc0, 0xc001d07020}, {0x7ff7c4da8dc0, 0xc001d07050})
	C:/a/ollama/ollama/kvcache/causal.go:559 +0x26a
github.com/ollama/ollama/kvcache.(*WrapperCache).Put(0x298e326dca0?, {0x7ff7c4d9bc70?, 0xc0000bc100?}, {0x7ff7c4da8dc0?, 0xc001d07020?}, {0x7ff7c4da8dc0?, 0xc001d07050?})
	C:/a/ollama/ollama/kvcache/wrapper.go:81 +0x4f
github.com/ollama/ollama/ml/nn.Attention({0x7ff7c4d9bc70, 0xc0000bc100}, {0x7ff7c4da8dc0, 0xc001d06f90}, {0x7ff7c4da8dc0, 0xc001d07020}, {0x7ff7c4da8dc0, 0xc001d07050}, 0x3ff0000000000000, {0x7ff7c4d9ac38, ...})
	C:/a/ollama/ollama/ml/nn/attention.go:39 +0x1c3
github.com/ollama/ollama/model/models/gemma3.(*TextSelfAttention).Forward(0xc001d724b0, {0x7ff7c4d9bc70, 0xc0000bc100}, 0x17, {0x7ff7c4da8dc0, 0xc001d06ed0}, {0x7ff7c4da8dc0, 0xc001cf64e0}, {0x7ff7c4d9ac38, 0xc000273520}, ...)
	C:/a/ollama/ollama/model/models/gemma3/model_text.go:104 +0x53e
github.com/ollama/ollama/model/models/gemma3.(*TextLayer).Forward(0xc000047890, {0x7ff7c4d9bc70, 0xc0000bc100}, 0x17, {0x7ff7c4da8dc0, 0xc001d06eb8}, {0x7ff7c4da8dc0, 0xc001cf64e0}, {0x0, 0x0}, ...)
	C:/a/ollama/ollama/model/models/gemma3/model_text.go:143 +0xda
github.com/ollama/ollama/model/models/gemma3.(*TextModel).Forward(0xc001cd6080, {0x7ff7c4d9bc70, 0xc0000bc100}, {0x7ff7c4da8dc0?, 0xc001cf6480?}, {0x7ff7c4da8dc0, 0xc001cf64e0}, {0x7ff7c4da8dc0, 0xc001cf64f8}, {{0x7ff7c4da8dc0, ...}, ...}, ...)
	C:/a/ollama/ollama/model/models/gemma3/model_text.go:197 +0x3be
github.com/ollama/ollama/model/models/gemma3.(*Model).Forward(0xc001ca6c60, {0x7ff7c4d9bc70, 0xc0000bc100}, {{0x7ff7c4da8dc0, 0xc001cf6480}, {0xc001d8a0c0, 0x1, 0x1}, {0xc00e896000, 0x200, ...}, ...})
	C:/a/ollama/ollama/model/models/gemma3/model.go:147 +0x1d7
github.com/ollama/ollama/runner/ollamarunner.(*Server).reserveWorstCaseGraph(0xc00016f9e0)
	C:/a/ollama/ollama/runner/ollamarunner/runner.go:821 +0xac5
github.com/ollama/ollama/runner/ollamarunner.(*Server).initModel(0xc00016f9e0, {0xc00003a150?, 0x0?}, {0x10, 0x0, 0x0, {0x0, 0x0, 0x0}, 0x0}, ...)
	C:/a/ollama/ollama/runner/ollamarunner/runner.go:865 +0x270
github.com/ollama/ollama/runner/ollamarunner.(*Server).load(0xc00016f9e0, {0x7ff7c4d93510, 0xc000158eb0}, {0xc00003a150?, 0x0?}, {0x10, 0x0, 0x0, {0x0, 0x0, ...}, ...}, ...)
	C:/a/ollama/ollama/runner/ollamarunner/runner.go:878 +0xb8
created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1
	C:/a/ollama/ollama/runner/ollamarunner/runner.go:959 +0xa11
time=2025-07-23T23:34:59.290+08:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server error"
time=2025-07-23T23:35:00.793+08:00 level=ERROR source=sched.go:489 msg="error loading llama server" error="llama runner process has terminated: exit status 2"
time=2025-07-23T23:35:00.793+08:00 level=DEBUG source=sched.go:491 msg="triggering expiration for failed load" runner.name=registry.ollama.ai/library/gemma3:latest runner.inference=cuda runner.devices=1 runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 runner.num_ctx=2048000
time=2025-07-23T23:35:00.793+08:00 level=DEBUG source=sched.go:364 msg="runner expired event received" runner.name=registry.ollama.ai/library/gemma3:latest runner.inference=cuda runner.devices=1 runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 runner.num_ctx=2048000
time=2025-07-23T23:35:00.793+08:00 level=DEBUG source=sched.go:379 msg="got lock to unload expired event" runner.name=registry.ollama.ai/library/gemma3:latest runner.inference=cuda runner.devices=1 runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 runner.num_ctx=2048000
time=2025-07-23T23:35:00.793+08:00 level=DEBUG source=sched.go:402 msg="starting background wait for VRAM recovery" runner.name=registry.ollama.ai/library/gemma3:latest runner.inference=cuda runner.devices=1 runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 runner.num_ctx=2048000
[GIN] 2025/07/23 - 23:35:00 | 500 |   14.4415483s |       127.0.0.1 | POST     "/api/generate"
time=2025-07-23T23:35:00.793+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.0 GiB" before.free_swap="78.9 GiB" now.total="95.6 GiB" now.free="83.4 GiB" now.free_swap="79.2 GiB"
time=2025-07-23T23:35:00.801+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB"
releasing nvml library
time=2025-07-23T23:35:00.802+08:00 level=DEBUG source=server.go:1031 msg="stopping llama server" pid=31340
time=2025-07-23T23:35:00.802+08:00 level=DEBUG source=sched.go:407 msg="runner terminated and removed from list, blocking for VRAM recovery" runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25
time=2025-07-23T23:35:01.052+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.4 GiB" before.free_swap="79.2 GiB" now.total="95.6 GiB" now.free="83.4 GiB" now.free_swap="79.2 GiB"
time=2025-07-23T23:35:01.060+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB"
releasing nvml library
time=2025-07-23T23:35:01.302+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.4 GiB" before.free_swap="79.2 GiB" now.total="95.6 GiB" now.free="83.4 GiB" now.free_swap="79.2 GiB"
time=2025-07-23T23:35:01.309+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB"
releasing nvml library
time=2025-07-23T23:35:01.552+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.4 GiB" before.free_swap="79.2 GiB" now.total="95.6 GiB" now.free="83.4 GiB" now.free_swap="79.2 GiB"
time=2025-07-23T23:35:01.559+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB"
releasing nvml library
time=2025-07-23T23:35:01.802+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.4 GiB" before.free_swap="79.2 GiB" now.total="95.6 GiB" now.free="83.4 GiB" now.free_swap="79.2 GiB"
time=2025-07-23T23:35:01.809+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB"
releasing nvml library
time=2025-07-23T23:35:01.809+08:00 level=DEBUG source=sched.go:700 msg="gpu VRAM free memory converged after 1.02 seconds" runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25
time=2025-07-23T23:35:01.809+08:00 level=DEBUG source=sched.go:410 msg="sending an unloaded event" runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25
time=2025-07-23T23:35:01.809+08:00 level=DEBUG source=sched.go:312 msg="ignoring unload event with no pending requests"
time=2025-07-23T23:36:00.231+08:00 level=DEBUG source=sched.go:322 msg="shutting down scheduler completed loop"

OS

Windows

GPU

Nvidia

CPU

AMD

Ollama version

0.9.6

Originally created by @Cope-233 on GitHub (Jul 23, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11504 ### What is the issue? I have already set the environment path: CUDA_VISIBLE_DEVICES=0 OLLAMA_GPU_LAYER=cuda But ollama cannot load the model to my RTX5090 Driver Version: 577.00 CUDA Version: 12.9 ### Relevant log output ```shell time=2025-07-23T23:30:49.433+08:00 level=INFO source=routes.go:1235 msg="server config" env="map[CUDA_VISIBLE_DEVICES:0 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:5 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\Sonic_Dong\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:500 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:]" time=2025-07-23T23:30:49.438+08:00 level=INFO source=images.go:476 msg="total blobs: 0" time=2025-07-23T23:30:49.438+08:00 level=INFO source=images.go:483 msg="total unused blobs removed: 0" time=2025-07-23T23:30:49.439+08:00 level=INFO source=routes.go:1288 msg="Listening on [::]:11434 (version 0.9.6)" time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=sched.go:108 msg="starting llm scheduler" time=2025-07-23T23:30:49.439+08:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-07-23T23:30:49.439+08:00 level=INFO source=gpu_windows.go:167 msg=packages count=1 time=2025-07-23T23:30:49.439+08:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=16 efficiency=0 threads=32 time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=nvml.dll time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvml.dll D:\\VMware\\bin\\nvml.dll C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\bin\\nvml.dll C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\libnvvp\\nvml.dll C:\\WINDOWS\\system32\\nvml.dll C:\\WINDOWS\\nvml.dll C:\\WINDOWS\\System32\\Wbem\\nvml.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\nvml.dll C:\\WINDOWS\\System32\\OpenSSH\\nvml.dll C:\\Program Files\\Zero Install\\nvml.dll C:\\Program Files\\dotnet\\nvml.dll C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR\\nvml.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvml.dll D:\\Git\\Git\\cmd\\nvml.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\nvml.dll D:\\MATLAB\\bin\\nvml.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Microsoft\\WindowsApps\\nvml.dll C:\\Users\\Sonic_Dong\\.lmstudio\\bin\\nvml.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\nvml.dll C:\\ProgramData\\anaconda3\\nvml.dll C:\\ProgramData\\anaconda3\\Scripts\\nvml.dll C:\\ProgramData\\anaconda3\\Library\\mingw-w64\\bin\\nvml.dll C:\\ProgramData\\anaconda3\\Library\\usr\\bin\\nvml.dll C:\\ProgramData\\anaconda3\\Library\\bin\\nvml.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\nvml.dll c:\\Windows\\System32\\nvml.dll]" time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=gpu.go:529 msg="skipping PhysX cuda library path" path="C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvml.dll" time=2025-07-23T23:30:49.439+08:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[C:\\WINDOWS\\system32\\nvml.dll c:\\Windows\\System32\\nvml.dll]" time=2025-07-23T23:30:49.449+08:00 level=DEBUG source=gpu.go:111 msg="nvidia-ml loaded" library=C:\WINDOWS\system32\nvml.dll time=2025-07-23T23:30:49.449+08:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=nvcuda.dll time=2025-07-23T23:30:49.449+08:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvcuda.dll D:\\VMware\\bin\\nvcuda.dll C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\bin\\nvcuda.dll C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\libnvvp\\nvcuda.dll C:\\WINDOWS\\system32\\nvcuda.dll C:\\WINDOWS\\nvcuda.dll C:\\WINDOWS\\System32\\Wbem\\nvcuda.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\nvcuda.dll C:\\WINDOWS\\System32\\OpenSSH\\nvcuda.dll C:\\Program Files\\Zero Install\\nvcuda.dll C:\\Program Files\\dotnet\\nvcuda.dll C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR\\nvcuda.dll C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvcuda.dll D:\\Git\\Git\\cmd\\nvcuda.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\nvcuda.dll D:\\MATLAB\\bin\\nvcuda.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Microsoft\\WindowsApps\\nvcuda.dll C:\\Users\\Sonic_Dong\\.lmstudio\\bin\\nvcuda.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll C:\\ProgramData\\anaconda3\\nvcuda.dll C:\\ProgramData\\anaconda3\\Scripts\\nvcuda.dll C:\\ProgramData\\anaconda3\\Library\\mingw-w64\\bin\\nvcuda.dll C:\\ProgramData\\anaconda3\\Library\\usr\\bin\\nvcuda.dll C:\\ProgramData\\anaconda3\\Library\\bin\\nvcuda.dll C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\nvcuda.dll c:\\windows\\system*\\nvcuda.dll]" time=2025-07-23T23:30:49.449+08:00 level=DEBUG source=gpu.go:529 msg="skipping PhysX cuda library path" path="C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common\\nvcuda.dll" time=2025-07-23T23:30:49.450+08:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[C:\WINDOWS\system32\nvcuda.dll] initializing C:\WINDOWS\system32\nvcuda.dll dlsym: cuInit - 00007FF985DA1F80 dlsym: cuDriverGetVersion - 00007FF985DA2020 dlsym: cuDeviceGetCount - 00007FF985DA2816 dlsym: cuDeviceGet - 00007FF985DA2810 dlsym: cuDeviceGetAttribute - 00007FF985DA2170 dlsym: cuDeviceGetUuid - 00007FF985DA2822 dlsym: cuDeviceGetName - 00007FF985DA281C dlsym: cuCtxCreate_v3 - 00007FF985DA2894 dlsym: cuMemGetInfo_v2 - 00007FF985DA2996 dlsym: cuCtxDestroy - 00007FF985DA28A6 calling cuInit calling cuDriverGetVersion raw version 0x2f3a CUDA driver version: 12.9 calling cuDeviceGetCount device count 1 time=2025-07-23T23:30:49.459+08:00 level=DEBUG source=gpu.go:125 msg="detected GPUs" count=1 library=C:\WINDOWS\system32\nvcuda.dll [GPU-cfb69b01-0098-7635-99b9-9e2c46877913] CUDA totalMem 32606mb [GPU-cfb69b01-0098-7635-99b9-9e2c46877913] CUDA freeMem 30843mb [GPU-cfb69b01-0098-7635-99b9-9e2c46877913] Compute Capability 12.0 time=2025-07-23T23:30:49.544+08:00 level=DEBUG source=amd_hip_windows.go:88 msg=hipDriverGetVersion version=60450101 time=2025-07-23T23:30:49.544+08:00 level=INFO source=amd_hip_windows.go:103 msg="AMD ROCm reports no devices found" time=2025-07-23T23:30:49.544+08:00 level=INFO source=amd_windows.go:49 msg="no compatible amdgpu devices detected" releasing cuda driver library releasing nvml library time=2025-07-23T23:30:49.545+08:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 library=cuda variant=v12 compute=12.0 driver=12.9 name="NVIDIA GeForce RTX 5090" total="31.8 GiB" available="30.1 GiB" [GIN] 2025/07/23 - 23:31:07 | 200 | 509.9µs | 127.0.0.1 | HEAD "/" [GIN] 2025/07/23 - 23:31:07 | 404 | 2.1248ms | 127.0.0.1 | POST "/api/show" time=2025-07-23T23:31:09.548+08:00 level=INFO source=download.go:177 msg="downloading aeda25e63ebd in 16 208 MB part(s)" time=2025-07-23T23:33:59.975+08:00 level=INFO source=download.go:295 msg="aeda25e63ebd part 1 attempt 0 failed: unexpected EOF, retrying in 1s" time=2025-07-23T23:34:37.564+08:00 level=INFO source=download.go:177 msg="downloading e0a42594d802 in 1 358 B part(s)" time=2025-07-23T23:34:39.186+08:00 level=INFO source=download.go:177 msg="downloading dd084c7d92a3 in 1 8.4 KB part(s)" time=2025-07-23T23:34:40.807+08:00 level=INFO source=download.go:177 msg="downloading 3116c5225075 in 1 77 B part(s)" time=2025-07-23T23:34:42.455+08:00 level=INFO source=download.go:177 msg="downloading b6ae5839783f in 1 489 B part(s)" [GIN] 2025/07/23 - 23:34:46 | 200 | 3m38s | 127.0.0.1 | POST "/api/pull" time=2025-07-23T23:34:46.342+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.alignment default=32 [GIN] 2025/07/23 - 23:34:46 | 200 | 67.9182ms | 127.0.0.1 | POST "/api/show" time=2025-07-23T23:34:46.387+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="80.3 GiB" before.free_swap="76.2 GiB" now.total="95.6 GiB" now.free="83.0 GiB" now.free_swap="78.9 GiB" time=2025-07-23T23:34:46.394+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="30.1 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB" releasing nvml library time=2025-07-23T23:34:46.412+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.alignment default=32 time=2025-07-23T23:34:46.448+08:00 level=DEBUG source=sched.go:228 msg="loading first model" model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 time=2025-07-23T23:34:46.448+08:00 level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=1 available="[29.9 GiB]" time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=memory.go:201 msg="gpu has too little memory to allocate any layers" id=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 library=cuda variant=v12 compute=12.0 driver=12.9 name="NVIDIA GeForce RTX 5090" total="31.8 GiB" available="29.9 GiB" minimum_memory=479199232 layer_size="2.0 GiB" gpu_zer_overhead="0 B" partial_offload="50.8 GiB" full_offload="35.2 GiB" time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=memory.go:351 msg="insufficient VRAM to load any model layers" time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=1 available="[29.9 GiB]" time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=memory.go:201 msg="gpu has too little memory to allocate any layers" id=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 library=cuda variant=v12 compute=12.0 driver=12.9 name="NVIDIA GeForce RTX 5090" total="31.8 GiB" available="29.9 GiB" minimum_memory=479199232 layer_size="2.0 GiB" gpu_zer_overhead="0 B" partial_offload="50.8 GiB" full_offload="35.2 GiB" time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=memory.go:351 msg="insufficient VRAM to load any model layers" time=2025-07-23T23:34:46.449+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.0 GiB" before.free_swap="78.9 GiB" now.total="95.6 GiB" now.free="83.0 GiB" now.free_swap="78.9 GiB" time=2025-07-23T23:34:46.457+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB" releasing nvml library time=2025-07-23T23:34:46.457+08:00 level=INFO source=server.go:135 msg="system memory" total="95.6 GiB" free="83.0 GiB" free_swap="78.9 GiB" time=2025-07-23T23:34:46.457+08:00 level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=1 available="[29.9 GiB]" time=2025-07-23T23:34:46.458+08:00 level=DEBUG source=memory.go:201 msg="gpu has too little memory to allocate any layers" id=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 library=cuda variant=v12 compute=12.0 driver=12.9 name="NVIDIA GeForce RTX 5090" total="31.8 GiB" available="29.9 GiB" minimum_memory=479199232 layer_size="2.0 GiB" gpu_zer_overhead="0 B" partial_offload="50.8 GiB" full_offload="35.2 GiB" time=2025-07-23T23:34:46.458+08:00 level=DEBUG source=memory.go:351 msg="insufficient VRAM to load any model layers" time=2025-07-23T23:34:46.458+08:00 level=INFO source=server.go:175 msg=offload library=cuda layers.requested=-1 layers.model=35 layers.offload=0 layers.split="" memory.available="[29.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="99.9 GiB" memory.required.partial="0 B" memory.required.kv="95.8 GiB" memory.required.allocations="[0 B]" memory.weights.total="2.3 GiB" memory.weights.repeating="1.8 GiB" memory.weights.nonrepeating="525.0 MiB" memory.graph.full="35.2 GiB" memory.graph.partial="50.8 GiB" projector.weights="795.9 MiB" projector.graph="1.0 GiB" time=2025-07-23T23:34:46.458+08:00 level=DEBUG source=server.go:291 msg="compatible gpu libraries" compatible=[] time=2025-07-23T23:34:46.487+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.alignment default=32 time=2025-07-23T23:34:46.487+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=tokenizer.ggml.eot_token_id default=106 time=2025-07-23T23:34:46.487+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}" time=2025-07-23T23:34:46.490+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07 time=2025-07-23T23:34:46.490+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.local.freq_base default=10000 time=2025-07-23T23:34:46.490+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.global.freq_base default=1e+06 time=2025-07-23T23:34:46.490+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.freq_scale default=1 time=2025-07-23T23:34:46.490+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.mm_tokens_per_image default=256 time=2025-07-23T23:34:46.494+08:00 level=DEBUG source=gpu.go:695 msg="no filter required for library cpu" time=2025-07-23T23:34:46.494+08:00 level=INFO source=server.go:438 msg="starting llama server" cmd="C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\Sonic_Dong\\.ollama\\models\\blobs\\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 --ctx-size 2048000 --batch-size 512 --threads 16 --no-mmap --parallel 500 --port 8037" time=2025-07-23T23:34:46.494+08:00 level=DEBUG source=server.go:439 msg=subprocess CUDA_VISIBLE_DEVICES=0 OLLAMA_GPU_LAYER=cuda OLLAMA_ORIGINS=* OLLAMA_NUM_PARALLEL=500 CUDA_PATH_V12_9="C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9" PATH="C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\lib\\ollama;D:\\VMware\\bin\\;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\bin;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\libnvvp;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Program Files\\Zero Install;C:\\Program Files\\dotnet\\;C:\\Program Files\\NVIDIA Corporation\\NVIDIA App\\NvDLISR;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;D:\\Git\\Git\\cmd;C:\\Program Files\\Docker\\Docker\\resources\\bin;D:\\MATLAB\\bin;C:\\Users\\Sonic_Dong\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\Sonic_Dong\\.lmstudio\\bin;C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama;C:\\ProgramData\\anaconda3;C:\\ProgramData\\anaconda3\\Scripts;C:\\ProgramData\\anaconda3\\Library\\mingw-w64\\bin;C:\\ProgramData\\anaconda3\\Library\\usr\\bin;C:\\ProgramData\\anaconda3\\Library\\bin;C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Microsoft VS Code\\bin;C:\\Users\\Sonic_Dong\\AppData\\Local\\Programs\\Ollama\\lib\\ollama" OLLAMA_MODELS=C:\Users\Sonic_Dong\.ollama\models CUDA_HOME="C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9" OLLAMA_HOST=0.0.0.0 OLLAMA_MAX_LOADED_MODELS=5 CUDA_PATH="C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9" OLLAMA_DEBUG=1 OLLAMA_LIBRARY_PATH=C:\Users\Sonic_Dong\AppData\Local\Programs\Ollama\lib\ollama time=2025-07-23T23:34:46.499+08:00 level=INFO source=sched.go:483 msg="loaded runners" count=1 time=2025-07-23T23:34:46.499+08:00 level=INFO source=server.go:598 msg="waiting for llama runner to start responding" time=2025-07-23T23:34:46.499+08:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server error" time=2025-07-23T23:34:46.524+08:00 level=INFO source=runner.go:925 msg="starting ollama engine" time=2025-07-23T23:34:46.529+08:00 level=INFO source=runner.go:983 msg="Server listening on 127.0.0.1:8037" time=2025-07-23T23:34:46.561+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.alignment default=32 time=2025-07-23T23:34:46.562+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.name default="" time=2025-07-23T23:34:46.562+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=general.description default="" time=2025-07-23T23:34:46.562+08:00 level=INFO source=ggml.go:92 msg="" architecture=gemma3 file_type=Q4_K_M name="" description="" num_tensors=883 num_key_values=36 time=2025-07-23T23:34:46.562+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=C:\Users\Sonic_Dong\AppData\Local\Programs\Ollama\lib\ollama time=2025-07-23T23:34:46.750+08:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server loading model" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes load_backend: loaded CUDA backend from C:\Users\Sonic_Dong\AppData\Local\Programs\Ollama\lib\ollama\ggml-cuda.dll load_backend: loaded CPU backend from C:\Users\Sonic_Dong\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-icelake.dll time=2025-07-23T23:34:53.848+08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2025-07-23T23:34:53.907+08:00 level=INFO source=ggml.go:359 msg="offloading 0 repeating layers to GPU" time=2025-07-23T23:34:53.907+08:00 level=INFO source=ggml.go:363 msg="offloading output layer to CPU" time=2025-07-23T23:34:53.907+08:00 level=INFO source=ggml.go:375 msg="offloaded 0/35 layers to GPU" time=2025-07-23T23:34:53.907+08:00 level=INFO source=ggml.go:377 msg="model weights" buffer=CPU size="3.6 GiB" time=2025-07-23T23:34:53.908+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=tokenizer.ggml.eot_token_id default=106 time=2025-07-23T23:34:53.908+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}" time=2025-07-23T23:34:53.909+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.attention.layer_norm_rms_epsilon default=9.999999974752427e-07 time=2025-07-23T23:34:53.909+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.local.freq_base default=10000 time=2025-07-23T23:34:53.909+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.global.freq_base default=1e+06 time=2025-07-23T23:34:53.909+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.rope.freq_scale default=1 time=2025-07-23T23:34:53.909+08:00 level=DEBUG source=ggml.go:206 msg="key with type not found" key=gemma3.mm_tokens_per_image default=256 time=2025-07-23T23:34:54.007+08:00 level=DEBUG source=ggml.go:648 msg="compute graph" nodes=972 splits=1 time=2025-07-23T23:34:54.007+08:00 level=INFO source=ggml.go:666 msg="compute graph" backend=CUDA0 buffer_type=CUDA0 size="0 B" time=2025-07-23T23:34:54.008+08:00 level=INFO source=ggml.go:666 msg="compute graph" backend=CPU buffer_type=CPU size="1.1 GiB" ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 4194304000 panic: insufficient memory - required allocations: {InputWeights:550502400A CPU:{Name:CPU UUID: Weights:[60561408A 60561408A 60561408A 60561408A 53803008A 53803008A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 53127168A 60561408A 53127168A 59885568A 60561408A 59885568A 59885568A 60561408A 1390946752A] Cache:[2099249152A 2099249152A 2099249152A 2099249152A 2099249152A 8388608000A 2099249152A 2099249152A 2099249152A 2099249152A 2099249152A 8388608000A 2099249152A 2099249152A 2099249152A 2099249152A 2099249152A 8388608000A 2099249152A 2099249152A 2099249152A 2099249152A 2099249152A 4194304000F 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U] Graph:1174011904A} GPUs:[{Name:CUDA0 UUID:GPU-cfb69b01-0098-7635-99b9-9e2c46877913 Weights:[0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U] Cache:[0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U 0U] Graph:0A}]} goroutine 51 [running]: github.com/ollama/ollama/ml/backend/ggml.(*Context).newTensor(0xc000416a00, 0x1?, {0xc001743248, 0x3, 0x7ff7c397b8c5?}) C:/a/ollama/ollama/ml/backend/ggml/ggml.go:742 +0x696 github.com/ollama/ollama/ml/backend/ggml.(*Context).Zeros(0x7ff7c4a8f780?, 0xc0017fd410?, {0xc001743248?, 0xc000606a80?, 0x300000002?}) C:/a/ollama/ollama/ml/backend/ggml/ggml.go:755 +0x1c github.com/ollama/ollama/kvcache.(*Causal).Put(0xc0017d4d20, {0x7ff7c4d9bc70, 0xc0000bc100}, {0x7ff7c4da8dc0, 0xc001d07020}, {0x7ff7c4da8dc0, 0xc001d07050}) C:/a/ollama/ollama/kvcache/causal.go:559 +0x26a github.com/ollama/ollama/kvcache.(*WrapperCache).Put(0x298e326dca0?, {0x7ff7c4d9bc70?, 0xc0000bc100?}, {0x7ff7c4da8dc0?, 0xc001d07020?}, {0x7ff7c4da8dc0?, 0xc001d07050?}) C:/a/ollama/ollama/kvcache/wrapper.go:81 +0x4f github.com/ollama/ollama/ml/nn.Attention({0x7ff7c4d9bc70, 0xc0000bc100}, {0x7ff7c4da8dc0, 0xc001d06f90}, {0x7ff7c4da8dc0, 0xc001d07020}, {0x7ff7c4da8dc0, 0xc001d07050}, 0x3ff0000000000000, {0x7ff7c4d9ac38, ...}) C:/a/ollama/ollama/ml/nn/attention.go:39 +0x1c3 github.com/ollama/ollama/model/models/gemma3.(*TextSelfAttention).Forward(0xc001d724b0, {0x7ff7c4d9bc70, 0xc0000bc100}, 0x17, {0x7ff7c4da8dc0, 0xc001d06ed0}, {0x7ff7c4da8dc0, 0xc001cf64e0}, {0x7ff7c4d9ac38, 0xc000273520}, ...) C:/a/ollama/ollama/model/models/gemma3/model_text.go:104 +0x53e github.com/ollama/ollama/model/models/gemma3.(*TextLayer).Forward(0xc000047890, {0x7ff7c4d9bc70, 0xc0000bc100}, 0x17, {0x7ff7c4da8dc0, 0xc001d06eb8}, {0x7ff7c4da8dc0, 0xc001cf64e0}, {0x0, 0x0}, ...) C:/a/ollama/ollama/model/models/gemma3/model_text.go:143 +0xda github.com/ollama/ollama/model/models/gemma3.(*TextModel).Forward(0xc001cd6080, {0x7ff7c4d9bc70, 0xc0000bc100}, {0x7ff7c4da8dc0?, 0xc001cf6480?}, {0x7ff7c4da8dc0, 0xc001cf64e0}, {0x7ff7c4da8dc0, 0xc001cf64f8}, {{0x7ff7c4da8dc0, ...}, ...}, ...) C:/a/ollama/ollama/model/models/gemma3/model_text.go:197 +0x3be github.com/ollama/ollama/model/models/gemma3.(*Model).Forward(0xc001ca6c60, {0x7ff7c4d9bc70, 0xc0000bc100}, {{0x7ff7c4da8dc0, 0xc001cf6480}, {0xc001d8a0c0, 0x1, 0x1}, {0xc00e896000, 0x200, ...}, ...}) C:/a/ollama/ollama/model/models/gemma3/model.go:147 +0x1d7 github.com/ollama/ollama/runner/ollamarunner.(*Server).reserveWorstCaseGraph(0xc00016f9e0) C:/a/ollama/ollama/runner/ollamarunner/runner.go:821 +0xac5 github.com/ollama/ollama/runner/ollamarunner.(*Server).initModel(0xc00016f9e0, {0xc00003a150?, 0x0?}, {0x10, 0x0, 0x0, {0x0, 0x0, 0x0}, 0x0}, ...) C:/a/ollama/ollama/runner/ollamarunner/runner.go:865 +0x270 github.com/ollama/ollama/runner/ollamarunner.(*Server).load(0xc00016f9e0, {0x7ff7c4d93510, 0xc000158eb0}, {0xc00003a150?, 0x0?}, {0x10, 0x0, 0x0, {0x0, 0x0, ...}, ...}, ...) C:/a/ollama/ollama/runner/ollamarunner/runner.go:878 +0xb8 created by github.com/ollama/ollama/runner/ollamarunner.Execute in goroutine 1 C:/a/ollama/ollama/runner/ollamarunner/runner.go:959 +0xa11 time=2025-07-23T23:34:59.290+08:00 level=INFO source=server.go:632 msg="waiting for server to become available" status="llm server error" time=2025-07-23T23:35:00.793+08:00 level=ERROR source=sched.go:489 msg="error loading llama server" error="llama runner process has terminated: exit status 2" time=2025-07-23T23:35:00.793+08:00 level=DEBUG source=sched.go:491 msg="triggering expiration for failed load" runner.name=registry.ollama.ai/library/gemma3:latest runner.inference=cuda runner.devices=1 runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 runner.num_ctx=2048000 time=2025-07-23T23:35:00.793+08:00 level=DEBUG source=sched.go:364 msg="runner expired event received" runner.name=registry.ollama.ai/library/gemma3:latest runner.inference=cuda runner.devices=1 runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 runner.num_ctx=2048000 time=2025-07-23T23:35:00.793+08:00 level=DEBUG source=sched.go:379 msg="got lock to unload expired event" runner.name=registry.ollama.ai/library/gemma3:latest runner.inference=cuda runner.devices=1 runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 runner.num_ctx=2048000 time=2025-07-23T23:35:00.793+08:00 level=DEBUG source=sched.go:402 msg="starting background wait for VRAM recovery" runner.name=registry.ollama.ai/library/gemma3:latest runner.inference=cuda runner.devices=1 runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 runner.num_ctx=2048000 [GIN] 2025/07/23 - 23:35:00 | 500 | 14.4415483s | 127.0.0.1 | POST "/api/generate" time=2025-07-23T23:35:00.793+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.0 GiB" before.free_swap="78.9 GiB" now.total="95.6 GiB" now.free="83.4 GiB" now.free_swap="79.2 GiB" time=2025-07-23T23:35:00.801+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB" releasing nvml library time=2025-07-23T23:35:00.802+08:00 level=DEBUG source=server.go:1031 msg="stopping llama server" pid=31340 time=2025-07-23T23:35:00.802+08:00 level=DEBUG source=sched.go:407 msg="runner terminated and removed from list, blocking for VRAM recovery" runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 time=2025-07-23T23:35:01.052+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.4 GiB" before.free_swap="79.2 GiB" now.total="95.6 GiB" now.free="83.4 GiB" now.free_swap="79.2 GiB" time=2025-07-23T23:35:01.060+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB" releasing nvml library time=2025-07-23T23:35:01.302+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.4 GiB" before.free_swap="79.2 GiB" now.total="95.6 GiB" now.free="83.4 GiB" now.free_swap="79.2 GiB" time=2025-07-23T23:35:01.309+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB" releasing nvml library time=2025-07-23T23:35:01.552+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.4 GiB" before.free_swap="79.2 GiB" now.total="95.6 GiB" now.free="83.4 GiB" now.free_swap="79.2 GiB" time=2025-07-23T23:35:01.559+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB" releasing nvml library time=2025-07-23T23:35:01.802+08:00 level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="95.6 GiB" before.free="83.4 GiB" before.free_swap="79.2 GiB" now.total="95.6 GiB" now.free="83.4 GiB" now.free_swap="79.2 GiB" time=2025-07-23T23:35:01.809+08:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-cfb69b01-0098-7635-99b9-9e2c46877913 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.8 GiB" before.free="29.9 GiB" now.total="31.8 GiB" now.free="29.9 GiB" now.used="2.0 GiB" releasing nvml library time=2025-07-23T23:35:01.809+08:00 level=DEBUG source=sched.go:700 msg="gpu VRAM free memory converged after 1.02 seconds" runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 time=2025-07-23T23:35:01.809+08:00 level=DEBUG source=sched.go:410 msg="sending an unloaded event" runner.size="99.9 GiB" runner.vram="0 B" runner.parallel=500 runner.pid=31340 runner.model=C:\Users\Sonic_Dong\.ollama\models\blobs\sha256-aeda25e63ebd698fab8638ffb778e68bed908b960d39d0becc650fa981609d25 time=2025-07-23T23:35:01.809+08:00 level=DEBUG source=sched.go:312 msg="ignoring unload event with no pending requests" time=2025-07-23T23:36:00.231+08:00 level=DEBUG source=sched.go:322 msg="shutting down scheduler completed loop" ``` ### OS Windows ### GPU Nvidia ### CPU AMD ### Ollama version 0.9.6
GiteaMirror added the bug label 2026-04-12 19:41:10 -05:00
GiteaMirror commented 2026-04-12 19:41:11 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (Jul 23, 2025):

OLLAMA_NUM_PARALLEL is set too high. Ollama should handle this more gracefully without crashing but this setting requires nearly 100G of VRAM and you have 32G. Leave it at the default (1) or make small increments above that.

<!-- gh-comment-id:3109534774 --> @jessegross commented on GitHub (Jul 23, 2025): OLLAMA_NUM_PARALLEL is set too high. Ollama should handle this more gracefully without crashing but this setting requires nearly 100G of VRAM and you have 32G. Leave it at the default (1) or make small increments above that.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#7600
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?