github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 21:55:43 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #14548] Qwen3-VL slow on RTX 5090 #87127

New Issue
Open
opened 2026-05-10 04:57:17 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-05-10 04:57:17 -05:00
Owner
Copy Link

Originally created by @Prads- on GitHub (Mar 2, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/14548

What is the issue?

Took 32.83 seconds to process a single 512x375 image on RTX 5090 with 32GB VRAM. If I'm not mistaken, it shouldn't be this slow on this hardware? I think for some reason the vision part is being offloaded to CPU instead of using GPU but I'm not fully sure. I think it's related to this issue: https://github.com/ggml-org/llama.cpp/issues/16895 and flash attention isn't being enabled for my GPU for some reason. I tried using $env:OLLAMA_FLASH_ATTENTION="1" but that didn't change things either.

Relevant log output

time=2026-03-02T21:39:01.825+11:00 level=INFO source=routes.go:1663 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:0 OLLAMA_DEBUG:INFO OLLAMA_EDITOR: OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:2 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\User\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NO_CLOUD:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:false ROCR_VISIBLE_DEVICES:]"
time=2026-03-02T21:39:01.825+11:00 level=INFO source=routes.go:1665 msg="Ollama cloud disabled: false"
time=2026-03-02T21:39:01.828+11:00 level=INFO source=images.go:473 msg="total blobs: 33"
time=2026-03-02T21:39:01.829+11:00 level=INFO source=images.go:480 msg="total unused blobs removed: 0"
time=2026-03-02T21:39:01.830+11:00 level=INFO source=routes.go:1718 msg="Listening on 127.0.0.1:11434 (version 0.17.4)"
time=2026-03-02T21:39:01.830+11:00 level=INFO source=runner.go:67 msg="discovering available GPUs..."
time=2026-03-02T21:39:01.840+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59404"
time=2026-03-02T21:39:01.976+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59408"
time=2026-03-02T21:39:02.100+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59412"
time=2026-03-02T21:39:02.329+11:00 level=INFO source=runner.go:106 msg="experimental Vulkan support disabled.  To enable, set OLLAMA_VULKAN=1"
time=2026-03-02T21:39:02.330+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59416"
time=2026-03-02T21:39:02.330+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59418"
time=2026-03-02T21:39:02.330+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59417"
time=2026-03-02T21:39:02.466+11:00 level=INFO source=types.go:42 msg="inference compute" id=GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 filter_id="" library=CUDA compute=12.0 name=CUDA0 description="NVIDIA GeForce RTX 5090" libdirs=ollama,cuda_v13 driver=13.0 pci_id=0000:01:00.0 type=discrete total="31.8 GiB" available="30.0 GiB"
time=2026-03-02T21:39:02.466+11:00 level=INFO source=routes.go:1768 msg="vram-based default context" total_vram="31.8 GiB" default_num_ctx=32768
time=2026-03-02T21:42:33.440+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 55770"
time=2026-03-02T21:42:33.569+11:00 level=INFO source=cpu_windows.go:148 msg=packages count=1
time=2026-03-02T21:42:33.569+11:00 level=INFO source=cpu_windows.go:195 msg="" package=0 cores=8 efficiency=0 threads=16
time=2026-03-02T21:42:33.609+11:00 level=INFO source=server.go:247 msg="enabling flash attention"
time=2026-03-02T21:42:33.610+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\User\\.ollama\\models\\blobs\\sha256-ed12a4674d727a74ac4816c906094ea9d3119fbea46ca93288c3ce4ffbe38c55 --port 55774"
time=2026-03-02T21:42:33.612+11:00 level=INFO source=sched.go:491 msg="system memory" total="61.6 GiB" free="46.1 GiB" free_swap="47.2 GiB"
time=2026-03-02T21:42:33.612+11:00 level=INFO source=sched.go:498 msg="gpu memory" id=GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 library=CUDA available="29.4 GiB" free="29.8 GiB" minimum="457.0 MiB" overhead="0 B"
time=2026-03-02T21:42:33.612+11:00 level=INFO source=server.go:757 msg="loading model" "model layers"=37 requested=99
time=2026-03-02T21:42:33.635+11:00 level=INFO source=runner.go:1411 msg="starting ollama engine"
time=2026-03-02T21:42:33.636+11:00 level=INFO source=runner.go:1446 msg="Server listening on 127.0.0.1:55774"
time=2026-03-02T21:42:33.645+11:00 level=INFO source=runner.go:1284 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:Enabled KvSize:8192 KvCacheType: NumThreads:8 GPULayers:37[ID:GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 Layers:37(0..36)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2026-03-02T21:42:33.661+11:00 level=INFO source=ggml.go:136 msg="" architecture=qwen3vl file_type=Q4_K_M name="" description="" num_tensors=858 num_key_values=40
load_backend: loaded CPU backend from C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-icelake.dll
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, ID: GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911
load_backend: loaded CUDA backend from C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama\cuda_v13\ggml-cuda.dll
time=2026-03-02T21:42:33.727+11:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=750,800,860,870,890,900,1000,1030,1100,1200,1210 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang)
time=2026-03-02T21:42:34.106+11:00 level=INFO source=runner.go:1284 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:Enabled KvSize:8192 KvCacheType: NumThreads:8 GPULayers:37[ID:GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 Layers:37(0..36)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=runner.go:1284 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:Enabled KvSize:8192 KvCacheType: NumThreads:8 GPULayers:37[ID:GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 Layers:37(0..36)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:482 msg="offloading 36 repeating layers to GPU"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:240 msg="model weights" device=CUDA0 size="5.4 GiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:489 msg="offloading output layer to GPU"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:494 msg="offloaded 37/37 layers to GPU"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:245 msg="model weights" device=CPU size="333.8 MiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:251 msg="kv cache" device=CUDA0 size="1.1 GiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:262 msg="compute graph" device=CUDA0 size="616.5 MiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:267 msg="compute graph" device=CPU size="63.3 MiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:272 msg="total memory" size="7.5 GiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=sched.go:566 msg="loaded runners" count=1
time=2026-03-02T21:42:34.328+11:00 level=INFO source=server.go:1350 msg="waiting for llama runner to start responding"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=server.go:1384 msg="waiting for server to become available" status="llm server loading model"
time=2026-03-02T21:42:35.080+11:00 level=INFO source=server.go:1388 msg="llama runner started in 1.47 seconds"
[GIN] 2026/03/02 - 21:43:06 | 200 |   32.8356467s |       127.0.0.1 | POST     "/api/generate"

OS

Windows

GPU

Nvidia

CPU

AMD

Ollama version

0.17.4

Originally created by @Prads- on GitHub (Mar 2, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/14548 ### What is the issue? Took 32.83 seconds to process a single 512x375 image on RTX 5090 with 32GB VRAM. If I'm not mistaken, it shouldn't be this slow on this hardware? I think for some reason the vision part is being offloaded to CPU instead of using GPU but I'm not fully sure. I think it's related to this issue: https://github.com/ggml-org/llama.cpp/issues/16895 and flash attention isn't being enabled for my GPU for some reason. I tried using `$env:OLLAMA_FLASH_ATTENTION="1"` but that didn't change things either. ### Relevant log output ```shell time=2026-03-02T21:39:01.825+11:00 level=INFO source=routes.go:1663 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:0 OLLAMA_DEBUG:INFO OLLAMA_EDITOR: OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:2 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\User\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NO_CLOUD:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:false ROCR_VISIBLE_DEVICES:]" time=2026-03-02T21:39:01.825+11:00 level=INFO source=routes.go:1665 msg="Ollama cloud disabled: false" time=2026-03-02T21:39:01.828+11:00 level=INFO source=images.go:473 msg="total blobs: 33" time=2026-03-02T21:39:01.829+11:00 level=INFO source=images.go:480 msg="total unused blobs removed: 0" time=2026-03-02T21:39:01.830+11:00 level=INFO source=routes.go:1718 msg="Listening on 127.0.0.1:11434 (version 0.17.4)" time=2026-03-02T21:39:01.830+11:00 level=INFO source=runner.go:67 msg="discovering available GPUs..." time=2026-03-02T21:39:01.840+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59404" time=2026-03-02T21:39:01.976+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59408" time=2026-03-02T21:39:02.100+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59412" time=2026-03-02T21:39:02.329+11:00 level=INFO source=runner.go:106 msg="experimental Vulkan support disabled. To enable, set OLLAMA_VULKAN=1" time=2026-03-02T21:39:02.330+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59416" time=2026-03-02T21:39:02.330+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59418" time=2026-03-02T21:39:02.330+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 59417" time=2026-03-02T21:39:02.466+11:00 level=INFO source=types.go:42 msg="inference compute" id=GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 filter_id="" library=CUDA compute=12.0 name=CUDA0 description="NVIDIA GeForce RTX 5090" libdirs=ollama,cuda_v13 driver=13.0 pci_id=0000:01:00.0 type=discrete total="31.8 GiB" available="30.0 GiB" time=2026-03-02T21:39:02.466+11:00 level=INFO source=routes.go:1768 msg="vram-based default context" total_vram="31.8 GiB" default_num_ctx=32768 time=2026-03-02T21:42:33.440+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --port 55770" time=2026-03-02T21:42:33.569+11:00 level=INFO source=cpu_windows.go:148 msg=packages count=1 time=2026-03-02T21:42:33.569+11:00 level=INFO source=cpu_windows.go:195 msg="" package=0 cores=8 efficiency=0 threads=16 time=2026-03-02T21:42:33.609+11:00 level=INFO source=server.go:247 msg="enabling flash attention" time=2026-03-02T21:42:33.610+11:00 level=INFO source=server.go:431 msg="starting runner" cmd="C:\\Users\\User\\AppData\\Local\\Programs\\Ollama\\ollama.exe runner --ollama-engine --model C:\\Users\\User\\.ollama\\models\\blobs\\sha256-ed12a4674d727a74ac4816c906094ea9d3119fbea46ca93288c3ce4ffbe38c55 --port 55774" time=2026-03-02T21:42:33.612+11:00 level=INFO source=sched.go:491 msg="system memory" total="61.6 GiB" free="46.1 GiB" free_swap="47.2 GiB" time=2026-03-02T21:42:33.612+11:00 level=INFO source=sched.go:498 msg="gpu memory" id=GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 library=CUDA available="29.4 GiB" free="29.8 GiB" minimum="457.0 MiB" overhead="0 B" time=2026-03-02T21:42:33.612+11:00 level=INFO source=server.go:757 msg="loading model" "model layers"=37 requested=99 time=2026-03-02T21:42:33.635+11:00 level=INFO source=runner.go:1411 msg="starting ollama engine" time=2026-03-02T21:42:33.636+11:00 level=INFO source=runner.go:1446 msg="Server listening on 127.0.0.1:55774" time=2026-03-02T21:42:33.645+11:00 level=INFO source=runner.go:1284 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:Enabled KvSize:8192 KvCacheType: NumThreads:8 GPULayers:37[ID:GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 Layers:37(0..36)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2026-03-02T21:42:33.661+11:00 level=INFO source=ggml.go:136 msg="" architecture=qwen3vl file_type=Q4_K_M name="" description="" num_tensors=858 num_key_values=40 load_backend: loaded CPU backend from C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama\ggml-cpu-icelake.dll ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, ID: GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 load_backend: loaded CUDA backend from C:\Users\User\AppData\Local\Programs\Ollama\lib\ollama\cuda_v13\ggml-cuda.dll time=2026-03-02T21:42:33.727+11:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=750,800,860,870,890,900,1000,1030,1100,1200,1210 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(clang) time=2026-03-02T21:42:34.106+11:00 level=INFO source=runner.go:1284 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:Enabled KvSize:8192 KvCacheType: NumThreads:8 GPULayers:37[ID:GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 Layers:37(0..36)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2026-03-02T21:42:34.328+11:00 level=INFO source=runner.go:1284 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:512 FlashAttention:Enabled KvSize:8192 KvCacheType: NumThreads:8 GPULayers:37[ID:GPU-96c4690a-d4d2-3e1b-32c4-9a20ae103911 Layers:37(0..36)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:482 msg="offloading 36 repeating layers to GPU" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:240 msg="model weights" device=CUDA0 size="5.4 GiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:489 msg="offloading output layer to GPU" time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:494 msg="offloaded 37/37 layers to GPU" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:245 msg="model weights" device=CPU size="333.8 MiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:251 msg="kv cache" device=CUDA0 size="1.1 GiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:262 msg="compute graph" device=CUDA0 size="616.5 MiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:267 msg="compute graph" device=CPU size="63.3 MiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:272 msg="total memory" size="7.5 GiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=sched.go:566 msg="loaded runners" count=1 time=2026-03-02T21:42:34.328+11:00 level=INFO source=server.go:1350 msg="waiting for llama runner to start responding" time=2026-03-02T21:42:34.328+11:00 level=INFO source=server.go:1384 msg="waiting for server to become available" status="llm server loading model" time=2026-03-02T21:42:35.080+11:00 level=INFO source=server.go:1388 msg="llama runner started in 1.47 seconds" [GIN] 2026/03/02 - 21:43:06 | 200 | 32.8356467s | 127.0.0.1 | POST "/api/generate" ``` ### OS Windows ### GPU Nvidia ### CPU AMD ### Ollama version 0.17.4
GiteaMirror added the bug label 2026-05-10 04:57:17 -05:00
GiteaMirror commented 2026-05-10 04:57:18 -05:00
Author
Owner
Copy Link

@Da-coding-pro commented on GitHub (Mar 2, 2026):

Please specify which qwen3-vl model you used and it could help to know if the Qwen model loads fully into your gpu. Ollama may sometimes offload the model onto CPU even if you have a capable gpu

<!-- gh-comment-id:3985358572 --> @Da-coding-pro commented on GitHub (Mar 2, 2026): Please specify which qwen3-vl model you used and it could help to know if the Qwen model loads fully into your gpu. Ollama may sometimes offload the model onto CPU even if you have a capable gpu
GiteaMirror commented 2026-05-10 04:57:19 -05:00
Author
Owner
Copy Link

@Prads- commented on GitHub (Mar 2, 2026):

I am using qwen3-vl:8b which should be able to fully load in my 32gb VRAM. I did check the GPU memory usage when the model was running and the usage was at 9gb with 23gb headroom.
From the logs, the interesting part are these lines:

time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:482 msg="offloading 36 repeating layers to GPU"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:240 msg="model weights" device=CUDA0 size="5.4 GiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:489 msg="offloading output layer to GPU"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:494 msg="offloaded 37/37 layers to GPU"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:245 msg="model weights" device=CPU size="333.8 MiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:251 msg="kv cache" device=CUDA0 size="1.1 GiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:262 msg="compute graph" device=CUDA0 size="616.5 MiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:267 msg="compute graph" device=CPU size="63.3 MiB"
time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:272 msg="total memory" size="7.5 GiB"

All the layers were offloaded to GPU but then some of the model weights around 333.8MB were stored in RAM. Not sure why that is the case when there is 23GB of VRAM that is still available.

Let me know if you need more log data, happy to provide them.

<!-- gh-comment-id:3987489786 --> @Prads- commented on GitHub (Mar 2, 2026): I am using `qwen3-vl:8b` which should be able to fully load in my 32gb VRAM. I did check the GPU memory usage when the model was running and the usage was at 9gb with 23gb headroom. From the logs, the interesting part are these lines: ``` time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:482 msg="offloading 36 repeating layers to GPU" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:240 msg="model weights" device=CUDA0 size="5.4 GiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:489 msg="offloading output layer to GPU" time=2026-03-02T21:42:34.328+11:00 level=INFO source=ggml.go:494 msg="offloaded 37/37 layers to GPU" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:245 msg="model weights" device=CPU size="333.8 MiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:251 msg="kv cache" device=CUDA0 size="1.1 GiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:262 msg="compute graph" device=CUDA0 size="616.5 MiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:267 msg="compute graph" device=CPU size="63.3 MiB" time=2026-03-02T21:42:34.328+11:00 level=INFO source=device.go:272 msg="total memory" size="7.5 GiB" ``` All the layers were offloaded to GPU but then some of the model weights around 333.8MB were stored in RAM. Not sure why that is the case when there is 23GB of VRAM that is still available. Let me know if you need more log data, happy to provide them.
GiteaMirror commented 2026-05-10 04:57:19 -05:00
Author
Owner
Copy Link

@g530040217 commented on GitHub (Mar 3, 2026):

same question.

Mar 03 02:00:23 ollama01 ollama[226033]: print_info: arch = qwen2
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: vocab_only = 1
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: no_alloc = 0
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: model type = ?B
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: model params = 32.76 B
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: general.name = DeepSeek R1 Distill Qwen 32B
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: vocab type = BPE
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: n_vocab = 152064
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: n_merges = 151387
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: BOS token = 151646 '<｜begin▁of▁sentence｜>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOS token = 151643 '<｜end▁of▁sentence｜>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOT token = 151643 '<｜end▁of▁sentence｜>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: PAD token = 151643 '<｜end▁of▁sentence｜>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: LF token = 198 'Ċ'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM PRE token = 151659 '<|fim_prefix|>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM SUF token = 151661 '<|fim_suffix|>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM MID token = 151660 '<|fim_middle|>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM PAD token = 151662 '<|fim_pad|>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM REP token = 151663 '<|repo_name|>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM SEP token = 151664 '<|file_sep|>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOG token = 151643 '<｜end▁of▁sentence｜>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOG token = 151662 '<|fim_pad|>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOG token = 151663 '<|repo_name|>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOG token = 151664 '<|file_sep|>'
Mar 03 02:00:23 ollama01 ollama[226033]: print_info: max token length = 256
Mar 03 02:00:23 ollama01 ollama[226033]: llama_model_load: vocab only - skipping tensors
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.166Z level=WARN source=server.go:169 msg="requested context size too large for model" num_ctx=262144 n_ctx_train=131072
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.167Z level=INFO source=server.go:431 msg="starting runner" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-6150cb382311b69f09cc0f9a1b69fc029cbd742b66bb8ec531aa5ecf5c613e93 --port 43513"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.167Z level=INFO source=sched.go:491 msg="system memory" total="62.8 GiB" free="61.2 GiB" free_swap="4.0 GiB"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.167Z level=INFO source=sched.go:498 msg="gpu memory" id=GPU-a18a2cd0-dee4-0f1a-449f-281310e4f7fb library=CUDA available="23.2 GiB" free="23.6 GiB" minimum="457.0 MiB" overhead="0 B"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.167Z level=INFO source=sched.go:498 msg="gpu memory" id=GPU-de485134-b66c-8080-ca42-c187b968b29f library=CUDA available="22.8 GiB" free="23.3 GiB" minimum="457.0 MiB" overhead="0 B"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.167Z level=INFO source=server.go:498 msg="loading model" "model layers"=65 requested=-1
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:240 msg="model weights" device=CUDA0 size="3.2 GiB"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:240 msg="model weights" device=CUDA1 size="3.1 GiB"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:245 msg="model weights" device=CPU size="11.7 GiB"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:251 msg="kv cache" device=CUDA0 size="6.0 GiB"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:251 msg="kv cache" device=CUDA1 size="5.5 GiB"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:256 msg="kv cache" device=CPU size="20.5 GiB"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:262 msg="compute graph" device=CUDA0 size="12.8 GiB"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:262 msg="compute graph" device=CUDA1 size="12.8 GiB"
Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:272 msg="total memory" size="75.6 GiB"

<!-- gh-comment-id:3988222358 --> @g530040217 commented on GitHub (Mar 3, 2026): same question. Mar 03 02:00:23 ollama01 ollama[226033]: print_info: arch = qwen2 Mar 03 02:00:23 ollama01 ollama[226033]: print_info: vocab_only = 1 Mar 03 02:00:23 ollama01 ollama[226033]: print_info: no_alloc = 0 Mar 03 02:00:23 ollama01 ollama[226033]: print_info: model type = ?B Mar 03 02:00:23 ollama01 ollama[226033]: print_info: model params = 32.76 B Mar 03 02:00:23 ollama01 ollama[226033]: print_info: general.name = DeepSeek R1 Distill Qwen 32B Mar 03 02:00:23 ollama01 ollama[226033]: print_info: vocab type = BPE Mar 03 02:00:23 ollama01 ollama[226033]: print_info: n_vocab = 152064 Mar 03 02:00:23 ollama01 ollama[226033]: print_info: n_merges = 151387 Mar 03 02:00:23 ollama01 ollama[226033]: print_info: BOS token = 151646 '<｜begin▁of▁sentence｜>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOS token = 151643 '<｜end▁of▁sentence｜>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOT token = 151643 '<｜end▁of▁sentence｜>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: PAD token = 151643 '<｜end▁of▁sentence｜>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: LF token = 198 'Ċ' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM PRE token = 151659 '<|fim_prefix|>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM SUF token = 151661 '<|fim_suffix|>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM MID token = 151660 '<|fim_middle|>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM PAD token = 151662 '<|fim_pad|>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM REP token = 151663 '<|repo_name|>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: FIM SEP token = 151664 '<|file_sep|>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOG token = 151643 '<｜end▁of▁sentence｜>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOG token = 151662 '<|fim_pad|>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOG token = 151663 '<|repo_name|>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: EOG token = 151664 '<|file_sep|>' Mar 03 02:00:23 ollama01 ollama[226033]: print_info: max token length = 256 Mar 03 02:00:23 ollama01 ollama[226033]: llama_model_load: vocab only - skipping tensors Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.166Z level=WARN source=server.go:169 msg="requested context size too large for model" num_ctx=262144 n_ctx_train=131072 Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.167Z level=INFO source=server.go:431 msg="starting runner" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-6150cb382311b69f09cc0f9a1b69fc029cbd742b66bb8ec531aa5ecf5c613e93 --port 43513" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.167Z level=INFO source=sched.go:491 msg="system memory" total="62.8 GiB" free="61.2 GiB" free_swap="4.0 GiB" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.167Z level=INFO source=sched.go:498 msg="gpu memory" id=GPU-a18a2cd0-dee4-0f1a-449f-281310e4f7fb library=CUDA available="23.2 GiB" free="23.6 GiB" minimum="457.0 MiB" overhead="0 B" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.167Z level=INFO source=sched.go:498 msg="gpu memory" id=GPU-de485134-b66c-8080-ca42-c187b968b29f library=CUDA available="22.8 GiB" free="23.3 GiB" minimum="457.0 MiB" overhead="0 B" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.167Z level=INFO source=server.go:498 msg="loading model" "model layers"=65 requested=-1 Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:240 msg="model weights" device=CUDA0 size="3.2 GiB" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:240 msg="model weights" device=CUDA1 size="3.1 GiB" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:245 msg="model weights" device=CPU size="11.7 GiB" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:251 msg="kv cache" device=CUDA0 size="6.0 GiB" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:251 msg="kv cache" device=CUDA1 size="5.5 GiB" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:256 msg="kv cache" device=CPU size="20.5 GiB" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:262 msg="compute graph" device=CUDA0 size="12.8 GiB" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:262 msg="compute graph" device=CUDA1 size="12.8 GiB" Mar 03 02:00:23 ollama01 ollama[226033]: time=2026-03-03T02:00:23.168Z level=INFO source=device.go:272 msg="total memory" size="75.6 GiB"
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#87127
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?