github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-23 06:01:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #13232] CPU/GPU usage remains high after response is completed #34508

New Issue
Closed
opened 2026-04-22 18:08:03 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-22 18:08:03 -05:00
Owner
Copy Link

Originally created by @moontato on GitHub (Nov 24, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/13232

What is the issue?

I am running v0.12.11 using a Radeon 680M iGPU on Ubuntu server 24.
Certain models, particularly the variants of qwen3-vl, seem to "hang" even after it finishes responding. GPU usage (and CPU, if partial offloading) remains ~100% until I manually restart the service.

gpt-oss, the model I've been using for months prior, has maybe had this issue once or twice? Either way, I could count it on my ten fingers. But with qwen3-vl (8b-thinking, 4b-thinking, both q4km), it seems to happen almost every time, and it is guaranteed to happen for me using this prompt:

**Q4:**  
Consider the recurrence:
T(n)=3T(n/2)+nT(n) = 3T(n/2) + nT(n)=3T(n/2)+n
Using the Master Theorem, what is the tight asymptotic bound?

When the message should be completed, the open-webui interface shows that the message is complete and I am able to send a new message, but obviously it never gets to it because it's still stuck on the previous request.

Here's how I'm running the ollama service:

[Service]
Environment="OLLAMA_HOST=http://myserverdomain:11434"
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"
Environment="OLLAMA_NUM_THREADS=8"
Environment="OPENBLAS_NUM_THREADS=8"
Environment="OMP_NUM_THREADS=8"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_NEW_ENGINE=1"

Environment="OLLAMA_DEBUG=1"

Note: I have also experimented with setting OLLAMA_VULKAN and removing Environment="OLLAMA_NEW_ENGINE=1" from the service file, but to no avail.

Also, I have tried updating to 0.13 to see if that'd fix the issue, but unfortunately there's a unified memory detection issue on that version that prevents qwen3-vl from even offloading most of its layers onto the GPU.

Edit: consistency fixes

Relevant log output

Nov 24 21:42:07 myserver systemd[1]: Started ollama.service - Ollama Service.
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.448Z level=INFO source=routes.go:1544 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION:10.3.0 HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://myserverdomain:11434 OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:true OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.450Z level=INFO source=images.go:522 msg="total blobs: 22"
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=INFO source=images.go:529 msg="total unused blobs removed: 0"
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=INFO source=routes.go:1597 msg="Listening on 100.100.10.2:11434 (version 0.12.11)"
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=DEBUG source=sched.go:120 msg="starting llm scheduler"
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=INFO source=runner.go:67 msg="discovering available GPUs..."
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 46727"
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v12 OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v12
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.468Z level=DEBUG source=runner.go:418 msg="bootstrap discovery took" duration=16.962393ms OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/cuda_v12]" extra_envs=map[]
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.468Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 46689"
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.468Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v13 OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v13
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.485Z level=DEBUG source=runner.go:418 msg="bootstrap discovery took" duration=16.691995ms OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/cuda_v13]" extra_envs=map[]
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.485Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 36975"
Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.485Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm
Nov 24 21:42:08 myserver ollama[162316]: time=2025-11-24T21:42:08.888Z level=DEBUG source=runner.go:418 msg="bootstrap discovery took" duration=1.403423063s OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/rocm]" extra_envs=map[]
Nov 24 21:42:08 myserver ollama[162316]: time=2025-11-24T21:42:08.888Z level=DEBUG source=runner.go:116 msg="evluating which if any devices to filter out" initial_count=1
Nov 24 21:42:08 myserver ollama[162316]: time=2025-11-24T21:42:08.888Z level=DEBUG source=runner.go:128 msg="verifying device is supported" library=/usr/local/lib/ollama/rocm description="AMD Radeon Graphics" compute=gfx1030 id=0 pci_id=0000:e9:00.0
Nov 24 21:42:08 myserver ollama[162316]: time=2025-11-24T21:42:08.889Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 39461"
Nov 24 21:42:08 myserver ollama[162316]: time=2025-11-24T21:42:08.889Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm ROCR_VISIBLE_DEVICES=0 GGML_CUDA_INIT=1
Nov 24 21:42:11 myserver ollama[162316]: time=2025-11-24T21:42:11.006Z level=DEBUG source=runner.go:418 msg="bootstrap discovery took" duration=2.118090349s OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/rocm]" extra_envs="map[GGML_CUDA_INIT:1 ROCR_VISIBLE_DEVICES:0]"
Nov 24 21:42:11 myserver ollama[162316]: time=2025-11-24T21:42:11.007Z level=DEBUG source=runner.go:175 msg="adjusting filtering IDs" FilterID=0 new_ID=0
Nov 24 21:42:11 myserver ollama[162316]: time=2025-11-24T21:42:11.007Z level=DEBUG source=runner.go:40 msg="GPU bootstrap discovery took" duration=3.555695961s
Nov 24 21:42:11 myserver ollama[162316]: time=2025-11-24T21:42:11.007Z level=INFO source=types.go:42 msg="inference compute" id=0 filter_id=0 library=ROCm compute=gfx1030 name=ROCm0 description="AMD Radeon Graphics" libdirs=ollama,rocm driver=60342.13 pci_id=0000:e9:00.0 type=iGPU total="21.5 GiB" available="21.5 GiB"
Nov 24 21:42:13 myserver ollama[162316]: [GIN] 2025/11/24 - 21:42:13 | 200 |    1.140065ms |    100.100.10.2 | GET      "/api/tags"
Nov 24 21:42:13 myserver ollama[162316]: [GIN] 2025/11/24 - 21:42:13 | 200 |     100.934µs |    100.100.10.2 | GET      "/api/ps"
Nov 24 21:42:13 myserver ollama[162316]: time=2025-11-24T21:42:13.672Z level=DEBUG source=runner.go:246 msg="refreshing free memory"
Nov 24 21:42:13 myserver ollama[162316]: time=2025-11-24T21:42:13.672Z level=DEBUG source=runner.go:310 msg="unable to refresh all GPUs with existing runners, performing bootstrap discovery"
Nov 24 21:42:13 myserver ollama[162316]: time=2025-11-24T21:42:13.672Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 39341"
Nov 24 21:42:13 myserver ollama[162316]: time=2025-11-24T21:42:13.672Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm ROCR_VISIBLE_DEVICES=0
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.100Z level=DEBUG source=runner.go:418 msg="bootstrap discovery took" duration=1.42850933s OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/rocm]" extra_envs=map[ROCR_VISIBLE_DEVICES:0]
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.101Z level=DEBUG source=runner.go:40 msg="overall device VRAM discovery took" duration=1.428618629s
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.101Z level=DEBUG source=sched.go:194 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=3 gpu_count=1
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.117Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.118Z level=DEBUG source=sched.go:211 msg="loading first model" model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.172Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.pooling_type default=0
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_bos_token default=false
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.bos_token_id default=0
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.rope.scaling.factor default=1
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_count default=0
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_used_count default=0
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.norm_top_k_prob default=true
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}"
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.vision.num_positional_embeddings default=2304
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}"
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=INFO source=server.go:209 msg="enabling flash attention"
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 --port 35507"
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm ROCR_VISIBLE_DEVICES=0
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=INFO source=sched.go:443 msg="system memory" total="27.2 GiB" free="25.2 GiB" free_swap="13.6 GiB"
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=INFO source=sched.go:450 msg="gpu memory" id=0 library=ROCm available="21.0 GiB" free="21.5 GiB" minimum="457.0 MiB" overhead="0 B"
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=INFO source=server.go:702 msg="loading model" "model layers"=37 requested=36
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.185Z level=INFO source=runner.go:1398 msg="starting ollama engine"
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.185Z level=INFO source=runner.go:1433 msg="Server listening on 127.0.0.1:35507"
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.196Z level=INFO source=runner.go:1271 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:256 FlashAttention:true KvSize:80000 KvCacheType: NumThreads:8 GPULayers:36[ID:0 Layers:36(0..35)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.222Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.223Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.name default=""
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.223Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.description default=""
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.223Z level=INFO source=ggml.go:136 msg="" architecture=qwen3vl file_type=Q4_K_M name="" description="" num_tensors=809 num_key_values=40
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.223Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/local/lib/ollama
Nov 24 21:42:15 myserver ollama[162316]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so
Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.227Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/local/lib/ollama/rocm
Nov 24 21:42:15 myserver ollama[162316]: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
Nov 24 21:42:16 myserver ollama[162316]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Nov 24 21:42:16 myserver ollama[162316]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Nov 24 21:42:16 myserver ollama[162316]: ggml_cuda_init: found 1 ROCm devices:
Nov 24 21:42:16 myserver ollama[162316]:   Device 0: AMD Radeon Graphics, gfx1030 (0x1030), VMM: no, Wave Size: 32, ID: 0
Nov 24 21:42:16 myserver ollama[162316]: load_backend: loaded ROCm backend from /usr/local/lib/ollama/rocm/libggml-hip.so
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.665Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 ROCm.0.NO_VMM=1 ROCm.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.pooling_type default=0
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_bos_token default=false
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.bos_token_id default=0
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.rope.scaling.factor default=1
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_count default=0
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_used_count default=0
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.norm_top_k_prob default=true
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}"
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.vision.num_positional_embeddings default=2304
Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}"
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.348Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1297 splits=412
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.899Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1243 splits=3
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1232 splits=3
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:240 msg="model weights" device=ROCm0 size="2.0 GiB"
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:245 msg="model weights" device=CPU size="1.3 GiB"
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:251 msg="kv cache" device=ROCm0 size="11.0 GiB"
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:262 msg="compute graph" device=ROCm0 size="4.3 GiB"
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:267 msg="compute graph" device=CPU size="126.6 MiB"
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:272 msg="total memory" size="18.8 GiB"
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=server.go:727 msg=memory success=true required.InputWeights=319065600 required.CPU.Weights="[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1118127616]" required.CPU.Graph=132710400 required.ROCm0.ID=0 required.ROCm0.Weights="[63888384 63888384 63888384 63888384 57467904 57467904 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 63212544 63212544 63888384 63212544 63212544 0]" required.ROCm0.Cache="[328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 0]" required.ROCm0.Graph=4608392192
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=server.go:921 msg="available gpu" id=0 library=ROCm "available layer vram"="16.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="4.3 GiB"
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.912Z level=DEBUG source=server.go:738 msg="new layout created" layers="36[ID:0 Layers:36(0..35)]"
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.912Z level=INFO source=runner.go:1271 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:256 FlashAttention:true KvSize:80000 KvCacheType: NumThreads:8 GPULayers:36[ID:0 Layers:36(0..35)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.937Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.pooling_type default=0
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_bos_token default=false
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.bos_token_id default=0
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.rope.scaling.factor default=1
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_count default=0
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_used_count default=0
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.norm_top_k_prob default=true
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}"
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.vision.num_positional_embeddings default=2304
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}"
Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.738Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1297 splits=412
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.829Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1243 splits=3
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.875Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1232 splits=3
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:240 msg="model weights" device=ROCm0 size="2.0 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:245 msg="model weights" device=CPU size="1.3 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:251 msg="kv cache" device=ROCm0 size="11.0 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:262 msg="compute graph" device=ROCm0 size="4.3 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:267 msg="compute graph" device=CPU size="126.6 MiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:272 msg="total memory" size="18.8 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=server.go:727 msg=memory success=true required.InputWeights=319065600 required.CPU.Weights="[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1118127616]" required.CPU.Graph=132710400 required.ROCm0.ID=0 required.ROCm0.Weights="[63888384 63888384 63888384 63888384 57467904 57467904 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 63212544 63212544 63888384 63212544 63212544 0]" required.ROCm0.Cache="[328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 0]" required.ROCm0.Graph=4608392192
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=server.go:921 msg="available gpu" id=0 library=ROCm "available layer vram"="16.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="4.3 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=server.go:738 msg="new layout created" layers="36[ID:0 Layers:36(0..35)]"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=runner.go:1271 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:256 FlashAttention:true KvSize:80000 KvCacheType: NumThreads:8 GPULayers:36[ID:0 Layers:36(0..35)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:240 msg="model weights" device=ROCm0 size="2.0 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:245 msg="model weights" device=CPU size="1.3 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:251 msg="kv cache" device=ROCm0 size="11.0 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=ggml.go:482 msg="offloading 36 repeating layers to GPU"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=ggml.go:486 msg="offloading output layer to CPU"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=ggml.go:494 msg="offloaded 36/37 layers to GPU"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:262 msg="compute graph" device=ROCm0 size="4.3 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:267 msg="compute graph" device=CPU size="126.6 MiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:272 msg="total memory" size="18.8 GiB"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=sched.go:517 msg="loaded runners" count=1
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=server.go:1294 msg="waiting for llama runner to start responding"
Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.877Z level=INFO source=server.go:1328 msg="waiting for server to become available" status="llm server loading model"
Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.128Z level=DEBUG source=server.go:1338 msg="model load progress 0.48"
Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.381Z level=DEBUG source=server.go:1338 msg="model load progress 0.79"
Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.632Z level=DEBUG source=server.go:1338 msg="model load progress 0.95"
Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.828Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.pooling_type default=0
Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.883Z level=INFO source=server.go:1332 msg="llama runner started in 6.71 seconds"
Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.883Z level=DEBUG source=sched.go:529 msg="finished setting up" runner.name=registry.ollama.ai/library/qwen3-vl:4b-thinking runner.inference="[{ID:0 Library:ROCm}]" runner.size="18.8 GiB" runner.vram="17.3 GiB" runner.parallel=1 runner.pid=162407 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 runner.num_ctx=80000
Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.883Z level=DEBUG source=server.go:1465 msg="completion request" images=0 prompt=201 format=""
Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.898Z level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=0 prompt=67 used=0 remaining=67
Nov 24 21:46:34 myserver ollama[162316]: [GIN] 2025/11/24 - 21:46:34 | 200 |         4m21s |    100.100.10.2 | POST     "/api/chat"
Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.839Z level=DEBUG source=sched.go:537 msg="context for request finished"
Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.839Z level=DEBUG source=sched.go:290 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3-vl:4b-thinking runner.inference="[{ID:0 Library:ROCm}]" runner.size="18.8 GiB" runner.vram="17.3 GiB" runner.parallel=1 runner.pid=162407 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 runner.num_ctx=80000 duration=2562047h47m16.854775807s
Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.839Z level=DEBUG source=sched.go:308 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3-vl:4b-thinking runner.inference="[{ID:0 Library:ROCm}]" runner.size="18.8 GiB" runner.vram="17.3 GiB" runner.parallel=1 runner.pid=162407 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 runner.num_ctx=80000 refCount=0
Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.984Z level=DEBUG source=sched.go:626 msg="evaluating already loaded" model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38
Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.984Z level=DEBUG source=server.go:1465 msg="completion request" images=0 prompt=3350 format=""
Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.987Z level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=4440 prompt=960 used=3 remaining=957
Nov 24 21:48:39 myserver systemd[1]: Stopping ollama.service - Ollama Service...
Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=sched.go:136 msg="shutting down scheduler pending loop"
Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=sched.go:269 msg="shutting down scheduler completed loop"
Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=sched.go:385 msg="context for request finished" runner.name=registry.ollama.ai/library/qwen3-vl:4b-thinking runner.inference="[{ID:0 Library:ROCm}]" runner.size="18.8 GiB" runner.vram="17.3 GiB" runner.parallel=1 runner.pid=162407 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 runner.num_ctx=80000
Nov 24 21:48:39 myserver ollama[162316]: [GIN] 2025/11/24 - 21:48:39 | 500 |          2m4s |    100.100.10.2 | POST     "/api/chat"
Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=sched.go:844 msg="shutting down runner" model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38
Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=server.go:1755 msg="stopping llama server" pid=162407
Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=server.go:1761 msg="waiting for llama server to exit" pid=162407
Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.206Z level=DEBUG source=server.go:1765 msg="llama server stopped" pid=162407
Nov 24 21:48:39 myserver systemd[1]: ollama.service: Deactivated successfully.
Nov 24 21:48:39 myserver systemd[1]: Stopped ollama.service - Ollama Service.

OS

Linux

GPU

AMD

CPU

AMD

Ollama version

0.12.11

Originally created by @moontato on GitHub (Nov 24, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/13232 ### What is the issue? I am running v0.12.11 using a Radeon 680M iGPU on Ubuntu server 24. Certain models, particularly the variants of `qwen3-vl`, seem to "hang" even after it finishes responding. GPU usage (and CPU, if partial offloading) remains ~100% until I manually restart the service. `gpt-oss`, the model I've been using for months prior, has maybe had this issue once or twice? Either way, I could count it on my ten fingers. But with `qwen3-vl` (8b-thinking, 4b-thinking, both q4km), it seems to happen almost every time, and it is guaranteed to happen for me using this prompt: ``` **Q4:** Consider the recurrence: T(n)=3T(n/2)+nT(n) = 3T(n/2) + nT(n)=3T(n/2)+n Using the Master Theorem, what is the tight asymptotic bound? ``` When the message should be completed, the open-webui interface shows that the message is complete and I am able to send a new message, but obviously it never gets to it because it's still stuck on the previous request. Here's how I'm running the ollama service: ``` [Service] Environment="OLLAMA_HOST=http://myserverdomain:11434" Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0" Environment="OLLAMA_NUM_THREADS=8" Environment="OPENBLAS_NUM_THREADS=8" Environment="OMP_NUM_THREADS=8" Environment="OLLAMA_FLASH_ATTENTION=1" Environment="OLLAMA_NEW_ENGINE=1" Environment="OLLAMA_DEBUG=1" ``` Note: I have also experimented with setting `OLLAMA_VULKAN` and removing `Environment="OLLAMA_NEW_ENGINE=1"` from the service file, but to no avail. Also, I have tried updating to 0.13 to see if that'd fix the issue, but unfortunately there's a unified memory detection issue on that version that prevents `qwen3-vl` from even offloading most of its layers onto the GPU. Edit: consistency fixes ### Relevant log output ```shell Nov 24 21:42:07 myserver systemd[1]: Started ollama.service - Ollama Service. Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.448Z level=INFO source=routes.go:1544 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION:10.3.0 HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG OLLAMA_FLASH_ATTENTION:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://myserverdomain:11434 OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:true OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.450Z level=INFO source=images.go:522 msg="total blobs: 22" Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=INFO source=images.go:529 msg="total unused blobs removed: 0" Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=INFO source=routes.go:1597 msg="Listening on 100.100.10.2:11434 (version 0.12.11)" Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=DEBUG source=sched.go:120 msg="starting llm scheduler" Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=INFO source=runner.go:67 msg="discovering available GPUs..." Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 46727" Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.451Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v12 OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v12 Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.468Z level=DEBUG source=runner.go:418 msg="bootstrap discovery took" duration=16.962393ms OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/cuda_v12]" extra_envs=map[] Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.468Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 46689" Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.468Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v13 OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v13 Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.485Z level=DEBUG source=runner.go:418 msg="bootstrap discovery took" duration=16.691995ms OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/cuda_v13]" extra_envs=map[] Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.485Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 36975" Nov 24 21:42:07 myserver ollama[162316]: time=2025-11-24T21:42:07.485Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm Nov 24 21:42:08 myserver ollama[162316]: time=2025-11-24T21:42:08.888Z level=DEBUG source=runner.go:418 msg="bootstrap discovery took" duration=1.403423063s OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/rocm]" extra_envs=map[] Nov 24 21:42:08 myserver ollama[162316]: time=2025-11-24T21:42:08.888Z level=DEBUG source=runner.go:116 msg="evluating which if any devices to filter out" initial_count=1 Nov 24 21:42:08 myserver ollama[162316]: time=2025-11-24T21:42:08.888Z level=DEBUG source=runner.go:128 msg="verifying device is supported" library=/usr/local/lib/ollama/rocm description="AMD Radeon Graphics" compute=gfx1030 id=0 pci_id=0000:e9:00.0 Nov 24 21:42:08 myserver ollama[162316]: time=2025-11-24T21:42:08.889Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 39461" Nov 24 21:42:08 myserver ollama[162316]: time=2025-11-24T21:42:08.889Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm ROCR_VISIBLE_DEVICES=0 GGML_CUDA_INIT=1 Nov 24 21:42:11 myserver ollama[162316]: time=2025-11-24T21:42:11.006Z level=DEBUG source=runner.go:418 msg="bootstrap discovery took" duration=2.118090349s OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/rocm]" extra_envs="map[GGML_CUDA_INIT:1 ROCR_VISIBLE_DEVICES:0]" Nov 24 21:42:11 myserver ollama[162316]: time=2025-11-24T21:42:11.007Z level=DEBUG source=runner.go:175 msg="adjusting filtering IDs" FilterID=0 new_ID=0 Nov 24 21:42:11 myserver ollama[162316]: time=2025-11-24T21:42:11.007Z level=DEBUG source=runner.go:40 msg="GPU bootstrap discovery took" duration=3.555695961s Nov 24 21:42:11 myserver ollama[162316]: time=2025-11-24T21:42:11.007Z level=INFO source=types.go:42 msg="inference compute" id=0 filter_id=0 library=ROCm compute=gfx1030 name=ROCm0 description="AMD Radeon Graphics" libdirs=ollama,rocm driver=60342.13 pci_id=0000:e9:00.0 type=iGPU total="21.5 GiB" available="21.5 GiB" Nov 24 21:42:13 myserver ollama[162316]: [GIN] 2025/11/24 - 21:42:13 | 200 | 1.140065ms | 100.100.10.2 | GET "/api/tags" Nov 24 21:42:13 myserver ollama[162316]: [GIN] 2025/11/24 - 21:42:13 | 200 | 100.934µs | 100.100.10.2 | GET "/api/ps" Nov 24 21:42:13 myserver ollama[162316]: time=2025-11-24T21:42:13.672Z level=DEBUG source=runner.go:246 msg="refreshing free memory" Nov 24 21:42:13 myserver ollama[162316]: time=2025-11-24T21:42:13.672Z level=DEBUG source=runner.go:310 msg="unable to refresh all GPUs with existing runners, performing bootstrap discovery" Nov 24 21:42:13 myserver ollama[162316]: time=2025-11-24T21:42:13.672Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --port 39341" Nov 24 21:42:13 myserver ollama[162316]: time=2025-11-24T21:42:13.672Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm ROCR_VISIBLE_DEVICES=0 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.100Z level=DEBUG source=runner.go:418 msg="bootstrap discovery took" duration=1.42850933s OLLAMA_LIBRARY_PATH="[/usr/local/lib/ollama /usr/local/lib/ollama/rocm]" extra_envs=map[ROCR_VISIBLE_DEVICES:0] Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.101Z level=DEBUG source=runner.go:40 msg="overall device VRAM discovery took" duration=1.428618629s Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.101Z level=DEBUG source=sched.go:194 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=3 gpu_count=1 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.117Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.118Z level=DEBUG source=sched.go:211 msg="loading first model" model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.172Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.pooling_type default=0 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_bos_token default=false Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.bos_token_id default=0 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.rope.scaling.factor default=1 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_count default=0 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_used_count default=0 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.norm_top_k_prob default=true Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.vision.num_positional_embeddings default=2304 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=INFO source=server.go:209 msg="enabling flash attention" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=INFO source=server.go:392 msg="starting runner" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 --port 35507" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=DEBUG source=server.go:393 msg=subprocess PATH=/home/user1/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=http://myserverdomain:11434 HSA_OVERRIDE_GFX_VERSION=10.3.0 OLLAMA_NUM_THREADS=8 OLLAMA_FLASH_ATTENTION=1 OLLAMA_NEW_ENGINE=1 OLLAMA_DEBUG=1 LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/rocm ROCR_VISIBLE_DEVICES=0 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=INFO source=sched.go:443 msg="system memory" total="27.2 GiB" free="25.2 GiB" free_swap="13.6 GiB" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=INFO source=sched.go:450 msg="gpu memory" id=0 library=ROCm available="21.0 GiB" free="21.5 GiB" minimum="457.0 MiB" overhead="0 B" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.173Z level=INFO source=server.go:702 msg="loading model" "model layers"=37 requested=36 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.185Z level=INFO source=runner.go:1398 msg="starting ollama engine" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.185Z level=INFO source=runner.go:1433 msg="Server listening on 127.0.0.1:35507" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.196Z level=INFO source=runner.go:1271 msg=load request="{Operation:fit LoraPath:[] Parallel:1 BatchSize:256 FlashAttention:true KvSize:80000 KvCacheType: NumThreads:8 GPULayers:36[ID:0 Layers:36(0..35)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.222Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.223Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.name default="" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.223Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.description default="" Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.223Z level=INFO source=ggml.go:136 msg="" architecture=qwen3vl file_type=Q4_K_M name="" description="" num_tensors=809 num_key_values=40 Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.223Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/local/lib/ollama Nov 24 21:42:15 myserver ollama[162316]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so Nov 24 21:42:15 myserver ollama[162316]: time=2025-11-24T21:42:15.227Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/local/lib/ollama/rocm Nov 24 21:42:15 myserver ollama[162316]: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory Nov 24 21:42:16 myserver ollama[162316]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Nov 24 21:42:16 myserver ollama[162316]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Nov 24 21:42:16 myserver ollama[162316]: ggml_cuda_init: found 1 ROCm devices: Nov 24 21:42:16 myserver ollama[162316]: Device 0: AMD Radeon Graphics, gfx1030 (0x1030), VMM: no, Wave Size: 32, ID: 0 Nov 24 21:42:16 myserver ollama[162316]: load_backend: loaded ROCm backend from /usr/local/lib/ollama/rocm/libggml-hip.so Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.665Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 ROCm.0.NO_VMM=1 ROCm.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.pooling_type default=0 Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_bos_token default=false Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.bos_token_id default=0 Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.rope.scaling.factor default=1 Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_count default=0 Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_used_count default=0 Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.norm_top_k_prob default=true Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}" Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.vision.num_positional_embeddings default=2304 Nov 24 21:42:16 myserver ollama[162316]: time=2025-11-24T21:42:16.667Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}" Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.348Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1297 splits=412 Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.899Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1243 splits=3 Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1232 splits=3 Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:240 msg="model weights" device=ROCm0 size="2.0 GiB" Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:245 msg="model weights" device=CPU size="1.3 GiB" Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:251 msg="kv cache" device=ROCm0 size="11.0 GiB" Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:262 msg="compute graph" device=ROCm0 size="4.3 GiB" Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:267 msg="compute graph" device=CPU size="126.6 MiB" Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=device.go:272 msg="total memory" size="18.8 GiB" Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=server.go:727 msg=memory success=true required.InputWeights=319065600 required.CPU.Weights="[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1118127616]" required.CPU.Graph=132710400 required.ROCm0.ID=0 required.ROCm0.Weights="[63888384 63888384 63888384 63888384 57467904 57467904 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 63212544 63212544 63888384 63212544 63212544 0]" required.ROCm0.Cache="[328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 0]" required.ROCm0.Graph=4608392192 Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.911Z level=DEBUG source=server.go:921 msg="available gpu" id=0 library=ROCm "available layer vram"="16.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="4.3 GiB" Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.912Z level=DEBUG source=server.go:738 msg="new layout created" layers="36[ID:0 Layers:36(0..35)]" Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.912Z level=INFO source=runner.go:1271 msg=load request="{Operation:alloc LoraPath:[] Parallel:1 BatchSize:256 FlashAttention:true KvSize:80000 KvCacheType: NumThreads:8 GPULayers:36[ID:0 Layers:36(0..35)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" Nov 24 21:42:17 myserver ollama[162316]: time=2025-11-24T21:42:17.937Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.pooling_type default=0 Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_bos_token default=false Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.bos_token_id default=0 Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.rope.scaling.factor default=1 Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_count default=0 Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.expert_used_count default=0 Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.norm_top_k_prob default=true Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}" Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.vision.num_positional_embeddings default=2304 Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.085Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.mrope_sections default="&{size:0 values:[24 20 20]}" Nov 24 21:42:18 myserver ollama[162316]: time=2025-11-24T21:42:18.738Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1297 splits=412 Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.829Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1243 splits=3 Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.875Z level=DEBUG source=ggml.go:853 msg="compute graph" nodes=1232 splits=3 Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:240 msg="model weights" device=ROCm0 size="2.0 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:245 msg="model weights" device=CPU size="1.3 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:251 msg="kv cache" device=ROCm0 size="11.0 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:262 msg="compute graph" device=ROCm0 size="4.3 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:267 msg="compute graph" device=CPU size="126.6 MiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=device.go:272 msg="total memory" size="18.8 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=server.go:727 msg=memory success=true required.InputWeights=319065600 required.CPU.Weights="[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1118127616]" required.CPU.Graph=132710400 required.ROCm0.ID=0 required.ROCm0.Weights="[63888384 63888384 63888384 63888384 57467904 57467904 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 56792064 56792064 63888384 63212544 63212544 63888384 63212544 63212544 0]" required.ROCm0.Cache="[328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 328204288 0]" required.ROCm0.Graph=4608392192 Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=server.go:921 msg="available gpu" id=0 library=ROCm "available layer vram"="16.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="4.3 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=DEBUG source=server.go:738 msg="new layout created" layers="36[ID:0 Layers:36(0..35)]" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=runner.go:1271 msg=load request="{Operation:commit LoraPath:[] Parallel:1 BatchSize:256 FlashAttention:true KvSize:80000 KvCacheType: NumThreads:8 GPULayers:36[ID:0 Layers:36(0..35)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:240 msg="model weights" device=ROCm0 size="2.0 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:245 msg="model weights" device=CPU size="1.3 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:251 msg="kv cache" device=ROCm0 size="11.0 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=ggml.go:482 msg="offloading 36 repeating layers to GPU" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=ggml.go:486 msg="offloading output layer to CPU" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=ggml.go:494 msg="offloaded 36/37 layers to GPU" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:262 msg="compute graph" device=ROCm0 size="4.3 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:267 msg="compute graph" device=CPU size="126.6 MiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=device.go:272 msg="total memory" size="18.8 GiB" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=sched.go:517 msg="loaded runners" count=1 Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.876Z level=INFO source=server.go:1294 msg="waiting for llama runner to start responding" Nov 24 21:42:20 myserver ollama[162316]: time=2025-11-24T21:42:20.877Z level=INFO source=server.go:1328 msg="waiting for server to become available" status="llm server loading model" Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.128Z level=DEBUG source=server.go:1338 msg="model load progress 0.48" Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.381Z level=DEBUG source=server.go:1338 msg="model load progress 0.79" Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.632Z level=DEBUG source=server.go:1338 msg="model load progress 0.95" Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.828Z level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3vl.pooling_type default=0 Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.883Z level=INFO source=server.go:1332 msg="llama runner started in 6.71 seconds" Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.883Z level=DEBUG source=sched.go:529 msg="finished setting up" runner.name=registry.ollama.ai/library/qwen3-vl:4b-thinking runner.inference="[{ID:0 Library:ROCm}]" runner.size="18.8 GiB" runner.vram="17.3 GiB" runner.parallel=1 runner.pid=162407 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 runner.num_ctx=80000 Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.883Z level=DEBUG source=server.go:1465 msg="completion request" images=0 prompt=201 format="" Nov 24 21:42:21 myserver ollama[162316]: time=2025-11-24T21:42:21.898Z level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=0 prompt=67 used=0 remaining=67 Nov 24 21:46:34 myserver ollama[162316]: [GIN] 2025/11/24 - 21:46:34 | 200 | 4m21s | 100.100.10.2 | POST "/api/chat" Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.839Z level=DEBUG source=sched.go:537 msg="context for request finished" Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.839Z level=DEBUG source=sched.go:290 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3-vl:4b-thinking runner.inference="[{ID:0 Library:ROCm}]" runner.size="18.8 GiB" runner.vram="17.3 GiB" runner.parallel=1 runner.pid=162407 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 runner.num_ctx=80000 duration=2562047h47m16.854775807s Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.839Z level=DEBUG source=sched.go:308 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3-vl:4b-thinking runner.inference="[{ID:0 Library:ROCm}]" runner.size="18.8 GiB" runner.vram="17.3 GiB" runner.parallel=1 runner.pid=162407 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 runner.num_ctx=80000 refCount=0 Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.984Z level=DEBUG source=sched.go:626 msg="evaluating already loaded" model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.984Z level=DEBUG source=server.go:1465 msg="completion request" images=0 prompt=3350 format="" Nov 24 21:46:34 myserver ollama[162316]: time=2025-11-24T21:46:34.987Z level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=4440 prompt=960 used=3 remaining=957 Nov 24 21:48:39 myserver systemd[1]: Stopping ollama.service - Ollama Service... Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=sched.go:136 msg="shutting down scheduler pending loop" Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=sched.go:269 msg="shutting down scheduler completed loop" Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=sched.go:385 msg="context for request finished" runner.name=registry.ollama.ai/library/qwen3-vl:4b-thinking runner.inference="[{ID:0 Library:ROCm}]" runner.size="18.8 GiB" runner.vram="17.3 GiB" runner.parallel=1 runner.pid=162407 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 runner.num_ctx=80000 Nov 24 21:48:39 myserver ollama[162316]: [GIN] 2025/11/24 - 21:48:39 | 500 | 2m4s | 100.100.10.2 | POST "/api/chat" Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=sched.go:844 msg="shutting down runner" model=/usr/share/ollama/.ollama/models/blobs/sha256-9c60bdd691c1897bbfe5ddbc67336848e18c346b7ee2ab8541b135f208e5bb38 Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=server.go:1755 msg="stopping llama server" pid=162407 Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.059Z level=DEBUG source=server.go:1761 msg="waiting for llama server to exit" pid=162407 Nov 24 21:48:39 myserver ollama[162316]: time=2025-11-24T21:48:39.206Z level=DEBUG source=server.go:1765 msg="llama server stopped" pid=162407 Nov 24 21:48:39 myserver systemd[1]: ollama.service: Deactivated successfully. Nov 24 21:48:39 myserver systemd[1]: Stopped ollama.service - Ollama Service. ``` ### OS Linux ### GPU AMD ### CPU AMD ### Ollama version 0.12.11
GiteaMirror added the needs more infobug labels 2026-04-22 18:08:03 -05:00
GiteaMirror commented 2026-04-22 18:08:06 -05:00
Author
Owner
Copy Link

@alpangolin commented on GitHub (Nov 25, 2025):

I'm experiencing a very much the same symptoms in 0.13.0 with AMD GPU and qwen3:8b. I'm calling ollama model from my app in a loop, and after few calls, ollama get stuck with no answer to app and 100% GPU usage.

There's 5 min memory keep-alive for model and when checking with ollama ps it keep counting like when idle. However, on timeout it moves to stopping state as usual, but seems to stay in that forever, until I restart ollama, which is only way to resolve the situation AFAIK.

<!-- gh-comment-id:3573884602 --> @alpangolin commented on GitHub (Nov 25, 2025): I'm experiencing a very much the same symptoms in 0.13.0 with AMD GPU and qwen3:8b. I'm calling ollama model from my app in a loop, and after few calls, ollama get stuck with no answer to app and 100% GPU usage. There's 5 min memory keep-alive for model and when checking with ollama ps it keep counting like when idle. However, on timeout it moves to stopping state as usual, but seems to stay in that forever, until I restart ollama, which is only way to resolve the situation AFAIK.
GiteaMirror commented 2026-04-22 18:08:07 -05:00
Author
Owner
Copy Link

@jessegross commented on GitHub (Nov 25, 2025):

@alpangolin This is almost always caused by the model losing coherence and going into a loop. Often this is triggered by a shift after exceeding the context window. The most definitive way to prevent it is to set the num_predict parameter to cap the number of output tokens. You can also try increasing the context length or switching models.

@moontato Your issue sounds similar, though you said that the messages appear to have finished. I see that the last message returned a 500 error. Most likely an error was encountered and returned to the client but the runner kept generating in a loop when it should have been stopped. As a workaround, the same num_predict parameter should be effective to stop the runner. My guess is that the error is triggered by a problem in the qwen3-vl parser. If possible, can you post the log with OLLAMA_DEBUG=2? Note that this will log user data.

<!-- gh-comment-id:3577795389 --> @jessegross commented on GitHub (Nov 25, 2025): @alpangolin This is almost always caused by the model losing coherence and going into a loop. Often this is triggered by a shift after exceeding the context window. The most definitive way to prevent it is to set the `num_predict` parameter to cap the number of output tokens. You can also try increasing the context length or switching models. @moontato Your issue sounds similar, though you said that the messages appear to have finished. I see that the last message returned a 500 error. Most likely an error was encountered and returned to the client but the runner kept generating in a loop when it should have been stopped. As a workaround, the same `num_predict` parameter should be effective to stop the runner. My guess is that the error is triggered by a problem in the qwen3-vl parser. If possible, can you post the log with OLLAMA_DEBUG=2? Note that this will log user data.
GiteaMirror commented 2026-04-22 18:08:07 -05:00
Author
Owner
Copy Link

@alpangolin commented on GitHub (Nov 26, 2025):

@jessegross ok, thank you for info. In my preliminary testing it seems that setting context length to 8000 solves the problem at least in my current case. I really appreciate your help!

<!-- gh-comment-id:3579855818 --> @alpangolin commented on GitHub (Nov 26, 2025): @jessegross ok, thank you for info. In my preliminary testing it seems that setting context length to 8000 solves the problem at least in my current case. I really appreciate your help!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug needs more info
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#34508
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?