github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-22 21:51:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #13372] Significant Slowdown in First Token Generation Time for qwen3:32b in Ollama 0.12.4~0.13.2 Compared to 0.12.3 #8830

New Issue
Closed
opened 2026-04-12 21:36:51 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-04-12 21:36:51 -05:00
Owner
Copy Link

Originally created by @mokeyish on GitHub (Dec 8, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/13372

Description
After upgrading from Ollama version 0.12.3 to 0.12.4~0.13.2, I observed a noticeable degradation in the first token generation speed when running the qwen3:32b model. The delay has increased significantly—by approximately 4–5 seconds or more—compared to the previous version.

Steps to Reproduce

  1. Run qwen3:32b on Ollama 0.11.5 and note the time to first token.
  2. Upgrade to Ollama 0.13.1.
  3. Run the same model (qwen3:32b) under identical conditions.
  4. Observe the increased latency for the first token generation.

Expected Behavior
First token generation time should remain consistent or improve with newer versions of Ollama.

Actual Behavior
First token generation time has increased by several seconds in version 0.12.4~0.13.2.

Environment

  • Ollama Version: 0.12.4~0.13.2 (regression observed from 0.12.3)
  • Model: qwen3:32b
  • Hardware: A800 GPU
  • Operating System: Debian bookworm

0.12.3

time=2025-12-09T14:28:02.866+08:00 level=INFO source=runner.go:1252 msg="starting ollama engine"
time=2025-12-09T14:28:02.866+08:00 level=INFO source=runner.go:1287 msg="Server listening on 127.0.0.1:39437"
time=2025-12-09T14:28:03.198+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 name="NVIDIA A800 80GB PCIe" overhead="0 B" before.total="79.1 GiB" before.free="24.2 GiB" now.total="79.1 GiB" now.free="24.2 GiB" now.used="54.9 GiB"
time=2025-12-09T14:28:03.518+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 name="NVIDIA A800 80GB PCIe" overhead="0 B" before.total="79.1 GiB" before.free="48.2 GiB" now.total="79.1 GiB" now.free="48.2 GiB" now.used="30.9 GiB"
time=2025-12-09T14:28:03.822+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 name="NVIDIA A800 80GB PCIe" overhead="0 B" before.total="79.1 GiB" before.free="54.7 GiB" now.total="79.1 GiB" now.free="54.7 GiB" now.used="24.5 GiB"
time=2025-12-09T14:28:04.318+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da name="NVIDIA A800 80GB PCIe" overhead="0 B" before.total="79.1 GiB" before.free="25.7 GiB" now.total="79.1 GiB" now.free="25.7 GiB" now.used="53.4 GiB"
releasing cuda driver library
time=2025-12-09T14:28:04.318+08:00 level=INFO source=server.go:678 msg="system memory" total="1007.5 GiB" free="829.6 GiB" free_swap="0 B"
time=2025-12-09T14:28:04.318+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 available="23.7 GiB" free="24.2 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-12-09T14:28:04.318+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 available="47.8 GiB" free="48.2 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-12-09T14:28:04.318+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 available="54.2 GiB" free="54.7 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-12-09T14:28:04.318+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da available="25.3 GiB" free="25.7 GiB" minimum="457.0 MiB" overhead="0 B"
time=2025-12-09T14:28:04.321+08:00 level=INFO source=runner.go:1171 msg=load request="{Operation:fit LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:false KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-12-09T14:28:04.359+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
time=2025-12-09T14:28:04.360+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.description default=""
time=2025-12-09T14:28:04.360+08:00 level=INFO source=ggml.go:131 msg="" architecture=qwen3 file_type=Q4_K_M name="Qwen3 32B" description="" num_tensors=707 num_key_values=28
time=2025-12-09T14:28:04.360+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/abc/.local/ollama-0.12.3/lib/ollama
load_backend: loaded CPU backend from /home/abc/.local/ollama-0.12.3/lib/ollama/libggml-cpu-icelake.so
time=2025-12-09T14:28:04.368+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/abc/.local/ollama-0.12.3/lib/ollama/cuda_v12
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3
  Device 1: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92
  Device 2: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5
  Device 3: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da
load_backend: loaded CUDA backend from /home/abc/.local/ollama-0.12.3/lib/ollama/cuda_v12/libggml-cuda.so
time=2025-12-09T14:28:04.484+08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0
time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_eos_token default=false
time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}"
time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.rope.scaling.factor default=1
time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_count default=0
time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_used_count default=0
time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.norm_top_k_prob default=true
time=2025-12-09T14:28:04.739+08:00 level=DEBUG source=ggml.go:794 msg="compute graph" nodes=2438 splits=2
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:310 msg="model weights" device=CUDA2 size="18.4 GiB"
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:315 msg="model weights" device=CPU size="417.3 MiB"
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:321 msg="kv cache" device=CUDA2 size="4.0 GiB"
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:332 msg="compute graph" device=CUDA2 size="2.1 GiB"
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:342 msg="total memory" size="24.9 GiB"
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=server.go:717 msg=memory success=true required.InputWeights=437575680U required.CPU.Graph=10485760U required.CUDA2.ID=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 required.CUDA2.Weights="[315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 638151680U]" required.CUDA2.Cache="[67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 0U]" required.CUDA2.Graph=2235566208U
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 "available layer vram"="52.1 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="2.1 GiB"
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 "available layer vram"="47.8 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da "available layer vram"="25.3 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 "available layer vram"="23.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:28:04.743+08:00 level=DEBUG source=server.go:728 msg="new layout created" layers="65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)]"
time=2025-12-09T14:28:04.743+08:00 level=INFO source=runner.go:1171 msg=load request="{Operation:alloc LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:false KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-12-09T14:28:04.802+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
time=2025-12-09T14:28:04.815+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0
time=2025-12-09T14:28:04.815+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_eos_token default=false
time=2025-12-09T14:28:04.815+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}"
time=2025-12-09T14:28:04.816+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.rope.scaling.factor default=1
time=2025-12-09T14:28:04.816+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_count default=0
time=2025-12-09T14:28:04.816+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_used_count default=0
time=2025-12-09T14:28:04.816+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.norm_top_k_prob default=true
time=2025-12-09T14:28:04.851+08:00 level=DEBUG source=ggml.go:794 msg="compute graph" nodes=2438 splits=2
time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:310 msg="model weights" device=CUDA2 size="18.4 GiB"
time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:315 msg="model weights" device=CPU size="417.3 MiB"
time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:321 msg="kv cache" device=CUDA2 size="4.0 GiB"
time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:332 msg="compute graph" device=CUDA2 size="2.1 GiB"
time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:342 msg="total memory" size="24.9 GiB"
time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=server.go:717 msg=memory success=true required.InputWeights=437575680A required.CPU.Graph=10485760A required.CUDA2.ID=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 required.CUDA2.Weights="[315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 638151680A]" required.CUDA2.Cache="[67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 0U]" required.CUDA2.Graph=2235566208A
time=2025-12-09T14:28:04.854+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 "available layer vram"="52.1 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="2.1 GiB"
time=2025-12-09T14:28:04.854+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 "available layer vram"="47.8 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:28:04.854+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da "available layer vram"="25.3 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:28:04.854+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 "available layer vram"="23.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:28:04.855+08:00 level=DEBUG source=server.go:728 msg="new layout created" layers="65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)]"
time=2025-12-09T14:28:04.855+08:00 level=INFO source=runner.go:1171 msg=load request="{Operation:commit LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:false KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-12-09T14:28:04.856+08:00 level=INFO source=ggml.go:487 msg="offloading 64 repeating layers to GPU"
time=2025-12-09T14:28:04.856+08:00 level=INFO source=ggml.go:493 msg="offloading output layer to GPU"
time=2025-12-09T14:28:04.856+08:00 level=INFO source=ggml.go:498 msg="offloaded 65/65 layers to GPU"
time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:310 msg="model weights" device=CUDA2 size="18.4 GiB"
time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:315 msg="model weights" device=CPU size="417.3 MiB"
time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:321 msg="kv cache" device=CUDA2 size="4.0 GiB"
time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:332 msg="compute graph" device=CUDA2 size="2.1 GiB"
time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:342 msg="total memory" size="24.9 GiB"
time=2025-12-09T14:28:04.858+08:00 level=INFO source=sched.go:470 msg="loaded runners" count=1
time=2025-12-09T14:28:04.858+08:00 level=INFO source=server.go:1251 msg="waiting for llama runner to start responding"
time=2025-12-09T14:28:04.870+08:00 level=INFO source=server.go:1285 msg="waiting for server to become available" status="llm server loading model"
time=2025-12-09T14:28:04.870+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.00"
time=2025-12-09T14:28:05.121+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.09"
time=2025-12-09T14:28:05.372+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.15"
time=2025-12-09T14:28:05.623+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.21"
time=2025-12-09T14:28:05.874+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.26"
time=2025-12-09T14:28:06.125+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.31"
time=2025-12-09T14:28:06.376+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.36"
time=2025-12-09T14:28:06.627+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.41"
time=2025-12-09T14:28:06.878+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.47"
time=2025-12-09T14:28:07.128+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.54"
time=2025-12-09T14:28:07.379+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.60"
time=2025-12-09T14:28:07.630+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.66"
time=2025-12-09T14:28:07.881+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.71"
time=2025-12-09T14:28:08.132+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.77"
time=2025-12-09T14:28:08.383+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.83"
time=2025-12-09T14:28:08.634+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.90"
time=2025-12-09T14:28:08.885+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.95"
time=2025-12-09T14:28:09.136+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.99"
time=2025-12-09T14:28:09.387+08:00 level=DEBUG source=server.go:1295 msg="model load progress 1.00"
[GIN] 2025/12/09 - 14:28:09 | 200 |    4.824238ms |  192.168.11.124 | GET      "/v1/models"
time=2025-12-09T14:28:09.638+08:00 level=DEBUG source=server.go:1298 msg="model load completed, waiting for server to become available" status="llm server loading model"
time=2025-12-09T14:28:09.937+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0
time=2025-12-09T14:28:10.142+08:00 level=INFO source=server.go:1289 msg="llama runner started in 7.29 seconds"
time=2025-12-09T14:28:10.142+08:00 level=DEBUG source=sched.go:482 msg="finished setting up" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096
time=2025-12-09T14:28:10.167+08:00 level=DEBUG source=server.go:1388 msg="completion request" images=0 prompt=369 format=""
time=2025-12-09T14:28:10.218+08:00 level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=0 prompt=72 used=0 remaining=72
[GIN] 2025/12/09 - 14:28:14 | 200 | 14.896581218s |    192.168.10.0 | POST     "/v1/chat/completions"
time=2025-12-09T14:28:14.598+08:00 level=DEBUG source=sched.go:490 msg="context for request finished"
time=2025-12-09T14:28:14.598+08:00 level=DEBUG source=sched.go:286 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 duration=72h0m0s
time=2025-12-09T14:28:14.598+08:00 level=DEBUG source=sched.go:304 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 refCount=0
[GIN] 2025/12/09 - 14:28:19 | 200 |    8.523088ms |  192.168.11.124 | GET      "/v1/models"
[GIN] 2025/12/09 - 14:28:29 | 200 |    8.604608ms |  192.168.11.124 | GET      "/v1/models"
[GIN] 2025/12/09 - 14:28:29 | 200 |    5.173909ms |  192.168.11.124 | GET      "/v1/models"
[GIN] 2025/12/09 - 14:28:39 | 200 |    6.666058ms |  192.168.11.124 | GET      "/v1/models"
time=2025-12-09T14:28:44.404+08:00 level=DEBUG source=sched.go:580 msg="evaluating already loaded" model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312
time=2025-12-09T14:28:44.405+08:00 level=DEBUG source=server.go:1388 msg="completion request" images=0 prompt=369 format=""
time=2025-12-09T14:28:44.407+08:00 level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=224 prompt=72 used=71 remaining=1
[GIN] 2025/12/09 - 14:28:49 | 200 |    7.090913ms |  192.168.11.124 | GET      "/v1/models"
[GIN] 2025/12/09 - 14:28:52 | 200 |  8.082864265s |    192.168.10.0 | POST     "/v1/chat/completions"
time=2025-12-09T14:28:52.346+08:00 level=DEBUG source=sched.go:377 msg="context for request finished" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096
time=2025-12-09T14:28:52.346+08:00 level=DEBUG source=sched.go:286 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 duration=72h0m0s
time=2025-12-09T14:28:52.346+08:00 level=DEBUG source=sched.go:304 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 refCount=0
[GIN] 2025/12/09 - 14:28:59 | 200 |    4.091623ms |  192.168.11.124 | GET      "/v1/models"

0.12.4

time=2025-12-09T14:30:37.103+08:00 level=INFO source=runner.go:1299 msg="starting ollama engine"
time=2025-12-09T14:30:37.103+08:00 level=INFO source=runner.go:1335 msg="Server listening on 127.0.0.1:35449"
time=2025-12-09T14:30:37.114+08:00 level=INFO source=runner.go:1172 msg=load request="{Operation:fit LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:true KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-12-09T14:30:37.152+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
time=2025-12-09T14:30:37.153+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.description default=""
time=2025-12-09T14:30:37.153+08:00 level=INFO source=ggml.go:133 msg="" architecture=qwen3 file_type=Q4_K_M name="Qwen3 32B" description="" num_tensors=707 num_key_values=28
time=2025-12-09T14:30:37.153+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/abc/.local/ollama-0.12.4/lib/ollama
load_backend: loaded CPU backend from /home/abc/.local/ollama-0.12.4/lib/ollama/libggml-cpu-icelake.so
time=2025-12-09T14:30:37.162+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/abc/.local/ollama-0.12.4/lib/ollama/cuda_v12
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3
  Device 1: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92
  Device 2: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5
  Device 3: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da
load_backend: loaded CUDA backend from /home/abc/.local/ollama-0.12.4/lib/ollama/cuda_v12/libggml-cuda.so
time=2025-12-09T14:30:37.387+08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
time=2025-12-09T14:30:37.395+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0
time=2025-12-09T14:30:37.395+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_eos_token default=false
time=2025-12-09T14:30:37.395+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}"
time=2025-12-09T14:30:37.396+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.rope.scaling.factor default=1
time=2025-12-09T14:30:37.396+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_count default=0
time=2025-12-09T14:30:37.396+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_used_count default=0
time=2025-12-09T14:30:37.396+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.norm_top_k_prob default=true
time=2025-12-09T14:30:37.729+08:00 level=DEBUG source=ggml.go:830 msg="compute graph" nodes=2182 splits=2
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:206 msg="model weights" device=CUDA2 size="18.4 GiB"
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:211 msg="model weights" device=CPU size="417.3 MiB"
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:217 msg="kv cache" device=CUDA2 size="4.0 GiB"
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:228 msg="compute graph" device=CUDA2 size="334.0 MiB"
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:233 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:238 msg="total memory" size="23.1 GiB"
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:720 msg=memory success=true required.InputWeights=437575680 required.CPU.Graph=10485760 required.CUDA2.ID=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 required.CUDA2.Weights="[315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 638151680]" required.CUDA2.Cache="[67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 0]" required.CUDA2.Graph=350226560
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 library=CUDA "available layer vram"="53.9 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="334.0 MiB"
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 library=CUDA "available layer vram"="47.8 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da library=CUDA "available layer vram"="25.3 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 library=CUDA "available layer vram"="23.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:731 msg="new layout created" layers="65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)]"
time=2025-12-09T14:30:37.731+08:00 level=INFO source=runner.go:1172 msg=load request="{Operation:alloc LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:true KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-12-09T14:30:37.775+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32
time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0
time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_eos_token default=false
time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}"
time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.rope.scaling.factor default=1
time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_count default=0
time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_used_count default=0
time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.norm_top_k_prob default=true
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=ggml.go:830 msg="compute graph" nodes=2182 splits=2
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:206 msg="model weights" device=CUDA2 size="18.4 GiB"
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:211 msg="model weights" device=CPU size="417.3 MiB"
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:217 msg="kv cache" device=CUDA2 size="4.0 GiB"
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:228 msg="compute graph" device=CUDA2 size="334.0 MiB"
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:233 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:238 msg="total memory" size="23.1 GiB"
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:720 msg=memory success=true required.InputWeights=437575680 required.CPU.Graph=10485760 required.CUDA2.ID=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 required.CUDA2.Weights="[315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 638151680]" required.CUDA2.Cache="[67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 0]" required.CUDA2.Graph=350226560
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 library=CUDA "available layer vram"="53.9 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="334.0 MiB"
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 library=CUDA "available layer vram"="47.8 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da library=CUDA "available layer vram"="25.3 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 library=CUDA "available layer vram"="23.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B"
time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:731 msg="new layout created" layers="65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)]"
time=2025-12-09T14:30:37.812+08:00 level=INFO source=runner.go:1172 msg=load request="{Operation:commit LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:true KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}"
time=2025-12-09T14:30:37.812+08:00 level=INFO source=ggml.go:477 msg="offloading 64 repeating layers to GPU"
time=2025-12-09T14:30:37.812+08:00 level=INFO source=ggml.go:483 msg="offloading output layer to GPU"
time=2025-12-09T14:30:37.812+08:00 level=INFO source=ggml.go:488 msg="offloaded 65/65 layers to GPU"
time=2025-12-09T14:30:37.812+08:00 level=INFO source=device.go:206 msg="model weights" device=CUDA2 size="18.4 GiB"
time=2025-12-09T14:30:37.812+08:00 level=INFO source=device.go:211 msg="model weights" device=CPU size="417.3 MiB"
time=2025-12-09T14:30:37.812+08:00 level=INFO source=device.go:217 msg="kv cache" device=CUDA2 size="4.0 GiB"
time=2025-12-09T14:30:37.813+08:00 level=INFO source=device.go:228 msg="compute graph" device=CUDA2 size="334.0 MiB"
time=2025-12-09T14:30:37.813+08:00 level=INFO source=device.go:233 msg="compute graph" device=CPU size="10.0 MiB"
time=2025-12-09T14:30:37.813+08:00 level=INFO source=device.go:238 msg="total memory" size="23.1 GiB"
time=2025-12-09T14:30:37.813+08:00 level=INFO source=sched.go:481 msg="loaded runners" count=1
time=2025-12-09T14:30:37.813+08:00 level=INFO source=server.go:1271 msg="waiting for llama runner to start responding"
time=2025-12-09T14:30:37.813+08:00 level=INFO source=server.go:1305 msg="waiting for server to become available" status="llm server loading model"
time=2025-12-09T14:30:37.813+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.00"
time=2025-12-09T14:30:38.064+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.08"
time=2025-12-09T14:30:38.315+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.15"
time=2025-12-09T14:30:38.566+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.22"
time=2025-12-09T14:30:38.816+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.29"
time=2025-12-09T14:30:39.067+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.36"
time=2025-12-09T14:30:39.318+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.43"
time=2025-12-09T14:30:39.569+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.51"
time=2025-12-09T14:30:39.820+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.59"
time=2025-12-09T14:30:40.071+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.67"
time=2025-12-09T14:30:40.322+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.74"
time=2025-12-09T14:30:40.573+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.82"
time=2025-12-09T14:30:40.823+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.91"
time=2025-12-09T14:30:41.074+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.98"
time=2025-12-09T14:30:41.325+08:00 level=DEBUG source=server.go:1315 msg="model load progress 1.00"
time=2025-12-09T14:30:41.576+08:00 level=DEBUG source=server.go:1318 msg="model load completed, waiting for server to become available" status="llm server loading model"
time=2025-12-09T14:30:42.012+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0
time=2025-12-09T14:30:42.079+08:00 level=INFO source=server.go:1309 msg="llama runner started in 5.00 seconds"
time=2025-12-09T14:30:42.079+08:00 level=DEBUG source=sched.go:493 msg="finished setting up" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096
time=2025-12-09T14:30:42.134+08:00 level=DEBUG source=server.go:1419 msg="completion request" images=0 prompt=376 format=""
time=2025-12-09T14:30:42.180+08:00 level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=0 prompt=74 used=0 remaining=74
[GIN] 2025/12/09 - 14:30:47 | 200 | 11.667528133s |    192.168.10.0 | POST     "/v1/chat/completions"
time=2025-12-09T14:30:47.181+08:00 level=DEBUG source=sched.go:501 msg="context for request finished"
time=2025-12-09T14:30:47.181+08:00 level=DEBUG source=sched.go:293 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 duration=72h0m0s
time=2025-12-09T14:30:47.182+08:00 level=DEBUG source=sched.go:311 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 refCount=0
[GIN] 2025/12/09 - 14:30:49 | 200 |    8.137969ms |  192.168.11.124 | GET      "/v1/models"
[GIN] 2025/12/09 - 14:30:49 | 200 |     8.18366ms |  192.168.11.124 | GET      "/v1/models"
time=2025-12-09T14:30:54.020+08:00 level=DEBUG source=sched.go:586 msg="evaluating already loaded" model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312
time=2025-12-09T14:30:54.021+08:00 level=DEBUG source=server.go:1419 msg="completion request" images=0 prompt=376 format=""
time=2025-12-09T14:30:54.023+08:00 level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=275 prompt=74 used=73 remaining=1
[GIN] 2025/12/09 - 14:30:59 | 200 |    9.333263ms |  192.168.11.124 | GET      "/v1/models"
[GIN] 2025/12/09 - 14:31:00 | 200 |  7.093718626s |    192.168.10.0 | POST     "/v1/chat/completions"
time=2025-12-09T14:31:00.906+08:00 level=DEBUG source=sched.go:388 msg="context for request finished" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096
time=2025-12-09T14:31:00.906+08:00 level=DEBUG source=sched.go:293 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 duration=72h0m0s
time=2025-12-09T14:31:00.906+08:00 level=DEBUG source=sched.go:311 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 refCount=0
Originally created by @mokeyish on GitHub (Dec 8, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/13372 **Description** After upgrading from Ollama version 0.12.3 to 0.12.4~0.13.2, I observed a noticeable degradation in the first token generation speed when running the `qwen3:32b` model. The delay has increased significantly—by approximately 4–5 seconds or more—compared to the previous version. **Steps to Reproduce** 1. Run `qwen3:32b` on Ollama 0.11.5 and note the time to first token. 2. Upgrade to Ollama 0.13.1. 3. Run the same model (`qwen3:32b`) under identical conditions. 4. Observe the increased latency for the first token generation. **Expected Behavior** First token generation time should remain consistent or improve with newer versions of Ollama. **Actual Behavior** First token generation time has increased by several seconds in version 0.12.4~0.13.2. **Environment** - Ollama Version: 0.12.4~0.13.2 (regression observed from 0.12.3) - Model: `qwen3:32b` - Hardware: A800 GPU - Operating System: Debian bookworm 0.12.3 ```log time=2025-12-09T14:28:02.866+08:00 level=INFO source=runner.go:1252 msg="starting ollama engine" time=2025-12-09T14:28:02.866+08:00 level=INFO source=runner.go:1287 msg="Server listening on 127.0.0.1:39437" time=2025-12-09T14:28:03.198+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 name="NVIDIA A800 80GB PCIe" overhead="0 B" before.total="79.1 GiB" before.free="24.2 GiB" now.total="79.1 GiB" now.free="24.2 GiB" now.used="54.9 GiB" time=2025-12-09T14:28:03.518+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 name="NVIDIA A800 80GB PCIe" overhead="0 B" before.total="79.1 GiB" before.free="48.2 GiB" now.total="79.1 GiB" now.free="48.2 GiB" now.used="30.9 GiB" time=2025-12-09T14:28:03.822+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 name="NVIDIA A800 80GB PCIe" overhead="0 B" before.total="79.1 GiB" before.free="54.7 GiB" now.total="79.1 GiB" now.free="54.7 GiB" now.used="24.5 GiB" time=2025-12-09T14:28:04.318+08:00 level=DEBUG source=gpu.go:460 msg="updating cuda memory data" gpu=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da name="NVIDIA A800 80GB PCIe" overhead="0 B" before.total="79.1 GiB" before.free="25.7 GiB" now.total="79.1 GiB" now.free="25.7 GiB" now.used="53.4 GiB" releasing cuda driver library time=2025-12-09T14:28:04.318+08:00 level=INFO source=server.go:678 msg="system memory" total="1007.5 GiB" free="829.6 GiB" free_swap="0 B" time=2025-12-09T14:28:04.318+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 available="23.7 GiB" free="24.2 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-12-09T14:28:04.318+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 available="47.8 GiB" free="48.2 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-12-09T14:28:04.318+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 available="54.2 GiB" free="54.7 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-12-09T14:28:04.318+08:00 level=INFO source=server.go:686 msg="gpu memory" id=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da available="25.3 GiB" free="25.7 GiB" minimum="457.0 MiB" overhead="0 B" time=2025-12-09T14:28:04.321+08:00 level=INFO source=runner.go:1171 msg=load request="{Operation:fit LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:false KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-12-09T14:28:04.359+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 time=2025-12-09T14:28:04.360+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.description default="" time=2025-12-09T14:28:04.360+08:00 level=INFO source=ggml.go:131 msg="" architecture=qwen3 file_type=Q4_K_M name="Qwen3 32B" description="" num_tensors=707 num_key_values=28 time=2025-12-09T14:28:04.360+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/abc/.local/ollama-0.12.3/lib/ollama load_backend: loaded CPU backend from /home/abc/.local/ollama-0.12.3/lib/ollama/libggml-cpu-icelake.so time=2025-12-09T14:28:04.368+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/abc/.local/ollama-0.12.3/lib/ollama/cuda_v12 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 4 CUDA devices: Device 0: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 Device 1: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 Device 2: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Device 3: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da load_backend: loaded CUDA backend from /home/abc/.local/ollama-0.12.3/lib/ollama/cuda_v12/libggml-cuda.so time=2025-12-09T14:28:04.484+08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0 time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_eos_token default=false time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}" time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.rope.scaling.factor default=1 time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_count default=0 time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_used_count default=0 time=2025-12-09T14:28:04.488+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.norm_top_k_prob default=true time=2025-12-09T14:28:04.739+08:00 level=DEBUG source=ggml.go:794 msg="compute graph" nodes=2438 splits=2 time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:310 msg="model weights" device=CUDA2 size="18.4 GiB" time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:315 msg="model weights" device=CPU size="417.3 MiB" time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:321 msg="kv cache" device=CUDA2 size="4.0 GiB" time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:332 msg="compute graph" device=CUDA2 size="2.1 GiB" time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=backend.go:342 msg="total memory" size="24.9 GiB" time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=server.go:717 msg=memory success=true required.InputWeights=437575680U required.CPU.Graph=10485760U required.CUDA2.ID=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 required.CUDA2.Weights="[315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 281846784U 281846784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 315638784U 638151680U]" required.CUDA2.Cache="[67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 67108864U 0U]" required.CUDA2.Graph=2235566208U time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 "available layer vram"="52.1 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="2.1 GiB" time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 "available layer vram"="47.8 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da "available layer vram"="25.3 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:28:04.742+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 "available layer vram"="23.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:28:04.743+08:00 level=DEBUG source=server.go:728 msg="new layout created" layers="65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)]" time=2025-12-09T14:28:04.743+08:00 level=INFO source=runner.go:1171 msg=load request="{Operation:alloc LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:false KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-12-09T14:28:04.802+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 time=2025-12-09T14:28:04.815+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0 time=2025-12-09T14:28:04.815+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_eos_token default=false time=2025-12-09T14:28:04.815+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}" time=2025-12-09T14:28:04.816+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.rope.scaling.factor default=1 time=2025-12-09T14:28:04.816+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_count default=0 time=2025-12-09T14:28:04.816+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_used_count default=0 time=2025-12-09T14:28:04.816+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.norm_top_k_prob default=true time=2025-12-09T14:28:04.851+08:00 level=DEBUG source=ggml.go:794 msg="compute graph" nodes=2438 splits=2 time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:310 msg="model weights" device=CUDA2 size="18.4 GiB" time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:315 msg="model weights" device=CPU size="417.3 MiB" time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:321 msg="kv cache" device=CUDA2 size="4.0 GiB" time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:332 msg="compute graph" device=CUDA2 size="2.1 GiB" time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=backend.go:342 msg="total memory" size="24.9 GiB" time=2025-12-09T14:28:04.853+08:00 level=DEBUG source=server.go:717 msg=memory success=true required.InputWeights=437575680A required.CPU.Graph=10485760A required.CUDA2.ID=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 required.CUDA2.Weights="[315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 281846784A 281846784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 315638784A 638151680A]" required.CUDA2.Cache="[67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 67108864A 0U]" required.CUDA2.Graph=2235566208A time=2025-12-09T14:28:04.854+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 "available layer vram"="52.1 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="2.1 GiB" time=2025-12-09T14:28:04.854+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 "available layer vram"="47.8 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:28:04.854+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da "available layer vram"="25.3 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:28:04.854+08:00 level=DEBUG source=server.go:894 msg="available gpu" id=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 "available layer vram"="23.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:28:04.855+08:00 level=DEBUG source=server.go:728 msg="new layout created" layers="65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)]" time=2025-12-09T14:28:04.855+08:00 level=INFO source=runner.go:1171 msg=load request="{Operation:commit LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:false KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-12-09T14:28:04.856+08:00 level=INFO source=ggml.go:487 msg="offloading 64 repeating layers to GPU" time=2025-12-09T14:28:04.856+08:00 level=INFO source=ggml.go:493 msg="offloading output layer to GPU" time=2025-12-09T14:28:04.856+08:00 level=INFO source=ggml.go:498 msg="offloaded 65/65 layers to GPU" time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:310 msg="model weights" device=CUDA2 size="18.4 GiB" time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:315 msg="model weights" device=CPU size="417.3 MiB" time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:321 msg="kv cache" device=CUDA2 size="4.0 GiB" time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:332 msg="compute graph" device=CUDA2 size="2.1 GiB" time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:337 msg="compute graph" device=CPU size="10.0 MiB" time=2025-12-09T14:28:04.858+08:00 level=INFO source=backend.go:342 msg="total memory" size="24.9 GiB" time=2025-12-09T14:28:04.858+08:00 level=INFO source=sched.go:470 msg="loaded runners" count=1 time=2025-12-09T14:28:04.858+08:00 level=INFO source=server.go:1251 msg="waiting for llama runner to start responding" time=2025-12-09T14:28:04.870+08:00 level=INFO source=server.go:1285 msg="waiting for server to become available" status="llm server loading model" time=2025-12-09T14:28:04.870+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.00" time=2025-12-09T14:28:05.121+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.09" time=2025-12-09T14:28:05.372+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.15" time=2025-12-09T14:28:05.623+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.21" time=2025-12-09T14:28:05.874+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.26" time=2025-12-09T14:28:06.125+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.31" time=2025-12-09T14:28:06.376+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.36" time=2025-12-09T14:28:06.627+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.41" time=2025-12-09T14:28:06.878+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.47" time=2025-12-09T14:28:07.128+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.54" time=2025-12-09T14:28:07.379+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.60" time=2025-12-09T14:28:07.630+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.66" time=2025-12-09T14:28:07.881+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.71" time=2025-12-09T14:28:08.132+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.77" time=2025-12-09T14:28:08.383+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.83" time=2025-12-09T14:28:08.634+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.90" time=2025-12-09T14:28:08.885+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.95" time=2025-12-09T14:28:09.136+08:00 level=DEBUG source=server.go:1295 msg="model load progress 0.99" time=2025-12-09T14:28:09.387+08:00 level=DEBUG source=server.go:1295 msg="model load progress 1.00" [GIN] 2025/12/09 - 14:28:09 | 200 | 4.824238ms | 192.168.11.124 | GET "/v1/models" time=2025-12-09T14:28:09.638+08:00 level=DEBUG source=server.go:1298 msg="model load completed, waiting for server to become available" status="llm server loading model" time=2025-12-09T14:28:09.937+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0 time=2025-12-09T14:28:10.142+08:00 level=INFO source=server.go:1289 msg="llama runner started in 7.29 seconds" time=2025-12-09T14:28:10.142+08:00 level=DEBUG source=sched.go:482 msg="finished setting up" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 time=2025-12-09T14:28:10.167+08:00 level=DEBUG source=server.go:1388 msg="completion request" images=0 prompt=369 format="" time=2025-12-09T14:28:10.218+08:00 level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=0 prompt=72 used=0 remaining=72 [GIN] 2025/12/09 - 14:28:14 | 200 | 14.896581218s | 192.168.10.0 | POST "/v1/chat/completions" time=2025-12-09T14:28:14.598+08:00 level=DEBUG source=sched.go:490 msg="context for request finished" time=2025-12-09T14:28:14.598+08:00 level=DEBUG source=sched.go:286 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 duration=72h0m0s time=2025-12-09T14:28:14.598+08:00 level=DEBUG source=sched.go:304 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 refCount=0 [GIN] 2025/12/09 - 14:28:19 | 200 | 8.523088ms | 192.168.11.124 | GET "/v1/models" [GIN] 2025/12/09 - 14:28:29 | 200 | 8.604608ms | 192.168.11.124 | GET "/v1/models" [GIN] 2025/12/09 - 14:28:29 | 200 | 5.173909ms | 192.168.11.124 | GET "/v1/models" [GIN] 2025/12/09 - 14:28:39 | 200 | 6.666058ms | 192.168.11.124 | GET "/v1/models" time=2025-12-09T14:28:44.404+08:00 level=DEBUG source=sched.go:580 msg="evaluating already loaded" model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 time=2025-12-09T14:28:44.405+08:00 level=DEBUG source=server.go:1388 msg="completion request" images=0 prompt=369 format="" time=2025-12-09T14:28:44.407+08:00 level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=224 prompt=72 used=71 remaining=1 [GIN] 2025/12/09 - 14:28:49 | 200 | 7.090913ms | 192.168.11.124 | GET "/v1/models" [GIN] 2025/12/09 - 14:28:52 | 200 | 8.082864265s | 192.168.10.0 | POST "/v1/chat/completions" time=2025-12-09T14:28:52.346+08:00 level=DEBUG source=sched.go:377 msg="context for request finished" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 time=2025-12-09T14:28:52.346+08:00 level=DEBUG source=sched.go:286 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 duration=72h0m0s time=2025-12-09T14:28:52.346+08:00 level=DEBUG source=sched.go:304 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference=cuda runner.devices=4 runner.size="24.9 GiB" runner.vram="24.9 GiB" runner.parallel=4 runner.pid=1198268 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 refCount=0 [GIN] 2025/12/09 - 14:28:59 | 200 | 4.091623ms | 192.168.11.124 | GET "/v1/models" ``` 0.12.4 ```log time=2025-12-09T14:30:37.103+08:00 level=INFO source=runner.go:1299 msg="starting ollama engine" time=2025-12-09T14:30:37.103+08:00 level=INFO source=runner.go:1335 msg="Server listening on 127.0.0.1:35449" time=2025-12-09T14:30:37.114+08:00 level=INFO source=runner.go:1172 msg=load request="{Operation:fit LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:true KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-12-09T14:30:37.152+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 time=2025-12-09T14:30:37.153+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.description default="" time=2025-12-09T14:30:37.153+08:00 level=INFO source=ggml.go:133 msg="" architecture=qwen3 file_type=Q4_K_M name="Qwen3 32B" description="" num_tensors=707 num_key_values=28 time=2025-12-09T14:30:37.153+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/abc/.local/ollama-0.12.4/lib/ollama load_backend: loaded CPU backend from /home/abc/.local/ollama-0.12.4/lib/ollama/libggml-cpu-icelake.so time=2025-12-09T14:30:37.162+08:00 level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/home/abc/.local/ollama-0.12.4/lib/ollama/cuda_v12 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 4 CUDA devices: Device 0: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 Device 1: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 Device 2: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Device 3: NVIDIA A800 80GB PCIe, compute capability 8.0, VMM: yes, ID: GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da load_backend: loaded CUDA backend from /home/abc/.local/ollama-0.12.4/lib/ollama/cuda_v12/libggml-cuda.so time=2025-12-09T14:30:37.387+08:00 level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 CUDA.0.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.0.USE_GRAPHS=1 CUDA.0.PEER_MAX_BATCH_SIZE=128 CUDA.1.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.1.USE_GRAPHS=1 CUDA.1.PEER_MAX_BATCH_SIZE=128 CUDA.2.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.2.USE_GRAPHS=1 CUDA.2.PEER_MAX_BATCH_SIZE=128 CUDA.3.ARCHS=500,600,610,700,750,800,860,870,890,900,1200 CUDA.3.USE_GRAPHS=1 CUDA.3.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) time=2025-12-09T14:30:37.395+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0 time=2025-12-09T14:30:37.395+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_eos_token default=false time=2025-12-09T14:30:37.395+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}" time=2025-12-09T14:30:37.396+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.rope.scaling.factor default=1 time=2025-12-09T14:30:37.396+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_count default=0 time=2025-12-09T14:30:37.396+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_used_count default=0 time=2025-12-09T14:30:37.396+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.norm_top_k_prob default=true time=2025-12-09T14:30:37.729+08:00 level=DEBUG source=ggml.go:830 msg="compute graph" nodes=2182 splits=2 time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:206 msg="model weights" device=CUDA2 size="18.4 GiB" time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:211 msg="model weights" device=CPU size="417.3 MiB" time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:217 msg="kv cache" device=CUDA2 size="4.0 GiB" time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:228 msg="compute graph" device=CUDA2 size="334.0 MiB" time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:233 msg="compute graph" device=CPU size="10.0 MiB" time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=device.go:238 msg="total memory" size="23.1 GiB" time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:720 msg=memory success=true required.InputWeights=437575680 required.CPU.Graph=10485760 required.CUDA2.ID=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 required.CUDA2.Weights="[315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 638151680]" required.CUDA2.Cache="[67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 0]" required.CUDA2.Graph=350226560 time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 library=CUDA "available layer vram"="53.9 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="334.0 MiB" time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 library=CUDA "available layer vram"="47.8 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da library=CUDA "available layer vram"="25.3 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 library=CUDA "available layer vram"="23.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:30:37.730+08:00 level=DEBUG source=server.go:731 msg="new layout created" layers="65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)]" time=2025-12-09T14:30:37.731+08:00 level=INFO source=runner.go:1172 msg=load request="{Operation:alloc LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:true KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-12-09T14:30:37.775+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=general.alignment default=32 time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0 time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.add_eos_token default=false time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=tokenizer.ggml.eos_token_ids default="&{size:0 values:[]}" time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.rope.scaling.factor default=1 time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_count default=0 time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.expert_used_count default=0 time=2025-12-09T14:30:37.781+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.norm_top_k_prob default=true time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=ggml.go:830 msg="compute graph" nodes=2182 splits=2 time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:206 msg="model weights" device=CUDA2 size="18.4 GiB" time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:211 msg="model weights" device=CPU size="417.3 MiB" time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:217 msg="kv cache" device=CUDA2 size="4.0 GiB" time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:228 msg="compute graph" device=CUDA2 size="334.0 MiB" time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:233 msg="compute graph" device=CPU size="10.0 MiB" time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=device.go:238 msg="total memory" size="23.1 GiB" time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:720 msg=memory success=true required.InputWeights=437575680 required.CPU.Graph=10485760 required.CUDA2.ID=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 required.CUDA2.Weights="[315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 281846784 281846784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 315638784 638151680]" required.CUDA2.Cache="[67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 67108864 0]" required.CUDA2.Graph=350226560 time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 library=CUDA "available layer vram"="53.9 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="334.0 MiB" time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-5b033b1c-5b13-a797-54f9-43db2f7b6b92 library=CUDA "available layer vram"="47.8 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-24358e7f-3aa7-bd48-32c2-dd910ad4b1da library=CUDA "available layer vram"="25.3 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:914 msg="available gpu" id=GPU-d68effe8-f2e2-a3a5-15ff-ffda541cd9e3 library=CUDA "available layer vram"="23.7 GiB" backoff=0.00 minimum="457.0 MiB" overhead="0 B" graph="0 B" time=2025-12-09T14:30:37.811+08:00 level=DEBUG source=server.go:731 msg="new layout created" layers="65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)]" time=2025-12-09T14:30:37.812+08:00 level=INFO source=runner.go:1172 msg=load request="{Operation:commit LoraPath:[] Parallel:4 BatchSize:512 FlashAttention:true KvSize:16384 KvCacheType: NumThreads:56 GPULayers:65[ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Layers:65(0..64)] MultiUserCache:false ProjectorPath: MainGPU:0 UseMmap:false}" time=2025-12-09T14:30:37.812+08:00 level=INFO source=ggml.go:477 msg="offloading 64 repeating layers to GPU" time=2025-12-09T14:30:37.812+08:00 level=INFO source=ggml.go:483 msg="offloading output layer to GPU" time=2025-12-09T14:30:37.812+08:00 level=INFO source=ggml.go:488 msg="offloaded 65/65 layers to GPU" time=2025-12-09T14:30:37.812+08:00 level=INFO source=device.go:206 msg="model weights" device=CUDA2 size="18.4 GiB" time=2025-12-09T14:30:37.812+08:00 level=INFO source=device.go:211 msg="model weights" device=CPU size="417.3 MiB" time=2025-12-09T14:30:37.812+08:00 level=INFO source=device.go:217 msg="kv cache" device=CUDA2 size="4.0 GiB" time=2025-12-09T14:30:37.813+08:00 level=INFO source=device.go:228 msg="compute graph" device=CUDA2 size="334.0 MiB" time=2025-12-09T14:30:37.813+08:00 level=INFO source=device.go:233 msg="compute graph" device=CPU size="10.0 MiB" time=2025-12-09T14:30:37.813+08:00 level=INFO source=device.go:238 msg="total memory" size="23.1 GiB" time=2025-12-09T14:30:37.813+08:00 level=INFO source=sched.go:481 msg="loaded runners" count=1 time=2025-12-09T14:30:37.813+08:00 level=INFO source=server.go:1271 msg="waiting for llama runner to start responding" time=2025-12-09T14:30:37.813+08:00 level=INFO source=server.go:1305 msg="waiting for server to become available" status="llm server loading model" time=2025-12-09T14:30:37.813+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.00" time=2025-12-09T14:30:38.064+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.08" time=2025-12-09T14:30:38.315+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.15" time=2025-12-09T14:30:38.566+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.22" time=2025-12-09T14:30:38.816+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.29" time=2025-12-09T14:30:39.067+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.36" time=2025-12-09T14:30:39.318+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.43" time=2025-12-09T14:30:39.569+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.51" time=2025-12-09T14:30:39.820+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.59" time=2025-12-09T14:30:40.071+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.67" time=2025-12-09T14:30:40.322+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.74" time=2025-12-09T14:30:40.573+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.82" time=2025-12-09T14:30:40.823+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.91" time=2025-12-09T14:30:41.074+08:00 level=DEBUG source=server.go:1315 msg="model load progress 0.98" time=2025-12-09T14:30:41.325+08:00 level=DEBUG source=server.go:1315 msg="model load progress 1.00" time=2025-12-09T14:30:41.576+08:00 level=DEBUG source=server.go:1318 msg="model load completed, waiting for server to become available" status="llm server loading model" time=2025-12-09T14:30:42.012+08:00 level=DEBUG source=ggml.go:276 msg="key with type not found" key=qwen3.pooling_type default=0 time=2025-12-09T14:30:42.079+08:00 level=INFO source=server.go:1309 msg="llama runner started in 5.00 seconds" time=2025-12-09T14:30:42.079+08:00 level=DEBUG source=sched.go:493 msg="finished setting up" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 time=2025-12-09T14:30:42.134+08:00 level=DEBUG source=server.go:1419 msg="completion request" images=0 prompt=376 format="" time=2025-12-09T14:30:42.180+08:00 level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=0 prompt=74 used=0 remaining=74 [GIN] 2025/12/09 - 14:30:47 | 200 | 11.667528133s | 192.168.10.0 | POST "/v1/chat/completions" time=2025-12-09T14:30:47.181+08:00 level=DEBUG source=sched.go:501 msg="context for request finished" time=2025-12-09T14:30:47.181+08:00 level=DEBUG source=sched.go:293 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 duration=72h0m0s time=2025-12-09T14:30:47.182+08:00 level=DEBUG source=sched.go:311 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 refCount=0 [GIN] 2025/12/09 - 14:30:49 | 200 | 8.137969ms | 192.168.11.124 | GET "/v1/models" [GIN] 2025/12/09 - 14:30:49 | 200 | 8.18366ms | 192.168.11.124 | GET "/v1/models" time=2025-12-09T14:30:54.020+08:00 level=DEBUG source=sched.go:586 msg="evaluating already loaded" model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 time=2025-12-09T14:30:54.021+08:00 level=DEBUG source=server.go:1419 msg="completion request" images=0 prompt=376 format="" time=2025-12-09T14:30:54.023+08:00 level=DEBUG source=cache.go:142 msg="loading cache slot" id=0 cache=275 prompt=74 used=73 remaining=1 [GIN] 2025/12/09 - 14:30:59 | 200 | 9.333263ms | 192.168.11.124 | GET "/v1/models" [GIN] 2025/12/09 - 14:31:00 | 200 | 7.093718626s | 192.168.10.0 | POST "/v1/chat/completions" time=2025-12-09T14:31:00.906+08:00 level=DEBUG source=sched.go:388 msg="context for request finished" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 time=2025-12-09T14:31:00.906+08:00 level=DEBUG source=sched.go:293 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 duration=72h0m0s time=2025-12-09T14:31:00.906+08:00 level=DEBUG source=sched.go:311 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3:32b runner.inference="[{ID:GPU-48d70bd0-25eb-03f4-2d88-889026ac56b5 Library:CUDA}]" runner.size="23.1 GiB" runner.vram="23.1 GiB" runner.parallel=4 runner.pid=1222175 runner.model=/home/abc/.ollama/models/blobs/sha256-3291abe70f16ee9682de7bfae08db5373ea9d6497e614aaad63340ad421d6312 runner.num_ctx=4096 refCount=0 ```
GiteaMirror added the bug label 2026-04-12 21:36:51 -05:00
GiteaMirror commented 2026-04-12 21:36:52 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 8, 2025):

Server log from both versions may help in debugging.

<!-- gh-comment-id:3626335835 --> @rick-github commented on GitHub (Dec 8, 2025): [Server log](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.mdx) from both versions may help in debugging.
GiteaMirror commented 2026-04-12 21:36:52 -05:00
Author
Owner
Copy Link

@mokeyish commented on GitHub (Dec 9, 2025):

Server log from both versions may help in debugging.

It has now been confirmed that, starting with version 0.12.4, all subsequent versions are slower than version 0.12.3. @rick-github

<!-- gh-comment-id:3630552145 --> @mokeyish commented on GitHub (Dec 9, 2025): > [Server log](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.mdx) from both versions may help in debugging. It has now been confirmed that, starting with version 0.12.4, all subsequent versions are slower than version 0.12.3. @rick-github
GiteaMirror commented 2026-04-12 21:36:53 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 9, 2025):

0.12.3

time=2025-12-09T14:28:10.142+08:00 level=INFO source=server.go:1289 msg="llama runner started in 7.29 seconds"
[GIN] 2025/12/09 - 14:28:14 | 200 | 14.896581218s |    192.168.10.0 | POST     "/v1/chat/completions"
[GIN] 2025/12/09 - 14:28:52 | 200 |  8.082864265s |    192.168.10.0 | POST     "/v1/chat/completions"

0.12.4

time=2025-12-09T14:30:42.079+08:00 level=INFO source=server.go:1309 msg="llama runner started in 5.00 seconds"
[GIN] 2025/12/09 - 14:30:47 | 200 | 11.667528133s |    192.168.10.0 | POST     "/v1/chat/completions"
[GIN] 2025/12/09 - 14:31:00 | 200 |  7.093718626s |    192.168.10.0 | POST     "/v1/chat/completions"

Logs show that model load and completions are faster in 0.12.4 than in 0.12.3. How are you measuring time to first token? Can you share the prompt that you used in the tests?

<!-- gh-comment-id:3632075526 --> @rick-github commented on GitHub (Dec 9, 2025): 0.12.3 ``` time=2025-12-09T14:28:10.142+08:00 level=INFO source=server.go:1289 msg="llama runner started in 7.29 seconds" [GIN] 2025/12/09 - 14:28:14 | 200 | 14.896581218s | 192.168.10.0 | POST "/v1/chat/completions" [GIN] 2025/12/09 - 14:28:52 | 200 | 8.082864265s | 192.168.10.0 | POST "/v1/chat/completions" ``` 0.12.4 ``` time=2025-12-09T14:30:42.079+08:00 level=INFO source=server.go:1309 msg="llama runner started in 5.00 seconds" [GIN] 2025/12/09 - 14:30:47 | 200 | 11.667528133s | 192.168.10.0 | POST "/v1/chat/completions" [GIN] 2025/12/09 - 14:31:00 | 200 | 7.093718626s | 192.168.10.0 | POST "/v1/chat/completions" ``` Logs show that model load and completions are faster in 0.12.4 than in 0.12.3. How are you measuring time to first token? Can you share the prompt that you used in the tests?
GiteaMirror commented 2026-04-12 21:36:53 -05:00
Author
Owner
Copy Link

@mokeyish commented on GitHub (Dec 9, 2025):

This log is a bit confusing. Currently, it appears to be the same on both the A800 and Nvidia DGX Spark, with 0.12.3 being the last faster version. The first request includes model startup time, so the main test is the second request to ensure the model is fully started before sending the next request. The prompt message is "Who are you?", in streaming mode.

<!-- gh-comment-id:3633416379 --> @mokeyish commented on GitHub (Dec 9, 2025): This log is a bit confusing. Currently, it appears to be the same on both the A800 and Nvidia DGX Spark, with 0.12.3 being the last faster version. The first request includes model startup time, so the main test is the second request to ensure the model is fully started before sending the next request. The prompt message is "Who are you?", in streaming mode.
GiteaMirror commented 2026-04-12 21:36:53 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 9, 2025):

How are you measuring time to first token? What client are you using too send the query "Who are you?"?

<!-- gh-comment-id:3633861619 --> @rick-github commented on GitHub (Dec 9, 2025): How are you measuring time to first token? What client are you using too send the query "Who are you?"?
GiteaMirror commented 2026-04-12 21:36:54 -05:00
Author
Owner
Copy Link

@mokeyish commented on GitHub (Dec 9, 2025):

How are you measuring time to first token? What client are you using too send the query "Who are you?"?

https://github.com/ChatGPTNextWeb/NextChat

<!-- gh-comment-id:3634739566 --> @mokeyish commented on GitHub (Dec 9, 2025): > How are you measuring time to first token? What client are you using too send the query "Who are you?"? https://github.com/ChatGPTNextWeb/NextChat
GiteaMirror commented 2026-04-12 21:36:55 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 10, 2025):

I suspect that what is happening is that the model is thinking and this is not being rendered in your client. Specifically, the client is searching for reasoning_content:

c3b8c1587c/app/client/platforms/openai.ts (L355)

But ollama returns reasoning:

$ curl -s localhost:11434/v1/chat/completions -d '{"model":"qwen3:32b","messages":[{"role":"user","content":"Who are you?"}]}' | jq
{
  "id": "chatcmpl-746",
  "object": "chat.completion",
  "created": 1765323910,
  "model": "qwen3:32b",
  "system_fingerprint": "fp_ollama",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I am Qwen3, a large language model developed by Alibaba Cloud. My purpose is to assist users in various ways, such as answering questions, creating content, and more. Here are some of my key features:\n\n1. **Multilingual Support**：I can communicate in multiple languages, including Chinese, English, French, Spanish, Portuguese, Russian, Arabic, Japanese, Korean, and many others.\n\n2. **Vast Knowledge**：I have learned a lot of information and can help with various topics, whether it be general knowledge, science and technology, or culture and society.\n\n3. **Versatile Creation**：I can help you write stories, emails, scripts, official documents, and more.\n\n4. **Logical Reasoning**：I can perform logical reasoning, coding, and solve complex problems.\n\n5. **Cultural Sensitivity**：I adapt well to different cultural contexts and can provide culturally relevant responses.\n\n6. **Dialogue Understanding**：I support multi-turn conversations and can remember the history to better understand your needs and provide personalized services.\n\n7. **Code Writing**：I can write and understand code in multiple programming languages.\n\n8. **Creative Thinking**：I can assist in coming up with creative ideas and solving problems from new angles.\n\nIf you have any questions or need help with anything, feel free to ask me anytime! I'm here to help you 😊",
        "reasoning": "Well, the user is asking, \"Who are you?\" I need to explain who I am, but in a simple and natural way. I should start by stating that I'm Qwen, a large language model developed by Alibaba Cloud. Next, I should mention my purpose: to assist users in answering questions, creating content, or other tasks. Then, I should list some of my capabilities, such as answering questions, writing stories, official documents, emails, scripts, logical reasoning, coding, and more. Also, I can express views and play games. However, since the user is asking about my identity, I should focus more on self-introduction, but also highlight my key functions. I need to maintain a friendly and helpful tone, avoiding overly technical terms so that beginners can understand easily. Additionally, the user might not just want a list of functions; they may also want to know how I can serve them, so I should encourage them to ask for further assistance and mention that I can provide information or perform tasks. I should keep the response structured and clear without using complex formatting, while ensuring the response is comprehensive but not too lengthy. Also, I should remember to translate my response into English if needed, but since the user is Chinese, the response should be in Chinese.\n"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 548,
    "total_tokens": 562
  }
}

See https://github.com/ollama/ollama/issues/13175 for an open issue about this.

The reason it became an issue in 0.12.4 is that thinking-capable models had thinking enabled in this and subsequent releases, while earlier releases had thinking disabled

The ollama API allows controlling the thinking but the OpenAI compatibility endpoint does not. A quick and dirty workaround is to disable thinking in the model, so it acts like pre-0.12.4:

$ ollama show --modelfile qwen3:32b | sed -e 's/\$.Think/(and & false)/' > Modelfile
$ ollama create qwen3:32b-nothink -f Modelfile

$ curl -s localhost:11434/v1/chat/completions -d '{"model":"qwen3:32b-nothink","messages":[{"role":"user","content":"Who are you?"}]}' | jq
{
  "id": "chatcmpl-286",
  "object": "chat.completion",
  "created": 1765325875,
  "model": "qwen3:32b-nothink",
  "system_fingerprint": "fp_ollama",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I am Qwen, a large-scale language model independently developed by the Tongyi Lab under Alibaba Group. I am designed to assist you with answering questions, creating text (such as stories, official documents, emails, and scripts), providing logical reasoning, programming, and other tasks. If you have any questions or need help, feel free to ask me anytime!"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 75,
    "total_tokens": 95
  }
}
<!-- gh-comment-id:3634833942 --> @rick-github commented on GitHub (Dec 10, 2025): I suspect that what is happening is that the model is thinking and this is not being rendered in your client. Specifically, the client is searching for `reasoning_content`: https://github.com/ChatGPTNextWeb/NextChat/blob/c3b8c1587c04fff05f7b42276a43016e87771527/app/client/platforms/openai.ts#L355 But ollama returns `reasoning`: ```console $ curl -s localhost:11434/v1/chat/completions -d '{"model":"qwen3:32b","messages":[{"role":"user","content":"Who are you?"}]}' | jq { "id": "chatcmpl-746", "object": "chat.completion", "created": 1765323910, "model": "qwen3:32b", "system_fingerprint": "fp_ollama", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hello! I am Qwen3, a large language model developed by Alibaba Cloud. My purpose is to assist users in various ways, such as answering questions, creating content, and more. Here are some of my key features:\n\n1. **Multilingual Support**：I can communicate in multiple languages, including Chinese, English, French, Spanish, Portuguese, Russian, Arabic, Japanese, Korean, and many others.\n\n2. **Vast Knowledge**：I have learned a lot of information and can help with various topics, whether it be general knowledge, science and technology, or culture and society.\n\n3. **Versatile Creation**：I can help you write stories, emails, scripts, official documents, and more.\n\n4. **Logical Reasoning**：I can perform logical reasoning, coding, and solve complex problems.\n\n5. **Cultural Sensitivity**：I adapt well to different cultural contexts and can provide culturally relevant responses.\n\n6. **Dialogue Understanding**：I support multi-turn conversations and can remember the history to better understand your needs and provide personalized services.\n\n7. **Code Writing**：I can write and understand code in multiple programming languages.\n\n8. **Creative Thinking**：I can assist in coming up with creative ideas and solving problems from new angles.\n\nIf you have any questions or need help with anything, feel free to ask me anytime! I'm here to help you 😊", "reasoning": "Well, the user is asking, \"Who are you?\" I need to explain who I am, but in a simple and natural way. I should start by stating that I'm Qwen, a large language model developed by Alibaba Cloud. Next, I should mention my purpose: to assist users in answering questions, creating content, or other tasks. Then, I should list some of my capabilities, such as answering questions, writing stories, official documents, emails, scripts, logical reasoning, coding, and more. Also, I can express views and play games. However, since the user is asking about my identity, I should focus more on self-introduction, but also highlight my key functions. I need to maintain a friendly and helpful tone, avoiding overly technical terms so that beginners can understand easily. Additionally, the user might not just want a list of functions; they may also want to know how I can serve them, so I should encourage them to ask for further assistance and mention that I can provide information or perform tasks. I should keep the response structured and clear without using complex formatting, while ensuring the response is comprehensive but not too lengthy. Also, I should remember to translate my response into English if needed, but since the user is Chinese, the response should be in Chinese.\n" }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 14, "completion_tokens": 548, "total_tokens": 562 } } ``` See https://github.com/ollama/ollama/issues/13175 for an open issue about this. The reason it became an issue in 0.12.4 is that thinking-capable models had thinking enabled in this and subsequent releases, while earlier releases had thinking disabled The ollama API allows controlling the thinking but the OpenAI compatibility endpoint does not. A quick and dirty workaround is to disable thinking in the model, so it acts like pre-0.12.4: ```console $ ollama show --modelfile qwen3:32b | sed -e 's/\$.Think/(and & false)/' > Modelfile $ ollama create qwen3:32b-nothink -f Modelfile ``` ```console $ curl -s localhost:11434/v1/chat/completions -d '{"model":"qwen3:32b-nothink","messages":[{"role":"user","content":"Who are you?"}]}' | jq { "id": "chatcmpl-286", "object": "chat.completion", "created": 1765325875, "model": "qwen3:32b-nothink", "system_fingerprint": "fp_ollama", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hello! I am Qwen, a large-scale language model independently developed by the Tongyi Lab under Alibaba Group. I am designed to assist you with answering questions, creating text (such as stories, official documents, emails, and scripts), providing logical reasoning, programming, and other tasks. If you have any questions or need help, feel free to ask me anytime!" }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 20, "completion_tokens": 75, "total_tokens": 95 } } ```
GiteaMirror commented 2026-04-12 21:36:55 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 10, 2025):

I just realized that thinking can be controlled by an OpenAI client, using the reasoning_effort field:

$ curl -s localhost:11434/v1/chat/completions -d '{"model":"qwen3:32b","messages":[{"role":"user","content":"Who are you?"}],"reasoning_effort":"none"}' | jq
{
  "id": "chatcmpl-847",
  "object": "chat.completion",
  "created": 1765328456,
  "model": "qwen3:32b",
  "system_fingerprint": "fp_ollama",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I am Qwen, a large-scale language model developed by Alibaba Cloud's Tongyi Lab. I am designed to answer questions, create text, support multiple languages, logical reasoning, coding, and more. I can also express views and play games. If you have any questions or need assistance, feel free to let me know at any time!"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 72,
    "total_tokens": 92
  }
}

Sadly this is not supported in NextChat.

<!-- gh-comment-id:3634924046 --> @rick-github commented on GitHub (Dec 10, 2025): I just realized that thinking can be controlled by an OpenAI client, using the `reasoning_effort` field: ```console $ curl -s localhost:11434/v1/chat/completions -d '{"model":"qwen3:32b","messages":[{"role":"user","content":"Who are you?"}],"reasoning_effort":"none"}' | jq { "id": "chatcmpl-847", "object": "chat.completion", "created": 1765328456, "model": "qwen3:32b", "system_fingerprint": "fp_ollama", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hello! I am Qwen, a large-scale language model developed by Alibaba Cloud's Tongyi Lab. I am designed to answer questions, create text, support multiple languages, logical reasoning, coding, and more. I can also express views and play games. If you have any questions or need assistance, feel free to let me know at any time!" }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 20, "completion_tokens": 72, "total_tokens": 92 } } ``` Sadly this is [not supported](https://github.com/ChatGPTNextWeb/NextChat/issues/6032) in NextChat.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#8830
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?