[GH-ISSUE #10999] 0.9.0 too slowly #7252

New Issue

GiteaMirror · 2026-04-12T19:17:50-05:00

GiteaMirror commented

2026-04-12 19:17:50 -05:00

Originally created by @goactiongo on GitHub (Jun 6, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10999

What is the issue?

0.9.0, with 3 L30 card.
testing with qwen3:0.6b,it's so slowly,

(base) root@ai001:/data/tools# journalctl -u ollama -f
Jun 06 17:19:31 ai001 ollama[518336]: [GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c] CUDA freeMem 43488mb
Jun 06 17:19:31 ai001 ollama[518336]: [GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c] Compute Capability 8.9
Jun 06 17:19:31 ai001 ollama[518336]: [GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90] CUDA totalMem 45457mb
Jun 06 17:19:31 ai001 ollama[518336]: [GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90] CUDA freeMem 45165mb
Jun 06 17:19:31 ai001 ollama[518336]: [GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90] Compute Capability 8.9
Jun 06 17:19:32 ai001 ollama[518336]: time=2025-06-06T17:19:32.055Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu"
Jun 06 17:19:32 ai001 ollama[518336]: releasing cuda driver library
Jun 06 17:19:32 ai001 ollama[518336]: time=2025-06-06T17:19:32.055Z level=INFO source=types.go:130 msg="inference compute" id=GPU-59d819ff-4712-49e4-5e5a-ba8f5c21e829 library=cuda variant=v12 compute=8.9 driver=12.9 name="NVIDIA L20" total="44.4 GiB" available="40.0 GiB"
Jun 06 17:19:32 ai001 ollama[518336]: time=2025-06-06T17:19:32.055Z level=INFO source=types.go:130 msg="inference compute" id=GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c library=cuda variant=v12 compute=8.9 driver=12.9 name="NVIDIA L20" total="44.4 GiB" available="42.5 GiB"
Jun 06 17:19:32 ai001 ollama[518336]: time=2025-06-06T17:19:32.055Z level=INFO source=types.go:130 msg="inference compute" id=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 library=cuda variant=v12 compute=8.9 driver=12.9 name="NVIDIA L20" total="44.4 GiB" available="44.1 GiB"






Jun 06 17:20:49 ai001 ollama[518336]: time=2025-06-06T17:20:49.642Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32
Jun 06 17:20:49 ai001 ollama[518336]: time=2025-06-06T17:20:49.643Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="376.6 GiB" before.free="364.9 GiB" before.free_swap="8.0 GiB" now.total="376.6 GiB" now.free="364.9 GiB" now.free_swap="8.0 GiB"
Jun 06 17:20:49 ai001 ollama[518336]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08
Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuInit - 0x7f0342af9680
Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDriverGetVersion - 0x7f0342af9740
Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDeviceGetCount - 0x7f0342af98c0
Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDeviceGet - 0x7f0342af9800
Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDeviceGetAttribute - 0x7f0342af9e00
Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDeviceGetUuid - 0x7f0342af9a40
Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDeviceGetName - 0x7f0342af9980
Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuCtxCreate_v3 - 0x7f0342afa940
Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuMemGetInfo_v2 - 0x7f0342afd5e0
Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuCtxDestroy - 0x7f0342b5f4e0
Jun 06 17:20:49 ai001 ollama[518336]: calling cuInit
Jun 06 17:20:49 ai001 ollama[518336]: calling cuDriverGetVersion
Jun 06 17:20:49 ai001 ollama[518336]: raw version 0x2f3a
Jun 06 17:20:49 ai001 ollama[518336]: CUDA driver version: 12.9
Jun 06 17:20:49 ai001 ollama[518336]: calling cuDeviceGetCount
Jun 06 17:20:49 ai001 ollama[518336]: device count 3
Jun 06 17:20:49 ai001 ollama[518336]: time=2025-06-06T17:20:49.819Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-59d819ff-4712-49e4-5e5a-ba8f5c21e829 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="40.0 GiB" now.total="44.4 GiB" now.free="40.0 GiB" now.used="4.4 GiB"
Jun 06 17:20:49 ai001 ollama[518336]: time=2025-06-06T17:20:49.975Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="42.5 GiB" now.total="44.4 GiB" now.free="42.5 GiB" now.used="1.9 GiB"
Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.584Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="44.1 GiB" now.total="44.4 GiB" now.free="44.1 GiB" now.used="292.2 MiB"
Jun 06 17:20:50 ai001 ollama[518336]: releasing cuda driver library
Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.584Z level=DEBUG source=sched.go:185 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=9 gpu_count=3
Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.612Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32
Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.636Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32
Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.637Z level=DEBUG source=sched.go:228 msg="loading first model" model=/usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa
Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.637Z level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=1 available="[44.1 GiB]"
Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.637Z level=DEBUG source=ggml.go:155 msg="key not found" key=qwen3.vision.block_count default=0
Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.637Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="376.6 GiB" before.free="364.9 GiB" before.free_swap="8.0 GiB" now.total="376.6 GiB" now.free="364.9 GiB" now.free_swap="8.0 GiB"
Jun 06 17:20:50 ai001 ollama[518336]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08
Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuInit - 0x7f0342af9680
Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDriverGetVersion - 0x7f0342af9740
Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDeviceGetCount - 0x7f0342af98c0
Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDeviceGet - 0x7f0342af9800
Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDeviceGetAttribute - 0x7f0342af9e00
Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDeviceGetUuid - 0x7f0342af9a40
Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDeviceGetName - 0x7f0342af9980
Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuCtxCreate_v3 - 0x7f0342afa940
Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuMemGetInfo_v2 - 0x7f0342afd5e0
Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuCtxDestroy - 0x7f0342b5f4e0
Jun 06 17:20:50 ai001 ollama[518336]: calling cuInit
Jun 06 17:20:50 ai001 ollama[518336]: calling cuDriverGetVersion
Jun 06 17:20:50 ai001 ollama[518336]: raw version 0x2f3a
Jun 06 17:20:50 ai001 ollama[518336]: CUDA driver version: 12.9
Jun 06 17:20:50 ai001 ollama[518336]: calling cuDeviceGetCount
Jun 06 17:20:50 ai001 ollama[518336]: device count 3
Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.801Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-59d819ff-4712-49e4-5e5a-ba8f5c21e829 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="40.0 GiB" now.total="44.4 GiB" now.free="40.0 GiB" now.used="4.4 GiB"
Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.952Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="42.5 GiB" now.total="44.4 GiB" now.free="42.5 GiB" now.used="1.9 GiB"
Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.097Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="44.1 GiB" now.total="44.4 GiB" now.free="44.1 GiB" now.used="292.2 MiB"
Jun 06 17:20:51 ai001 ollama[518336]: releasing cuda driver library
Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.097Z level=INFO source=sched.go:788 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa gpu=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 parallel=2 available=47359262720 required="2.1 GiB"
Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.097Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="376.6 GiB" before.free="364.9 GiB" before.free_swap="8.0 GiB" now.total="376.6 GiB" now.free="364.9 GiB" now.free_swap="8.0 GiB"
Jun 06 17:20:51 ai001 ollama[518336]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuInit - 0x7f0342af9680
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDriverGetVersion - 0x7f0342af9740
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetCount - 0x7f0342af98c0
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGet - 0x7f0342af9800
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetAttribute - 0x7f0342af9e00
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetUuid - 0x7f0342af9a40
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetName - 0x7f0342af9980
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuCtxCreate_v3 - 0x7f0342afa940
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuMemGetInfo_v2 - 0x7f0342afd5e0
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuCtxDestroy - 0x7f0342b5f4e0
Jun 06 17:20:51 ai001 ollama[518336]: calling cuInit
Jun 06 17:20:51 ai001 ollama[518336]: calling cuDriverGetVersion
Jun 06 17:20:51 ai001 ollama[518336]: raw version 0x2f3a
Jun 06 17:20:51 ai001 ollama[518336]: CUDA driver version: 12.9
Jun 06 17:20:51 ai001 ollama[518336]: calling cuDeviceGetCount
Jun 06 17:20:51 ai001 ollama[518336]: device count 3
Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.248Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-59d819ff-4712-49e4-5e5a-ba8f5c21e829 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="40.0 GiB" now.total="44.4 GiB" now.free="40.0 GiB" now.used="4.4 GiB"
Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.394Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="42.5 GiB" now.total="44.4 GiB" now.free="42.5 GiB" now.used="1.9 GiB"
Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.544Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="44.1 GiB" now.total="44.4 GiB" now.free="44.1 GiB" now.used="292.2 MiB"
Jun 06 17:20:51 ai001 ollama[518336]: releasing cuda driver library
Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.544Z level=INFO source=server.go:135 msg="system memory" total="376.6 GiB" free="364.9 GiB" free_swap="8.0 GiB"
Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.544Z level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=1 available="[44.1 GiB]"
Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.544Z level=DEBUG source=ggml.go:155 msg="key not found" key=qwen3.vision.block_count default=0
Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.545Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="376.6 GiB" before.free="364.9 GiB" before.free_swap="8.0 GiB" now.total="376.6 GiB" now.free="364.9 GiB" now.free_swap="8.0 GiB"
Jun 06 17:20:51 ai001 ollama[518336]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuInit - 0x7f0342af9680
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDriverGetVersion - 0x7f0342af9740
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetCount - 0x7f0342af98c0
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGet - 0x7f0342af9800
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetAttribute - 0x7f0342af9e00
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetUuid - 0x7f0342af9a40
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetName - 0x7f0342af9980
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuCtxCreate_v3 - 0x7f0342afa940
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuMemGetInfo_v2 - 0x7f0342afd5e0
Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuCtxDestroy - 0x7f0342b5f4e0
Jun 06 17:20:51 ai001 ollama[518336]: calling cuInit
Jun 06 17:20:51 ai001 ollama[518336]: calling cuDriverGetVersion
Jun 06 17:20:51 ai001 ollama[518336]: raw version 0x2f3a
Jun 06 17:20:51 ai001 ollama[518336]: CUDA driver version: 12.9
Jun 06 17:20:51 ai001 ollama[518336]: calling cuDeviceGetCount
Jun 06 17:20:51 ai001 ollama[518336]: device count 3
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.145Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-59d819ff-4712-49e4-5e5a-ba8f5c21e829 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="40.0 GiB" now.total="44.4 GiB" now.free="40.0 GiB" now.used="4.4 GiB"
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.291Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="42.5 GiB" now.total="44.4 GiB" now.free="42.5 GiB" now.used="1.9 GiB"
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.432Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="44.1 GiB" now.total="44.4 GiB" now.free="44.1 GiB" now.used="292.2 MiB"
Jun 06 17:20:52 ai001 ollama[518336]: releasing cuda driver library
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.432Z level=INFO source=server.go:168 msg=offload library=cuda layers.requested=-1 layers.model=29 layers.offload=29 layers.split="" memory.available="[44.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="2.1 GiB" memory.required.partial="2.1 GiB" memory.required.kv="896.0 MiB" memory.required.allocations="[2.1 GiB]" memory.weights.total="409.3 MiB" memory.weights.repeating="287.6 MiB" memory.weights.nonrepeating="121.7 MiB" memory.graph.full="298.7 MiB" memory.graph.partial="298.7 MiB"
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.432Z level=INFO source=server.go:211 msg="enabling flash attention"
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.432Z level=WARN source=server.go:219 msg="kv cache type not supported by model" type=""
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.432Z level=DEBUG source=server.go:284 msg="compatible gpu libraries" compatible=[]
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: loaded meta data with 28 key-value pairs and 311 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa (version GGUF V3 (latest))
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   0:                       general.architecture str              = qwen3
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   1:                               general.type str              = model
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   2:                               general.name str              = Qwen3 0.6B
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   3:                           general.basename str              = Qwen3
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   4:                         general.size_label str              = 0.6B
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   5:                            general.license str              = apache-2.0
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   6:                          qwen3.block_count u32              = 28
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   7:                       qwen3.context_length u32              = 40960
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   8:                     qwen3.embedding_length u32              = 1024
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   9:                  qwen3.feed_forward_length u32              = 3072
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  10:                 qwen3.attention.head_count u32              = 16
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  11:              qwen3.attention.head_count_kv u32              = 8
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  12:                       qwen3.rope.freq_base f32              = 1000000.000000
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  13:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  14:                 qwen3.attention.key_length u32              = 128
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  15:               qwen3.attention.value_length u32              = 128
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  16:                       tokenizer.ggml.model str              = gpt2
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  17:                         tokenizer.ggml.pre str              = qwen2
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  18:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  19:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  20:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  21:                tokenizer.ggml.eos_token_id u32              = 151645
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 151643
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 151643
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  24:               tokenizer.ggml.add_bos_token bool             = false
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  25:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  26:               general.quantization_version u32              = 2
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  27:                          general.file_type u32              = 15
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type  f32:  113 tensors
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type  f16:   28 tensors
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type q4_K:  155 tensors
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type q6_K:   15 tensors
Jun 06 17:20:52 ai001 ollama[518336]: print_info: file format = GGUF V3 (latest)
Jun 06 17:20:52 ai001 ollama[518336]: print_info: file type   = Q4_K - Medium
Jun 06 17:20:52 ai001 ollama[518336]: print_info: file size   = 492.75 MiB (5.50 BPW)
Jun 06 17:20:52 ai001 ollama[518336]: init_tokenizer: initializing tokenizer for type 2
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151660 '<|fim_middle|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151659 '<|fim_prefix|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151653 '<|vision_end|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151648 '<|box_start|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151646 '<|object_ref_start|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151649 '<|box_end|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151655 '<|image_pad|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151651 '<|quad_end|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151647 '<|object_ref_end|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151652 '<|vision_start|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151654 '<|vision_pad|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151656 '<|video_pad|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151644 '<|im_start|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151661 '<|fim_suffix|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151650 '<|quad_start|>' is not marked as EOG
Jun 06 17:20:52 ai001 ollama[518336]: load: special tokens cache size = 26
Jun 06 17:20:52 ai001 ollama[518336]: load: token to piece cache size = 0.9311 MB
Jun 06 17:20:52 ai001 ollama[518336]: print_info: arch             = qwen3
Jun 06 17:20:52 ai001 ollama[518336]: print_info: vocab_only       = 1
Jun 06 17:20:52 ai001 ollama[518336]: print_info: model type       = ?B
Jun 06 17:20:52 ai001 ollama[518336]: print_info: model params     = 751.63 M
Jun 06 17:20:52 ai001 ollama[518336]: print_info: general.name     = Qwen3 0.6B
Jun 06 17:20:52 ai001 ollama[518336]: print_info: vocab type       = BPE
Jun 06 17:20:52 ai001 ollama[518336]: print_info: n_vocab          = 151936
Jun 06 17:20:52 ai001 ollama[518336]: print_info: n_merges         = 151387
Jun 06 17:20:52 ai001 ollama[518336]: print_info: BOS token        = 151643 '<|endoftext|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOS token        = 151645 '<|im_end|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOT token        = 151645 '<|im_end|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: PAD token        = 151643 '<|endoftext|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: LF token         = 198 'Ċ'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM MID token    = 151660 '<|fim_middle|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM PAD token    = 151662 '<|fim_pad|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM REP token    = 151663 '<|repo_name|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM SEP token    = 151664 '<|file_sep|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOG token        = 151643 '<|endoftext|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOG token        = 151645 '<|im_end|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOG token        = 151662 '<|fim_pad|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOG token        = 151663 '<|repo_name|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOG token        = 151664 '<|file_sep|>'
Jun 06 17:20:52 ai001 ollama[518336]: print_info: max token length = 256
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_load: vocab only - skipping tensors
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.717Z level=INFO source=server.go:431 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa --ctx-size 8192 --batch-size 512 --n-gpu-layers 29 --threads 64 --flash-attn --parallel 2 --port 43101"
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.718Z level=DEBUG source=server.go:432 msg=subprocess PATH=/root/anaconda3/bin:/root/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=0.0.0.0 OLLAMA_DEBUG=1 OLLAMA_NUM_PARALLEL=2 OLLAMA_FLASH_ATTENTION=1 GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 OLLAMA_MAX_LOADED_MODELS=9 OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama CUDA_VISIBLE_DEVICES=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.718Z level=INFO source=sched.go:483 msg="loaded runners" count=1
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.718Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.718Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding"
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.751Z level=INFO source=runner.go:815 msg="starting go runner"
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.751Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/local/lib/ollama
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.751Z level=INFO source=ggml.go:104 msg=system CPU.0.LLAMAFILE=1 compiler=cgo(gcc)
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.782Z level=INFO source=runner.go:874 msg="Server listening on 127.0.0.1:43101"
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: loaded meta data with 28 key-value pairs and 311 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa (version GGUF V3 (latest))
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   0:                       general.architecture str              = qwen3
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   1:                               general.type str              = model
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   2:                               general.name str              = Qwen3 0.6B
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   3:                           general.basename str              = Qwen3
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   4:                         general.size_label str              = 0.6B
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   5:                            general.license str              = apache-2.0
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   6:                          qwen3.block_count u32              = 28
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   7:                       qwen3.context_length u32              = 40960
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   8:                     qwen3.embedding_length u32              = 1024
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv   9:                  qwen3.feed_forward_length u32              = 3072
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  10:                 qwen3.attention.head_count u32              = 16
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  11:              qwen3.attention.head_count_kv u32              = 8
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  12:                       qwen3.rope.freq_base f32              = 1000000.000000
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  13:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  14:                 qwen3.attention.key_length u32              = 128
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  15:               qwen3.attention.value_length u32              = 128
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  16:                       tokenizer.ggml.model str              = gpt2
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  17:                         tokenizer.ggml.pre str              = qwen2
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  18:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  19:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  20:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  21:                tokenizer.ggml.eos_token_id u32              = 151645
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 151643
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 151643
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  24:               tokenizer.ggml.add_bos_token bool             = false
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  25:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  26:               general.quantization_version u32              = 2
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv  27:                          general.file_type u32              = 15
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type  f32:  113 tensors
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type  f16:   28 tensors
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type q4_K:  155 tensors
Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type q6_K:   15 tensors
Jun 06 17:20:52 ai001 ollama[518336]: print_info: file format = GGUF V3 (latest)
Jun 06 17:20:52 ai001 ollama[518336]: print_info: file type   = Q4_K - Medium
Jun 06 17:20:52 ai001 ollama[518336]: print_info: file size   = 492.75 MiB (5.50 BPW)
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.970Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
Jun 06 17:20:53 ai001 ollama[518336]: init_tokenizer: initializing tokenizer for type 2
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151660 '<|fim_middle|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151659 '<|fim_prefix|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151653 '<|vision_end|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151648 '<|box_start|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151646 '<|object_ref_start|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151649 '<|box_end|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151655 '<|image_pad|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151651 '<|quad_end|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151647 '<|object_ref_end|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151652 '<|vision_start|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151654 '<|vision_pad|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151656 '<|video_pad|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151644 '<|im_start|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151661 '<|fim_suffix|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151650 '<|quad_start|>' is not marked as EOG
Jun 06 17:20:53 ai001 ollama[518336]: load: special tokens cache size = 26
Jun 06 17:20:53 ai001 ollama[518336]: load: token to piece cache size = 0.9311 MB
Jun 06 17:20:53 ai001 ollama[518336]: print_info: arch             = qwen3
Jun 06 17:20:53 ai001 ollama[518336]: print_info: vocab_only       = 0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_ctx_train      = 40960
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_embd           = 1024
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_layer          = 28
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_head           = 16
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_head_kv        = 8
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_rot            = 128
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_swa            = 0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_swa_pattern    = 1
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_embd_head_k    = 128
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_embd_head_v    = 128
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_gqa            = 2
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_embd_k_gqa     = 1024
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_embd_v_gqa     = 1024
Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_norm_eps       = 0.0e+00
Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_norm_rms_eps   = 1.0e-06
Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_clamp_kqv      = 0.0e+00
Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_max_alibi_bias = 0.0e+00
Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_logit_scale    = 0.0e+00
Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_attn_scale     = 0.0e+00
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_ff             = 3072
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_expert         = 0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_expert_used    = 0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: causal attn      = 1
Jun 06 17:20:53 ai001 ollama[518336]: print_info: pooling type     = 0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: rope type        = 2
Jun 06 17:20:53 ai001 ollama[518336]: print_info: rope scaling     = linear
Jun 06 17:20:53 ai001 ollama[518336]: print_info: freq_base_train  = 1000000.0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: freq_scale_train = 1
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_ctx_orig_yarn  = 40960
Jun 06 17:20:53 ai001 ollama[518336]: print_info: rope_finetuned   = unknown
Jun 06 17:20:53 ai001 ollama[518336]: print_info: ssm_d_conv       = 0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: ssm_d_inner      = 0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: ssm_d_state      = 0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: ssm_dt_rank      = 0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: ssm_dt_b_c_rms   = 0
Jun 06 17:20:53 ai001 ollama[518336]: print_info: model type       = 0.6B
Jun 06 17:20:53 ai001 ollama[518336]: print_info: model params     = 751.63 M
Jun 06 17:20:53 ai001 ollama[518336]: print_info: general.name     = Qwen3 0.6B
Jun 06 17:20:53 ai001 ollama[518336]: print_info: vocab type       = BPE
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_vocab          = 151936
Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_merges         = 151387
Jun 06 17:20:53 ai001 ollama[518336]: print_info: BOS token        = 151643 '<|endoftext|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOS token        = 151645 '<|im_end|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOT token        = 151645 '<|im_end|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: PAD token        = 151643 '<|endoftext|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: LF token         = 198 'Ċ'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM MID token    = 151660 '<|fim_middle|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM PAD token    = 151662 '<|fim_pad|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM REP token    = 151663 '<|repo_name|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM SEP token    = 151664 '<|file_sep|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOG token        = 151643 '<|endoftext|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOG token        = 151645 '<|im_end|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOG token        = 151662 '<|fim_pad|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOG token        = 151663 '<|repo_name|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOG token        = 151664 '<|file_sep|>'
Jun 06 17:20:53 ai001 ollama[518336]: print_info: max token length = 256
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: loading model tensors, this can take a while... (mmap = true)
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer   0 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer   1 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer   2 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer   3 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer   4 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer   5 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer   6 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer   7 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer   8 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer   9 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  10 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  11 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  12 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  13 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  14 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  15 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  16 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  17 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  18 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  19 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  20 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  21 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  22 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  23 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  24 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  25 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  26 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  27 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer  28 assigned to device CPU, is_swa = 0
Jun 06 17:20:53 ai001 ollama[518336]: load_tensors:   CPU_Mapped model buffer size =   492.75 MiB
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: constructing llama_context
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_seq_max     = 2
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_ctx         = 8192
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_ctx_per_seq = 4096
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_batch       = 1024
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_ubatch      = 512
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: causal_attn   = 1
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: flash_attn    = 1
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: freq_base     = 1000000.0
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: freq_scale    = 1
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_ctx_per_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
Jun 06 17:20:53 ai001 ollama[518336]: set_abort_callback: call
Jun 06 17:20:53 ai001 ollama[518336]: llama_context:        CPU  output buffer size =     1.17 MiB
Jun 06 17:20:53 ai001 ollama[518336]: create_memory: n_ctx = 8192 (padded)
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: kv_size = 8192, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1, padding = 256
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer   0: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer   1: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer   2: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer   3: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer   4: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer   5: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer   6: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer   7: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer   8: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer   9: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  10: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  11: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  12: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  13: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  14: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  15: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  16: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  17: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  18: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  19: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  20: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  21: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  22: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  23: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  24: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  25: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  26: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer  27: dev = CPU
Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.222Z level=DEBUG source=server.go:636 msg="model load progress 1.00"
Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.473Z level=DEBUG source=server.go:639 msg="model load completed, waiting for server to become available" status="llm server loading model"
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified:        CPU KV buffer size =   896.00 MiB
Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: KV self size  =  896.00 MiB, K (f16):  448.00 MiB, V (f16):  448.00 MiB
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: enumerating backends
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: backend_ptrs.size() = 1
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: max_nodes = 65536
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 0
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: reserving graph for n_tokens = 512, n_seqs = 1
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: reserving graph for n_tokens = 1, n_seqs = 1
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: reserving graph for n_tokens = 512, n_seqs = 1
Jun 06 17:20:53 ai001 ollama[518336]: llama_context:        CPU compute buffer size =   298.75 MiB
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: graph nodes  = 959
Jun 06 17:20:53 ai001 ollama[518336]: llama_context: graph splits = 1
Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.724Z level=INFO source=server.go:630 msg="llama runner started in 1.01 seconds"
Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.724Z level=DEBUG source=sched.go:495 msg="finished setting up" runner.name=registry.ollama.ai/library/qwen3:0.6b runner.inference=cuda runner.devices=1 runner.size="2.1 GiB" runner.vram="2.1 GiB" runner.parallel=2 runner.pid=518390 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa runner.num_ctx=8192
Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.724Z level=DEBUG source=server.go:729 msg="completion request" images=0 prompt=52 format=""
Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.731Z level=DEBUG source=cache.go:104 msg="loading cache slot" id=0 cache=0 prompt=9 used=0 remaining=9
Jun 06 17:20:57 ai001 ollama[518336]: [GIN] 2025/06/06 - 17:20:57 | 200 |  8.056269519s |    172.16.1.219 | POST     "/api/chat"
Jun 06 17:20:57 ai001 ollama[518336]: time=2025-06-06T17:20:57.656Z level=DEBUG source=sched.go:503 msg="context for request finished"
Jun 06 17:20:57 ai001 ollama[518336]: time=2025-06-06T17:20:57.656Z level=DEBUG source=sched.go:343 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3:0.6b runner.inference=cuda runner.devices=1 runner.size="2.1 GiB" runner.vram="2.1 GiB" runner.parallel=2 runner.pid=518390 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa runner.num_ctx=8192 duration=5m0s
Jun 06 17:20:57 ai001 ollama[518336]: time=2025-06-06T17:20:57.656Z level=DEBUG source=sched.go:361 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3:0.6b runner.inference=cuda runner.devices=1 runner.size="2.1 GiB" runner.vram="2.1 GiB" runner.parallel=2 runner.pid=518390 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa runner.num_ctx=8192 refCount=0

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 504474 C /root/miniconda3/bin/python3 4198MiB |
+-----------------------------------------------------------------------------------------+
(base) root@ai001:~#

Relevant log output

OS

Ubuntu 22.04.5 LTS

GPU

Nvidia L30 *3

CPU

x86_64

Ollama version

0.9.0

Originally created by @goactiongo on GitHub (Jun 6, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10999 ### What is the issue? 0.9.0, with 3 L30 card. testing with qwen3:0.6b,it's so slowly, ``` (base) root@ai001:/data/tools# journalctl -u ollama -f Jun 06 17:19:31 ai001 ollama[518336]: [GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c] CUDA freeMem 43488mb Jun 06 17:19:31 ai001 ollama[518336]: [GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c] Compute Capability 8.9 Jun 06 17:19:31 ai001 ollama[518336]: [GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90] CUDA totalMem 45457mb Jun 06 17:19:31 ai001 ollama[518336]: [GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90] CUDA freeMem 45165mb Jun 06 17:19:31 ai001 ollama[518336]: [GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90] Compute Capability 8.9 Jun 06 17:19:32 ai001 ollama[518336]: time=2025-06-06T17:19:32.055Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu" Jun 06 17:19:32 ai001 ollama[518336]: releasing cuda driver library Jun 06 17:19:32 ai001 ollama[518336]: time=2025-06-06T17:19:32.055Z level=INFO source=types.go:130 msg="inference compute" id=GPU-59d819ff-4712-49e4-5e5a-ba8f5c21e829 library=cuda variant=v12 compute=8.9 driver=12.9 name="NVIDIA L20" total="44.4 GiB" available="40.0 GiB" Jun 06 17:19:32 ai001 ollama[518336]: time=2025-06-06T17:19:32.055Z level=INFO source=types.go:130 msg="inference compute" id=GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c library=cuda variant=v12 compute=8.9 driver=12.9 name="NVIDIA L20" total="44.4 GiB" available="42.5 GiB" Jun 06 17:19:32 ai001 ollama[518336]: time=2025-06-06T17:19:32.055Z level=INFO source=types.go:130 msg="inference compute" id=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 library=cuda variant=v12 compute=8.9 driver=12.9 name="NVIDIA L20" total="44.4 GiB" available="44.1 GiB" Jun 06 17:20:49 ai001 ollama[518336]: time=2025-06-06T17:20:49.642Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 Jun 06 17:20:49 ai001 ollama[518336]: time=2025-06-06T17:20:49.643Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="376.6 GiB" before.free="364.9 GiB" before.free_swap="8.0 GiB" now.total="376.6 GiB" now.free="364.9 GiB" now.free_swap="8.0 GiB" Jun 06 17:20:49 ai001 ollama[518336]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08 Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuInit - 0x7f0342af9680 Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDriverGetVersion - 0x7f0342af9740 Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDeviceGetCount - 0x7f0342af98c0 Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDeviceGet - 0x7f0342af9800 Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDeviceGetAttribute - 0x7f0342af9e00 Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDeviceGetUuid - 0x7f0342af9a40 Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuDeviceGetName - 0x7f0342af9980 Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuCtxCreate_v3 - 0x7f0342afa940 Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuMemGetInfo_v2 - 0x7f0342afd5e0 Jun 06 17:20:49 ai001 ollama[518336]: dlsym: cuCtxDestroy - 0x7f0342b5f4e0 Jun 06 17:20:49 ai001 ollama[518336]: calling cuInit Jun 06 17:20:49 ai001 ollama[518336]: calling cuDriverGetVersion Jun 06 17:20:49 ai001 ollama[518336]: raw version 0x2f3a Jun 06 17:20:49 ai001 ollama[518336]: CUDA driver version: 12.9 Jun 06 17:20:49 ai001 ollama[518336]: calling cuDeviceGetCount Jun 06 17:20:49 ai001 ollama[518336]: device count 3 Jun 06 17:20:49 ai001 ollama[518336]: time=2025-06-06T17:20:49.819Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-59d819ff-4712-49e4-5e5a-ba8f5c21e829 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="40.0 GiB" now.total="44.4 GiB" now.free="40.0 GiB" now.used="4.4 GiB" Jun 06 17:20:49 ai001 ollama[518336]: time=2025-06-06T17:20:49.975Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="42.5 GiB" now.total="44.4 GiB" now.free="42.5 GiB" now.used="1.9 GiB" Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.584Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="44.1 GiB" now.total="44.4 GiB" now.free="44.1 GiB" now.used="292.2 MiB" Jun 06 17:20:50 ai001 ollama[518336]: releasing cuda driver library Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.584Z level=DEBUG source=sched.go:185 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=9 gpu_count=3 Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.612Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.636Z level=DEBUG source=ggml.go:155 msg="key not found" key=general.alignment default=32 Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.637Z level=DEBUG source=sched.go:228 msg="loading first model" model=/usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.637Z level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=1 available="[44.1 GiB]" Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.637Z level=DEBUG source=ggml.go:155 msg="key not found" key=qwen3.vision.block_count default=0 Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.637Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="376.6 GiB" before.free="364.9 GiB" before.free_swap="8.0 GiB" now.total="376.6 GiB" now.free="364.9 GiB" now.free_swap="8.0 GiB" Jun 06 17:20:50 ai001 ollama[518336]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08 Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuInit - 0x7f0342af9680 Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDriverGetVersion - 0x7f0342af9740 Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDeviceGetCount - 0x7f0342af98c0 Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDeviceGet - 0x7f0342af9800 Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDeviceGetAttribute - 0x7f0342af9e00 Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDeviceGetUuid - 0x7f0342af9a40 Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuDeviceGetName - 0x7f0342af9980 Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuCtxCreate_v3 - 0x7f0342afa940 Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuMemGetInfo_v2 - 0x7f0342afd5e0 Jun 06 17:20:50 ai001 ollama[518336]: dlsym: cuCtxDestroy - 0x7f0342b5f4e0 Jun 06 17:20:50 ai001 ollama[518336]: calling cuInit Jun 06 17:20:50 ai001 ollama[518336]: calling cuDriverGetVersion Jun 06 17:20:50 ai001 ollama[518336]: raw version 0x2f3a Jun 06 17:20:50 ai001 ollama[518336]: CUDA driver version: 12.9 Jun 06 17:20:50 ai001 ollama[518336]: calling cuDeviceGetCount Jun 06 17:20:50 ai001 ollama[518336]: device count 3 Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.801Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-59d819ff-4712-49e4-5e5a-ba8f5c21e829 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="40.0 GiB" now.total="44.4 GiB" now.free="40.0 GiB" now.used="4.4 GiB" Jun 06 17:20:50 ai001 ollama[518336]: time=2025-06-06T17:20:50.952Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="42.5 GiB" now.total="44.4 GiB" now.free="42.5 GiB" now.used="1.9 GiB" Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.097Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="44.1 GiB" now.total="44.4 GiB" now.free="44.1 GiB" now.used="292.2 MiB" Jun 06 17:20:51 ai001 ollama[518336]: releasing cuda driver library Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.097Z level=INFO source=sched.go:788 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa gpu=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 parallel=2 available=47359262720 required="2.1 GiB" Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.097Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="376.6 GiB" before.free="364.9 GiB" before.free_swap="8.0 GiB" now.total="376.6 GiB" now.free="364.9 GiB" now.free_swap="8.0 GiB" Jun 06 17:20:51 ai001 ollama[518336]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuInit - 0x7f0342af9680 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDriverGetVersion - 0x7f0342af9740 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetCount - 0x7f0342af98c0 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGet - 0x7f0342af9800 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetAttribute - 0x7f0342af9e00 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetUuid - 0x7f0342af9a40 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetName - 0x7f0342af9980 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuCtxCreate_v3 - 0x7f0342afa940 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuMemGetInfo_v2 - 0x7f0342afd5e0 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuCtxDestroy - 0x7f0342b5f4e0 Jun 06 17:20:51 ai001 ollama[518336]: calling cuInit Jun 06 17:20:51 ai001 ollama[518336]: calling cuDriverGetVersion Jun 06 17:20:51 ai001 ollama[518336]: raw version 0x2f3a Jun 06 17:20:51 ai001 ollama[518336]: CUDA driver version: 12.9 Jun 06 17:20:51 ai001 ollama[518336]: calling cuDeviceGetCount Jun 06 17:20:51 ai001 ollama[518336]: device count 3 Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.248Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-59d819ff-4712-49e4-5e5a-ba8f5c21e829 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="40.0 GiB" now.total="44.4 GiB" now.free="40.0 GiB" now.used="4.4 GiB" Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.394Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="42.5 GiB" now.total="44.4 GiB" now.free="42.5 GiB" now.used="1.9 GiB" Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.544Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="44.1 GiB" now.total="44.4 GiB" now.free="44.1 GiB" now.used="292.2 MiB" Jun 06 17:20:51 ai001 ollama[518336]: releasing cuda driver library Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.544Z level=INFO source=server.go:135 msg="system memory" total="376.6 GiB" free="364.9 GiB" free_swap="8.0 GiB" Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.544Z level=DEBUG source=memory.go:111 msg=evaluating library=cuda gpu_count=1 available="[44.1 GiB]" Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.544Z level=DEBUG source=ggml.go:155 msg="key not found" key=qwen3.vision.block_count default=0 Jun 06 17:20:51 ai001 ollama[518336]: time=2025-06-06T17:20:51.545Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="376.6 GiB" before.free="364.9 GiB" before.free_swap="8.0 GiB" now.total="376.6 GiB" now.free="364.9 GiB" now.free_swap="8.0 GiB" Jun 06 17:20:51 ai001 ollama[518336]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuInit - 0x7f0342af9680 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDriverGetVersion - 0x7f0342af9740 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetCount - 0x7f0342af98c0 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGet - 0x7f0342af9800 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetAttribute - 0x7f0342af9e00 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetUuid - 0x7f0342af9a40 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuDeviceGetName - 0x7f0342af9980 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuCtxCreate_v3 - 0x7f0342afa940 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuMemGetInfo_v2 - 0x7f0342afd5e0 Jun 06 17:20:51 ai001 ollama[518336]: dlsym: cuCtxDestroy - 0x7f0342b5f4e0 Jun 06 17:20:51 ai001 ollama[518336]: calling cuInit Jun 06 17:20:51 ai001 ollama[518336]: calling cuDriverGetVersion Jun 06 17:20:51 ai001 ollama[518336]: raw version 0x2f3a Jun 06 17:20:51 ai001 ollama[518336]: CUDA driver version: 12.9 Jun 06 17:20:51 ai001 ollama[518336]: calling cuDeviceGetCount Jun 06 17:20:51 ai001 ollama[518336]: device count 3 Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.145Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-59d819ff-4712-49e4-5e5a-ba8f5c21e829 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="40.0 GiB" now.total="44.4 GiB" now.free="40.0 GiB" now.used="4.4 GiB" Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.291Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-a2bee386-03d6-6ecb-7a1e-74be9074150c name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="42.5 GiB" now.total="44.4 GiB" now.free="42.5 GiB" now.used="1.9 GiB" Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.432Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 name="NVIDIA L20" overhead="0 B" before.total="44.4 GiB" before.free="44.1 GiB" now.total="44.4 GiB" now.free="44.1 GiB" now.used="292.2 MiB" Jun 06 17:20:52 ai001 ollama[518336]: releasing cuda driver library Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.432Z level=INFO source=server.go:168 msg=offload library=cuda layers.requested=-1 layers.model=29 layers.offload=29 layers.split="" memory.available="[44.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="2.1 GiB" memory.required.partial="2.1 GiB" memory.required.kv="896.0 MiB" memory.required.allocations="[2.1 GiB]" memory.weights.total="409.3 MiB" memory.weights.repeating="287.6 MiB" memory.weights.nonrepeating="121.7 MiB" memory.graph.full="298.7 MiB" memory.graph.partial="298.7 MiB" Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.432Z level=INFO source=server.go:211 msg="enabling flash attention" Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.432Z level=WARN source=server.go:219 msg="kv cache type not supported by model" type="" Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.432Z level=DEBUG source=server.go:284 msg="compatible gpu libraries" compatible=[] Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: loaded meta data with 28 key-value pairs and 311 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa (version GGUF V3 (latest)) Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 0: general.architecture str = qwen3 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 1: general.type str = model Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 2: general.name str = Qwen3 0.6B Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 3: general.basename str = Qwen3 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 4: general.size_label str = 0.6B Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 5: general.license str = apache-2.0 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 6: qwen3.block_count u32 = 28 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 7: qwen3.context_length u32 = 40960 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 8: qwen3.embedding_length u32 = 1024 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 9: qwen3.feed_forward_length u32 = 3072 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 10: qwen3.attention.head_count u32 = 16 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 11: qwen3.attention.head_count_kv u32 = 8 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 12: qwen3.rope.freq_base f32 = 1000000.000000 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 13: qwen3.attention.layer_norm_rms_epsilon f32 = 0.000001 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 14: qwen3.attention.key_length u32 = 128 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 15: qwen3.attention.value_length u32 = 128 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 16: tokenizer.ggml.model str = gpt2 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 17: tokenizer.ggml.pre str = qwen2 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 18: tokenizer.ggml.tokens arr[str,151936] = ["!", "\"", "#", "$", "%", "&", "'", ... Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 19: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 20: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 21: tokenizer.ggml.eos_token_id u32 = 151645 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 22: tokenizer.ggml.padding_token_id u32 = 151643 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 23: tokenizer.ggml.bos_token_id u32 = 151643 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 24: tokenizer.ggml.add_bos_token bool = false Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 25: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 26: general.quantization_version u32 = 2 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 27: general.file_type u32 = 15 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type f32: 113 tensors Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type f16: 28 tensors Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type q4_K: 155 tensors Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type q6_K: 15 tensors Jun 06 17:20:52 ai001 ollama[518336]: print_info: file format = GGUF V3 (latest) Jun 06 17:20:52 ai001 ollama[518336]: print_info: file type = Q4_K - Medium Jun 06 17:20:52 ai001 ollama[518336]: print_info: file size = 492.75 MiB (5.50 BPW) Jun 06 17:20:52 ai001 ollama[518336]: init_tokenizer: initializing tokenizer for type 2 Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151660 '<|fim_middle|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151659 '<|fim_prefix|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151653 '<|vision_end|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151648 '<|box_start|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151646 '<|object_ref_start|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151649 '<|box_end|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151655 '<|image_pad|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151651 '<|quad_end|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151647 '<|object_ref_end|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151652 '<|vision_start|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151654 '<|vision_pad|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151656 '<|video_pad|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151644 '<|im_start|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151661 '<|fim_suffix|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: control token: 151650 '<|quad_start|>' is not marked as EOG Jun 06 17:20:52 ai001 ollama[518336]: load: special tokens cache size = 26 Jun 06 17:20:52 ai001 ollama[518336]: load: token to piece cache size = 0.9311 MB Jun 06 17:20:52 ai001 ollama[518336]: print_info: arch = qwen3 Jun 06 17:20:52 ai001 ollama[518336]: print_info: vocab_only = 1 Jun 06 17:20:52 ai001 ollama[518336]: print_info: model type = ?B Jun 06 17:20:52 ai001 ollama[518336]: print_info: model params = 751.63 M Jun 06 17:20:52 ai001 ollama[518336]: print_info: general.name = Qwen3 0.6B Jun 06 17:20:52 ai001 ollama[518336]: print_info: vocab type = BPE Jun 06 17:20:52 ai001 ollama[518336]: print_info: n_vocab = 151936 Jun 06 17:20:52 ai001 ollama[518336]: print_info: n_merges = 151387 Jun 06 17:20:52 ai001 ollama[518336]: print_info: BOS token = 151643 '<|endoftext|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOS token = 151645 '<|im_end|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOT token = 151645 '<|im_end|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: PAD token = 151643 '<|endoftext|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: LF token = 198 'Ċ' Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM PRE token = 151659 '<|fim_prefix|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM SUF token = 151661 '<|fim_suffix|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM MID token = 151660 '<|fim_middle|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM PAD token = 151662 '<|fim_pad|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM REP token = 151663 '<|repo_name|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: FIM SEP token = 151664 '<|file_sep|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOG token = 151643 '<|endoftext|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOG token = 151645 '<|im_end|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOG token = 151662 '<|fim_pad|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOG token = 151663 '<|repo_name|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: EOG token = 151664 '<|file_sep|>' Jun 06 17:20:52 ai001 ollama[518336]: print_info: max token length = 256 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_load: vocab only - skipping tensors Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.717Z level=INFO source=server.go:431 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa --ctx-size 8192 --batch-size 512 --n-gpu-layers 29 --threads 64 --flash-attn --parallel 2 --port 43101" Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.718Z level=DEBUG source=server.go:432 msg=subprocess PATH=/root/anaconda3/bin:/root/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_HOST=0.0.0.0 OLLAMA_DEBUG=1 OLLAMA_NUM_PARALLEL=2 OLLAMA_FLASH_ATTENTION=1 GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 OLLAMA_MAX_LOADED_MODELS=9 OLLAMA_LIBRARY_PATH=/usr/local/lib/ollama LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama CUDA_VISIBLE_DEVICES=GPU-ad7c6ece-c9bd-b8b9-2a1c-23f9aea08b90 Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.718Z level=INFO source=sched.go:483 msg="loaded runners" count=1 Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.718Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding" Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.718Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.751Z level=INFO source=runner.go:815 msg="starting go runner" Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.751Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/local/lib/ollama Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.751Z level=INFO source=ggml.go:104 msg=system CPU.0.LLAMAFILE=1 compiler=cgo(gcc) Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.782Z level=INFO source=runner.go:874 msg="Server listening on 127.0.0.1:43101" Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: loaded meta data with 28 key-value pairs and 311 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa (version GGUF V3 (latest)) Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 0: general.architecture str = qwen3 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 1: general.type str = model Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 2: general.name str = Qwen3 0.6B Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 3: general.basename str = Qwen3 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 4: general.size_label str = 0.6B Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 5: general.license str = apache-2.0 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 6: qwen3.block_count u32 = 28 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 7: qwen3.context_length u32 = 40960 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 8: qwen3.embedding_length u32 = 1024 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 9: qwen3.feed_forward_length u32 = 3072 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 10: qwen3.attention.head_count u32 = 16 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 11: qwen3.attention.head_count_kv u32 = 8 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 12: qwen3.rope.freq_base f32 = 1000000.000000 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 13: qwen3.attention.layer_norm_rms_epsilon f32 = 0.000001 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 14: qwen3.attention.key_length u32 = 128 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 15: qwen3.attention.value_length u32 = 128 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 16: tokenizer.ggml.model str = gpt2 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 17: tokenizer.ggml.pre str = qwen2 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 18: tokenizer.ggml.tokens arr[str,151936] = ["!", "\"", "#", "$", "%", "&", "'", ... Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 19: tokenizer.ggml.token_type arr[i32,151936] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 20: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 21: tokenizer.ggml.eos_token_id u32 = 151645 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 22: tokenizer.ggml.padding_token_id u32 = 151643 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 23: tokenizer.ggml.bos_token_id u32 = 151643 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 24: tokenizer.ggml.add_bos_token bool = false Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 25: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 26: general.quantization_version u32 = 2 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - kv 27: general.file_type u32 = 15 Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type f32: 113 tensors Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type f16: 28 tensors Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type q4_K: 155 tensors Jun 06 17:20:52 ai001 ollama[518336]: llama_model_loader: - type q6_K: 15 tensors Jun 06 17:20:52 ai001 ollama[518336]: print_info: file format = GGUF V3 (latest) Jun 06 17:20:52 ai001 ollama[518336]: print_info: file type = Q4_K - Medium Jun 06 17:20:52 ai001 ollama[518336]: print_info: file size = 492.75 MiB (5.50 BPW) Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.970Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" Jun 06 17:20:53 ai001 ollama[518336]: init_tokenizer: initializing tokenizer for type 2 Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151660 '<|fim_middle|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151659 '<|fim_prefix|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151653 '<|vision_end|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151648 '<|box_start|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151646 '<|object_ref_start|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151649 '<|box_end|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151655 '<|image_pad|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151651 '<|quad_end|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151647 '<|object_ref_end|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151652 '<|vision_start|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151654 '<|vision_pad|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151656 '<|video_pad|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151644 '<|im_start|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151661 '<|fim_suffix|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: control token: 151650 '<|quad_start|>' is not marked as EOG Jun 06 17:20:53 ai001 ollama[518336]: load: special tokens cache size = 26 Jun 06 17:20:53 ai001 ollama[518336]: load: token to piece cache size = 0.9311 MB Jun 06 17:20:53 ai001 ollama[518336]: print_info: arch = qwen3 Jun 06 17:20:53 ai001 ollama[518336]: print_info: vocab_only = 0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_ctx_train = 40960 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_embd = 1024 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_layer = 28 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_head = 16 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_head_kv = 8 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_rot = 128 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_swa_pattern = 1 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_embd_head_k = 128 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_embd_head_v = 128 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_gqa = 2 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_embd_k_gqa = 1024 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_embd_v_gqa = 1024 Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_norm_eps = 0.0e+00 Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_norm_rms_eps = 1.0e-06 Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_clamp_kqv = 0.0e+00 Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_max_alibi_bias = 0.0e+00 Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_logit_scale = 0.0e+00 Jun 06 17:20:53 ai001 ollama[518336]: print_info: f_attn_scale = 0.0e+00 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_ff = 3072 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_expert = 0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_expert_used = 0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: causal attn = 1 Jun 06 17:20:53 ai001 ollama[518336]: print_info: pooling type = 0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: rope type = 2 Jun 06 17:20:53 ai001 ollama[518336]: print_info: rope scaling = linear Jun 06 17:20:53 ai001 ollama[518336]: print_info: freq_base_train = 1000000.0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: freq_scale_train = 1 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_ctx_orig_yarn = 40960 Jun 06 17:20:53 ai001 ollama[518336]: print_info: rope_finetuned = unknown Jun 06 17:20:53 ai001 ollama[518336]: print_info: ssm_d_conv = 0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: ssm_d_inner = 0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: ssm_d_state = 0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: ssm_dt_rank = 0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: ssm_dt_b_c_rms = 0 Jun 06 17:20:53 ai001 ollama[518336]: print_info: model type = 0.6B Jun 06 17:20:53 ai001 ollama[518336]: print_info: model params = 751.63 M Jun 06 17:20:53 ai001 ollama[518336]: print_info: general.name = Qwen3 0.6B Jun 06 17:20:53 ai001 ollama[518336]: print_info: vocab type = BPE Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_vocab = 151936 Jun 06 17:20:53 ai001 ollama[518336]: print_info: n_merges = 151387 Jun 06 17:20:53 ai001 ollama[518336]: print_info: BOS token = 151643 '<|endoftext|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOS token = 151645 '<|im_end|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOT token = 151645 '<|im_end|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: PAD token = 151643 '<|endoftext|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: LF token = 198 'Ċ' Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM PRE token = 151659 '<|fim_prefix|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM SUF token = 151661 '<|fim_suffix|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM MID token = 151660 '<|fim_middle|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM PAD token = 151662 '<|fim_pad|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM REP token = 151663 '<|repo_name|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: FIM SEP token = 151664 '<|file_sep|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOG token = 151643 '<|endoftext|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOG token = 151645 '<|im_end|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOG token = 151662 '<|fim_pad|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOG token = 151663 '<|repo_name|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: EOG token = 151664 '<|file_sep|>' Jun 06 17:20:53 ai001 ollama[518336]: print_info: max token length = 256 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: loading model tensors, this can take a while... (mmap = true) Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 0 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 1 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 2 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 3 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 4 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 5 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 6 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 7 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 8 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 9 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 10 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 11 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 12 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 13 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 14 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 15 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 16 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 17 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 18 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 19 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 20 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 21 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 22 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 23 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 24 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 25 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 26 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 27 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: layer 28 assigned to device CPU, is_swa = 0 Jun 06 17:20:53 ai001 ollama[518336]: load_tensors: CPU_Mapped model buffer size = 492.75 MiB Jun 06 17:20:53 ai001 ollama[518336]: llama_context: constructing llama_context Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_seq_max = 2 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_ctx = 8192 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_ctx_per_seq = 4096 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_batch = 1024 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_ubatch = 512 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: causal_attn = 1 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: flash_attn = 1 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: freq_base = 1000000.0 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: freq_scale = 1 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: n_ctx_per_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized Jun 06 17:20:53 ai001 ollama[518336]: set_abort_callback: call Jun 06 17:20:53 ai001 ollama[518336]: llama_context: CPU output buffer size = 1.17 MiB Jun 06 17:20:53 ai001 ollama[518336]: create_memory: n_ctx = 8192 (padded) Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: kv_size = 8192, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1, padding = 256 Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 0: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 1: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 2: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 3: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 4: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 5: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 6: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 7: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 8: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 9: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 10: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 11: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 12: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 13: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 14: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 15: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 16: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 17: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 18: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 19: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 20: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 21: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 22: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 23: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 24: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 25: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 26: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: layer 27: dev = CPU Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.222Z level=DEBUG source=server.go:636 msg="model load progress 1.00" Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.473Z level=DEBUG source=server.go:639 msg="model load completed, waiting for server to become available" status="llm server loading model" Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: CPU KV buffer size = 896.00 MiB Jun 06 17:20:53 ai001 ollama[518336]: llama_kv_cache_unified: KV self size = 896.00 MiB, K (f16): 448.00 MiB, V (f16): 448.00 MiB Jun 06 17:20:53 ai001 ollama[518336]: llama_context: enumerating backends Jun 06 17:20:53 ai001 ollama[518336]: llama_context: backend_ptrs.size() = 1 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: max_nodes = 65536 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 0 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: reserving graph for n_tokens = 512, n_seqs = 1 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: reserving graph for n_tokens = 1, n_seqs = 1 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: reserving graph for n_tokens = 512, n_seqs = 1 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: CPU compute buffer size = 298.75 MiB Jun 06 17:20:53 ai001 ollama[518336]: llama_context: graph nodes = 959 Jun 06 17:20:53 ai001 ollama[518336]: llama_context: graph splits = 1 Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.724Z level=INFO source=server.go:630 msg="llama runner started in 1.01 seconds" Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.724Z level=DEBUG source=sched.go:495 msg="finished setting up" runner.name=registry.ollama.ai/library/qwen3:0.6b runner.inference=cuda runner.devices=1 runner.size="2.1 GiB" runner.vram="2.1 GiB" runner.parallel=2 runner.pid=518390 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa runner.num_ctx=8192 Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.724Z level=DEBUG source=server.go:729 msg="completion request" images=0 prompt=52 format="" Jun 06 17:20:53 ai001 ollama[518336]: time=2025-06-06T17:20:53.731Z level=DEBUG source=cache.go:104 msg="loading cache slot" id=0 cache=0 prompt=9 used=0 remaining=9 Jun 06 17:20:57 ai001 ollama[518336]: [GIN] 2025/06/06 - 17:20:57 | 200 | 8.056269519s | 172.16.1.219 | POST "/api/chat" Jun 06 17:20:57 ai001 ollama[518336]: time=2025-06-06T17:20:57.656Z level=DEBUG source=sched.go:503 msg="context for request finished" Jun 06 17:20:57 ai001 ollama[518336]: time=2025-06-06T17:20:57.656Z level=DEBUG source=sched.go:343 msg="runner with non-zero duration has gone idle, adding timer" runner.name=registry.ollama.ai/library/qwen3:0.6b runner.inference=cuda runner.devices=1 runner.size="2.1 GiB" runner.vram="2.1 GiB" runner.parallel=2 runner.pid=518390 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa runner.num_ctx=8192 duration=5m0s Jun 06 17:20:57 ai001 ollama[518336]: time=2025-06-06T17:20:57.656Z level=DEBUG source=sched.go:361 msg="after processing request finished event" runner.name=registry.ollama.ai/library/qwen3:0.6b runner.inference=cuda runner.devices=1 runner.size="2.1 GiB" runner.vram="2.1 GiB" runner.parallel=2 runner.pid=518390 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa runner.num_ctx=8192 refCount=0 ``` nvidia-smi Fri Jun 6 17:24:29 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA L20 Off | 00000000:38:00.0 Off | 0 | | N/A 43C P0 78W / 350W | 4207MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA L20 Off | 00000000:A8:00.0 Off | 0 | | N/A 43C P0 69W / 350W | 3MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA L20 Off | 00000000:B8:00.0 Off | 0 | | N/A 31C P0 63W / 350W | 3MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 504474 C /root/miniconda3/bin/python3 4198MiB | +-----------------------------------------------------------------------------------------+ (base) root@ai001:~# ### Relevant log output ```shell ``` ### OS Ubuntu 22.04.5 LTS ### GPU Nvidia L30 *3 ### CPU x86_64 ### Ollama version 0.9.0

GiteaMirror added the bug label 2026-04-12 19:17:50 -05:00

GiteaMirror closed this issue

2026-04-12 19:17:50 -05:00

GiteaMirror commented

2026-04-12 19:17:52 -05:00

@rick-github commented on GitHub (Jun 6, 2025):

Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.751Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/local/lib/ollama
Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.751Z level=INFO source=ggml.go:104 msg=system CPU.0.LLAMAFILE=1 compiler=cgo(gcc)

No CPU or GPU backends found. How did you install ollama?

@rick-github commented on GitHub (Jun 6, 2025): ``` Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.751Z level=DEBUG source=ggml.go:94 msg="ggml backend load all from path" path=/usr/local/lib/ollama Jun 06 17:20:52 ai001 ollama[518336]: time=2025-06-06T17:20:52.751Z level=INFO source=ggml.go:104 msg=system CPU.0.LLAMAFILE=1 compiler=cgo(gcc) ``` No CPU or GPU backends found. How did you install ollama?

GiteaMirror commented

2026-04-12 19:17:53 -05:00

@goactiongo commented on GitHub (Jun 6, 2025):

After installing version 0.9.0 using the command curl -fsSL https://ollama.com/install.sh | sh, there were some issues. I originally planned to reinstall using the script again, but since the installation via the script is very slow (curl -fsSL https://ollama.com/install.sh | sh), I instead downloaded the ollama-linux-amd64.tgz file for version 0.9.0 from GitHub, and then extracted it using tar xvzf ollama-linux-amd64.tgz.

@goactiongo commented on GitHub (Jun 6, 2025): After installing version 0.9.0 using the command `curl -fsSL https://ollama.com/install.sh | sh`, there were some issues. I originally planned to reinstall using the script again, but since the installation via the script is very slow (`curl -fsSL https://ollama.com/install.sh | sh`), I instead downloaded the `ollama-linux-amd64.tgz` file for version 0.9.0 from GitHub, and then extracted it using `tar xvzf ollama-linux-amd64.tgz`. ![Image](https://github.com/user-attachments/assets/ab09c0a2-0568-46ee-a34e-c76c86b37988)

GiteaMirror commented

2026-04-12 19:17:54 -05:00

@rick-github commented on GitHub (Jun 6, 2025):

Where did you extract it to? What commands, exactly, did you use to install ollama?

@rick-github commented on GitHub (Jun 6, 2025): Where did you extract it to? What commands, exactly, did you use to install ollama?

GiteaMirror commented

2026-04-12 19:17:55 -05:00

@goactiongo commented on GitHub (Jun 6, 2025):

(base) root@ai001:/data# ls
bin docker docker-compose.yml lib lost+found metahuman ollama-linux-amd64.tgz tools

The command is
tar xvzf ollama-linux-amd64.tgz
sudo mv bin/ollama /usr/local/bin/

@goactiongo commented on GitHub (Jun 6, 2025): (base) root@ai001:/data# ls bin docker docker-compose.yml lib lost+found metahuman ollama-linux-amd64.tgz tools The command is tar xvzf ollama-linux-amd64.tgz sudo mv bin/ollama /usr/local/bin/

GiteaMirror commented

2026-04-12 19:17:56 -05:00

@rick-github commented on GitHub (Jun 6, 2025):

https://github.com/ollama/ollama/issues/8532#issuecomment-2616281903

@rick-github commented on GitHub (Jun 6, 2025): https://github.com/ollama/ollama/issues/8532#issuecomment-2616281903

GiteaMirror referenced this issue

2026-04-22 09:56:25 -05:00

[GH-ISSUE #7252] add h2ovl-mississippi-800m and h2ovl-mississippi-2b #30368

GiteaMirror referenced this issue

2026-04-28 18:26:45 -05:00

[GH-ISSUE #7252] add h2ovl-mississippi-800m and h2ovl-mississippi-2b #51119

GiteaMirror referenced this issue

2026-05-04 07:45:08 -05:00

[GH-ISSUE #7252] add h2ovl-mississippi-800m and h2ovl-mississippi-2b #66664

Sign in to join this conversation.

Branches Tags

main

dhiltgen/ci

hoyyeva/editor-config-repair

parth-launch-codex-app

hoyyeva/fix-codex-model-metadata-warning

hoyyeva/qwen

hoyyeva/launch-backup-ux

parth/hide-claude-desktop-till-release

hoyyeva/opencode-image-modality

parth-mlx-decode-checkpoints

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

hoyyeva/opencode-thinking

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

brucemacd/download-before-remove

parth/update-claude-docs

parth-anthropic-reference-images-path

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#7252