github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-08 00:51:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #10319] ps show 100% usage but actually running on CPU #6775

New Issue
Closed
opened 2026-04-12 18:32:16 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-12 18:32:16 -05:00
Owner
Copy Link

Originally created by @zhkchen on GitHub (Apr 17, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10319

What is the issue?

What is the issue?
The LLM model is running on CPU instead of GPU

What happened? What did you expect to happen?
I try to run qwen2.5:latest by ollama. Although ollama showed that the model was running on GPU but I find it was running on CPU after I check nvidia-smi or nvtop

Relevant log output
Apr 17 07:53:50 softusing systemd[1]: Stopping ollama.service - Ollama Service...
Apr 17 07:53:51 softusing systemd[1]: ollama.service: Deactivated successfully.
Apr 17 07:53:51 softusing systemd[1]: Stopped ollama.service - Ollama Service.
Apr 17 07:53:51 softusing systemd[1]: ollama.service: Consumed 1h 13min 18.925s CPU time.
Apr 17 07:53:51 softusing systemd[1]: Started ollama.service - Ollama Service.
Apr 17 07:53:51 softusing ollama[34414]: 2025/04/17 07:53:51 routes.go:1231: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Apr 17 07:53:51 softusing ollama[34414]: time=2025-04-17T07:53:51.342Z level=INFO source=images.go:458 msg="total blobs: 12"
Apr 17 07:53:51 softusing ollama[34414]: time=2025-04-17T07:53:51.342Z level=INFO source=images.go:465 msg="total unused blobs removed: 0"
Apr 17 07:53:51 softusing ollama[34414]: time=2025-04-17T07:53:51.342Z level=INFO source=routes.go:1298 msg="Listening on 127.0.0.1:11434 (version 0.6.5)"
Apr 17 07:53:51 softusing ollama[34414]: time=2025-04-17T07:53:51.342Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-72fba784-c975-ec0a-07e4-9b39fe95e55c library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB"
Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-e5a3f463-44a5-7024-83b8-60fdf6eca8c7 library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB"
Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-b0a3114e-93d6-68ba-a256-fd04f167c5fc library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB"
Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-c92d92a2-66e0-0ce1-2dae-fb0e638c2d3e library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB"
Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-32d195bd-0c3b-8ddb-5ee6-d1863f7215e4 library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB"
Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-1db39bfb-17e5-ca63-be7a-f070ff74213e library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB"
Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-93187c83-c546-4502-bc3a-2fab6d2a1c9a library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB"
Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.309Z level=INFO source=types.go:130 msg="inference compute" id=GPU-1d265e93-d723-3779-de69-6d7cde52c1ff library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB"
Apr 17 07:54:02 softusing ollama[34414]: [GIN] 2025/04/17 - 07:54:02 | 200 | 199.218µs | 127.0.0.1 | HEAD "/"
Apr 17 07:54:02 softusing ollama[34414]: [GIN] 2025/04/17 - 07:54:02 | 200 | 32.162891ms | 127.0.0.1 | POST "/api/show"
Apr 17 07:54:03 softusing ollama[34414]: time=2025-04-17T07:54:03.886Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.vision.block_count default=0
Apr 17 07:54:03 softusing ollama[34414]: time=2025-04-17T07:54:03.886Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.key_length default=128
Apr 17 07:54:03 softusing ollama[34414]: time=2025-04-17T07:54:03.886Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.value_length default=128
Apr 17 07:54:03 softusing ollama[34414]: time=2025-04-17T07:54:03.887Z level=INFO source=sched.go:716 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730 gpu=GPU-72fba784-c975-ec0a-07e4-9b39fe95e55c parallel=4 available=84474396672 required="5.6 GiB"
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.653Z level=INFO source=server.go:105 msg="system memory" total="2015.4 GiB" free="1998.2 GiB" free_swap="8.0 GiB"
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.653Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.vision.block_count default=0
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.653Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.key_length default=128
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.653Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.value_length default=128
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.653Z level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=29 layers.offload=29 layers.split="" memory.available="[78.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.6 GiB" memory.required.partial="5.6 GiB" memory.required.kv="448.0 MiB" memory.required.allocations="[5.6 GiB]" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="478.0 MiB" memory.graph.partial="730.4 MiB"
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: loaded meta data with 34 key-value pairs and 339 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730 (version GGUF V3 (latest))
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 0: general.architecture str = qwen2
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 1: general.type str = model
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 2: general.name str = Qwen2.5 7B Instruct
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 3: general.finetune str = Instruct
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 4: general.basename str = Qwen2.5
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 5: general.size_label str = 7B
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 6: general.license str = apache-2.0
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-7...
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 8: general.base_model.count u32 = 1
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 7B
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-7B
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 12: general.tags arr[str,2] = ["chat", "text-generation"]
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"]
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 14: qwen2.block_count u32 = 28
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 15: qwen2.context_length u32 = 32768
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 16: qwen2.embedding_length u32 = 3584
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 18944
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 28
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 4
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 22: general.file_type u32 = 15
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", """, "#", "$", "%", "&", "'", ...
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 33: general.quantization_version u32 = 2
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type f32: 141 tensors
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type q4_K: 169 tensors
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type q6_K: 29 tensors
Apr 17 07:54:05 softusing ollama[34414]: print_info: file format = GGUF V3 (latest)
Apr 17 07:54:05 softusing ollama[34414]: print_info: file type = Q4_K - Medium
Apr 17 07:54:05 softusing ollama[34414]: print_info: file size = 4.36 GiB (4.91 BPW)
Apr 17 07:54:05 softusing ollama[34414]: load: special tokens cache size = 22
Apr 17 07:54:05 softusing ollama[34414]: load: token to piece cache size = 0.9310 MB
Apr 17 07:54:05 softusing ollama[34414]: print_info: arch = qwen2
Apr 17 07:54:05 softusing ollama[34414]: print_info: vocab_only = 1
Apr 17 07:54:05 softusing ollama[34414]: print_info: model type = ?B
Apr 17 07:54:05 softusing ollama[34414]: print_info: model params = 7.62 B
Apr 17 07:54:05 softusing ollama[34414]: print_info: general.name = Qwen2.5 7B Instruct
Apr 17 07:54:05 softusing ollama[34414]: print_info: vocab type = BPE
Apr 17 07:54:05 softusing ollama[34414]: print_info: n_vocab = 152064
Apr 17 07:54:05 softusing ollama[34414]: print_info: n_merges = 151387
Apr 17 07:54:05 softusing ollama[34414]: print_info: BOS token = 151643 '<|endoftext|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: EOS token = 151645 '<|im_end|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: EOT token = 151645 '<|im_end|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: PAD token = 151643 '<|endoftext|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: LF token = 198 'Ċ'
Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM PRE token = 151659 '<|fim_prefix|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM SUF token = 151661 '<|fim_suffix|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM MID token = 151660 '<|fim_middle|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM PAD token = 151662 '<|fim_pad|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM REP token = 151663 '<|repo_name|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM SEP token = 151664 '<|file_sep|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: EOG token = 151643 '<|endoftext|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: EOG token = 151645 '<|im_end|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: EOG token = 151662 '<|fim_pad|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: EOG token = 151663 '<|repo_name|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: EOG token = 151664 '<|file_sep|>'
Apr 17 07:54:05 softusing ollama[34414]: print_info: max token length = 256
Apr 17 07:54:05 softusing ollama[34414]: llama_model_load: vocab only - skipping tensors
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.854Z level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730 --ctx-size 8192 --batch-size 512 --n-gpu-layers 29 --threads 112 --parallel 4 --port 34391"
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.855Z level=INFO source=sched.go:451 msg="loaded runners" count=1
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.855Z level=INFO source=server.go:580 msg="waiting for llama runner to start responding"
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.856Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error"
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.872Z level=INFO source=runner.go:853 msg="starting go runner"
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.873Z level=INFO source=ggml.go:109 msg=system CPU.0.LLAMAFILE=1 compiler=cgo(gcc)
Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.875Z level=INFO source=runner.go:913 msg="Server listening on 127.0.0.1:34391"
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: loaded meta data with 34 key-value pairs and 339 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730 (version GGUF V3 (latest))
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 0: general.architecture str = qwen2
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 1: general.type str = model
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 2: general.name str = Qwen2.5 7B Instruct
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 3: general.finetune str = Instruct
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 4: general.basename str = Qwen2.5
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 5: general.size_label str = 7B
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 6: general.license str = apache-2.0
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-7...
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 8: general.base_model.count u32 = 1
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 7B
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-7B
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 12: general.tags arr[str,2] = ["chat", "text-generation"]
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"]
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 14: qwen2.block_count u32 = 28
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 15: qwen2.context_length u32 = 32768
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 16: qwen2.embedding_length u32 = 3584
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 18944
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 28
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 4
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 22: general.file_type u32 = 15
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", """, "#", "$", "%", "&", "'", ...
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 33: general.quantization_version u32 = 2
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type f32: 141 tensors
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type q4_K: 169 tensors
Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type q6_K: 29 tensors
Apr 17 07:54:05 softusing ollama[34414]: print_info: file format = GGUF V3 (latest)
Apr 17 07:54:05 softusing ollama[34414]: print_info: file type = Q4_K - Medium
Apr 17 07:54:05 softusing ollama[34414]: print_info: file size = 4.36 GiB (4.91 BPW)
Apr 17 07:54:06 softusing ollama[34414]: load: special tokens cache size = 22
Apr 17 07:54:06 softusing ollama[34414]: load: token to piece cache size = 0.9310 MB
Apr 17 07:54:06 softusing ollama[34414]: print_info: arch = qwen2
Apr 17 07:54:06 softusing ollama[34414]: print_info: vocab_only = 0
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_ctx_train = 32768
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_embd = 3584
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_layer = 28
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_head = 28
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_head_kv = 4
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_rot = 128
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_swa = 0
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_embd_head_k = 128
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_embd_head_v = 128
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_gqa = 7
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_embd_k_gqa = 512
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_embd_v_gqa = 512
Apr 17 07:54:06 softusing ollama[34414]: print_info: f_norm_eps = 0.0e+00
Apr 17 07:54:06 softusing ollama[34414]: print_info: f_norm_rms_eps = 1.0e-06
Apr 17 07:54:06 softusing ollama[34414]: print_info: f_clamp_kqv = 0.0e+00
Apr 17 07:54:06 softusing ollama[34414]: print_info: f_max_alibi_bias = 0.0e+00
Apr 17 07:54:06 softusing ollama[34414]: print_info: f_logit_scale = 0.0e+00
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_ff = 18944
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_expert = 0
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_expert_used = 0
Apr 17 07:54:06 softusing ollama[34414]: print_info: causal attn = 1
Apr 17 07:54:06 softusing ollama[34414]: print_info: pooling type = 0
Apr 17 07:54:06 softusing ollama[34414]: print_info: rope type = 2
Apr 17 07:54:06 softusing ollama[34414]: print_info: rope scaling = linear
Apr 17 07:54:06 softusing ollama[34414]: print_info: freq_base_train = 1000000.0
Apr 17 07:54:06 softusing ollama[34414]: print_info: freq_scale_train = 1
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_ctx_orig_yarn = 32768
Apr 17 07:54:06 softusing ollama[34414]: print_info: rope_finetuned = unknown
Apr 17 07:54:06 softusing ollama[34414]: print_info: ssm_d_conv = 0
Apr 17 07:54:06 softusing ollama[34414]: print_info: ssm_d_inner = 0
Apr 17 07:54:06 softusing ollama[34414]: print_info: ssm_d_state = 0
Apr 17 07:54:06 softusing ollama[34414]: print_info: ssm_dt_rank = 0
Apr 17 07:54:06 softusing ollama[34414]: print_info: ssm_dt_b_c_rms = 0
Apr 17 07:54:06 softusing ollama[34414]: print_info: model type = 7B
Apr 17 07:54:06 softusing ollama[34414]: print_info: model params = 7.62 B
Apr 17 07:54:06 softusing ollama[34414]: print_info: general.name = Qwen2.5 7B Instruct
Apr 17 07:54:06 softusing ollama[34414]: print_info: vocab type = BPE
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_vocab = 152064
Apr 17 07:54:06 softusing ollama[34414]: print_info: n_merges = 151387
Apr 17 07:54:06 softusing ollama[34414]: print_info: BOS token = 151643 '<|endoftext|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: EOS token = 151645 '<|im_end|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: EOT token = 151645 '<|im_end|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: PAD token = 151643 '<|endoftext|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: LF token = 198 'Ċ'
Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM PRE token = 151659 '<|fim_prefix|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM SUF token = 151661 '<|fim_suffix|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM MID token = 151660 '<|fim_middle|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM PAD token = 151662 '<|fim_pad|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM REP token = 151663 '<|repo_name|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM SEP token = 151664 '<|file_sep|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: EOG token = 151643 '<|endoftext|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: EOG token = 151645 '<|im_end|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: EOG token = 151662 '<|fim_pad|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: EOG token = 151663 '<|repo_name|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: EOG token = 151664 '<|file_sep|>'
Apr 17 07:54:06 softusing ollama[34414]: print_info: max token length = 256
Apr 17 07:54:06 softusing ollama[34414]: load_tensors: loading model tensors, this can take a while... (mmap = true)
Apr 17 07:54:06 softusing ollama[34414]: load_tensors: CPU_Mapped model buffer size = 4460.45 MiB
Apr 17 07:54:06 softusing ollama[34414]: time=2025-04-17T07:54:06.236Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model"
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_seq_max = 4
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_ctx = 8192
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_ctx_per_seq = 2048
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_batch = 2048
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_ubatch = 512
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: flash_attn = 0
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: freq_base = 1000000.0
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: freq_scale = 1
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_ctx_per_seq (2048) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
Apr 17 07:54:06 softusing ollama[34414]: llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1
Apr 17 07:54:06 softusing ollama[34414]: llama_kv_cache_init: CPU KV buffer size = 448.00 MiB
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: KV self size = 448.00 MiB, K (f16): 224.00 MiB, V (f16): 224.00 MiB
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: CPU output buffer size = 2.38 MiB
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: CPU compute buffer size = 492.01 MiB
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: graph nodes = 986
Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: graph splits = 1
Apr 17 07:54:06 softusing ollama[34414]: time=2025-04-17T07:54:06.487Z level=INFO source=server.go:619 msg="llama runner started in 0.63 seconds"
Apr 17 07:54:06 softusing ollama[34414]: [GIN] 2025/04/17 - 07:54:06 | 200 | 4.422513851s | 127.0.0.1 | POST "/api/generate"

OS
Ubuntu 24.04.2 LTS

GPU
8xNvidia H100

CPU
Intel(R) Xeon(R) Platinum 8480+

Ollama version
0.6.5

Relevant log output

OS

No response

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @zhkchen on GitHub (Apr 17, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10319 ### What is the issue? What is the issue? The LLM model is running on CPU instead of GPU What happened? What did you expect to happen? I try to run qwen2.5:latest by ollama. Although ollama showed that the model was running on GPU but I find it was running on CPU after I check nvidia-smi or nvtop Relevant log output Apr 17 07:53:50 softusing systemd[1]: Stopping ollama.service - Ollama Service... Apr 17 07:53:51 softusing systemd[1]: ollama.service: Deactivated successfully. Apr 17 07:53:51 softusing systemd[1]: Stopped ollama.service - Ollama Service. Apr 17 07:53:51 softusing systemd[1]: ollama.service: Consumed 1h 13min 18.925s CPU time. Apr 17 07:53:51 softusing systemd[1]: Started ollama.service - Ollama Service. Apr 17 07:53:51 softusing ollama[34414]: 2025/04/17 07:53:51 routes.go:1231: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" Apr 17 07:53:51 softusing ollama[34414]: time=2025-04-17T07:53:51.342Z level=INFO source=images.go:458 msg="total blobs: 12" Apr 17 07:53:51 softusing ollama[34414]: time=2025-04-17T07:53:51.342Z level=INFO source=images.go:465 msg="total unused blobs removed: 0" Apr 17 07:53:51 softusing ollama[34414]: time=2025-04-17T07:53:51.342Z level=INFO source=routes.go:1298 msg="Listening on 127.0.0.1:11434 (version 0.6.5)" Apr 17 07:53:51 softusing ollama[34414]: time=2025-04-17T07:53:51.342Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-72fba784-c975-ec0a-07e4-9b39fe95e55c library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB" Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-e5a3f463-44a5-7024-83b8-60fdf6eca8c7 library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB" Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-b0a3114e-93d6-68ba-a256-fd04f167c5fc library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB" Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-c92d92a2-66e0-0ce1-2dae-fb0e638c2d3e library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB" Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-32d195bd-0c3b-8ddb-5ee6-d1863f7215e4 library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB" Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-1db39bfb-17e5-ca63-be7a-f070ff74213e library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB" Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.308Z level=INFO source=types.go:130 msg="inference compute" id=GPU-93187c83-c546-4502-bc3a-2fab6d2a1c9a library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB" Apr 17 07:53:55 softusing ollama[34414]: time=2025-04-17T07:53:55.309Z level=INFO source=types.go:130 msg="inference compute" id=GPU-1d265e93-d723-3779-de69-6d7cde52c1ff library=cuda variant=v12 compute=9.0 driver=12.8 name="NVIDIA H100 80GB HBM3" total="79.2 GiB" available="78.7 GiB" Apr 17 07:54:02 softusing ollama[34414]: [GIN] 2025/04/17 - 07:54:02 | 200 | 199.218µs | 127.0.0.1 | HEAD "/" Apr 17 07:54:02 softusing ollama[34414]: [GIN] 2025/04/17 - 07:54:02 | 200 | 32.162891ms | 127.0.0.1 | POST "/api/show" Apr 17 07:54:03 softusing ollama[34414]: time=2025-04-17T07:54:03.886Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.vision.block_count default=0 Apr 17 07:54:03 softusing ollama[34414]: time=2025-04-17T07:54:03.886Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.key_length default=128 Apr 17 07:54:03 softusing ollama[34414]: time=2025-04-17T07:54:03.886Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.value_length default=128 Apr 17 07:54:03 softusing ollama[34414]: time=2025-04-17T07:54:03.887Z level=INFO source=sched.go:716 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730 gpu=GPU-72fba784-c975-ec0a-07e4-9b39fe95e55c parallel=4 available=84474396672 required="5.6 GiB" Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.653Z level=INFO source=server.go:105 msg="system memory" total="2015.4 GiB" free="1998.2 GiB" free_swap="8.0 GiB" Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.653Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.vision.block_count default=0 Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.653Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.key_length default=128 Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.653Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.value_length default=128 Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.653Z level=INFO source=server.go:138 msg=offload library=cuda layers.requested=-1 layers.model=29 layers.offload=29 layers.split="" memory.available="[78.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.6 GiB" memory.required.partial="5.6 GiB" memory.required.kv="448.0 MiB" memory.required.allocations="[5.6 GiB]" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="478.0 MiB" memory.graph.partial="730.4 MiB" Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: loaded meta data with 34 key-value pairs and 339 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730 (version GGUF V3 (latest)) Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 0: general.architecture str = qwen2 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 1: general.type str = model Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 2: general.name str = Qwen2.5 7B Instruct Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 3: general.finetune str = Instruct Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 4: general.basename str = Qwen2.5 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 5: general.size_label str = 7B Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 6: general.license str = apache-2.0 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-7... Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 8: general.base_model.count u32 = 1 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 7B Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-7B Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 12: general.tags arr[str,2] = ["chat", "text-generation"] Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"] Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 14: qwen2.block_count u32 = 28 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 15: qwen2.context_length u32 = 32768 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 16: qwen2.embedding_length u32 = 3584 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 18944 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 28 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 4 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 22: general.file_type u32 = 15 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 33: general.quantization_version u32 = 2 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type f32: 141 tensors Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type q4_K: 169 tensors Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type q6_K: 29 tensors Apr 17 07:54:05 softusing ollama[34414]: print_info: file format = GGUF V3 (latest) Apr 17 07:54:05 softusing ollama[34414]: print_info: file type = Q4_K - Medium Apr 17 07:54:05 softusing ollama[34414]: print_info: file size = 4.36 GiB (4.91 BPW) Apr 17 07:54:05 softusing ollama[34414]: load: special tokens cache size = 22 Apr 17 07:54:05 softusing ollama[34414]: load: token to piece cache size = 0.9310 MB Apr 17 07:54:05 softusing ollama[34414]: print_info: arch = qwen2 Apr 17 07:54:05 softusing ollama[34414]: print_info: vocab_only = 1 Apr 17 07:54:05 softusing ollama[34414]: print_info: model type = ?B Apr 17 07:54:05 softusing ollama[34414]: print_info: model params = 7.62 B Apr 17 07:54:05 softusing ollama[34414]: print_info: general.name = Qwen2.5 7B Instruct Apr 17 07:54:05 softusing ollama[34414]: print_info: vocab type = BPE Apr 17 07:54:05 softusing ollama[34414]: print_info: n_vocab = 152064 Apr 17 07:54:05 softusing ollama[34414]: print_info: n_merges = 151387 Apr 17 07:54:05 softusing ollama[34414]: print_info: BOS token = 151643 '<|endoftext|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: EOS token = 151645 '<|im_end|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: EOT token = 151645 '<|im_end|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: PAD token = 151643 '<|endoftext|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: LF token = 198 'Ċ' Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM PRE token = 151659 '<|fim_prefix|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM SUF token = 151661 '<|fim_suffix|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM MID token = 151660 '<|fim_middle|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM PAD token = 151662 '<|fim_pad|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM REP token = 151663 '<|repo_name|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: FIM SEP token = 151664 '<|file_sep|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: EOG token = 151643 '<|endoftext|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: EOG token = 151645 '<|im_end|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: EOG token = 151662 '<|fim_pad|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: EOG token = 151663 '<|repo_name|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: EOG token = 151664 '<|file_sep|>' Apr 17 07:54:05 softusing ollama[34414]: print_info: max token length = 256 Apr 17 07:54:05 softusing ollama[34414]: llama_model_load: vocab only - skipping tensors Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.854Z level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730 --ctx-size 8192 --batch-size 512 --n-gpu-layers 29 --threads 112 --parallel 4 --port 34391" Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.855Z level=INFO source=sched.go:451 msg="loaded runners" count=1 Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.855Z level=INFO source=server.go:580 msg="waiting for llama runner to start responding" Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.856Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error" Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.872Z level=INFO source=runner.go:853 msg="starting go runner" Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.873Z level=INFO source=ggml.go:109 msg=system CPU.0.LLAMAFILE=1 compiler=cgo(gcc) Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.875Z level=INFO source=runner.go:913 msg="Server listening on 127.0.0.1:34391" Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: loaded meta data with 34 key-value pairs and 339 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-2bada8a7450677000f678be90653b85d364de7db25eb5ea54136ada5f3933730 (version GGUF V3 (latest)) Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 0: general.architecture str = qwen2 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 1: general.type str = model Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 2: general.name str = Qwen2.5 7B Instruct Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 3: general.finetune str = Instruct Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 4: general.basename str = Qwen2.5 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 5: general.size_label str = 7B Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 6: general.license str = apache-2.0 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-7... Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 8: general.base_model.count u32 = 1 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 7B Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-7B Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 12: general.tags arr[str,2] = ["chat", "text-generation"] Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"] Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 14: qwen2.block_count u32 = 28 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 15: qwen2.context_length u32 = 32768 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 16: qwen2.embedding_length u32 = 3584 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 18944 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 28 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 4 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 22: general.file_type u32 = 15 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>... Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - kv 33: general.quantization_version u32 = 2 Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type f32: 141 tensors Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type q4_K: 169 tensors Apr 17 07:54:05 softusing ollama[34414]: llama_model_loader: - type q6_K: 29 tensors Apr 17 07:54:05 softusing ollama[34414]: print_info: file format = GGUF V3 (latest) Apr 17 07:54:05 softusing ollama[34414]: print_info: file type = Q4_K - Medium Apr 17 07:54:05 softusing ollama[34414]: print_info: file size = 4.36 GiB (4.91 BPW) Apr 17 07:54:06 softusing ollama[34414]: load: special tokens cache size = 22 Apr 17 07:54:06 softusing ollama[34414]: load: token to piece cache size = 0.9310 MB Apr 17 07:54:06 softusing ollama[34414]: print_info: arch = qwen2 Apr 17 07:54:06 softusing ollama[34414]: print_info: vocab_only = 0 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_ctx_train = 32768 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_embd = 3584 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_layer = 28 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_head = 28 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_head_kv = 4 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_rot = 128 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_swa = 0 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_embd_head_k = 128 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_embd_head_v = 128 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_gqa = 7 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_embd_k_gqa = 512 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_embd_v_gqa = 512 Apr 17 07:54:06 softusing ollama[34414]: print_info: f_norm_eps = 0.0e+00 Apr 17 07:54:06 softusing ollama[34414]: print_info: f_norm_rms_eps = 1.0e-06 Apr 17 07:54:06 softusing ollama[34414]: print_info: f_clamp_kqv = 0.0e+00 Apr 17 07:54:06 softusing ollama[34414]: print_info: f_max_alibi_bias = 0.0e+00 Apr 17 07:54:06 softusing ollama[34414]: print_info: f_logit_scale = 0.0e+00 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_ff = 18944 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_expert = 0 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_expert_used = 0 Apr 17 07:54:06 softusing ollama[34414]: print_info: causal attn = 1 Apr 17 07:54:06 softusing ollama[34414]: print_info: pooling type = 0 Apr 17 07:54:06 softusing ollama[34414]: print_info: rope type = 2 Apr 17 07:54:06 softusing ollama[34414]: print_info: rope scaling = linear Apr 17 07:54:06 softusing ollama[34414]: print_info: freq_base_train = 1000000.0 Apr 17 07:54:06 softusing ollama[34414]: print_info: freq_scale_train = 1 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_ctx_orig_yarn = 32768 Apr 17 07:54:06 softusing ollama[34414]: print_info: rope_finetuned = unknown Apr 17 07:54:06 softusing ollama[34414]: print_info: ssm_d_conv = 0 Apr 17 07:54:06 softusing ollama[34414]: print_info: ssm_d_inner = 0 Apr 17 07:54:06 softusing ollama[34414]: print_info: ssm_d_state = 0 Apr 17 07:54:06 softusing ollama[34414]: print_info: ssm_dt_rank = 0 Apr 17 07:54:06 softusing ollama[34414]: print_info: ssm_dt_b_c_rms = 0 Apr 17 07:54:06 softusing ollama[34414]: print_info: model type = 7B Apr 17 07:54:06 softusing ollama[34414]: print_info: model params = 7.62 B Apr 17 07:54:06 softusing ollama[34414]: print_info: general.name = Qwen2.5 7B Instruct Apr 17 07:54:06 softusing ollama[34414]: print_info: vocab type = BPE Apr 17 07:54:06 softusing ollama[34414]: print_info: n_vocab = 152064 Apr 17 07:54:06 softusing ollama[34414]: print_info: n_merges = 151387 Apr 17 07:54:06 softusing ollama[34414]: print_info: BOS token = 151643 '<|endoftext|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: EOS token = 151645 '<|im_end|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: EOT token = 151645 '<|im_end|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: PAD token = 151643 '<|endoftext|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: LF token = 198 'Ċ' Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM PRE token = 151659 '<|fim_prefix|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM SUF token = 151661 '<|fim_suffix|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM MID token = 151660 '<|fim_middle|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM PAD token = 151662 '<|fim_pad|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM REP token = 151663 '<|repo_name|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: FIM SEP token = 151664 '<|file_sep|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: EOG token = 151643 '<|endoftext|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: EOG token = 151645 '<|im_end|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: EOG token = 151662 '<|fim_pad|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: EOG token = 151663 '<|repo_name|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: EOG token = 151664 '<|file_sep|>' Apr 17 07:54:06 softusing ollama[34414]: print_info: max token length = 256 Apr 17 07:54:06 softusing ollama[34414]: load_tensors: loading model tensors, this can take a while... (mmap = true) Apr 17 07:54:06 softusing ollama[34414]: load_tensors: CPU_Mapped model buffer size = 4460.45 MiB Apr 17 07:54:06 softusing ollama[34414]: time=2025-04-17T07:54:06.236Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model" Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_seq_max = 4 Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_ctx = 8192 Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_ctx_per_seq = 2048 Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_batch = 2048 Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_ubatch = 512 Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: flash_attn = 0 Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: freq_base = 1000000.0 Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: freq_scale = 1 Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: n_ctx_per_seq (2048) < n_ctx_train (32768) -- the full capacity of the model will not be utilized Apr 17 07:54:06 softusing ollama[34414]: llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1 Apr 17 07:54:06 softusing ollama[34414]: llama_kv_cache_init: CPU KV buffer size = 448.00 MiB Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: KV self size = 448.00 MiB, K (f16): 224.00 MiB, V (f16): 224.00 MiB Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: CPU output buffer size = 2.38 MiB Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: CPU compute buffer size = 492.01 MiB Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: graph nodes = 986 Apr 17 07:54:06 softusing ollama[34414]: llama_init_from_model: graph splits = 1 Apr 17 07:54:06 softusing ollama[34414]: time=2025-04-17T07:54:06.487Z level=INFO source=server.go:619 msg="llama runner started in 0.63 seconds" Apr 17 07:54:06 softusing ollama[34414]: [GIN] 2025/04/17 - 07:54:06 | 200 | 4.422513851s | 127.0.0.1 | POST "/api/generate" OS Ubuntu 24.04.2 LTS GPU 8xNvidia H100 CPU Intel(R) Xeon(R) Platinum 8480+ Ollama version 0.6.5 ### Relevant log output ```shell ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_
GiteaMirror added the needs more infobug labels 2026-04-12 18:32:16 -05:00
GiteaMirror commented 2026-04-12 18:32:21 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 17, 2025):

Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.873Z level=INFO source=ggml.go:109 msg=system CPU.0.LLAMAFILE=1 compiler=cgo(gcc)

No backends were found. What's the output of

find /usr/local/lib/ollama /usr/lib/ollama

If you add OLLAMA_DEBUG=1 to the server environment there will be more information about backends in the log.

<!-- gh-comment-id:2812259732 --> @rick-github commented on GitHub (Apr 17, 2025): ``` Apr 17 07:54:05 softusing ollama[34414]: time=2025-04-17T07:54:05.873Z level=INFO source=ggml.go:109 msg=system CPU.0.LLAMAFILE=1 compiler=cgo(gcc) ``` No backends were found. What's the output of ``` find /usr/local/lib/ollama /usr/lib/ollama ``` If you add `OLLAMA_DEBUG=1` to the server environment there will be more information about backends in the log.
GiteaMirror commented 2026-04-12 18:32:21 -05:00
Author
Owner
Copy Link

@TY-teo commented on GitHub (Apr 23, 2025):

how to soleve it，please

<!-- gh-comment-id:2822825841 --> @TY-teo commented on GitHub (Apr 23, 2025): how to soleve it，please
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug needs more info
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6775
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?