github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #10449] GPU not being used in Linux by Ollama > 0.5.1 #53382

New Issue
Closed
opened 2026-04-29 02:49:37 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-29 02:49:37 -05:00
Owner
Copy Link

Originally created by @leikareipa on GitHub (Apr 28, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10449

What is the issue?

Ollama 0.5.1 from half a year ago is about the last version where the GPU takes part. In later versions, including the most recent, not so. Using Ubuntu Mate 20.04 and kernel 5.15, RTX 4070 and probably the newest Nvidia drivers.

ollama ps reports "50%/50% CPU/GPU" but nvidia-smi shows ~0% GPU usage and no evidence that weights have been loaded into VRAM. Ollama runs dog slow as you'd expect when doing 100% on the CPU. The same weights properly use the GPU in Ollama 0.5.1, but of course new models no longer support this version.

Relevant log output

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.6.7

Originally created by @leikareipa on GitHub (Apr 28, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10449 ### What is the issue? Ollama 0.5.1 from half a year ago is about the last version where the GPU takes part. In later versions, including the most recent, not so. Using Ubuntu Mate 20.04 and kernel 5.15, RTX 4070 and probably the newest Nvidia drivers. `ollama ps` reports "50%/50% CPU/GPU" but `nvidia-smi` shows ~0% GPU usage and no evidence that weights have been loaded into VRAM. Ollama runs dog slow as you'd expect when doing 100% on the CPU. The same weights properly use the GPU in Ollama 0.5.1, but of course new models no longer support this version. ### Relevant log output ```shell ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.6.7
GiteaMirror added the bug label 2026-04-29 02:49:37 -05:00
GiteaMirror commented 2026-04-29 02:49:41 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 28, 2025):

Server logs may aid in debugging.

<!-- gh-comment-id:2837031870 --> @rick-github commented on GitHub (Apr 28, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) may aid in debugging.
GiteaMirror commented 2026-04-29 02:49:43 -05:00
Author
Owner
Copy Link

@tsteffek commented on GitHub (Apr 29, 2025):

I'm also running into the issue that newer ollama versions don't use my GPU.

Unlike OP ollama ps shows 100% CPU, even though the logging clearly states it detected my GPU. I've tried 0.6.5, 0.6.6, 0.6.7-rc0, and whatever latest currently is (it's 5 days younger than 0.6.6?). I misremembered the version in this issue and tried out 0.5.7, that is picking up and using my GPU.

No idea whether it's related to this issue, but maybe it is.

Server logs with debug

Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.
Your new public key is:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKXYfUlo6JXxbDaVPxmczYlS8k/v8EVcKgh5MbubxkqM

2025/04/29 08:22:06 routes.go:1231: INFO server config env="map[CUDA_VISIBLE_DEVICES:0 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:16384 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/ollama-cache/ OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:32 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-04-29T08:22:06.256Z level=INFO source=images.go:458 msg="total blobs: 21"
time=2025-04-29T08:22:06.257Z level=INFO source=images.go:465 msg="total unused blobs removed: 0"
time=2025-04-29T08:22:06.257Z level=INFO source=routes.go:1298 msg="Listening on 0.0.0.0:11434 (version 0.6.5)"
time=2025-04-29T08:22:06.257Z level=DEBUG source=sched.go:107 msg="starting llm scheduler"
time=2025-04-29T08:22:06.257Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-04-29T08:22:06.292Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-04-29T08:22:06.292Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so
time=2025-04-29T08:22:06.292Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets//lib/libcuda.so /usr/lib/-linux-gnu/nvidia/current/libcuda.so /usr/lib/-linux-gnu/libcuda.so /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers//libcuda.so /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2025-04-29T08:22:06.293Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15]
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fd597e76e00
dlsym: cuDriverGetVersion - 0x7fd597e76e20
dlsym: cuDeviceGetCount - 0x7fd597e76e60
dlsym: cuDeviceGet - 0x7fd597e76e40
dlsym: cuDeviceGetAttribute - 0x7fd597e76f40
dlsym: cuDeviceGetUuid - 0x7fd597e76ea0
dlsym: cuDeviceGetName - 0x7fd597e76e80
dlsym: cuCtxCreate_v3 - 0x7fd597e77120
dlsym: cuMemGetInfo_v2 - 0x7fd597e778a0
dlsym: cuCtxDestroy - 0x7fd597ed59f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 1
time=2025-04-29T08:22:06.367Z level=DEBUG source=gpu.go:125 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
[GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50] CUDA totalMem 40442 mb
[GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50] CUDA freeMem 40019 mb
[GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50] Compute Capability 8.0
time=2025-04-29T08:22:06.659Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
time=2025-04-29T08:22:06.659Z level=INFO source=types.go:130 msg="inference compute" id=GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50 library=cuda variant=v12 compute=8.0 driver=12.8 name="NVIDIA A100-SXM4-40GB" total="39.5 GiB" available="39.1 GiB"
time=2025-04-29T08:22:41.680Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="1007.7 GiB" before.free="987.2 GiB" before.free_swap="0 B" now.total="1007.7 GiB" now.free="987.2 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fd597e76e00
dlsym: cuDriverGetVersion - 0x7fd597e76e20
dlsym: cuDeviceGetCount - 0x7fd597e76e60
dlsym: cuDeviceGet - 0x7fd597e76e40
dlsym: cuDeviceGetAttribute - 0x7fd597e76f40
dlsym: cuDeviceGetUuid - 0x7fd597e76ea0
dlsym: cuDeviceGetName - 0x7fd597e76e80
dlsym: cuCtxCreate_v3 - 0x7fd597e77120
dlsym: cuMemGetInfo_v2 - 0x7fd597e778a0
dlsym: cuCtxDestroy - 0x7fd597ed59f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 1
time=2025-04-29T08:22:41.973Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50 name="NVIDIA A100-SXM4-40GB" overhead="0 B" before.total="39.5 GiB" before.free="39.1 GiB" now.total="39.5 GiB" now.free="39.1 GiB" now.used="423.0 MiB"
releasing cuda driver library
time=2025-04-29T08:22:41.974Z level=DEBUG source=sched.go:183 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=3 gpu_count=1
time=2025-04-29T08:22:41.984Z level=DEBUG source=sched.go:226 msg="loading first model" model=/ollama-cache/blobs/sha256-2ce1208c5e84b3ab44af8faeb793ce2ea09923c79ea157225cd1b74044ee28a9
time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[39.1 GiB]"
time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.vision.block_count default=0
time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.attention.key_length default=96
time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.attention.value_length default=96
time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:194 msg="gpu has too little memory to allocate any layers" id=GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50 library=cuda variant=v12 compute=8.0 driver=12.8 name="NVIDIA A100-SXM4-40GB" total="39.5 GiB" available="39.1 GiB" minimum_memory=479199232 layer_size="6.2 GiB" gpu_zer_overhead="0 B" partial_offload="32.0 GiB" full_offload="32.0 GiB"
time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:338 msg="insufficient VRAM to load any model layers"
time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[39.1 GiB]"
time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.vision.block_count default=0
time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.attention.key_length default=96
time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.attention.value_length default=96
time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:194 msg="gpu has too little memory to allocate any layers" id=GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50 library=cuda variant=v12 compute=8.0 driver=12.8 name="NVIDIA A100-SXM4-40GB" total="39.5 GiB" available="39.1 GiB" minimum_memory=479199232 layer_size="6.2 GiB" gpu_zer_overhead="0 B" partial_offload="32.0 GiB" full_offload="32.0 GiB"
time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:338 msg="insufficient VRAM to load any model layers"
time=2025-04-29T08:22:41.985Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="1007.7 GiB" before.free="987.2 GiB" before.free_swap="0 B" now.total="1007.7 GiB" now.free="987.2 GiB" now.free_swap="0 B"
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15
dlsym: cuInit - 0x7fd597e76e00
dlsym: cuDriverGetVersion - 0x7fd597e76e20
dlsym: cuDeviceGetCount - 0x7fd597e76e60
dlsym: cuDeviceGet - 0x7fd597e76e40
dlsym: cuDeviceGetAttribute - 0x7fd597e76f40
dlsym: cuDeviceGetUuid - 0x7fd597e76ea0
dlsym: cuDeviceGetName - 0x7fd597e76e80
dlsym: cuCtxCreate_v3 - 0x7fd597e77120
dlsym: cuMemGetInfo_v2 - 0x7fd597e778a0
dlsym: cuCtxDestroy - 0x7fd597ed59f0
calling cuInit
calling cuDriverGetVersion
raw version 0x2f30
CUDA driver version: 12.8
calling cuDeviceGetCount
device count 1
init_tokenizer: initializing tokenizer for type 1
load: control token: 32008 '<|placeholder5|>' is not marked as EOG
load: control token: 32006 '<|system|>' is not marked as EOG
load: control token: 32002 '<|placeholder1|>' is not marked as EOG
load: control token: 32001 '<|assistant|>' is not marked as EOG
load: control token: 32004 '<|placeholder3|>' is not marked as EOG
load: control token: 32003 '<|placeholder2|>' is not marked as EOG
load: control token: 0 '' is not marked as EOG
load: control token: 32005 '<|placeholder4|>' is not marked as EOG
load: control token: 32010 '<|user|>' is not marked as EOG
load: control token: 32009 '<|placeholder6|>' is not marked as EOG
load: control token: 1 '' is not marked as EOG
load: special tokens cache size = 14
load: token to piece cache size = 0.1685 MB
print_info: arch = phi3
print_info: vocab_only = 0
print_info: n_ctx_train = 131072
print_info: n_embd = 3072
print_info: n_layer = 32
print_info: n_head = 32
print_info: n_head_kv = 32
print_info: n_rot = 96
print_info: n_swa = 262144
print_info: n_embd_head_k = 96
print_info: n_embd_head_v = 96
print_info: n_gqa = 1
print_info: n_embd_k_gqa = 3072
print_info: n_embd_v_gqa = 3072
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: n_ff = 8192
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 2
print_info: rope scaling = linear
print_info: freq_base_train = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 4096
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 3B
print_info: model params = 3.82 B
print_info: general.name = Phi 3 Mini 128k Instruct
print_info: vocab type = SPM
print_info: n_vocab = 32064
print_info: n_merges = 0
print_info: BOS token = 1 ''
print_info: EOS token = 32000 '<|endoftext|>'
print_info: EOT token = 32007 '<|end|>'
print_info: UNK token = 0 ''
print_info: PAD token = 32000 '<|endoftext|>'
print_info: LF token = 13 '<0x0A>'
print_info: EOG token = 32000 '<|endoftext|>'
print_info: EOG token = 32007 '<|end|>'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: layer 0 assigned to device CPU
load_tensors: layer 1 assigned to device CPU
load_tensors: layer 2 assigned to device CPU
load_tensors: layer 3 assigned to device CPU
load_tensors: layer 4 assigned to device CPU
load_tensors: layer 5 assigned to device CPU
load_tensors: layer 6 assigned to device CPU
load_tensors: layer 7 assigned to device CPU
load_tensors: layer 8 assigned to device CPU
load_tensors: layer 9 assigned to device CPU
load_tensors: layer 10 assigned to device CPU
load_tensors: layer 11 assigned to device CPU
load_tensors: layer 12 assigned to device CPU
load_tensors: layer 13 assigned to device CPU
load_tensors: layer 14 assigned to device CPU
load_tensors: layer 15 assigned to device CPU
load_tensors: layer 16 assigned to device CPU
load_tensors: layer 17 assigned to device CPU
load_tensors: layer 18 assigned to device CPU
load_tensors: layer 19 assigned to device CPU
load_tensors: layer 20 assigned to device CPU
load_tensors: layer 21 assigned to device CPU
load_tensors: layer 22 assigned to device CPU
load_tensors: layer 23 assigned to device CPU
load_tensors: layer 24 assigned to device CPU
load_tensors: layer 25 assigned to device CPU
load_tensors: layer 26 assigned to device CPU
load_tensors: layer 27 assigned to device CPU
load_tensors: layer 28 assigned to device CPU
load_tensors: layer 29 assigned to device CPU
load_tensors: layer 30 assigned to device CPU
load_tensors: layer 31 assigned to device CPU
load_tensors: layer 32 assigned to device CPU
load_tensors: CPU model buffer size = 7288.51 MiB
load_all_data: no device found for buffer type CPU for async uploads
time=2025-04-29T08:22:42.708Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model"
time=2025-04-29T08:22:43.210Z level=DEBUG source=server.go:625 msg="model load progress 0.03"
...
time=2025-04-29T08:23:14.290Z level=DEBUG source=server.go:625 msg="model load progress 0.99"
llama_init_from_model: n_seq_max = 32
llama_init_from_model: n_ctx = 524288
llama_init_from_model: n_ctx_per_seq = 16384
llama_init_from_model: n_batch = 16384
llama_init_from_model: n_ubatch = 512
llama_init_from_model: flash_attn = 0
llama_init_from_model: freq_base = 10000.0
llama_init_from_model: freq_scale = 1
llama_init_from_model: n_ctx_per_seq (16384) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 524288, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1
llama_kv_cache_init: layer 0: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 1: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 2: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 3: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 4: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 5: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 6: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 7: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 8: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 9: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 10: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 11: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 12: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 13: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 14: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 15: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 16: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 17: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 18: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 19: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 20: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 21: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 22: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 23: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 24: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 25: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 26: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 27: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 28: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 29: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 30: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
llama_kv_cache_init: layer 31: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072
time=2025-04-29T08:23:14.541Z level=DEBUG source=server.go:625 msg="model load progress 1.00"
time=2025-04-29T08:23:14.791Z level=DEBUG source=server.go:628 msg="model load completed, waiting for server to become available" status="llm server loading model"

<!-- gh-comment-id:2838038897 --> @tsteffek commented on GitHub (Apr 29, 2025): I'm also running into the issue that newer ollama versions don't use my GPU. Unlike OP ollama ps shows 100% CPU, even though the logging clearly states it detected my GPU. I've tried 0.6.5, 0.6.6, 0.6.7-rc0, and whatever latest currently is (it's 5 days younger than 0.6.6?). I misremembered the version in this issue and tried out 0.5.7, that is picking up and using my GPU. No idea whether it's related to this issue, but maybe it is. <details> <summary>Server logs with debug</summary> Couldn't find '/root/.ollama/id_ed25519'. Generating new private key. Your new public key is: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKXYfUlo6JXxbDaVPxmczYlS8k/v8EVcKgh5MbubxkqM 2025/04/29 08:22:06 routes.go:1231: INFO server config env="map[CUDA_VISIBLE_DEVICES:0 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:16384 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/ollama-cache/ OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:32 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-04-29T08:22:06.256Z level=INFO source=images.go:458 msg="total blobs: 21" time=2025-04-29T08:22:06.257Z level=INFO source=images.go:465 msg="total unused blobs removed: 0" time=2025-04-29T08:22:06.257Z level=INFO source=routes.go:1298 msg="Listening on 0.0.0.0:11434 (version 0.6.5)" time=2025-04-29T08:22:06.257Z level=DEBUG source=sched.go:107 msg="starting llm scheduler" time=2025-04-29T08:22:06.257Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-04-29T08:22:06.292Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2025-04-29T08:22:06.292Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so* time=2025-04-29T08:22:06.292Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2025-04-29T08:22:06.293Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15] initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fd597e76e00 dlsym: cuDriverGetVersion - 0x7fd597e76e20 dlsym: cuDeviceGetCount - 0x7fd597e76e60 dlsym: cuDeviceGet - 0x7fd597e76e40 dlsym: cuDeviceGetAttribute - 0x7fd597e76f40 dlsym: cuDeviceGetUuid - 0x7fd597e76ea0 dlsym: cuDeviceGetName - 0x7fd597e76e80 dlsym: cuCtxCreate_v3 - 0x7fd597e77120 dlsym: cuMemGetInfo_v2 - 0x7fd597e778a0 dlsym: cuCtxDestroy - 0x7fd597ed59f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 1 time=2025-04-29T08:22:06.367Z level=DEBUG source=gpu.go:125 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 [GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50] CUDA totalMem 40442 mb [GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50] CUDA freeMem 40019 mb [GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50] Compute Capability 8.0 time=2025-04-29T08:22:06.659Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2025-04-29T08:22:06.659Z level=INFO source=types.go:130 msg="inference compute" id=GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50 library=cuda variant=v12 compute=8.0 driver=12.8 name="NVIDIA A100-SXM4-40GB" total="39.5 GiB" available="39.1 GiB" time=2025-04-29T08:22:41.680Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="1007.7 GiB" before.free="987.2 GiB" before.free_swap="0 B" now.total="1007.7 GiB" now.free="987.2 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fd597e76e00 dlsym: cuDriverGetVersion - 0x7fd597e76e20 dlsym: cuDeviceGetCount - 0x7fd597e76e60 dlsym: cuDeviceGet - 0x7fd597e76e40 dlsym: cuDeviceGetAttribute - 0x7fd597e76f40 dlsym: cuDeviceGetUuid - 0x7fd597e76ea0 dlsym: cuDeviceGetName - 0x7fd597e76e80 dlsym: cuCtxCreate_v3 - 0x7fd597e77120 dlsym: cuMemGetInfo_v2 - 0x7fd597e778a0 dlsym: cuCtxDestroy - 0x7fd597ed59f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 1 time=2025-04-29T08:22:41.973Z level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50 name="NVIDIA A100-SXM4-40GB" overhead="0 B" before.total="39.5 GiB" before.free="39.1 GiB" now.total="39.5 GiB" now.free="39.1 GiB" now.used="423.0 MiB" releasing cuda driver library time=2025-04-29T08:22:41.974Z level=DEBUG source=sched.go:183 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=3 gpu_count=1 time=2025-04-29T08:22:41.984Z level=DEBUG source=sched.go:226 msg="loading first model" model=/ollama-cache/blobs/sha256-2ce1208c5e84b3ab44af8faeb793ce2ea09923c79ea157225cd1b74044ee28a9 time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[39.1 GiB]" time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.vision.block_count default=0 time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.attention.key_length default=96 time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.attention.value_length default=96 time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:194 msg="gpu has too little memory to allocate any layers" id=GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50 library=cuda variant=v12 compute=8.0 driver=12.8 name="NVIDIA A100-SXM4-40GB" total="39.5 GiB" available="39.1 GiB" minimum_memory=479199232 layer_size="6.2 GiB" gpu_zer_overhead="0 B" partial_offload="32.0 GiB" full_offload="32.0 GiB" time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:338 msg="insufficient VRAM to load any model layers" time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:108 msg=evaluating library=cuda gpu_count=1 available="[39.1 GiB]" time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.vision.block_count default=0 time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.attention.key_length default=96 time=2025-04-29T08:22:41.984Z level=WARN source=ggml.go:152 msg="key not found" key=phi3.attention.value_length default=96 time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:194 msg="gpu has too little memory to allocate any layers" id=GPU-104eac07-8ef6-faf7-38bf-a960bfa65d50 library=cuda variant=v12 compute=8.0 driver=12.8 name="NVIDIA A100-SXM4-40GB" total="39.5 GiB" available="39.1 GiB" minimum_memory=479199232 layer_size="6.2 GiB" gpu_zer_overhead="0 B" partial_offload="32.0 GiB" full_offload="32.0 GiB" time=2025-04-29T08:22:41.984Z level=DEBUG source=memory.go:338 msg="insufficient VRAM to load any model layers" time=2025-04-29T08:22:41.985Z level=DEBUG source=gpu.go:391 msg="updating system memory data" before.total="1007.7 GiB" before.free="987.2 GiB" before.free_swap="0 B" now.total="1007.7 GiB" now.free="987.2 GiB" now.free_swap="0 B" initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.86.15 dlsym: cuInit - 0x7fd597e76e00 dlsym: cuDriverGetVersion - 0x7fd597e76e20 dlsym: cuDeviceGetCount - 0x7fd597e76e60 dlsym: cuDeviceGet - 0x7fd597e76e40 dlsym: cuDeviceGetAttribute - 0x7fd597e76f40 dlsym: cuDeviceGetUuid - 0x7fd597e76ea0 dlsym: cuDeviceGetName - 0x7fd597e76e80 dlsym: cuCtxCreate_v3 - 0x7fd597e77120 dlsym: cuMemGetInfo_v2 - 0x7fd597e778a0 dlsym: cuCtxDestroy - 0x7fd597ed59f0 calling cuInit calling cuDriverGetVersion raw version 0x2f30 CUDA driver version: 12.8 calling cuDeviceGetCount device count 1 init_tokenizer: initializing tokenizer for type 1 load: control token: 32008 '<|placeholder5|>' is not marked as EOG load: control token: 32006 '<|system|>' is not marked as EOG load: control token: 32002 '<|placeholder1|>' is not marked as EOG load: control token: 32001 '<|assistant|>' is not marked as EOG load: control token: 32004 '<|placeholder3|>' is not marked as EOG load: control token: 32003 '<|placeholder2|>' is not marked as EOG load: control token: 0 '<unk>' is not marked as EOG load: control token: 32005 '<|placeholder4|>' is not marked as EOG load: control token: 32010 '<|user|>' is not marked as EOG load: control token: 32009 '<|placeholder6|>' is not marked as EOG load: control token: 1 '<s>' is not marked as EOG load: special tokens cache size = 14 load: token to piece cache size = 0.1685 MB print_info: arch = phi3 print_info: vocab_only = 0 print_info: n_ctx_train = 131072 print_info: n_embd = 3072 print_info: n_layer = 32 print_info: n_head = 32 print_info: n_head_kv = 32 print_info: n_rot = 96 print_info: n_swa = 262144 print_info: n_embd_head_k = 96 print_info: n_embd_head_v = 96 print_info: n_gqa = 1 print_info: n_embd_k_gqa = 3072 print_info: n_embd_v_gqa = 3072 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-05 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: n_ff = 8192 print_info: n_expert = 0 print_info: n_expert_used = 0 print_info: causal attn = 1 print_info: pooling type = 0 print_info: rope type = 2 print_info: rope scaling = linear print_info: freq_base_train = 10000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 4096 print_info: rope_finetuned = unknown print_info: ssm_d_conv = 0 print_info: ssm_d_inner = 0 print_info: ssm_d_state = 0 print_info: ssm_dt_rank = 0 print_info: ssm_dt_b_c_rms = 0 print_info: model type = 3B print_info: model params = 3.82 B print_info: general.name = Phi 3 Mini 128k Instruct print_info: vocab type = SPM print_info: n_vocab = 32064 print_info: n_merges = 0 print_info: BOS token = 1 '<s>' print_info: EOS token = 32000 '<|endoftext|>' print_info: EOT token = 32007 '<|end|>' print_info: UNK token = 0 '<unk>' print_info: PAD token = 32000 '<|endoftext|>' print_info: LF token = 13 '<0x0A>' print_info: EOG token = 32000 '<|endoftext|>' print_info: EOG token = 32007 '<|end|>' print_info: max token length = 48 load_tensors: loading model tensors, this can take a while... (mmap = false) load_tensors: layer 0 assigned to device CPU load_tensors: layer 1 assigned to device CPU load_tensors: layer 2 assigned to device CPU load_tensors: layer 3 assigned to device CPU load_tensors: layer 4 assigned to device CPU load_tensors: layer 5 assigned to device CPU load_tensors: layer 6 assigned to device CPU load_tensors: layer 7 assigned to device CPU load_tensors: layer 8 assigned to device CPU load_tensors: layer 9 assigned to device CPU load_tensors: layer 10 assigned to device CPU load_tensors: layer 11 assigned to device CPU load_tensors: layer 12 assigned to device CPU load_tensors: layer 13 assigned to device CPU load_tensors: layer 14 assigned to device CPU load_tensors: layer 15 assigned to device CPU load_tensors: layer 16 assigned to device CPU load_tensors: layer 17 assigned to device CPU load_tensors: layer 18 assigned to device CPU load_tensors: layer 19 assigned to device CPU load_tensors: layer 20 assigned to device CPU load_tensors: layer 21 assigned to device CPU load_tensors: layer 22 assigned to device CPU load_tensors: layer 23 assigned to device CPU load_tensors: layer 24 assigned to device CPU load_tensors: layer 25 assigned to device CPU load_tensors: layer 26 assigned to device CPU load_tensors: layer 27 assigned to device CPU load_tensors: layer 28 assigned to device CPU load_tensors: layer 29 assigned to device CPU load_tensors: layer 30 assigned to device CPU load_tensors: layer 31 assigned to device CPU load_tensors: layer 32 assigned to device CPU load_tensors: CPU model buffer size = 7288.51 MiB load_all_data: no device found for buffer type CPU for async uploads time=2025-04-29T08:22:42.708Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model" time=2025-04-29T08:22:43.210Z level=DEBUG source=server.go:625 msg="model load progress 0.03" ... time=2025-04-29T08:23:14.290Z level=DEBUG source=server.go:625 msg="model load progress 0.99" llama_init_from_model: n_seq_max = 32 llama_init_from_model: n_ctx = 524288 llama_init_from_model: n_ctx_per_seq = 16384 llama_init_from_model: n_batch = 16384 llama_init_from_model: n_ubatch = 512 llama_init_from_model: flash_attn = 0 llama_init_from_model: freq_base = 10000.0 llama_init_from_model: freq_scale = 1 llama_init_from_model: n_ctx_per_seq (16384) < n_ctx_train (131072) -- the full capacity of the model will not be utilized llama_kv_cache_init: kv_size = 524288, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1 llama_kv_cache_init: layer 0: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 1: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 2: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 3: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 4: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 5: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 6: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 7: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 8: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 9: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 10: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 11: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 12: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 13: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 14: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 15: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 16: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 17: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 18: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 19: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 20: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 21: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 22: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 23: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 24: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 25: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 26: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 27: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 28: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 29: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 30: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 llama_kv_cache_init: layer 31: n_embd_k_gqa = 3072, n_embd_v_gqa = 3072 time=2025-04-29T08:23:14.541Z level=DEBUG source=server.go:625 msg="model load progress 1.00" time=2025-04-29T08:23:14.791Z level=DEBUG source=server.go:628 msg="model load completed, waiting for server to become available" status="llm server loading model" </details>
GiteaMirror commented 2026-04-29 02:49:46 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 29, 2025):

You have OLLAMA_CONTEXT_LENGTH=16384 and OLLAMA_NUM_PARALLEL=32. The amount of KV cache required for those parameters (192G) will not fit in the available VRAM so the model is run in system RAM.

<!-- gh-comment-id:2838232837 --> @rick-github commented on GitHub (Apr 29, 2025): You have `OLLAMA_CONTEXT_LENGTH=16384` and `OLLAMA_NUM_PARALLEL=32`. The amount of KV cache required for those parameters (192G) will not fit in the available VRAM so the model is run in system RAM.
GiteaMirror commented 2026-04-29 02:49:48 -05:00
Author
Owner
Copy Link

@tsteffek commented on GitHub (Apr 29, 2025):

Ah that makes sense, thank you. But what changed that going back a few version allows the same configuration to work on the GPU? Does it now preemptively optimize, while earlier it would wait for actually 32 requests (which I never reach btw) to come?

<!-- gh-comment-id:2839452487 --> @tsteffek commented on GitHub (Apr 29, 2025): Ah that makes sense, thank you. But what changed that going back a few version allows the same configuration to work on the GPU? Does it now preemptively optimize, while earlier it would wait for actually 32 requests (which I never reach btw) to come?
GiteaMirror commented 2026-04-29 02:49:49 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 29, 2025):

How do you set the context length of 0.5.7? OLLAMA_CONTEXT_LENGTH was introduced in 0.5.13, so if you were using the environment variable with 0.5.7, the context length would have been the default 2048.

<!-- gh-comment-id:2839509613 --> @rick-github commented on GitHub (Apr 29, 2025): How do you set the context length of 0.5.7? `OLLAMA_CONTEXT_LENGTH` was introduced in 0.5.13, so if you were using the environment variable with 0.5.7, the context length would have been the default 2048.
GiteaMirror commented 2026-04-29 02:49:50 -05:00
Author
Owner
Copy Link

@leikareipa commented on GitHub (May 13, 2025):

For the record the issue is still there with the latest version of ollama that came out four hours ago. Probably some library expecting a newer system, unlikely to be fixed at this point.

<!-- gh-comment-id:2875045848 --> @leikareipa commented on GitHub (May 13, 2025): For the record the issue is still there with the latest version of ollama that came out four hours ago. Probably some library expecting a newer system, unlikely to be fixed at this point.
GiteaMirror commented 2026-04-29 02:49:51 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (May 13, 2025):

If you provide logs then it could be debugged.

<!-- gh-comment-id:2875453265 --> @rick-github commented on GitHub (May 13, 2025): If you provide logs then it could be debugged.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#53382
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?