github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 04:34:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8532] ollama only using cpu even with gpu found #5503

New Issue
Closed
opened 2026-04-12 16:44:41 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-04-12 16:44:41 -05:00
Owner
Copy Link

Originally created by @nyllewin on GitHub (Jan 22, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8532

What is the issue?

hello,
this has been reported in the past at least two times, I am here to report it a third time because something doesnt seem right.

relevant issues:
https://github.com/ollama/ollama/issues/8485
https://github.com/ollama/ollama/issues/8467

same error, same fix with ' just reinstalling within the same session no reboot no nothing just magically fixes it.

installing from official arch repos also causes the gpu not to be used, any way to .. fix that?

`

2025/01/22 09:42:17 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/nylle/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-01-22T09:42:17.713+01:00 level=INFO source=images.go:432 msg="total blobs: 30"
time=2025-01-22T09:42:17.713+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

  • using env: export GIN_MODE=release
  • using code: gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)
[GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
[GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
[GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers)
[GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
[GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers)
[GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers)
[GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers)
[GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers)
[GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers)
[GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
[GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
[GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers)
[GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
[GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers)
[GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers)
[GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers)
[GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers)
[GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
[GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
[GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
[GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
time=2025-01-22T09:42:17.714+01:00 level=INFO source=routes.go:1238 msg="Listening on 127.0.0.1:11434 (version 0.5.7)"
time=2025-01-22T09:42:17.714+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu]
time=2025-01-22T09:42:17.715+01:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
time=2025-01-22T09:42:17.991+01:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-f16d0b56-1989-0c40-d33b-480f8247ae00 library=cuda variant=v12 compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4070" total="11.6 GiB" available="10.9 GiB"
[GIN] 2025/01/22 - 09:42:29 | 200 | 2.178715ms | 127.0.0.1 | GET "/api/tags"
[GIN] 2025/01/22 - 09:42:29 | 200 | 2.279349ms | 127.0.0.1 | GET "/api/tags"
[GIN] 2025/01/22 - 09:42:29 | 200 | 569.677µs | 127.0.0.1 | GET "/api/tags"
[GIN] 2025/01/22 - 09:42:29 | 200 | 641.407µs | 127.0.0.1 | GET "/api/tags"
[GIN] 2025/01/22 - 09:42:29 | 200 | 952.749µs | 127.0.0.1 | GET "/api/tags"
[GIN] 2025/01/22 - 09:42:29 | 404 | 2.06483ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 404 | 3.234779ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 404 | 708.286µs | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 404 | 4.649411ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 404 | 507.699µs | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 15.227486ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 14.713616ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 16.811971ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 14.913503ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 16.80457ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 11.736522ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 14.276927ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 15.946584ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 16.378602ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 16.905593ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 19.674409ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 9.267617ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 18.489808ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 9.548855ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 14.015642ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 18.927918ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 10.819718ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 20.026506ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 20.160486ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 20.971805ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 22.430443ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 20.434382ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 23.104035ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 19.600649ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 12.181043ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 9.689575ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 11.553348ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 23.011692ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 24.279914ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 24.367486ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 21.774185ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 23.247884ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 17.658695ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 23.333246ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 19.259493ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 14.342186ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 14.479045ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 26.379969ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 12.077609ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 24.800855ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 17.678988ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 27.778669ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 28.465122ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 21.794806ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 33.221807ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 26.157838ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 6.808224ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 7.428048ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 7.468563ms | 127.0.0.1 | POST "/api/show"
[GIN] 2025/01/22 - 09:42:29 | 200 | 7.715287ms | 127.0.0.1 | POST "/api/show"
time=2025-01-22T09:42:34.091+01:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 gpu=GPU-f16d0b56-1989-0c40-d33b-480f8247ae00 parallel=1 available=11592138752 required="9.1 GiB"
time=2025-01-22T09:42:34.212+01:00 level=INFO source=server.go:104 msg="system memory" total="31.3 GiB" free="26.3 GiB" free_swap="32.0 GiB"
time=2025-01-22T09:42:34.213+01:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[10.8 GiB]" memory.gpu_overhead="0 B" memory.required.full="9.1 GiB" memory.required.partial="9.1 GiB" memory.required.kv="4.0 GiB" memory.required.allocations="[9.1 GiB]" memory.weights.total="7.7 GiB" memory.weights.repeating="7.6 GiB" memory.weights.nonrepeating="103.4 MiB" memory.graph.full="553.8 MiB" memory.graph.partial="673.3 MiB"
time=2025-01-22T09:42:34.213+01:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 --ctx-size 8096 --batch-size 512 --n-gpu-layers 33 --threads 16 --parallel 1 --port 42953"
time=2025-01-22T09:42:34.213+01:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-01-22T09:42:34.213+01:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
time=2025-01-22T09:42:34.214+01:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
time=2025-01-22T09:42:34.223+01:00 level=INFO source=runner.go:936 msg="starting go runner"
time=2025-01-22T09:42:34.223+01:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | AARCH64_REPACK = 1 | CPU : LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=16
time=2025-01-22T09:42:34.223+01:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:42953"
llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from /home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = deepseek-ai
llama_model_loader: - kv 2: llama.context_length u32 = 16384
llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
llama_model_loader: - kv 4: llama.block_count u32 = 32
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 11008
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 32
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 10: llama.rope.freq_base f32 = 100000.000000
llama_model_loader: - kv 11: llama.rope.scaling.type str = linear
llama_model_loader: - kv 12: llama.rope.scaling.factor f32 = 4.000000
llama_model_loader: - kv 13: general.file_type u32 = 3
llama_model_loader: - kv 14: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32256] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32256] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,31757] = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 32013
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 32014
llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 32014
llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 24: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_1: 225 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default'
llm_load_vocab: control-looking token: 32015 '<｜fim▁hole｜>' was not control-type; this is probably a bug in the model. its type will be overridden
llm_load_vocab: control-looking token: 32017 '<｜fim▁end｜>' was not control-type; this is probably a bug in the model. its type will be overridden
llm_load_vocab: control-looking token: 32016 '<｜fim▁begin｜>' was not control-type; this is probably a bug in the model. its type will be overridden
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.1792 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 32256
llm_load_print_meta: n_merges = 31757
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 16384
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 4096
llm_load_print_meta: n_embd_v_gqa = 4096
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 11008
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 100000.0
llm_load_print_meta: freq_scale_train = 0.25
llm_load_print_meta: n_ctx_orig_yarn = 16384
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: ssm_dt_b_c_rms = 0
llm_load_print_meta: model type = 7B
llm_load_print_meta: model ftype = Q4_1
llm_load_print_meta: model params = 6.74 B
llm_load_print_meta: model size = 3.95 GiB (5.03 BPW)
llm_load_print_meta: general.name = deepseek-ai
llm_load_print_meta: BOS token = 32013 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token = 32014 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOT token = 32014 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token = 32014 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token = 126 'Ä'
llm_load_print_meta: FIM PRE token = 32016 '<｜fim▁begin｜>'
llm_load_print_meta: FIM SUF token = 32015 '<｜fim▁hole｜>'
llm_load_print_meta: FIM MID token = 32017 '<｜fim▁end｜>'
llm_load_print_meta: EOG token = 32014 '<｜end▁of▁sentence｜>'
llm_load_print_meta: max token length = 128
time=2025-01-22T09:42:34.465+01:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model"
llm_load_tensors: CPU_Mapped model buffer size = 4043.12 MiB
llama_new_context_with_model: n_seq_max = 1
llama_new_context_with_model: n_ctx = 8096
llama_new_context_with_model: n_ctx_per_seq = 8096
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 100000.0
llama_new_context_with_model: freq_scale = 0.25
llama_new_context_with_model: n_ctx_per_seq (8096) < n_ctx_train (16384) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 8096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1
llama_kv_cache_init: CPU KV buffer size = 4048.00 MiB
llama_new_context_with_model: KV self size = 4048.00 MiB, K (f16): 2024.00 MiB, V (f16): 2024.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.14 MiB
llama_new_context_with_model: CPU compute buffer size = 553.82 MiB
llama_new_context_with_model: graph nodes = 1030
llama_new_context_with_model: graph splits = 1
time=2025-01-22T09:42:38.977+01:00 level=INFO source=server.go:594 msg="llama runner started in 4.76 seconds"
llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from /home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = deepseek-ai
llama_model_loader: - kv 2: llama.context_length u32 = 16384
llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
llama_model_loader: - kv 4: llama.block_count u32 = 32
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 11008
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 32
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 10: llama.rope.freq_base f32 = 100000.000000
llama_model_loader: - kv 11: llama.rope.scaling.type str = linear
llama_model_loader: - kv 12: llama.rope.scaling.factor f32 = 4.000000
llama_model_loader: - kv 13: general.file_type u32 = 3
llama_model_loader: - kv 14: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32256] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32256] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,31757] = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 32013
llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 32014
llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 32014
llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 24: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_1: 225 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default'
llm_load_vocab: control-looking token: 32015 '<｜fim▁hole｜>' was not control-type; this is probably a bug in the model. its type will be overridden
llm_load_vocab: control-looking token: 32017 '<｜fim▁end｜>' was not control-type; this is probably a bug in the model. its type will be overridden
llm_load_vocab: control-looking token: 32016 '<｜fim▁begin｜>' was not control-type; this is probably a bug in the model. its type will be overridden
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.1792 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 32256
llm_load_print_meta: n_merges = 31757
llm_load_print_meta: vocab_only = 1
llm_load_print_meta: model type = ?B
llm_load_print_meta: model ftype = all F32
llm_load_print_meta: model params = 6.74 B
llm_load_print_meta: model size = 3.95 GiB (5.03 BPW)
llm_load_print_meta: general.name = deepseek-ai
llm_load_print_meta: BOS token = 32013 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token = 32014 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOT token = 32014 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token = 32014 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token = 126 'Ä'
llm_load_print_meta: FIM PRE token = 32016 '<｜fim▁begin｜>'
llm_load_print_meta: FIM SUF token = 32015 '<｜fim▁hole｜>'
llm_load_print_meta: FIM MID token = 32017 '<｜fim▁end｜>'
llm_load_print_meta: EOG token = 32014 '<｜end▁of▁sentence｜>'
llm_load_print_meta: max token length = 128
llama_model_load: vocab only - skipping tensors
[GIN] 2025/01/22 - 09:43:28 | 200 | 54.425610491s | 127.0.0.1 | POST "/api/chat"

`

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.5.7

Originally created by @nyllewin on GitHub (Jan 22, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8532 ### What is the issue? hello, this has been reported in the past at least two times, I am here to report it a third time because something doesnt seem right. relevant issues: https://github.com/ollama/ollama/issues/8485 https://github.com/ollama/ollama/issues/8467 same error, same fix with ' just reinstalling within the same session no reboot no nothing just magically fixes it. installing from official arch repos also causes the gpu not to be used, any way to .. fix that? ` > 2025/01/22 09:42:17 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/nylle/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" > time=2025-01-22T09:42:17.713+01:00 level=INFO source=images.go:432 msg="total blobs: 30" > time=2025-01-22T09:42:17.713+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" > [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. > > [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. > - using env: export GIN_MODE=release > - using code: gin.SetMode(gin.ReleaseMode) > > [GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers) > [GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers) > [GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers) > [GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers) > [GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers) > [GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers) > [GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers) > [GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers) > [GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers) > [GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers) > [GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers) > [GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers) > [GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers) > [GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers) > [GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers) > [GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers) > [GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers) > [GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers) > [GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) > [GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) > [GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) > [GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) > [GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) > [GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) > time=2025-01-22T09:42:17.714+01:00 level=INFO source=routes.go:1238 msg="Listening on 127.0.0.1:11434 (version 0.5.7)" > time=2025-01-22T09:42:17.714+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu] > time=2025-01-22T09:42:17.715+01:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs" > time=2025-01-22T09:42:17.991+01:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-f16d0b56-1989-0c40-d33b-480f8247ae00 library=cuda variant=v12 compute=8.9 driver=12.7 name="NVIDIA GeForce RTX 4070" total="11.6 GiB" available="10.9 GiB" > [GIN] 2025/01/22 - 09:42:29 | 200 | 2.178715ms | 127.0.0.1 | GET "/api/tags" > [GIN] 2025/01/22 - 09:42:29 | 200 | 2.279349ms | 127.0.0.1 | GET "/api/tags" > [GIN] 2025/01/22 - 09:42:29 | 200 | 569.677µs | 127.0.0.1 | GET "/api/tags" > [GIN] 2025/01/22 - 09:42:29 | 200 | 641.407µs | 127.0.0.1 | GET "/api/tags" > [GIN] 2025/01/22 - 09:42:29 | 200 | 952.749µs | 127.0.0.1 | GET "/api/tags" > [GIN] 2025/01/22 - 09:42:29 | 404 | 2.06483ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 404 | 3.234779ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 404 | 708.286µs | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 404 | 4.649411ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 404 | 507.699µs | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 15.227486ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 14.713616ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 16.811971ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 14.913503ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 16.80457ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 11.736522ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 14.276927ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 15.946584ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 16.378602ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 16.905593ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 19.674409ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 9.267617ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 18.489808ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 9.548855ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 14.015642ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 18.927918ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 10.819718ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 20.026506ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 20.160486ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 20.971805ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 22.430443ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 20.434382ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 23.104035ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 19.600649ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 12.181043ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 9.689575ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 11.553348ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 23.011692ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 24.279914ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 24.367486ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 21.774185ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 23.247884ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 17.658695ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 23.333246ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 19.259493ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 14.342186ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 14.479045ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 26.379969ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 12.077609ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 24.800855ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 17.678988ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 27.778669ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 28.465122ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 21.794806ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 33.221807ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 26.157838ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 6.808224ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 7.428048ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 7.468563ms | 127.0.0.1 | POST "/api/show" > [GIN] 2025/01/22 - 09:42:29 | 200 | 7.715287ms | 127.0.0.1 | POST "/api/show" > time=2025-01-22T09:42:34.091+01:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 gpu=GPU-f16d0b56-1989-0c40-d33b-480f8247ae00 parallel=1 available=11592138752 required="9.1 GiB" > time=2025-01-22T09:42:34.212+01:00 level=INFO source=server.go:104 msg="system memory" total="31.3 GiB" free="26.3 GiB" free_swap="32.0 GiB" > time=2025-01-22T09:42:34.213+01:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[10.8 GiB]" memory.gpu_overhead="0 B" memory.required.full="9.1 GiB" memory.required.partial="9.1 GiB" memory.required.kv="4.0 GiB" memory.required.allocations="[9.1 GiB]" memory.weights.total="7.7 GiB" memory.weights.repeating="7.6 GiB" memory.weights.nonrepeating="103.4 MiB" memory.graph.full="553.8 MiB" memory.graph.partial="673.3 MiB" > time=2025-01-22T09:42:34.213+01:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 --ctx-size 8096 --batch-size 512 --n-gpu-layers 33 --threads 16 --parallel 1 --port 42953" > time=2025-01-22T09:42:34.213+01:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 > time=2025-01-22T09:42:34.213+01:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding" > time=2025-01-22T09:42:34.214+01:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error" > time=2025-01-22T09:42:34.223+01:00 level=INFO source=runner.go:936 msg="starting go runner" > time=2025-01-22T09:42:34.223+01:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | AARCH64_REPACK = 1 | CPU : LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=16 > time=2025-01-22T09:42:34.223+01:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:42953" > llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from /home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 (version GGUF V3 (latest)) > llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. > llama_model_loader: - kv 0: general.architecture str = llama > llama_model_loader: - kv 1: general.name str = deepseek-ai > llama_model_loader: - kv 2: llama.context_length u32 = 16384 > llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 > llama_model_loader: - kv 4: llama.block_count u32 = 32 > llama_model_loader: - kv 5: llama.feed_forward_length u32 = 11008 > llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 > llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 > llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 32 > llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000001 > llama_model_loader: - kv 10: llama.rope.freq_base f32 = 100000.000000 > llama_model_loader: - kv 11: llama.rope.scaling.type str = linear > llama_model_loader: - kv 12: llama.rope.scaling.factor f32 = 4.000000 > llama_model_loader: - kv 13: general.file_type u32 = 3 > llama_model_loader: - kv 14: tokenizer.ggml.model str = gpt2 > llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32256] = ["!", "\"", "#", "$", "%", "&", "'", ... > llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32256] = [0.000000, 0.000000, 0.000000, 0.0000... > llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... > llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,31757] = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e... > llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 32013 > llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 32014 > llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 32014 > llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true > llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false > llama_model_loader: - kv 24: general.quantization_version u32 = 2 > llama_model_loader: - type f32: 65 tensors > llama_model_loader: - type q4_1: 225 tensors > llama_model_loader: - type q6_K: 1 tensors > llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default' > llm_load_vocab: control-looking token: 32015 '<｜fim▁hole｜>' was not control-type; this is probably a bug in the model. its type will be overridden > llm_load_vocab: control-looking token: 32017 '<｜fim▁end｜>' was not control-type; this is probably a bug in the model. its type will be overridden > llm_load_vocab: control-looking token: 32016 '<｜fim▁begin｜>' was not control-type; this is probably a bug in the model. its type will be overridden > llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect > llm_load_vocab: special tokens cache size = 256 > llm_load_vocab: token to piece cache size = 0.1792 MB > llm_load_print_meta: format = GGUF V3 (latest) > llm_load_print_meta: arch = llama > llm_load_print_meta: vocab type = BPE > llm_load_print_meta: n_vocab = 32256 > llm_load_print_meta: n_merges = 31757 > llm_load_print_meta: vocab_only = 0 > llm_load_print_meta: n_ctx_train = 16384 > llm_load_print_meta: n_embd = 4096 > llm_load_print_meta: n_layer = 32 > llm_load_print_meta: n_head = 32 > llm_load_print_meta: n_head_kv = 32 > llm_load_print_meta: n_rot = 128 > llm_load_print_meta: n_swa = 0 > llm_load_print_meta: n_embd_head_k = 128 > llm_load_print_meta: n_embd_head_v = 128 > llm_load_print_meta: n_gqa = 1 > llm_load_print_meta: n_embd_k_gqa = 4096 > llm_load_print_meta: n_embd_v_gqa = 4096 > llm_load_print_meta: f_norm_eps = 0.0e+00 > llm_load_print_meta: f_norm_rms_eps = 1.0e-06 > llm_load_print_meta: f_clamp_kqv = 0.0e+00 > llm_load_print_meta: f_max_alibi_bias = 0.0e+00 > llm_load_print_meta: f_logit_scale = 0.0e+00 > llm_load_print_meta: n_ff = 11008 > llm_load_print_meta: n_expert = 0 > llm_load_print_meta: n_expert_used = 0 > llm_load_print_meta: causal attn = 1 > llm_load_print_meta: pooling type = 0 > llm_load_print_meta: rope type = 0 > llm_load_print_meta: rope scaling = linear > llm_load_print_meta: freq_base_train = 100000.0 > llm_load_print_meta: freq_scale_train = 0.25 > llm_load_print_meta: n_ctx_orig_yarn = 16384 > llm_load_print_meta: rope_finetuned = unknown > llm_load_print_meta: ssm_d_conv = 0 > llm_load_print_meta: ssm_d_inner = 0 > llm_load_print_meta: ssm_d_state = 0 > llm_load_print_meta: ssm_dt_rank = 0 > llm_load_print_meta: ssm_dt_b_c_rms = 0 > llm_load_print_meta: model type = 7B > llm_load_print_meta: model ftype = Q4_1 > llm_load_print_meta: model params = 6.74 B > llm_load_print_meta: model size = 3.95 GiB (5.03 BPW) > llm_load_print_meta: general.name = deepseek-ai > llm_load_print_meta: BOS token = 32013 '<｜begin▁of▁sentence｜>' > llm_load_print_meta: EOS token = 32014 '<｜end▁of▁sentence｜>' > llm_load_print_meta: EOT token = 32014 '<｜end▁of▁sentence｜>' > llm_load_print_meta: PAD token = 32014 '<｜end▁of▁sentence｜>' > llm_load_print_meta: LF token = 126 'Ä' > llm_load_print_meta: FIM PRE token = 32016 '<｜fim▁begin｜>' > llm_load_print_meta: FIM SUF token = 32015 '<｜fim▁hole｜>' > llm_load_print_meta: FIM MID token = 32017 '<｜fim▁end｜>' > llm_load_print_meta: EOG token = 32014 '<｜end▁of▁sentence｜>' > llm_load_print_meta: max token length = 128 > time=2025-01-22T09:42:34.465+01:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model" > llm_load_tensors: CPU_Mapped model buffer size = 4043.12 MiB > llama_new_context_with_model: n_seq_max = 1 > llama_new_context_with_model: n_ctx = 8096 > llama_new_context_with_model: n_ctx_per_seq = 8096 > llama_new_context_with_model: n_batch = 512 > llama_new_context_with_model: n_ubatch = 512 > llama_new_context_with_model: flash_attn = 0 > llama_new_context_with_model: freq_base = 100000.0 > llama_new_context_with_model: freq_scale = 0.25 > llama_new_context_with_model: n_ctx_per_seq (8096) < n_ctx_train (16384) -- the full capacity of the model will not be utilized > llama_kv_cache_init: kv_size = 8096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1 > llama_kv_cache_init: CPU KV buffer size = 4048.00 MiB > llama_new_context_with_model: KV self size = 4048.00 MiB, K (f16): 2024.00 MiB, V (f16): 2024.00 MiB > llama_new_context_with_model: CPU output buffer size = 0.14 MiB > llama_new_context_with_model: CPU compute buffer size = 553.82 MiB > llama_new_context_with_model: graph nodes = 1030 > llama_new_context_with_model: graph splits = 1 > time=2025-01-22T09:42:38.977+01:00 level=INFO source=server.go:594 msg="llama runner started in 4.76 seconds" > llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from /home/nylle/.ollama/models/blobs/sha256-e04bbddd58d9290a89af21ef484ce1113ff34ef35e822e95e52ff1045bac17f5 (version GGUF V3 (latest)) > llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. > llama_model_loader: - kv 0: general.architecture str = llama > llama_model_loader: - kv 1: general.name str = deepseek-ai > llama_model_loader: - kv 2: llama.context_length u32 = 16384 > llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 > llama_model_loader: - kv 4: llama.block_count u32 = 32 > llama_model_loader: - kv 5: llama.feed_forward_length u32 = 11008 > llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 > llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 > llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 32 > llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000001 > llama_model_loader: - kv 10: llama.rope.freq_base f32 = 100000.000000 > llama_model_loader: - kv 11: llama.rope.scaling.type str = linear > llama_model_loader: - kv 12: llama.rope.scaling.factor f32 = 4.000000 > llama_model_loader: - kv 13: general.file_type u32 = 3 > llama_model_loader: - kv 14: tokenizer.ggml.model str = gpt2 > llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32256] = ["!", "\"", "#", "$", "%", "&", "'", ... > llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32256] = [0.000000, 0.000000, 0.000000, 0.0000... > llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... > llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,31757] = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e... > llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 32013 > llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 32014 > llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 32014 > llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true > llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false > llama_model_loader: - kv 24: general.quantization_version u32 = 2 > llama_model_loader: - type f32: 65 tensors > llama_model_loader: - type q4_1: 225 tensors > llama_model_loader: - type q6_K: 1 tensors > llm_load_vocab: missing or unrecognized pre-tokenizer type, using: 'default' > llm_load_vocab: control-looking token: 32015 '<｜fim▁hole｜>' was not control-type; this is probably a bug in the model. its type will be overridden > llm_load_vocab: control-looking token: 32017 '<｜fim▁end｜>' was not control-type; this is probably a bug in the model. its type will be overridden > llm_load_vocab: control-looking token: 32016 '<｜fim▁begin｜>' was not control-type; this is probably a bug in the model. its type will be overridden > llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect > llm_load_vocab: special tokens cache size = 256 > llm_load_vocab: token to piece cache size = 0.1792 MB > llm_load_print_meta: format = GGUF V3 (latest) > llm_load_print_meta: arch = llama > llm_load_print_meta: vocab type = BPE > llm_load_print_meta: n_vocab = 32256 > llm_load_print_meta: n_merges = 31757 > llm_load_print_meta: vocab_only = 1 > llm_load_print_meta: model type = ?B > llm_load_print_meta: model ftype = all F32 > llm_load_print_meta: model params = 6.74 B > llm_load_print_meta: model size = 3.95 GiB (5.03 BPW) > llm_load_print_meta: general.name = deepseek-ai > llm_load_print_meta: BOS token = 32013 '<｜begin▁of▁sentence｜>' > llm_load_print_meta: EOS token = 32014 '<｜end▁of▁sentence｜>' > llm_load_print_meta: EOT token = 32014 '<｜end▁of▁sentence｜>' > llm_load_print_meta: PAD token = 32014 '<｜end▁of▁sentence｜>' > llm_load_print_meta: LF token = 126 'Ä' > llm_load_print_meta: FIM PRE token = 32016 '<｜fim▁begin｜>' > llm_load_print_meta: FIM SUF token = 32015 '<｜fim▁hole｜>' > llm_load_print_meta: FIM MID token = 32017 '<｜fim▁end｜>' > llm_load_print_meta: EOG token = 32014 '<｜end▁of▁sentence｜>' > llm_load_print_meta: max token length = 128 > llama_model_load: vocab only - skipping tensors > [GIN] 2025/01/22 - 09:43:28 | 200 | 54.425610491s | 127.0.0.1 | POST "/api/chat" ` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.5.7
GiteaMirror added the bug label 2026-04-12 16:44:41 -05:00
GiteaMirror commented 2026-04-12 16:44:42 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 22, 2025):

time=2025-01-22T09:42:17.714+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu]

Your installation doesn't have GPU enabled runners. How did you install it?

<!-- gh-comment-id:2606688361 --> @rick-github commented on GitHub (Jan 22, 2025): ``` time=2025-01-22T09:42:17.714+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu] ``` Your installation doesn't have GPU enabled runners. How did you install it?
GiteaMirror commented 2026-04-12 16:44:43 -05:00
Author
Owner
Copy Link

@nyllewin commented on GitHub (Jan 22, 2025):

time=2025-01-22T09:42:17.714+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu]

Your installation doesn't have GPU enabled runners. How did you install it?

this happened with two install methods, one being sudo pacman -S ollama ollama-cuda, the other one being the bash script, only after a second or third time installing it via bash script it gave me the gpu.
package manager install never gave me gpus

<!-- gh-comment-id:2607657705 --> @nyllewin commented on GitHub (Jan 22, 2025): > ``` > time=2025-01-22T09:42:17.714+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu] > ``` > > Your installation doesn't have GPU enabled runners. How did you install it? this happened with two install methods, one being sudo pacman -S ollama ollama-cuda, the other one being the bash script, only after a second or third time installing it via bash script it gave me the gpu. package manager install never gave me gpus
GiteaMirror commented 2026-04-12 16:44:43 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 22, 2025):

Try setting OLLAMA_DEBUG=1 in the server environment, it might give some insight in to why the runner detection is failing.

<!-- gh-comment-id:2607689340 --> @rick-github commented on GitHub (Jan 22, 2025): Try setting `OLLAMA_DEBUG=1` in the server environment, it might give some insight in to why the runner detection is failing.
GiteaMirror commented 2026-04-12 16:44:44 -05:00
Author
Owner
Copy Link

@troed commented on GitHub (Jan 27, 2025):

I have the same issue. Installed manually by copying the lib and binary into the paths I wanted and my own system daemon to execute them. I see the [cpu] runner but also everything I would expect regarding my GPU. Using nvidia-smi ollama is clearly not making use of the GPU at all during inference.

jan 27 17:06:44 desktop systemd[1]: Started ollama.service - Ollama Service.
jan 27 17:06:44 desktop ollama[1646734]: 2025/01/27 17:06:44 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://192.168.0.2:* http://192.168.0.6:* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.420+01:00 level=INFO source=images.go:432 msg="total blobs: 13"
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.420+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
jan 27 17:06:44 desktop ollama[1646734]:  - using env:        export GIN_MODE=release
jan 27 17:06:44 desktop ollama[1646734]:  - using code:        gin.SetMode(gin.ReleaseMode)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /api/pull                 --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /api/generate             --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /api/chat                 --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /api/embed                --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /api/embeddings           --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /api/create               --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /api/push                 --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /api/copy                 --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] DELETE /api/delete               --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /api/show                 --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] HEAD   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET    /api/ps                   --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /v1/chat/completions      --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /v1/completions           --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST   /v1/embeddings            --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET    /v1/models                --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET    /v1/models/:model         --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET    /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET    /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET    /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] HEAD   /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] HEAD   /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] HEAD   /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.420+01:00 level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7)"
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=DEBUG source=common.go:85 msg="no dynamic runners detected, using only built-in"
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu]
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=DEBUG source=routes.go:1268 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler"
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.422+01:00 level=DEBUG source=gpu.go:99 msg="searching for GPU discovery libraries for NVIDIA"
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.422+01:00 level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=libcuda.so*
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.422+01:00 level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[libcuda.so* /libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.428+01:00 level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths="[/usr/lib/i386-linux-gnu/libcuda.so.560.35.03 /usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03]"
jan 27 17:06:44 desktop ollama[1646734]: initializing /usr/lib/i386-linux-gnu/libcuda.so.560.35.03
jan 27 17:06:44 desktop ollama[1646734]: library /usr/lib/i386-linux-gnu/libcuda.so.560.35.03 load err: /usr/lib/i386-linux-gnu/libcuda.so.560.35.03: wrong ELF class: ELFCLASS32
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.428+01:00 level=DEBUG source=gpu.go:627 msg="skipping 32bit library" library=/usr/lib/i386-linux-gnu/libcuda.so.560.35.03
jan 27 17:06:44 desktop ollama[1646734]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03
jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuInit - 0x714a94c5d7f0
jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDriverGetVersion - 0x714a94c5d810
jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDeviceGetCount - 0x714a94c5d850
jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDeviceGet - 0x714a94c5d830
jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDeviceGetAttribute - 0x714a94c5d930
jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDeviceGetUuid - 0x714a94c5d890
jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDeviceGetName - 0x714a94c5d870
jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuCtxCreate_v3 - 0x714a94c68060
jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuMemGetInfo_v2 - 0x714a94c73520
jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuCtxDestroy - 0x714a94cce380
jan 27 17:06:44 desktop ollama[1646734]: calling cuInit
jan 27 17:06:44 desktop ollama[1646734]: calling cuDriverGetVersion
jan 27 17:06:44 desktop ollama[1646734]: raw version 0x2f1c
jan 27 17:06:44 desktop ollama[1646734]: CUDA driver version: 12.6
jan 27 17:06:44 desktop ollama[1646734]: calling cuDeviceGetCount
jan 27 17:06:44 desktop ollama[1646734]: device count 1
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.442+01:00 level=DEBUG source=gpu.go:134 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03
jan 27 17:06:44 desktop ollama[1646734]: [GPU-d1756887-9770-5b36-7763-1b6bc048de4d] CUDA totalMem 11901 mb
jan 27 17:06:44 desktop ollama[1646734]: [GPU-d1756887-9770-5b36-7763-1b6bc048de4d] CUDA freeMem 8208 mb
jan 27 17:06:44 desktop ollama[1646734]: [GPU-d1756887-9770-5b36-7763-1b6bc048de4d] Compute Capability 8.6
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.657+01:00 level=DEBUG source=amd_linux.go:421 msg="amdgpu driver not detected /sys/module/amdgpu"
jan 27 17:06:44 desktop ollama[1646734]: releasing cuda driver library
jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.657+01:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-d1756887-9770-5b36-7763-1b6bc048de4d library=cuda variant=v12 compute=8.6 driver=12.6 name="NVIDIA RTX A2000 12GB" total="11.6 GiB" available="8.0 GiB"

jan 27 17:08:19 desktop ollama[1646734]: [GIN] 2025/01/27 - 17:08:19 | 200 |     140.818µs |       127.0.0.1 | HEAD     "/"
jan 27 17:08:19 desktop ollama[1646734]: [GIN] 2025/01/27 - 17:08:19 | 200 |   28.825457ms |       127.0.0.1 | POST     "/api/show"
jan 27 17:08:19 desktop ollama[1646734]: time=2025-01-27T17:08:19.907+01:00 level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="94.0 GiB" before.free="71.1 GiB" before.free_swap="1.9 GiB" now.total="94.0 GiB" now.free="71.1 GiB" now.free_swap="1.9 GiB"
jan 27 17:08:19 desktop ollama[1646734]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03
jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuInit - 0x714a94c5d7f0
jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDriverGetVersion - 0x714a94c5d810
jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDeviceGetCount - 0x714a94c5d850
jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDeviceGet - 0x714a94c5d830
jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDeviceGetAttribute - 0x714a94c5d930
jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDeviceGetUuid - 0x714a94c5d890
jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDeviceGetName - 0x714a94c5d870
jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuCtxCreate_v3 - 0x714a94c68060
jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuMemGetInfo_v2 - 0x714a94c73520
jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuCtxDestroy - 0x714a94cce380
jan 27 17:08:19 desktop ollama[1646734]: calling cuInit
jan 27 17:08:19 desktop ollama[1646734]: calling cuDriverGetVersion
jan 27 17:08:19 desktop ollama[1646734]: raw version 0x2f1c
jan 27 17:08:19 desktop ollama[1646734]: CUDA driver version: 12.6
jan 27 17:08:19 desktop ollama[1646734]: calling cuDeviceGetCount
jan 27 17:08:19 desktop ollama[1646734]: device count 1
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.086+01:00 level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-d1756887-9770-5b36-7763-1b6bc048de4d name="NVIDIA RTX A2000 12GB" overhead="0 B" before.total="11.6 GiB" before.free="8.0 GiB" now.total="11.6 GiB" now.free="8.1 GiB" now.used="3.5 GiB"
jan 27 17:08:20 desktop ollama[1646734]: releasing cuda driver library
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.086+01:00 level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x5aae4df874c0 gpu_count=1
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.109+01:00 level=DEBUG source=sched.go:224 msg="loading first model" model=/usr/share/ollama/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.109+01:00 level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[8.1 GiB]"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.110+01:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be gpu=GPU-d1756887-9770-5b36-7763-1b6bc048de4d parallel=4 available=8691253248 required="6.5 GiB"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.110+01:00 level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="94.0 GiB" before.free="71.1 GiB" before.free_swap="1.9 GiB" now.total="94.0 GiB" now.free="71.1 GiB" now.free_swap="1.9 GiB"
jan 27 17:08:20 desktop ollama[1646734]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03
jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuInit - 0x714a94c5d7f0
jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDriverGetVersion - 0x714a94c5d810
jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDeviceGetCount - 0x714a94c5d850
jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDeviceGet - 0x714a94c5d830
jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDeviceGetAttribute - 0x714a94c5d930
jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDeviceGetUuid - 0x714a94c5d890
jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDeviceGetName - 0x714a94c5d870
jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuCtxCreate_v3 - 0x714a94c68060
jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuMemGetInfo_v2 - 0x714a94c73520
jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuCtxDestroy - 0x714a94cce380
jan 27 17:08:20 desktop ollama[1646734]: calling cuInit
jan 27 17:08:20 desktop ollama[1646734]: calling cuDriverGetVersion
jan 27 17:08:20 desktop ollama[1646734]: raw version 0x2f1c
jan 27 17:08:20 desktop ollama[1646734]: CUDA driver version: 12.6
jan 27 17:08:20 desktop ollama[1646734]: calling cuDeviceGetCount
jan 27 17:08:20 desktop ollama[1646734]: device count 1
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-d1756887-9770-5b36-7763-1b6bc048de4d name="NVIDIA RTX A2000 12GB" overhead="0 B" before.total="11.6 GiB" before.free="8.1 GiB" now.total="11.6 GiB" now.free="8.1 GiB" now.used="3.5 GiB"
jan 27 17:08:20 desktop ollama[1646734]: releasing cuda driver library
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=INFO source=server.go:104 msg="system memory" total="94.0 GiB" free="71.1 GiB" free_swap="1.9 GiB"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[8.1 GiB]"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[8.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.9 GiB" memory.weights.repeating="4.5 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=DEBUG source=gpu.go:713 msg="no filter required for library cpu"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/opt/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --verbose --threads 8 --parallel 4 --port 35083"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=DEBUG source=server.go:393 msg=subprocess environment="[PATH=/home/troed/.local/bin:/home/troed/bin:/opt/conda/bin:/opt/conda/condabin:/home/troed/.local/bin:/home/troed/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/opt/local/bin:/usr/local/cuda-12.0/bin:/home/troed/.npmglobal/bin:/opt/local/bin:/usr/local/cuda-12.0/bin:/home/troed/.npmglobal/bin LD_LIBRARY_PATH=.:/opt/local/bin]"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.281+01:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.281+01:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.281+01:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.289+01:00 level=INFO source=runner.go:936 msg="starting go runner"
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.289+01:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | AARCH64_REPACK = 1 | CPU : LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8
jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.290+01:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:35083"
<!-- gh-comment-id:2616194111 --> @troed commented on GitHub (Jan 27, 2025): I have the same issue. Installed manually by copying the lib and binary into the paths I wanted and my own system daemon to execute them. I see the [cpu] runner but also everything I would expect regarding my GPU. Using nvidia-smi ollama is clearly not making use of the GPU at all during inference. ``` jan 27 17:06:44 desktop systemd[1]: Started ollama.service - Ollama Service. jan 27 17:06:44 desktop ollama[1646734]: 2025/01/27 17:06:44 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://192.168.0.2:* http://192.168.0.6:* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.420+01:00 level=INFO source=images.go:432 msg="total blobs: 13" jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.420+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. jan 27 17:06:44 desktop ollama[1646734]: - using env: export GIN_MODE=release jan 27 17:06:44 desktop ollama[1646734]: - using code: gin.SetMode(gin.ReleaseMode) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /v1/completions --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] POST /v1/embeddings --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET /v1/models --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET /v1/models/:model --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: [GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers) jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.420+01:00 level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7)" jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=DEBUG source=common.go:85 msg="no dynamic runners detected, using only built-in" jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu] jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=DEBUG source=routes.go:1268 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler" jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=INFO source=gpu.go:226 msg="looking for compatible GPUs" jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.422+01:00 level=DEBUG source=gpu.go:99 msg="searching for GPU discovery libraries for NVIDIA" jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.422+01:00 level=DEBUG source=gpu.go:517 msg="Searching for GPU library" name=libcuda.so* jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.422+01:00 level=DEBUG source=gpu.go:543 msg="gpu library search" globs="[libcuda.so* /libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.428+01:00 level=DEBUG source=gpu.go:576 msg="discovered GPU libraries" paths="[/usr/lib/i386-linux-gnu/libcuda.so.560.35.03 /usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03]" jan 27 17:06:44 desktop ollama[1646734]: initializing /usr/lib/i386-linux-gnu/libcuda.so.560.35.03 jan 27 17:06:44 desktop ollama[1646734]: library /usr/lib/i386-linux-gnu/libcuda.so.560.35.03 load err: /usr/lib/i386-linux-gnu/libcuda.so.560.35.03: wrong ELF class: ELFCLASS32 jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.428+01:00 level=DEBUG source=gpu.go:627 msg="skipping 32bit library" library=/usr/lib/i386-linux-gnu/libcuda.so.560.35.03 jan 27 17:06:44 desktop ollama[1646734]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03 jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuInit - 0x714a94c5d7f0 jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDriverGetVersion - 0x714a94c5d810 jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDeviceGetCount - 0x714a94c5d850 jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDeviceGet - 0x714a94c5d830 jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDeviceGetAttribute - 0x714a94c5d930 jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDeviceGetUuid - 0x714a94c5d890 jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuDeviceGetName - 0x714a94c5d870 jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuCtxCreate_v3 - 0x714a94c68060 jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuMemGetInfo_v2 - 0x714a94c73520 jan 27 17:06:44 desktop ollama[1646734]: dlsym: cuCtxDestroy - 0x714a94cce380 jan 27 17:06:44 desktop ollama[1646734]: calling cuInit jan 27 17:06:44 desktop ollama[1646734]: calling cuDriverGetVersion jan 27 17:06:44 desktop ollama[1646734]: raw version 0x2f1c jan 27 17:06:44 desktop ollama[1646734]: CUDA driver version: 12.6 jan 27 17:06:44 desktop ollama[1646734]: calling cuDeviceGetCount jan 27 17:06:44 desktop ollama[1646734]: device count 1 jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.442+01:00 level=DEBUG source=gpu.go:134 msg="detected GPUs" count=1 library=/usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03 jan 27 17:06:44 desktop ollama[1646734]: [GPU-d1756887-9770-5b36-7763-1b6bc048de4d] CUDA totalMem 11901 mb jan 27 17:06:44 desktop ollama[1646734]: [GPU-d1756887-9770-5b36-7763-1b6bc048de4d] CUDA freeMem 8208 mb jan 27 17:06:44 desktop ollama[1646734]: [GPU-d1756887-9770-5b36-7763-1b6bc048de4d] Compute Capability 8.6 jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.657+01:00 level=DEBUG source=amd_linux.go:421 msg="amdgpu driver not detected /sys/module/amdgpu" jan 27 17:06:44 desktop ollama[1646734]: releasing cuda driver library jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.657+01:00 level=INFO source=types.go:131 msg="inference compute" id=GPU-d1756887-9770-5b36-7763-1b6bc048de4d library=cuda variant=v12 compute=8.6 driver=12.6 name="NVIDIA RTX A2000 12GB" total="11.6 GiB" available="8.0 GiB" ``` ``` jan 27 17:08:19 desktop ollama[1646734]: [GIN] 2025/01/27 - 17:08:19 | 200 | 140.818µs | 127.0.0.1 | HEAD "/" jan 27 17:08:19 desktop ollama[1646734]: [GIN] 2025/01/27 - 17:08:19 | 200 | 28.825457ms | 127.0.0.1 | POST "/api/show" jan 27 17:08:19 desktop ollama[1646734]: time=2025-01-27T17:08:19.907+01:00 level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="94.0 GiB" before.free="71.1 GiB" before.free_swap="1.9 GiB" now.total="94.0 GiB" now.free="71.1 GiB" now.free_swap="1.9 GiB" jan 27 17:08:19 desktop ollama[1646734]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03 jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuInit - 0x714a94c5d7f0 jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDriverGetVersion - 0x714a94c5d810 jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDeviceGetCount - 0x714a94c5d850 jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDeviceGet - 0x714a94c5d830 jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDeviceGetAttribute - 0x714a94c5d930 jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDeviceGetUuid - 0x714a94c5d890 jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuDeviceGetName - 0x714a94c5d870 jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuCtxCreate_v3 - 0x714a94c68060 jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuMemGetInfo_v2 - 0x714a94c73520 jan 27 17:08:19 desktop ollama[1646734]: dlsym: cuCtxDestroy - 0x714a94cce380 jan 27 17:08:19 desktop ollama[1646734]: calling cuInit jan 27 17:08:19 desktop ollama[1646734]: calling cuDriverGetVersion jan 27 17:08:19 desktop ollama[1646734]: raw version 0x2f1c jan 27 17:08:19 desktop ollama[1646734]: CUDA driver version: 12.6 jan 27 17:08:19 desktop ollama[1646734]: calling cuDeviceGetCount jan 27 17:08:19 desktop ollama[1646734]: device count 1 jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.086+01:00 level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-d1756887-9770-5b36-7763-1b6bc048de4d name="NVIDIA RTX A2000 12GB" overhead="0 B" before.total="11.6 GiB" before.free="8.0 GiB" now.total="11.6 GiB" now.free="8.1 GiB" now.used="3.5 GiB" jan 27 17:08:20 desktop ollama[1646734]: releasing cuda driver library jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.086+01:00 level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x5aae4df874c0 gpu_count=1 jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.109+01:00 level=DEBUG source=sched.go:224 msg="loading first model" model=/usr/share/ollama/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.109+01:00 level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[8.1 GiB]" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.110+01:00 level=INFO source=sched.go:714 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be gpu=GPU-d1756887-9770-5b36-7763-1b6bc048de4d parallel=4 available=8691253248 required="6.5 GiB" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.110+01:00 level=DEBUG source=gpu.go:406 msg="updating system memory data" before.total="94.0 GiB" before.free="71.1 GiB" before.free_swap="1.9 GiB" now.total="94.0 GiB" now.free="71.1 GiB" now.free_swap="1.9 GiB" jan 27 17:08:20 desktop ollama[1646734]: initializing /usr/lib/x86_64-linux-gnu/libcuda.so.560.35.03 jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuInit - 0x714a94c5d7f0 jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDriverGetVersion - 0x714a94c5d810 jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDeviceGetCount - 0x714a94c5d850 jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDeviceGet - 0x714a94c5d830 jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDeviceGetAttribute - 0x714a94c5d930 jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDeviceGetUuid - 0x714a94c5d890 jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuDeviceGetName - 0x714a94c5d870 jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuCtxCreate_v3 - 0x714a94c68060 jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuMemGetInfo_v2 - 0x714a94c73520 jan 27 17:08:20 desktop ollama[1646734]: dlsym: cuCtxDestroy - 0x714a94cce380 jan 27 17:08:20 desktop ollama[1646734]: calling cuInit jan 27 17:08:20 desktop ollama[1646734]: calling cuDriverGetVersion jan 27 17:08:20 desktop ollama[1646734]: raw version 0x2f1c jan 27 17:08:20 desktop ollama[1646734]: CUDA driver version: 12.6 jan 27 17:08:20 desktop ollama[1646734]: calling cuDeviceGetCount jan 27 17:08:20 desktop ollama[1646734]: device count 1 jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=DEBUG source=gpu.go:456 msg="updating cuda memory data" gpu=GPU-d1756887-9770-5b36-7763-1b6bc048de4d name="NVIDIA RTX A2000 12GB" overhead="0 B" before.total="11.6 GiB" before.free="8.1 GiB" now.total="11.6 GiB" now.free="8.1 GiB" now.used="3.5 GiB" jan 27 17:08:20 desktop ollama[1646734]: releasing cuda driver library jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=INFO source=server.go:104 msg="system memory" total="94.0 GiB" free="71.1 GiB" free_swap="1.9 GiB" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=DEBUG source=memory.go:107 msg=evaluating library=cuda gpu_count=1 available="[8.1 GiB]" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[8.1 GiB]" memory.gpu_overhead="0 B" memory.required.full="6.5 GiB" memory.required.partial="6.5 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.5 GiB]" memory.weights.total="4.9 GiB" memory.weights.repeating="4.5 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=DEBUG source=gpu.go:713 msg="no filter required for library cpu" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/opt/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-6340dc3229b0d08ea9cc49b75d4098702983e17b4c096d57afbbf2ffc813f2be --ctx-size 8192 --batch-size 512 --n-gpu-layers 33 --verbose --threads 8 --parallel 4 --port 35083" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.280+01:00 level=DEBUG source=server.go:393 msg=subprocess environment="[PATH=/home/troed/.local/bin:/home/troed/bin:/opt/conda/bin:/opt/conda/condabin:/home/troed/.local/bin:/home/troed/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/opt/local/bin:/usr/local/cuda-12.0/bin:/home/troed/.npmglobal/bin:/opt/local/bin:/usr/local/cuda-12.0/bin:/home/troed/.npmglobal/bin LD_LIBRARY_PATH=.:/opt/local/bin]" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.281+01:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.281+01:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.281+01:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.289+01:00 level=INFO source=runner.go:936 msg="starting go runner" jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.289+01:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | AARCH64_REPACK = 1 | CPU : LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=8 jan 27 17:08:20 desktop ollama[1646734]: time=2025-01-27T17:08:20.290+01:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:35083" ```
GiteaMirror commented 2026-04-12 16:44:45 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 27, 2025):

jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=DEBUG source=common.go:85 msg="no dynamic runners detected, using only built-in"

Installed manually by copying the lib and binary into the paths I wanted

ollama finds runners relative to the path to the binary. If you haven't preserved that relationship, ollama can't find the runners.
Binary: /xxx/yyy/bin/ollama
Runners: /xxx/yyy/lib/ollama/runners/

A better way than installing into different paths is leave the ollama files where they were installed and use a symlink.

$ sudo ln -s /xxx/yyy/bin/ollama /my/special/location/ollama

Alternatively, add the path to ollama to your PATH environment variable:

$ PATH=$PATH:/xxx/yyy/bin
<!-- gh-comment-id:2616281903 --> @rick-github commented on GitHub (Jan 27, 2025): ``` jan 27 17:06:44 desktop ollama[1646734]: time=2025-01-27T17:06:44.421+01:00 level=DEBUG source=common.go:85 msg="no dynamic runners detected, using only built-in" ``` > Installed manually by copying the lib and binary into the paths I wanted ollama finds runners relative to the path to the binary. If you haven't preserved that relationship, ollama can't find the runners. Binary: /xxx/yyy/bin/ollama Runners: /xxx/yyy/lib/ollama/runners/ A better way than installing into different paths is leave the ollama files where they were installed and use a symlink. ```sh $ sudo ln -s /xxx/yyy/bin/ollama /my/special/location/ollama ``` Alternatively, add the path to ollama to your PATH environment variable: ```console $ PATH=$PATH:/xxx/yyy/bin ```
GiteaMirror commented 2026-04-12 16:44:45 -05:00
Author
Owner
Copy Link

@troed commented on GitHub (Jan 27, 2025):

edit: Actually you might be correct, sorry. I don't think I have the ollama-directory under lib. I'll check shortly.

That's good to know, Linux users probably expect ldconfig to be used. Unfortunately it doesn't help in my case since the paths are /opt/local/bin/ollama with the libraries in /opt/local/lib/

Do I understand it correctly that even though all the GPU detection and cuda libs are written out in the log it's really the runners=[cpu] that I need to figure out?

<!-- gh-comment-id:2616306843 --> @troed commented on GitHub (Jan 27, 2025): edit: Actually you might be correct, sorry. I don't think I have the ollama-directory under lib. I'll check shortly. That's good to know, Linux users probably expect ldconfig to be used. Unfortunately it doesn't help in my case since the paths are /opt/local/bin/ollama with the libraries in /opt/local/lib/ Do I understand it correctly that even though all the GPU detection and cuda libs are written out in the log it's really the runners=[cpu] that I need to figure out?
GiteaMirror commented 2026-04-12 16:44:46 -05:00
Author
Owner
Copy Link

@troed commented on GitHub (Jan 27, 2025):

Thank you - ollama not finding the lib/ollama/runners directory was indeed the cause of my problem. Correcting the directory structure according to your hint solved the issue for me.

<!-- gh-comment-id:2616809114 --> @troed commented on GitHub (Jan 27, 2025): Thank you - ollama not finding the lib/ollama/runners directory was indeed the cause of my problem. Correcting the directory structure according to your hint solved the issue for me.
GiteaMirror commented 2026-04-12 16:44:46 -05:00
Author
Owner
Copy Link

@Alexandero89 commented on GitHub (Jan 29, 2025):

i had the same issue.
by moving the ollama binary to the correct place it worked again.

@nyllewin could you please close the issue if it also worked for you.

<!-- gh-comment-id:2621566767 --> @Alexandero89 commented on GitHub (Jan 29, 2025): i had the same issue. by moving the ollama binary to the correct place it worked again. @nyllewin could you please close the issue if it also worked for you.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5503
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?