github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 08:02:14 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #2641] GPU sometimes detected, sometimes not (Windows beta) #48076

New Issue
Closed
opened 2026-04-28 06:43:27 -05:00 by GiteaMirror · 13 comments
GiteaMirror commented 2026-04-28 06:43:27 -05:00
Owner
Copy Link

Originally created by @CrispStrobe on GitHub (Feb 21, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2641

Originally assigned to: @dhiltgen on GitHub.

Using ollama 0.1.25 under Windows, sometimes my GPU (A1000) is detected:
From server.log:
time=2024-02-21T17:04:44.912+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-21T17:04:44.912+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
time=2024-02-21T17:04:44.912+01:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2603M available memory"
time=2024-02-21T17:04:44.912+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-21T17:04:44.912+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
time=2024-02-21T17:04:44.912+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-21T17:04:44.912+01:00 level=DEBUG source=payload_common.go:93 msg="ordered list of LLM libraries to try [...]
time=2024-02-21T17:04:44.912+01:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: [...]"
time=2024-02-21T17:04:44.913+01:00 level=INFO source=dyn_ext_server.go:145 msg="Initializing llama server"
[1708531484] system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 |
[1708531484] Performing pre-initialization of GPU

Sometimes not/nonworking:
time=2024-02-21T16:51:03.026+01:00 level=INFO source=gpu.go:94 msg="Detecting GPU type"
time=2024-02-21T16:51:03.026+01:00 level=INFO source=gpu.go:262 msg="Searching for GPU management library nvml.dll"
time=2024-02-21T16:51:03.026+01:00 level=DEBUG source=gpu.go:280 msg="gpu management search paths: [...]
time=2024-02-21T16:51:03.748+01:00 level=INFO source=gpu.go:99 msg="Nvidia GPU detected"
time=2024-02-21T16:51:03.749+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-21T16:51:03.755+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
time=2024-02-21T16:51:03.755+01:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2956M available memory"
time=2024-02-21T16:51:03.755+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-21T16:51:03.755+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
time=2024-02-21T16:51:03.755+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-21T16:51:03.755+01:00 level=DEBUG source=payload_common.go:93 msg="ordered list of LLM libraries to try [...]
time=2024-02-21T16:51:03.761+01:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: [...]"
time=2024-02-21T16:51:03.761+01:00 level=INFO source=dyn_ext_server.go:145 msg="Initializing llama server"
[1708530663] system info: AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 |

note the discrepancies in AVX(2) etc. this is "almost" reproducable, that is, when i run 10 starts, in between quitting ollama, i can be sure that some will be with working GPU and some not.

Originally created by @CrispStrobe on GitHub (Feb 21, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2641 Originally assigned to: @dhiltgen on GitHub. Using ollama 0.1.25 under Windows, sometimes my GPU (A1000) is detected: From server.log: time=2024-02-21T17:04:44.912+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-02-21T17:04:44.912+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" time=2024-02-21T17:04:44.912+01:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2603M available memory" time=2024-02-21T17:04:44.912+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-02-21T17:04:44.912+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" time=2024-02-21T17:04:44.912+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-02-21T17:04:44.912+01:00 level=DEBUG source=payload_common.go:93 msg="ordered list of LLM libraries to try [...] time=2024-02-21T17:04:44.912+01:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: [...]" time=2024-02-21T17:04:44.913+01:00 level=INFO source=dyn_ext_server.go:145 msg="Initializing llama server" [1708531484] system info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | [1708531484] Performing pre-initialization of GPU Sometimes not/nonworking: time=2024-02-21T16:51:03.026+01:00 level=INFO source=gpu.go:94 msg="Detecting GPU type" time=2024-02-21T16:51:03.026+01:00 level=INFO source=gpu.go:262 msg="Searching for GPU management library nvml.dll" time=2024-02-21T16:51:03.026+01:00 level=DEBUG source=gpu.go:280 msg="gpu management search paths: [...] time=2024-02-21T16:51:03.748+01:00 level=INFO source=gpu.go:99 msg="Nvidia GPU detected" time=2024-02-21T16:51:03.749+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-02-21T16:51:03.755+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" time=2024-02-21T16:51:03.755+01:00 level=DEBUG source=gpu.go:251 msg="cuda detected 1 devices with 2956M available memory" time=2024-02-21T16:51:03.755+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-02-21T16:51:03.755+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" time=2024-02-21T16:51:03.755+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-02-21T16:51:03.755+01:00 level=DEBUG source=payload_common.go:93 msg="ordered list of LLM libraries to try [...] time=2024-02-21T16:51:03.761+01:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: [...]" time=2024-02-21T16:51:03.761+01:00 level=INFO source=dyn_ext_server.go:145 msg="Initializing llama server" [1708530663] system info: AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | note the discrepancies in AVX(2) etc. this is "almost" reproducable, that is, when i run 10 starts, in between quitting ollama, i can be sure that some will be with working GPU and some not.
GiteaMirror added the bug label 2026-04-28 06:43:27 -05:00
GiteaMirror commented 2026-04-28 06:43:34 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Feb 26, 2024):

We fixed a bug recently where the cuda LLM library was incorrectly compiled with AVX2 instead of AVX. The AVX=0 is unexpected though. Can you include a bit more of the log output for the latest version you've tried so we can see which llm library it's loading, and what version it is?

<!-- gh-comment-id:1964774469 --> @dhiltgen commented on GitHub (Feb 26, 2024): We fixed a bug recently where the cuda LLM library was incorrectly compiled with AVX2 instead of AVX. The AVX=0 is unexpected though. Can you include a bit more of the log output for the latest version you've tried so we can see which llm library it's loading, and what version it is?
GiteaMirror commented 2026-04-28 06:43:35 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 12, 2024):

If you're still having problems after upgrading to the latest version, please share your server log and I'll re-open the defect, but I believe this should be resolved now.

<!-- gh-comment-id:1992329433 --> @dhiltgen commented on GitHub (Mar 12, 2024): If you're still having problems after upgrading to the latest version, please share your server log and I'll re-open the defect, but I believe this should be resolved now.
GiteaMirror commented 2026-04-28 06:43:36 -05:00
Author
Owner
Copy Link

@CrispStrobe commented on GitHub (Mar 12, 2024):

thank you. i think it actually did work once, but just tested it and it is not working, token generation speed is veeeeerrrrrryyyy sllloooooow... from the logs:
time=2024-03-12T21:41:12.613+01:00 level=INFO source=images.go:710 msg="total blobs: 4"
time=2024-03-12T21:41:12.740+01:00 level=INFO source=images.go:717 msg="total unused blobs removed: 0"
time=2024-03-12T21:41:12.744+01:00 level=INFO source=routes.go:1021 msg="Listening on 127.0.0.1:11434 (version 0.1.28)"
time=2024-03-12T21:41:12.744+01:00 level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
time=2024-03-12T21:41:13.349+01:00 level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [cuda_v11.3 cpu cpu_avx cpu_avx2]"
[GIN] 2024/03/12 - 21:41:22 | 200 | 3.8204ms | 127.0.0.1 | HEAD "/"
[GIN] 2024/03/12 - 21:41:22 | 200 | 18.2656ms | 127.0.0.1 | GET "/api/tags"
[GIN] 2024/03/12 - 21:41:28 | 200 | 4.9278ms | 127.0.0.1 | HEAD "/"
[GIN] 2024/03/12 - 21:41:28 | 200 | 5.2552ms | 127.0.0.1 | POST "/api/show"
[GIN] 2024/03/12 - 21:41:28 | 200 | 3.1001ms | 127.0.0.1 | POST "/api/show"
time=2024-03-12T21:41:29.192+01:00 level=INFO source=gpu.go:94 msg="Detecting GPU type"
time=2024-03-12T21:41:29.192+01:00 level=INFO source=gpu.go:265 msg="Searching for GPU management library nvml.dll"
time=2024-03-12T21:41:29.353+01:00 level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [c:\Windows\System32\nvml.dll C:\WINDOWS\system32\nvml.dll]"
time=2024-03-12T21:41:30.023+01:00 level=INFO source=gpu.go:99 msg="Nvidia GPU detected"
time=2024-03-12T21:41:30.023+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-12T21:41:30.056+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
time=2024-03-12T21:41:30.056+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-12T21:41:30.056+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
time=2024-03-12T21:41:30.056+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-12T21:41:30.057+01:00 level=INFO source=dyn_ext_server.go:385 msg="Updating PATH to ..."
time=2024-03-12T21:41:30.603+01:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server"
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA RTX A1000 Laptop GPU, compute capability 8.6, VMM: yes
llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from C:\Users\stc.ollama\models\blobs\sha256-5035734e13f09a626430a25d011d1f2d2b39cfbf3b2b784165d2d4d7cc6290e6 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = llama.cpp
llama_model_loader: - kv 2: llama.context_length u32 = 32768
llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
llama_model_loader: - kv 4: llama.block_count u32 = 32
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 11: general.file_type u32 = 15
llama_model_loader: - kv 12: tokenizer.ggml.model str = llama
llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["", "", "", "<0x00>", "<...
llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 19: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 20: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 21: tokenizer.chat_template str = {% for message in messages %}{{'<|im_...
llama_model_loader: - kv 22: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32000
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: model type = 7B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 7.24 B
llm_load_print_meta: model size = 4.07 GiB (4.83 BPW)
llm_load_print_meta: general.name = llama.cpp
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 2 ''
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_tensors: ggml ctx size = 0.22 MiB
llm_load_tensors: offloading 20 repeating layers to GPU
llm_load_tensors: offloaded 20/33 layers to GPU
llm_load_tensors: CPU buffer size = 3946.32 MiB
llm_load_tensors: CUDA0 buffer size = 2480.88 MiB
................................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA_Host KV buffer size = 96.00 MiB
llama_kv_cache_init: CUDA0 KV buffer size = 160.00 MiB
llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB
llama_new_context_with_model: CUDA_Host input buffer size = 13.02 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 164.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 168.00 MiB
llama_new_context_with_model: graph splits (measure): 3
{"function":"initialize","level":"INFO","line":433,"msg":"initializing slots","n_slots":1,"tid":"15528","timestamp":1710276148}
{"function":"initialize","level":"INFO","line":445,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"15528","timestamp":1710276148}
time=2024-03-12T21:42:28.864+01:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop"
[GIN] 2024/03/12 - 21:42:28 | 200 | 1m0s | 127.0.0.1 | POST "/api/chat"
{"function":"update_slots","level":"INFO","line":1565,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"13624","timestamp":1710276148}
[GIN] 2024/03/12 - 21:42:28 | 200 | 0s | 127.0.0.1 | HEAD "/"
{"function":"launch_slot_with_data","level":"INFO","line":826,"msg":"slot is processing task","slot_id":0,"task_id":0,"tid":"13624","timestamp":1710276155}
{"function":"update_slots","level":"INFO","line":1801,"msg":"slot progression","n_past":0,"n_prompt_tokens_processed":33,"slot_id":0,"task_id":0,"tid":"13624","timestamp":1710276155}
{"function":"update_slots","level":"INFO","line":1825,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":0,"tid":"13624","timestamp":1710276155}

<!-- gh-comment-id:1992550822 --> @CrispStrobe commented on GitHub (Mar 12, 2024): thank you. i think it actually did work once, but just tested it and it is not working, token generation speed is veeeeerrrrrryyyy sllloooooow... from the logs: time=2024-03-12T21:41:12.613+01:00 level=INFO source=images.go:710 msg="total blobs: 4" time=2024-03-12T21:41:12.740+01:00 level=INFO source=images.go:717 msg="total unused blobs removed: 0" time=2024-03-12T21:41:12.744+01:00 level=INFO source=routes.go:1021 msg="Listening on 127.0.0.1:11434 (version 0.1.28)" time=2024-03-12T21:41:12.744+01:00 level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..." time=2024-03-12T21:41:13.349+01:00 level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [cuda_v11.3 cpu cpu_avx cpu_avx2]" [GIN] 2024/03/12 - 21:41:22 | 200 | 3.8204ms | 127.0.0.1 | HEAD "/" [GIN] 2024/03/12 - 21:41:22 | 200 | 18.2656ms | 127.0.0.1 | GET "/api/tags" [GIN] 2024/03/12 - 21:41:28 | 200 | 4.9278ms | 127.0.0.1 | HEAD "/" [GIN] 2024/03/12 - 21:41:28 | 200 | 5.2552ms | 127.0.0.1 | POST "/api/show" [GIN] 2024/03/12 - 21:41:28 | 200 | 3.1001ms | 127.0.0.1 | POST "/api/show" time=2024-03-12T21:41:29.192+01:00 level=INFO source=gpu.go:94 msg="Detecting GPU type" time=2024-03-12T21:41:29.192+01:00 level=INFO source=gpu.go:265 msg="Searching for GPU management library nvml.dll" time=2024-03-12T21:41:29.353+01:00 level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [c:\\Windows\\System32\\nvml.dll C:\\WINDOWS\\system32\\nvml.dll]" time=2024-03-12T21:41:30.023+01:00 level=INFO source=gpu.go:99 msg="Nvidia GPU detected" time=2024-03-12T21:41:30.023+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-12T21:41:30.056+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" time=2024-03-12T21:41:30.056+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-12T21:41:30.056+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" time=2024-03-12T21:41:30.056+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-12T21:41:30.057+01:00 level=INFO source=dyn_ext_server.go:385 msg="Updating PATH to ..." time=2024-03-12T21:41:30.603+01:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server" ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA RTX A1000 Laptop GPU, compute capability 8.6, VMM: yes llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from C:\Users\stc\.ollama\models\blobs\sha256-5035734e13f09a626430a25d011d1f2d2b39cfbf3b2b784165d2d4d7cc6290e6 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = llama.cpp llama_model_loader: - kv 2: llama.context_length u32 = 32768 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 11: general.file_type u32 = 15 llama_model_loader: - kv 12: tokenizer.ggml.model str = llama llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 19: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 20: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 21: tokenizer.chat_template str = {% for message in messages %}{{'<|im_... llama_model_loader: - kv 22: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors llm_load_vocab: special tokens definition check successful ( 259/32000 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 7.24 B llm_load_print_meta: model size = 4.07 GiB (4.83 BPW) llm_load_print_meta: general.name = llama.cpp llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_tensors: ggml ctx size = 0.22 MiB llm_load_tensors: offloading 20 repeating layers to GPU llm_load_tensors: offloaded 20/33 layers to GPU llm_load_tensors: CPU buffer size = 3946.32 MiB llm_load_tensors: CUDA0 buffer size = 2480.88 MiB ................................................................................................ llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA_Host KV buffer size = 96.00 MiB llama_kv_cache_init: CUDA0 KV buffer size = 160.00 MiB llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB llama_new_context_with_model: CUDA_Host input buffer size = 13.02 MiB llama_new_context_with_model: CUDA0 compute buffer size = 164.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 168.00 MiB llama_new_context_with_model: graph splits (measure): 3 {"function":"initialize","level":"INFO","line":433,"msg":"initializing slots","n_slots":1,"tid":"15528","timestamp":1710276148} {"function":"initialize","level":"INFO","line":445,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"15528","timestamp":1710276148} time=2024-03-12T21:42:28.864+01:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop" [GIN] 2024/03/12 - 21:42:28 | 200 | 1m0s | 127.0.0.1 | POST "/api/chat" {"function":"update_slots","level":"INFO","line":1565,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"13624","timestamp":1710276148} [GIN] 2024/03/12 - 21:42:28 | 200 | 0s | 127.0.0.1 | HEAD "/" {"function":"launch_slot_with_data","level":"INFO","line":826,"msg":"slot is processing task","slot_id":0,"task_id":0,"tid":"13624","timestamp":1710276155} {"function":"update_slots","level":"INFO","line":1801,"msg":"slot progression","n_past":0,"n_prompt_tokens_processed":33,"slot_id":0,"task_id":0,"tid":"13624","timestamp":1710276155} {"function":"update_slots","level":"INFO","line":1825,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":0,"tid":"13624","timestamp":1710276155}
GiteaMirror commented 2026-04-28 06:43:36 -05:00
Author
Owner
Copy Link

@CrispStrobe commented on GitHub (Mar 12, 2024):

i actually had one video tdr failure blue screen when testing this now. anyway, i can quite ollama (from the icon in taskbar) and restart, and, with some luck, sometime it will run correctly. the log then will look like this:
time=2024-03-12T21:58:03.215+01:00 level=INFO source=gpu.go:94 msg="Detecting GPU type"
time=2024-03-12T21:58:03.215+01:00 level=INFO source=gpu.go:265 msg="Searching for GPU management library nvml.dll"
time=2024-03-12T21:58:03.254+01:00 level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [c:\Windows\System32\nvml.dll C:\WINDOWS\system32\nvml.dll]"
time=2024-03-12T21:58:04.435+01:00 level=INFO source=gpu.go:99 msg="Nvidia GPU detected"
time=2024-03-12T21:58:04.435+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-12T21:58:04.467+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
time=2024-03-12T21:58:04.467+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-12T21:58:04.467+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6"
time=2024-03-12T21:58:04.467+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-12T21:58:04.467+01:00 level=INFO source=dyn_ext_server.go:385 msg="Updating PATH to ..."
time=2024-03-12T21:58:04.530+01:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: C:\Users\stc\AppData\Local\Temp\ollama939529883\cuda_v11.3\ext_server.dll"
time=2024-03-12T21:58:04.530+01:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server"
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA RTX A1000 Laptop GPU, compute capability 8.6, VMM: yes
llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from C:\Users\stc.ollama\models\blobs\sha256-5035734e13f09a626430a25d011d1f2d2b39cfbf3b2b784165d2d4d7cc6290e6 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = llama.cpp
llama_model_loader: - kv 2: llama.context_length u32 = 32768
llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
llama_model_loader: - kv 4: llama.block_count u32 = 32
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 11: general.file_type u32 = 15
llama_model_loader: - kv 12: tokenizer.ggml.model str = llama
llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["", "", "", "<0x00>", "<...
llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 19: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 20: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 21: tokenizer.chat_template str = {% for message in messages %}{{'<|im_...
llama_model_loader: - kv 22: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32000
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: model type = 7B
llm_load_print_meta: model ftype = Q4_K - Medium
llm_load_print_meta: model params = 7.24 B
llm_load_print_meta: model size = 4.07 GiB (4.83 BPW)
llm_load_print_meta: general.name = llama.cpp
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 2 ''
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_tensors: ggml ctx size = 0.22 MiB
llm_load_tensors: offloading 20 repeating layers to GPU
llm_load_tensors: offloaded 20/33 layers to GPU
llm_load_tensors: CPU buffer size = 3946.32 MiB
llm_load_tensors: CUDA0 buffer size = 2480.88 MiB
................................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA_Host KV buffer size = 96.00 MiB
llama_kv_cache_init: CUDA0 KV buffer size = 160.00 MiB
llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB
llama_new_context_with_model: CUDA_Host input buffer size = 13.02 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 164.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 168.00 MiB
llama_new_context_with_model: graph splits (measure): 3
{"function":"initialize","level":"INFO","line":433,"msg":"initializing slots","n_slots":1,"tid":"17112","timestamp":1710277089}
{"function":"initialize","level":"INFO","line":445,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"17112","timestamp":1710277089}
time=2024-03-12T21:58:09.643+01:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop"
{"function":"update_slots","level":"INFO","line":1565,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"18812","timestamp":1710277089}
[GIN] 2024/03/12 - 21:58:09 | 200 | 6.9585749s | 127.0.0.1 | POST "/api/chat"
{"function":"launch_slot_with_data","level":"INFO","line":826,"msg":"slot is processing task","slot_id":0,"task_id":0,"tid":"18812","timestamp":1710277093}
{"function":"update_slots","level":"INFO","line":1801,"msg":"slot progression","n_past":0,"n_prompt_tokens_processed":33,"slot_id":0,"task_id":0,"tid":"18812","timestamp":1710277093}
{"function":"update_slots","level":"INFO","line":1825,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":0,"tid":"18812","timestamp":1710277093}
{"function":"print_timings","level":"INFO","line":264,"msg":"prompt eval time = 1352.45 ms / 33 tokens ( 40.98 ms per token, 24.40 tokens per second)","n_prompt_tokens_processed":33,"n_tokens_second":24.40012658489913,"slot_id":0,"t_prompt_processing":1352.452,"t_token":40.98339393939394,"task_id":0,"tid":"18812","timestamp":1710277103}

<!-- gh-comment-id:1992583082 --> @CrispStrobe commented on GitHub (Mar 12, 2024): i actually had one video tdr failure blue screen when testing this now. anyway, i can quite ollama (from the icon in taskbar) and restart, and, with some luck, sometime it will run correctly. the log then will look like this: time=2024-03-12T21:58:03.215+01:00 level=INFO source=gpu.go:94 msg="Detecting GPU type" time=2024-03-12T21:58:03.215+01:00 level=INFO source=gpu.go:265 msg="Searching for GPU management library nvml.dll" time=2024-03-12T21:58:03.254+01:00 level=INFO source=gpu.go:311 msg="Discovered GPU libraries: [c:\\Windows\\System32\\nvml.dll C:\\WINDOWS\\system32\\nvml.dll]" time=2024-03-12T21:58:04.435+01:00 level=INFO source=gpu.go:99 msg="Nvidia GPU detected" time=2024-03-12T21:58:04.435+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-12T21:58:04.467+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" time=2024-03-12T21:58:04.467+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-12T21:58:04.467+01:00 level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 8.6" time=2024-03-12T21:58:04.467+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-12T21:58:04.467+01:00 level=INFO source=dyn_ext_server.go:385 msg="Updating PATH to ..." time=2024-03-12T21:58:04.530+01:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: C:\\Users\\stc\\AppData\\Local\\Temp\\ollama939529883\\cuda_v11.3\\ext_server.dll" time=2024-03-12T21:58:04.530+01:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server" ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA RTX A1000 Laptop GPU, compute capability 8.6, VMM: yes llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from C:\Users\stc\.ollama\models\blobs\sha256-5035734e13f09a626430a25d011d1f2d2b39cfbf3b2b784165d2d4d7cc6290e6 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = llama.cpp llama_model_loader: - kv 2: llama.context_length u32 = 32768 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 11: general.file_type u32 = 15 llama_model_loader: - kv 12: tokenizer.ggml.model str = llama llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 19: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 20: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 21: tokenizer.chat_template str = {% for message in messages %}{{'<|im_... llama_model_loader: - kv 22: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q4_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors llm_load_vocab: special tokens definition check successful ( 259/32000 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 7.24 B llm_load_print_meta: model size = 4.07 GiB (4.83 BPW) llm_load_print_meta: general.name = llama.cpp llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_tensors: ggml ctx size = 0.22 MiB llm_load_tensors: offloading 20 repeating layers to GPU llm_load_tensors: offloaded 20/33 layers to GPU llm_load_tensors: CPU buffer size = 3946.32 MiB llm_load_tensors: CUDA0 buffer size = 2480.88 MiB ................................................................................................ llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA_Host KV buffer size = 96.00 MiB llama_kv_cache_init: CUDA0 KV buffer size = 160.00 MiB llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB llama_new_context_with_model: CUDA_Host input buffer size = 13.02 MiB llama_new_context_with_model: CUDA0 compute buffer size = 164.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 168.00 MiB llama_new_context_with_model: graph splits (measure): 3 {"function":"initialize","level":"INFO","line":433,"msg":"initializing slots","n_slots":1,"tid":"17112","timestamp":1710277089} {"function":"initialize","level":"INFO","line":445,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"17112","timestamp":1710277089} time=2024-03-12T21:58:09.643+01:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop" {"function":"update_slots","level":"INFO","line":1565,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"18812","timestamp":1710277089} [GIN] 2024/03/12 - 21:58:09 | 200 | 6.9585749s | 127.0.0.1 | POST "/api/chat" {"function":"launch_slot_with_data","level":"INFO","line":826,"msg":"slot is processing task","slot_id":0,"task_id":0,"tid":"18812","timestamp":1710277093} {"function":"update_slots","level":"INFO","line":1801,"msg":"slot progression","n_past":0,"n_prompt_tokens_processed":33,"slot_id":0,"task_id":0,"tid":"18812","timestamp":1710277093} {"function":"update_slots","level":"INFO","line":1825,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":0,"tid":"18812","timestamp":1710277093} {"function":"print_timings","level":"INFO","line":264,"msg":"prompt eval time = 1352.45 ms / 33 tokens ( 40.98 ms per token, 24.40 tokens per second)","n_prompt_tokens_processed":33,"n_tokens_second":24.40012658489913,"slot_id":0,"t_prompt_processing":1352.452,"t_token":40.98339393939394,"task_id":0,"tid":"18812","timestamp":1710277103}
GiteaMirror commented 2026-04-28 06:43:37 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 13, 2024):

Hmm... both log snippets you shared show it loading on the GPU, at least as much as will fit in your VRAM offloaded 20/33 layers to GPU

You'll see best performance if the whole model fits in GPU, so you can try loading a smaller model, but it does appear to be detecting the GPU correctly. The GPU will be "waiting" on the CPU for the processing of those remaining 13 layers, and if your host is "busy" that could potentially explain inconsistent performance.

The first snippet doesn't contain the final tokens per second line (cut off maybe?) Can you share what the best case vs. worst case performance you're seeing on the same model?

<!-- gh-comment-id:1994643250 --> @dhiltgen commented on GitHub (Mar 13, 2024): Hmm... both log snippets you shared show it loading on the GPU, at least as much as will fit in your VRAM `offloaded 20/33 layers to GPU` You'll see best performance if the whole model fits in GPU, so you can try loading a smaller model, but it does appear to be detecting the GPU correctly. The GPU will be "waiting" on the CPU for the processing of those remaining 13 layers, and if your host is "busy" that could potentially explain inconsistent performance. The first snippet doesn't contain the final `tokens per second` line (cut off maybe?) Can you share what the best case vs. worst case performance you're seeing on the same model?
GiteaMirror commented 2026-04-28 06:43:38 -05:00
Author
Owner
Copy Link

@CrispStrobe commented on GitHub (Mar 13, 2024):

thanks for having a look at this!
indeed, since level=DEBUG is off per default now, we no longer see the "system info:" where there was a difference visible in both cases. i can try and wait for one answer completion, but the speed in those cases will be 1 token per several minutes probably, don't think the precise number will be very discriminative ;)

ok here is the log for this
................................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CUDA_Host KV buffer size = 96.00 MiB
llama_kv_cache_init: CUDA0 KV buffer size = 160.00 MiB
llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB
llama_new_context_with_model: CUDA_Host input buffer size = 13.02 MiB
llama_new_context_with_model: CUDA0 compute buffer size = 164.00 MiB
llama_new_context_with_model: CUDA_Host compute buffer size = 168.00 MiB
llama_new_context_with_model: graph splits (measure): 3
{"function":"initialize","level":"INFO","line":433,"msg":"initializing slots","n_slots":1,"tid":"19980","timestamp":1710346509}
{"function":"initialize","level":"INFO","line":445,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"19980","timestamp":1710346509}
time=2024-03-13T17:15:09.627+01:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop"
[GIN] 2024/03/13 - 17:15:09 | 200 | 56.3419613s | 127.0.0.1 | POST "/api/chat"
{"function":"update_slots","level":"INFO","line":1565,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"8284","timestamp":1710346509}
{"function":"launch_slot_with_data","level":"INFO","line":826,"msg":"slot is processing task","slot_id":0,"task_id":0,"tid":"8284","timestamp":1710346540}
{"function":"update_slots","level":"INFO","line":1801,"msg":"slot progression","n_past":0,"n_prompt_tokens_processed":32,"slot_id":0,"task_id":0,"tid":"8284","timestamp":1710346540}
{"function":"update_slots","level":"INFO","line":1825,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":0,"tid":"8284","timestamp":1710346540}
{"function":"print_timings","level":"INFO","line":264,"msg":"prompt eval time = 50175.68 ms / 32 tokens ( 1567.99 ms per token, 0.64 tokens per second)","n_prompt_tokens_processed":32,"n_tokens_second":0.6377591439614114,"slot_id":0,"t_prompt_processing":50175.682,"t_token":1567.9900625,"task_id":0,"tid":"8284","timestamp":1710349498}
{"function":"print_timings","level":"INFO","line":278,"msg":"generation eval time = 2908629.24 ms / 85 runs (34219.17 ms per token, 0.03 tokens per second)","n_decoded":85,"n_tokens_second":0.02922338767491262,"slot_id":0,"t_token":34219.16757647059,"t_token_generation":2908629.244,"task_id":0,"tid":"8284","timestamp":1710349498}
{"function":"print_timings","level":"INFO","line":287,"msg":" total time = 2958804.93 ms","slot_id":0,"t_prompt_processing":50175.682,"t_token_generation":2908629.244,"t_total":2958804.926,"task_id":0,"tid":"8284","timestamp":1710349498}
{"function":"update_slots","level":"INFO","line":1635,"msg":"slot released","n_cache_tokens":117,"n_ctx":2048,"n_past":116,"n_system_tokens":0,"slot_id":0,"task_id":0,"tid":"8284","timestamp":1710349498,"truncated":false}
[GIN] 2024/03/13 - 18:04:58 | 200 | 49m18s | 127.0.0.1 | POST "/api/chat"

according to windows task manager, gpu uses 3.2 of 4.0 gb but is at 0% usage.
this behavior is reproducable, and it is not specific to some model, and the same models run fine on the same machine eg im lm studio.

will try with
$env:OLLAMA_DEBUG="1"
next... any other ideas?

<!-- gh-comment-id:1994882476 --> @CrispStrobe commented on GitHub (Mar 13, 2024): thanks for having a look at this! indeed, since level=DEBUG is off per default now, we no longer see the "system info:" where there was a difference visible in both cases. i can try and wait for one answer completion, but the speed in those cases will be 1 token per several minutes probably, don't think the precise number will be very discriminative ;) ok here is the log for this ................................................................................................ llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA_Host KV buffer size = 96.00 MiB llama_kv_cache_init: CUDA0 KV buffer size = 160.00 MiB llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB llama_new_context_with_model: CUDA_Host input buffer size = 13.02 MiB llama_new_context_with_model: CUDA0 compute buffer size = 164.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 168.00 MiB llama_new_context_with_model: graph splits (measure): 3 {"function":"initialize","level":"INFO","line":433,"msg":"initializing slots","n_slots":1,"tid":"19980","timestamp":1710346509} {"function":"initialize","level":"INFO","line":445,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"19980","timestamp":1710346509} time=2024-03-13T17:15:09.627+01:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop" [GIN] 2024/03/13 - 17:15:09 | 200 | 56.3419613s | 127.0.0.1 | POST "/api/chat" {"function":"update_slots","level":"INFO","line":1565,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"8284","timestamp":1710346509} {"function":"launch_slot_with_data","level":"INFO","line":826,"msg":"slot is processing task","slot_id":0,"task_id":0,"tid":"8284","timestamp":1710346540} {"function":"update_slots","level":"INFO","line":1801,"msg":"slot progression","n_past":0,"n_prompt_tokens_processed":32,"slot_id":0,"task_id":0,"tid":"8284","timestamp":1710346540} {"function":"update_slots","level":"INFO","line":1825,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":0,"tid":"8284","timestamp":1710346540} {"function":"print_timings","level":"INFO","line":264,"msg":"prompt eval time = 50175.68 ms / 32 tokens ( 1567.99 ms per token, 0.64 tokens per second)","n_prompt_tokens_processed":32,"n_tokens_second":0.6377591439614114,"slot_id":0,"t_prompt_processing":50175.682,"t_token":1567.9900625,"task_id":0,"tid":"8284","timestamp":1710349498} {"function":"print_timings","level":"INFO","line":278,"msg":"generation eval time = 2908629.24 ms / 85 runs (34219.17 ms per token, 0.03 tokens per second)","n_decoded":85,"n_tokens_second":0.02922338767491262,"slot_id":0,"t_token":34219.16757647059,"t_token_generation":2908629.244,"task_id":0,"tid":"8284","timestamp":1710349498} {"function":"print_timings","level":"INFO","line":287,"msg":" total time = 2958804.93 ms","slot_id":0,"t_prompt_processing":50175.682,"t_token_generation":2908629.244,"t_total":2958804.926,"task_id":0,"tid":"8284","timestamp":1710349498} {"function":"update_slots","level":"INFO","line":1635,"msg":"slot released","n_cache_tokens":117,"n_ctx":2048,"n_past":116,"n_system_tokens":0,"slot_id":0,"task_id":0,"tid":"8284","timestamp":1710349498,"truncated":false} [GIN] 2024/03/13 - 18:04:58 | 200 | 49m18s | 127.0.0.1 | POST "/api/chat" according to windows task manager, gpu uses 3.2 of 4.0 gb but is at 0% usage. this behavior is reproducable, and it is not specific to some model, and the same models run fine on the same machine eg im lm studio. will try with $env:OLLAMA_DEBUG="1" next... any other ideas?
GiteaMirror commented 2026-04-28 06:43:39 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 13, 2024):

In the CLI you can run /set verbose and it will show some information at the end of the response, including tokens per second.

From the log you just posted I see 0.03 tokens per second which is a lot slower than I'd expect from loading 20/33 layers to the GPU when you sometimes get 24 TPS.

Can you try running a smaller model that fully fits in the GPU to test to see if this is specific to some layers loading on the CPU?

Are there any AV products that might be present and doing deeper analysis sporadically? In taskmanager, do you see any other processes/components using lots of CPU when ollama is processing a prompt?

<!-- gh-comment-id:1996054791 --> @dhiltgen commented on GitHub (Mar 13, 2024): In the CLI you can run `/set verbose` and it will show some information at the end of the response, including tokens per second. From the log you just posted I see `0.03 tokens per second` which is a lot slower than I'd expect from loading 20/33 layers to the GPU when you sometimes get 24 TPS. Can you try running a smaller model that fully fits in the GPU to test to see if this is specific to some layers loading on the CPU? Are there any AV products that might be present and doing deeper analysis sporadically? In taskmanager, do you see any other processes/components using lots of CPU when ollama is processing a prompt?
GiteaMirror commented 2026-04-28 06:43:41 -05:00
Author
Owner
Copy Link

@CrispStrobe commented on GitHub (Mar 14, 2024):

thanks.
probably i described the problem in too unprecise words, sorry for that.
is is dependend upon how ollama starts and will keep this way forever, not be faster some time during the same session.
the only way to get it to work is to quit ollama completely (i do this by rightclick/ quit on the icon in the taskbar) and start it again. then, after some tries, sometimes it will run correctly and eg wits 24 tps. other times, it will run at say 0.03 tps, and that for the whole session.
there are no other av processes running.
i can try with a <4gb model of course. but note that sometimes it works as it should. and the behavior i described is the same for other 7-13b models. and they all work in lm studio eg also with partial offloading, which also invoked llama.cpp as i understand it.
also, i sometimes get bsods.

<!-- gh-comment-id:1996524356 --> @CrispStrobe commented on GitHub (Mar 14, 2024): thanks. probably i described the problem in too unprecise words, sorry for that. is is dependend upon how ollama starts and will keep this way forever, not be faster some time during the same session. the only way to get it to work is to quit ollama completely (i do this by rightclick/ quit on the icon in the taskbar) and start it again. then, after some tries, sometimes it will run correctly and eg wits 24 tps. other times, it will run at say 0.03 tps, and that for the whole session. there are no other av processes running. i can try with a <4gb model of course. but note that sometimes it works as it should. and the behavior i described is the same for other 7-13b models. and they all work in lm studio eg also with partial offloading, which also invoked llama.cpp as i understand it. also, i sometimes get bsods.
GiteaMirror commented 2026-04-28 06:43:42 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 18, 2024):

Interesting. If BSODs are involved, that usually points to driver bugs or hardware faults. Can you make sure you're running the latet nvidia driver on your system?

<!-- gh-comment-id:2003080695 --> @dhiltgen commented on GitHub (Mar 18, 2024): Interesting. If BSODs are involved, that usually points to driver bugs or hardware faults. Can you make sure you're running the latet nvidia driver on your system?
GiteaMirror commented 2026-04-28 06:43:43 -05:00
Author
Owner
Copy Link

@CrispStrobe commented on GitHub (Mar 18, 2024):

thanks, version is 546.12, and i had no bsods yet with eg lm studio

<!-- gh-comment-id:2004252577 --> @CrispStrobe commented on GitHub (Mar 18, 2024): thanks, version is 546.12, and i had no bsods yet with eg lm studio
GiteaMirror commented 2026-04-28 06:43:44 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 15, 2024):

You might want to take a look at #3511 and see if the problem (and workarounds) described there apply to your scenario.

<!-- gh-comment-id:2057921828 --> @dhiltgen commented on GitHub (Apr 15, 2024): You might want to take a look at #3511 and see if the problem (and workarounds) described there apply to your scenario.
GiteaMirror commented 2026-04-28 06:43:45 -05:00
Author
Owner
Copy Link

@CrispStrobe commented on GitHub (Apr 16, 2024):

thank you. it seems to work well at the moment (and after updating to version 0.1.31).

<!-- gh-comment-id:2058588623 --> @CrispStrobe commented on GitHub (Apr 16, 2024): thank you. it seems to work well at the moment (and after updating to version 0.1.31).
GiteaMirror commented 2026-04-28 06:43:47 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 16, 2024):

That's great to hear!

<!-- gh-comment-id:2059461310 --> @dhiltgen commented on GitHub (Apr 16, 2024): That's great to hear!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#48076
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?