github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #7745] gpu VRAM usage didn't recover within timeout on llama3.2-vision:90b #82628

New Issue
Closed
opened 2026-05-09 15:25:22 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-05-09 15:25:22 -05:00
Owner
Copy Link

Originally created by @ergosumdre on GitHub (Nov 19, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7745

What is the issue?

I'm encountering a 'timed out waiting for llama runner to start' error when executing the following command:
ollama run llama3.2-vision:90b

I have 64GB of VRAM, and I’m able to run other models without any issues. However, this specific model doesn’t seem to work.

Here are the server logs:

dre@dre-slimserver-3t:~$ journalctl -u ollama --no-pager | tail -200
Nov 19 11:54:18 dre-slimserver-3t ollama[569358]: llm_load_tensors: CUDA3 buffer size = 6520.80 MiB
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: n_ctx = 8192
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: n_batch = 2048
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: n_ubatch = 512
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: flash_attn = 0
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: freq_base = 500000.0
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: freq_scale = 1
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_kv_cache_init: CUDA0 KV buffer size = 672.00 MiB
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_kv_cache_init: CUDA1 KV buffer size = 640.00 MiB
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_kv_cache_init: CUDA2 KV buffer size = 640.00 MiB
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_kv_cache_init: CUDA3 KV buffer size = 608.00 MiB
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: KV self size = 2560.00 MiB, K (f16): 1280.00 MiB, V (f16): 1280.00 MiB
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA_Host output buffer size = 2.08 MiB
Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: pipeline parallelism enabled (n_copies=4)
Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA0 compute buffer size = 1216.01 MiB
Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA1 compute buffer size = 1216.01 MiB
Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA2 compute buffer size = 1216.01 MiB
Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA3 compute buffer size = 1216.02 MiB
Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA_Host compute buffer size = 80.02 MiB
Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: graph nodes = 2566
Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: graph splits = 5
Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:54:24.228-06:00 level=INFO source=server.go:601 msg="llama runner started in 201.30 seconds"
Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:54:24 | 200 | 3m22s | 100.100.70.109 | POST "/api/generate"
Nov 19 11:54:55 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:54:55 | 200 | 3.021552133s | 100.100.70.109 | POST "/api/chat"
Nov 19 11:55:02 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:02 | 200 | 37.298µs | 100.100.70.109 | HEAD "/"
Nov 19 11:55:02 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:02 | 200 | 53.04444ms | 100.100.70.109 | POST "/api/show"
Nov 19 11:55:02 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:02 | 200 | 55.176054ms | 100.100.70.109 | POST "/api/generate"
Nov 19 11:55:07 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:07 | 200 | 1.876809867s | 100.100.70.109 | POST "/api/chat"
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: loaded meta data with 22 key-value pairs and 723 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-a09affdc2dffd0db03e7f0d1344c374d85add72ea4c715aa69b7b427f03d35d3 (version GGUF V3 (latest))
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 0: general.architecture str = llama
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 1: general.name str = Meta-Llama-3-70B-Instruct
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 2: llama.block_count u32 = 80
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 3: llama.context_length u32 = 8192
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 4: llama.embedding_length u32 = 8192
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 5: llama.feed_forward_length u32 = 28672
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 6: llama.attention.head_count u32 = 64
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 8: llama.rope.freq_base f32 = 500000.000000
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 10: general.file_type u32 = 10
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 11: llama.vocab_size u32 = 128256
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 14: tokenizer.ggml.pre str = llama-bpe
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,128256] = ["!", """, "#", "$", "%", "&", "'", ...
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 128000
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 128009
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 20: tokenizer.chat_template str = {% set loop_messages = messages %}{% ...
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 21: general.quantization_version u32 = 2
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - type f32: 161 tensors
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q2_K: 321 tensors
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q3_K: 160 tensors
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q5_K: 80 tensors
Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q6_K: 1 tensors
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_vocab: special tokens cache size = 256
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_vocab: token to piece cache size = 0.8000 MB
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: format = GGUF V3 (latest)
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: arch = llama
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: vocab type = BPE
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_vocab = 128256
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_merges = 280147
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: vocab_only = 1
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model type = ?B
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model ftype = all F32
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model params = 70.55 B
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model size = 24.56 GiB (2.99 BPW)
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: general.name = Meta-Llama-3-70B-Instruct
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOS token = 128009 '<|eot_id|>'
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: LF token = 128 'Ä'
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOG token = 128009 '<|eot_id|>'
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: max token length = 256
Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llama_model_load: vocab only - skipping tensors
Nov 19 11:55:42 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:42 | 200 | 28.364µs | 100.100.70.109 | HEAD "/"
Nov 19 11:55:42 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:42 | 200 | 26.459µs | 100.100.70.109 | GET "/api/ps"
Nov 19 11:55:47 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:47 | 200 | 33.907µs | 100.100.70.109 | HEAD "/"
Nov 19 11:55:47 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:47 | 200 | 37.95µs | 100.100.70.109 | GET "/api/ps"
Nov 19 11:56:32 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:56:32 | 200 | 1m19s | 100.100.70.109 | POST "/api/chat"
Nov 19 11:57:02 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:57:02 | 200 | 31.326µs | 100.100.70.109 | HEAD "/"
Nov 19 11:57:02 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:57:02 | 200 | 36.847003ms | 100.100.70.109 | POST "/api/show"
Nov 19 11:57:02 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:02.933-06:00 level=WARN source=sched.go:137 msg="mllama doesn't support parallel requests yet"
Nov 19 11:57:03 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:03.647-06:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-3b28212c-2887-4f2a-6f4b-65a53d0f55ad library=cuda total="15.8 GiB" available="6.5 GiB"
Nov 19 11:57:03 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:03.647-06:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-a9fe4c2f-a87c-c645-82ef-2da178797d63 library=cuda total="15.8 GiB" available="6.6 GiB"
Nov 19 11:57:03 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:03.647-06:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-8e52c368-ae0c-66f4-0f9f-d9b1b5682d23 library=cuda total="15.8 GiB" available="7.4 GiB"
Nov 19 11:57:03 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:03.647-06:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-bb673042-8563-32d8-a436-fddbe6b0de03 library=cuda total="15.8 GiB" available="7.4 GiB"
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.079-06:00 level=INFO source=server.go:105 msg="system memory" total="125.7 GiB" free="120.9 GiB" free_swap="2.0 GiB"
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.083-06:00 level=INFO source=memory.go:343 msg="offload to cuda" projector.weights="1.9 GiB" projector.graph="2.8 GiB" layers.requested=-1 layers.model=101 layers.offload=96 layers.split=16,27,27,26 memory.available="[15.0 GiB 15.5 GiB 15.5 GiB 15.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="62.9 GiB" memory.required.partial="60.0 GiB" memory.required.kv="1.6 GiB" memory.required.allocations="[14.5 GiB 15.2 GiB 15.2 GiB 15.1 GiB]" memory.weights.total="49.3 GiB" memory.weights.repeating="48.5 GiB" memory.weights.nonrepeating="822.0 MiB" memory.graph.full="1.1 GiB" memory.graph.partial="1.1 GiB"
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.085-06:00 level=INFO source=server.go:383 msg="starting llama server" cmd="/tmp/ollama1922152689/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-da63a910e34997d50c9f21cc7f16996d1e76e1c128b13319edd68348f760ecc7 --ctx-size 2048 --batch-size 512 --n-gpu-layers 96 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-6b6c374d159e097509b33e9fda648c178c903959fc0c7dbfae487cc8d958093e --threads 10 --parallel 1 --tensor-split 16,27,27,26 --port 34707"
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.085-06:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.085-06:00 level=INFO source=server.go:562 msg="waiting for llama runner to start responding"
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.085-06:00 level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server error"
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.134-06:00 level=INFO source=runner.go:883 msg="starting go runner"
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.135-06:00 level=INFO source=runner.go:884 msg=system info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | cgo(gcc)" threads=10
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.135-06:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:34707"
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: loaded meta data with 27 key-value pairs and 984 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-da63a910e34997d50c9f21cc7f16996d1e76e1c128b13319edd68348f760ecc7 (version GGUF V3 (latest))
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 0: general.architecture str = mllama
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 1: general.type str = model
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 2: general.name str = Model
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 3: general.size_label str = 88B
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 4: mllama.block_count u32 = 100
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 5: mllama.context_length u32 = 131072
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 6: mllama.embedding_length u32 = 8192
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 7: mllama.feed_forward_length u32 = 28672
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 8: mllama.attention.head_count u32 = 64
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 9: mllama.attention.head_count_kv u32 = 8
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 10: mllama.rope.freq_base f32 = 500000.000000
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 11: mllama.attention.layer_norm_rms_epsilon f32 = 0.000010
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 12: general.file_type u32 = 15
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 13: mllama.vocab_size u32 = 128256
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 14: mllama.rope.dimension_count u32 = 128
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 15: mllama.attention.cross_attention_layers arr[i32,20] = [3, 8, 13, 18, 23, 28, 33, 38, 43, 48...
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 16: tokenizer.ggml.add_bos_token bool = true
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 17: tokenizer.ggml.model str = gpt2
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 18: tokenizer.ggml.pre str = llama-bpe
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 19: tokenizer.ggml.tokens arr[str,128257] = ["!", """, "#", "$", "%", "&", "'", ...
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 20: tokenizer.ggml.token_type arr[i32,128257] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.337-06:00 level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server loading model"
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 21: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 22: tokenizer.ggml.bos_token_id u32 = 128000
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 23: tokenizer.ggml.eos_token_id u32 = 128009
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 24: tokenizer.ggml.padding_token_id u32 = 128004
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 25: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ...
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 26: general.quantization_version u32 = 2
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - type f32: 282 tensors
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q4_K: 611 tensors
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q6_K: 91 tensors
Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llm_load_vocab: special tokens cache size = 257
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_vocab: token to piece cache size = 0.7999 MB
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: format = GGUF V3 (latest)
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: arch = mllama
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: vocab type = BPE
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_vocab = 128256
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_merges = 280147
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: vocab_only = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_ctx_train = 131072
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_embd = 8192
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_layer = 100
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_head = 64
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_head_kv = 8
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_rot = 128
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_swa = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_embd_head_k = 128
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_embd_head_v = 128
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_gqa = 8
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_embd_k_gqa = 1024
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_embd_v_gqa = 1024
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: f_norm_eps = 0.0e+00
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: f_norm_rms_eps = 1.0e-05
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: f_clamp_kqv = 0.0e+00
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: f_logit_scale = 0.0e+00
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_ff = 28672
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_expert = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_expert_used = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: causal attn = 1
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: pooling type = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: rope type = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: rope scaling = linear
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: freq_base_train = 500000.0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: freq_scale_train = 1
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_ctx_orig_yarn = 131072
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: rope_finetuned = unknown
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: ssm_d_conv = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: ssm_d_inner = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: ssm_d_state = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: ssm_dt_rank = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: ssm_dt_b_c_rms = 0
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model type = ?B
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model ftype = Q4_K - Medium
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model params = 87.67 B
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model size = 49.08 GiB (4.81 BPW)
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: general.name = Model
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOS token = 128009 '<|eot_id|>'
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: PAD token = 128004 '<|finetune_right_pad_id|>'
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: LF token = 128 'Ä'
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOM token = 128008 '<|eom_id|>'
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOG token = 128008 '<|eom_id|>'
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOG token = 128009 '<|eot_id|>'
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: max token length = 256
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llama_model_load: vocab mismatch 128256 !- 128257 ...
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: ggml_cuda_init: found 4 CUDA devices:
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: Device 0: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: Device 1: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: Device 2: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: Device 3: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes
Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_tensors: ggml ctx size = 2.25 MiB
Nov 19 12:02:09 dre-slimserver-3t ollama[569358]: time=2024-11-19T12:02:07.166-06:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="timed out waiting for llama runner to start - progress 0.00 - "
Nov 19 12:02:09 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 12:02:07 | 500 | 5m4s | 100.100.70.109 | POST "/api/generate"
Nov 19 12:02:12 dre-slimserver-3t ollama[569358]: time=2024-11-19T12:02:12.413-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.247257499 model=/usr/share/ollama/.ollama/models/blobs/sha256-da63a910e34997d50c9f21cc7f16996d1e76e1c128b13319edd68348f760ecc7
Nov 19 12:02:13 dre-slimserver-3t ollama[569358]: time=2024-11-19T12:02:13.171-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.005048646 model=/usr/share/ollama/.ollama/models/blobs/sha256-da63a910e34997d50c9f21cc7f16996d1e76e1c128b13319edd68348f760ecc7
Nov 19 12:02:13 dre-slimserver-3t ollama[569358]: time=2024-11-19T12:02:13.802-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.635933639 model=/usr/share/ollama/.ollama/models/blobs/sha256-da63a910e34997d50c9f21cc7f16996d1e76e1c128b13319edd68348f760ecc7

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.4.2

Originally created by @ergosumdre on GitHub (Nov 19, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7745 ### What is the issue? I'm encountering a 'timed out waiting for llama runner to start' error when executing the following command: `ollama run llama3.2-vision:90b` I have 64GB of VRAM, and I’m able to run other models without any issues. However, this specific model doesn’t seem to work. Here are the server logs: > dre@dre-slimserver-3t:~$ journalctl -u ollama --no-pager | tail -200 > Nov 19 11:54:18 dre-slimserver-3t ollama[569358]: llm_load_tensors: CUDA3 buffer size = 6520.80 MiB > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: n_ctx = 8192 > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: n_batch = 2048 > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: n_ubatch = 512 > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: flash_attn = 0 > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: freq_base = 500000.0 > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: freq_scale = 1 > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_kv_cache_init: CUDA0 KV buffer size = 672.00 MiB > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_kv_cache_init: CUDA1 KV buffer size = 640.00 MiB > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_kv_cache_init: CUDA2 KV buffer size = 640.00 MiB > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_kv_cache_init: CUDA3 KV buffer size = 608.00 MiB > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: KV self size = 2560.00 MiB, K (f16): 1280.00 MiB, V (f16): 1280.00 MiB > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA_Host output buffer size = 2.08 MiB > Nov 19 11:54:23 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: pipeline parallelism enabled (n_copies=4) > Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA0 compute buffer size = 1216.01 MiB > Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA1 compute buffer size = 1216.01 MiB > Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA2 compute buffer size = 1216.01 MiB > Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA3 compute buffer size = 1216.02 MiB > Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: CUDA_Host compute buffer size = 80.02 MiB > Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: graph nodes = 2566 > Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: llama_new_context_with_model: graph splits = 5 > Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:54:24.228-06:00 level=INFO source=server.go:601 msg="llama runner started in 201.30 seconds" > Nov 19 11:54:24 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:54:24 | 200 | 3m22s | 100.100.70.109 | POST "/api/generate" > Nov 19 11:54:55 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:54:55 | 200 | 3.021552133s | 100.100.70.109 | POST "/api/chat" > Nov 19 11:55:02 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:02 | 200 | 37.298µs | 100.100.70.109 | HEAD "/" > Nov 19 11:55:02 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:02 | 200 | 53.04444ms | 100.100.70.109 | POST "/api/show" > Nov 19 11:55:02 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:02 | 200 | 55.176054ms | 100.100.70.109 | POST "/api/generate" > Nov 19 11:55:07 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:07 | 200 | 1.876809867s | 100.100.70.109 | POST "/api/chat" > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: loaded meta data with 22 key-value pairs and 723 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-a09affdc2dffd0db03e7f0d1344c374d85add72ea4c715aa69b7b427f03d35d3 (version GGUF V3 (latest)) > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 0: general.architecture str = llama > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 1: general.name str = Meta-Llama-3-70B-Instruct > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 2: llama.block_count u32 = 80 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 3: llama.context_length u32 = 8192 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 4: llama.embedding_length u32 = 8192 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 5: llama.feed_forward_length u32 = 28672 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 6: llama.attention.head_count u32 = 64 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 8: llama.rope.freq_base f32 = 500000.000000 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 10: general.file_type u32 = 10 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 11: llama.vocab_size u32 = 128256 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 14: tokenizer.ggml.pre str = llama-bpe > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 128000 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 128009 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 20: tokenizer.chat_template str = {% set loop_messages = messages %}{% ... > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 21: general.quantization_version u32 = 2 > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - type f32: 161 tensors > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q2_K: 321 tensors > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q3_K: 160 tensors > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q5_K: 80 tensors > Nov 19 11:55:13 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q6_K: 1 tensors > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_vocab: special tokens cache size = 256 > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_vocab: token to piece cache size = 0.8000 MB > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: format = GGUF V3 (latest) > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: arch = llama > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: vocab type = BPE > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_vocab = 128256 > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_merges = 280147 > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: vocab_only = 1 > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model type = ?B > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model ftype = all F32 > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model params = 70.55 B > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model size = 24.56 GiB (2.99 BPW) > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: general.name = Meta-Llama-3-70B-Instruct > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>' > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOS token = 128009 '<|eot_id|>' > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: LF token = 128 'Ä' > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOT token = 128009 '<|eot_id|>' > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOG token = 128009 '<|eot_id|>' > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llm_load_print_meta: max token length = 256 > Nov 19 11:55:14 dre-slimserver-3t ollama[569358]: llama_model_load: vocab only - skipping tensors > Nov 19 11:55:42 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:42 | 200 | 28.364µs | 100.100.70.109 | HEAD "/" > Nov 19 11:55:42 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:42 | 200 | 26.459µs | 100.100.70.109 | GET "/api/ps" > Nov 19 11:55:47 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:47 | 200 | 33.907µs | 100.100.70.109 | HEAD "/" > Nov 19 11:55:47 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:55:47 | 200 | 37.95µs | 100.100.70.109 | GET "/api/ps" > Nov 19 11:56:32 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:56:32 | 200 | 1m19s | 100.100.70.109 | POST "/api/chat" > Nov 19 11:57:02 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:57:02 | 200 | 31.326µs | 100.100.70.109 | HEAD "/" > Nov 19 11:57:02 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 11:57:02 | 200 | 36.847003ms | 100.100.70.109 | POST "/api/show" > Nov 19 11:57:02 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:02.933-06:00 level=WARN source=sched.go:137 msg="mllama doesn't support parallel requests yet" > Nov 19 11:57:03 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:03.647-06:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-3b28212c-2887-4f2a-6f4b-65a53d0f55ad library=cuda total="15.8 GiB" available="6.5 GiB" > Nov 19 11:57:03 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:03.647-06:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-a9fe4c2f-a87c-c645-82ef-2da178797d63 library=cuda total="15.8 GiB" available="6.6 GiB" > Nov 19 11:57:03 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:03.647-06:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-8e52c368-ae0c-66f4-0f9f-d9b1b5682d23 library=cuda total="15.8 GiB" available="7.4 GiB" > Nov 19 11:57:03 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:03.647-06:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-bb673042-8563-32d8-a436-fddbe6b0de03 library=cuda total="15.8 GiB" available="7.4 GiB" > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.079-06:00 level=INFO source=server.go:105 msg="system memory" total="125.7 GiB" free="120.9 GiB" free_swap="2.0 GiB" > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.083-06:00 level=INFO source=memory.go:343 msg="offload to cuda" projector.weights="1.9 GiB" projector.graph="2.8 GiB" layers.requested=-1 layers.model=101 layers.offload=96 layers.split=16,27,27,26 memory.available="[15.0 GiB 15.5 GiB 15.5 GiB 15.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="62.9 GiB" memory.required.partial="60.0 GiB" memory.required.kv="1.6 GiB" memory.required.allocations="[14.5 GiB 15.2 GiB 15.2 GiB 15.1 GiB]" memory.weights.total="49.3 GiB" memory.weights.repeating="48.5 GiB" memory.weights.nonrepeating="822.0 MiB" memory.graph.full="1.1 GiB" memory.graph.partial="1.1 GiB" > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.085-06:00 level=INFO source=server.go:383 msg="starting llama server" cmd="/tmp/ollama1922152689/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-da63a910e34997d50c9f21cc7f16996d1e76e1c128b13319edd68348f760ecc7 --ctx-size 2048 --batch-size 512 --n-gpu-layers 96 --mmproj /usr/share/ollama/.ollama/models/blobs/sha256-6b6c374d159e097509b33e9fda648c178c903959fc0c7dbfae487cc8d958093e --threads 10 --parallel 1 --tensor-split 16,27,27,26 --port 34707" > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.085-06:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.085-06:00 level=INFO source=server.go:562 msg="waiting for llama runner to start responding" > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.085-06:00 level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server error" > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.134-06:00 level=INFO source=runner.go:883 msg="starting go runner" > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.135-06:00 level=INFO source=runner.go:884 msg=system info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | cgo(gcc)" threads=10 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.135-06:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:34707" > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: loaded meta data with 27 key-value pairs and 984 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-da63a910e34997d50c9f21cc7f16996d1e76e1c128b13319edd68348f760ecc7 (version GGUF V3 (latest)) > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 0: general.architecture str = mllama > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 1: general.type str = model > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 2: general.name str = Model > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 3: general.size_label str = 88B > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 4: mllama.block_count u32 = 100 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 5: mllama.context_length u32 = 131072 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 6: mllama.embedding_length u32 = 8192 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 7: mllama.feed_forward_length u32 = 28672 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 8: mllama.attention.head_count u32 = 64 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 9: mllama.attention.head_count_kv u32 = 8 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 10: mllama.rope.freq_base f32 = 500000.000000 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 11: mllama.attention.layer_norm_rms_epsilon f32 = 0.000010 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 12: general.file_type u32 = 15 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 13: mllama.vocab_size u32 = 128256 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 14: mllama.rope.dimension_count u32 = 128 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 15: mllama.attention.cross_attention_layers arr[i32,20] = [3, 8, 13, 18, 23, 28, 33, 38, 43, 48... > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 16: tokenizer.ggml.add_bos_token bool = true > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 17: tokenizer.ggml.model str = gpt2 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 18: tokenizer.ggml.pre str = llama-bpe > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 19: tokenizer.ggml.tokens arr[str,128257] = ["!", "\"", "#", "$", "%", "&", "'", ... > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 20: tokenizer.ggml.token_type arr[i32,128257] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: time=2024-11-19T11:57:07.337-06:00 level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server loading model" > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 21: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 22: tokenizer.ggml.bos_token_id u32 = 128000 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 23: tokenizer.ggml.eos_token_id u32 = 128009 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 24: tokenizer.ggml.padding_token_id u32 = 128004 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 25: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ... > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - kv 26: general.quantization_version u32 = 2 > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - type f32: 282 tensors > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q4_K: 611 tensors > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llama_model_loader: - type q6_K: 91 tensors > Nov 19 11:57:07 dre-slimserver-3t ollama[569358]: llm_load_vocab: special tokens cache size = 257 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_vocab: token to piece cache size = 0.7999 MB > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: format = GGUF V3 (latest) > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: arch = mllama > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: vocab type = BPE > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_vocab = 128256 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_merges = 280147 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: vocab_only = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_ctx_train = 131072 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_embd = 8192 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_layer = 100 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_head = 64 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_head_kv = 8 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_rot = 128 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_swa = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_embd_head_k = 128 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_embd_head_v = 128 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_gqa = 8 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_embd_k_gqa = 1024 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_embd_v_gqa = 1024 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: f_norm_eps = 0.0e+00 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: f_norm_rms_eps = 1.0e-05 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: f_logit_scale = 0.0e+00 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_ff = 28672 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_expert = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_expert_used = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: causal attn = 1 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: pooling type = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: rope type = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: rope scaling = linear > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: freq_base_train = 500000.0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: freq_scale_train = 1 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: n_ctx_orig_yarn = 131072 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: rope_finetuned = unknown > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: ssm_d_conv = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: ssm_d_inner = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: ssm_d_state = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: ssm_dt_rank = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: ssm_dt_b_c_rms = 0 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model type = ?B > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model ftype = Q4_K - Medium > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model params = 87.67 B > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: model size = 49.08 GiB (4.81 BPW) > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: general.name = Model > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>' > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOS token = 128009 '<|eot_id|>' > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: PAD token = 128004 '<|finetune_right_pad_id|>' > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: LF token = 128 'Ä' > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOT token = 128009 '<|eot_id|>' > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOM token = 128008 '<|eom_id|>' > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOG token = 128008 '<|eom_id|>' > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: EOG token = 128009 '<|eot_id|>' > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_print_meta: max token length = 256 > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llama_model_load: vocab mismatch 128256 !- 128257 ... > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: ggml_cuda_init: found 4 CUDA devices: > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: Device 0: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: Device 1: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: Device 2: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: Device 3: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes > Nov 19 11:57:08 dre-slimserver-3t ollama[569358]: llm_load_tensors: ggml ctx size = 2.25 MiB > Nov 19 12:02:09 dre-slimserver-3t ollama[569358]: time=2024-11-19T12:02:07.166-06:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="timed out waiting for llama runner to start - progress 0.00 - " > Nov 19 12:02:09 dre-slimserver-3t ollama[569358]: [GIN] 2024/11/19 - 12:02:07 | 500 | 5m4s | 100.100.70.109 | POST "/api/generate" > Nov 19 12:02:12 dre-slimserver-3t ollama[569358]: time=2024-11-19T12:02:12.413-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.247257499 model=/usr/share/ollama/.ollama/models/blobs/sha256-da63a910e34997d50c9f21cc7f16996d1e76e1c128b13319edd68348f760ecc7 > Nov 19 12:02:13 dre-slimserver-3t ollama[569358]: time=2024-11-19T12:02:13.171-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.005048646 model=/usr/share/ollama/.ollama/models/blobs/sha256-da63a910e34997d50c9f21cc7f16996d1e76e1c128b13319edd68348f760ecc7 > Nov 19 12:02:13 dre-slimserver-3t ollama[569358]: time=2024-11-19T12:02:13.802-06:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.635933639 model=/usr/share/ollama/.ollama/models/blobs/sha256-da63a910e34997d50c9f21cc7f16996d1e76e1c128b13319edd68348f760ecc7 ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.4.2
GiteaMirror added the bug label 2026-05-09 15:25:22 -05:00
GiteaMirror commented 2026-05-09 15:25:25 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 19, 2024):

gpu VRAM usage didn't recover within timeout is just a warning. The problem is timed out waiting for llama runner to start. Set OLLAMA_LOAD_TIMEOUT=10m in the server environment.

<!-- gh-comment-id:2486524088 --> @rick-github commented on GitHub (Nov 19, 2024): `gpu VRAM usage didn't recover within timeout` is just a warning. The problem is `timed out waiting for llama runner to start`. Set `OLLAMA_LOAD_TIMEOUT=10m` in the [server environment](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server).
GiteaMirror commented 2026-05-09 15:25:25 -05:00
Author
Owner
Copy Link

@ergosumdre commented on GitHub (Nov 19, 2024):

Thanks! I set the the variable and re-pulled the model. It seems to be working now.

<!-- gh-comment-id:2486882652 --> @ergosumdre commented on GitHub (Nov 19, 2024): Thanks! I set the the variable and re-pulled the model. It seems to be working now.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#82628
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?