github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-08 08:59:13 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #5826] Azurefile (NFS) causes very slow model loads - Mixtral 22B isn't loaded on an A100 (80GB VRAM) #65667

New Issue
Closed
opened 2026-05-03 22:09:20 -05:00 by GiteaMirror · 10 comments
GiteaMirror commented 2026-05-03 22:09:20 -05:00
Owner
Copy Link

Originally created by @juangon on GitHub (Jul 21, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5826

What is the issue?

Trying to load Mixtral 8x22B model using an A100 GPU as a deployment in Kubernetes, but it isn't loaded after 6 minutes.

Mistral 7B model is loaded fine.

Here is the debug log:

time=2024-07-21T07:39:07.407Z level=DEBUG source=gpu.go:358 msg="updating system memory data" before.total="216.3 GiB" before.free="212.5 GiB" before.free_swap="0 B" now.total="216.3 GiB" now.free="212.5 GiB" now.free_swap="0 B" CUDA driver version: 12.4 time=2024-07-21T07:39:07.762Z level=DEBUG source=gpu.go:406 msg="updating cuda memory data" gpu=GPU-4101ce7d-41d5-c2ee-2fe9-927eb4440974 name="NVIDIA A100 80GB PCIe" overhead="0 B" before.total="79.1 GiB" before.free="78.7 GiB" now.total="79.1 GiB" now.free="78.7 GiB" now.used="426.1 MiB" releasing cuda driver library time=2024-07-21T07:39:07.768Z level=DEBUG source=sched.go:214 msg="loading first model" model=/root/.ollama/models/blobs/sha256-85bbeb31e9a57b841db2386003d8b057acbe0dce01e1939711cd533ccbc69bca time=2024-07-21T07:39:07.768Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[78.7 GiB]" time=2024-07-21T07:39:07.768Z level=INFO source=sched.go:701 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-85bbeb31e9a57b841db2386003d8b057acbe0dce01e1939711cd533ccbc69bca gpu=GPU-4101ce7d-41d5-c2ee-2fe9-927eb4440974 parallel=4 available=84527415296 required="67.3 GiB" time=2024-07-21T07:39:07.769Z level=DEBUG source=server.go:100 msg="system memory" total="216.3 GiB" free="212.5 GiB" free_swap="0 B" time=2024-07-21T07:39:07.769Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[78.7 GiB]" time=2024-07-21T07:39:07.769Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=57 layers.offload=57 layers.split="" memory.available="[78.7 GiB]" memory.required.full="67.3 GiB" memory.required.partial="67.3 GiB" memory.required.kv="1.8 GiB" memory.required.allocations="[67.3 GiB]" memory.weights.total="64.7 GiB" memory.weights.repeating="64.5 GiB" memory.weights.nonrepeating="157.5 MiB" memory.graph.full="832.0 MiB" memory.graph.partial="1.1 GiB" time=2024-07-21T07:39:07.769Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu/ollama_llama_server time=2024-07-21T07:39:07.769Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu_avx/ollama_llama_server time=2024-07-21T07:39:07.769Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu_avx2/ollama_llama_server time=2024-07-21T07:39:07.769Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cuda_v11/ollama_llama_server time=2024-07-21T07:39:07.769Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/rocm_v60102/ollama_llama_server time=2024-07-21T07:39:07.770Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu/ollama_llama_server time=2024-07-21T07:39:07.770Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu_avx/ollama_llama_server time=2024-07-21T07:39:07.770Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu_avx2/ollama_llama_server time=2024-07-21T07:39:07.770Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cuda_v11/ollama_llama_server time=2024-07-21T07:39:07.770Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/rocm_v60102/ollama_llama_server time=2024-07-21T07:39:07.770Z level=INFO source=server.go:383 msg="starting llama server" cmd="/tmp/ollama232821427/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-85bbeb31e9a57b841db2386003d8b057acbe0dce01e1939711cd533ccbc69bca --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 57 --verbose --parallel 4 --port 37297" time=2024-07-21T07:39:07.770Z level=DEBUG source=server.go:398 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/tmp/ollama232821427/runners/cuda_v11:/tmp/ollama232821427/runners:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_VISIBLE_DEVICES=GPU-4101ce7d-41d5-c2ee-2fe9-927eb4440974]" time=2024-07-21T07:39:07.770Z level=INFO source=sched.go:437 msg="loaded runners" count=1 time=2024-07-21T07:39:07.770Z level=INFO source=server.go:571 msg="waiting for llama runner to start responding" time=2024-07-21T07:39:07.770Z level=INFO source=server.go:612 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=1 commit="a8db2a9" tid="139752639410176" timestamp=1721547547 INFO [main] system info | n_threads=24 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="139752639410176" timestamp=1721547547 total_threads=24 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="23" port="37297" tid="139752639410176" timestamp=1721547547 llama_model_loader: loaded meta data with 28 key-value pairs and 563 tensors from /root/.ollama/models/blobs/sha256-85bbeb31e9a57b841db2386003d8b057acbe0dce01e1939711cd533ccbc69bca (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = Mixtral-8x22B-Instruct-v0.1 llama_model_loader: - kv 2: llama.block_count u32 = 56 llama_model_loader: - kv 3: llama.context_length u32 = 65536 llama_model_loader: - kv 4: llama.embedding_length u32 = 6144 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 16384 llama_model_loader: - kv 6: llama.attention.head_count u32 = 48 llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 8: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: llama.expert_count u32 = 8 llama_model_loader: - kv 11: llama.expert_used_count u32 = 2 llama_model_loader: - kv 12: general.file_type u32 = 12 llama_model_loader: - kv 13: llama.vocab_size u32 = 32768 llama_model_loader: - kv 14: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 15: tokenizer.ggml.model str = llama llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,32768] = ["<unk>", "<s>", "</s>", "[INST]", "[... llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,32768] = [-1000.000000, -1000.000000, -1000.00... llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,32768] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 24: tokenizer.chat_template.tool_use str = {{bos_token}}{% set user_messages = m... llama_model_loader: - kv 25: tokenizer.chat_templates arr[str,1] = ["tool_use"] llama_model_loader: - kv 26: tokenizer.chat_template str = {{bos_token}}{% for message in messag... llama_model_loader: - kv 27: general.quantization_version u32 = 2 llama_model_loader: - type f32: 113 tensors llama_model_loader: - type f16: 56 tensors llama_model_loader: - type q8_0: 112 tensors llama_model_loader: - type q3_K: 169 tensors llama_model_loader: - type q4_K: 53 tensors llama_model_loader: - type q5_K: 59 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens cache size = 259 llm_load_vocab: token to piece cache size = 0.1732 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32768 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 65536 llm_load_print_meta: n_embd = 6144 llm_load_print_meta: n_layer = 56 llm_load_print_meta: n_head = 48 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 6 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 16384 llm_load_print_meta: n_expert = 8 llm_load_print_meta: n_expert_used = 2 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 65536 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 8x22B llm_load_print_meta: model ftype = Q3_K - Medium llm_load_print_meta: model params = 140.63 B llm_load_print_meta: model size = 63.14 GiB (3.86 BPW) llm_load_print_meta: general.name = Mixtral-8x22B-Instruct-v0.1 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 781 '<0x0A>' llm_load_print_meta: max token length = 48 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA A100 80GB PCIe, compute capability 8.0, VMM: yes time=2024-07-21T07:39:08.021Z level=INFO source=server.go:612 msg="waiting for server to become available" status="llm server loading model" llm_load_tensors: ggml ctx size = 0.51 MiB

OS

Linux

GPU

Nvidia

CPU

No response

Ollama version

2.6 (docker image 2.7)

Originally created by @juangon on GitHub (Jul 21, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5826 ### What is the issue? Trying to load Mixtral 8x22B model using an A100 GPU as a deployment in Kubernetes, but it isn't loaded after 6 minutes. Mistral 7B model is loaded fine. Here is the debug log: `time=2024-07-21T07:39:07.407Z level=DEBUG source=gpu.go:358 msg="updating system memory data" before.total="216.3 GiB" before.free="212.5 GiB" before.free_swap="0 B" now.total="216.3 GiB" now.free="212.5 GiB" now.free_swap="0 B" CUDA driver version: 12.4 time=2024-07-21T07:39:07.762Z level=DEBUG source=gpu.go:406 msg="updating cuda memory data" gpu=GPU-4101ce7d-41d5-c2ee-2fe9-927eb4440974 name="NVIDIA A100 80GB PCIe" overhead="0 B" before.total="79.1 GiB" before.free="78.7 GiB" now.total="79.1 GiB" now.free="78.7 GiB" now.used="426.1 MiB" releasing cuda driver library time=2024-07-21T07:39:07.768Z level=DEBUG source=sched.go:214 msg="loading first model" model=/root/.ollama/models/blobs/sha256-85bbeb31e9a57b841db2386003d8b057acbe0dce01e1939711cd533ccbc69bca time=2024-07-21T07:39:07.768Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[78.7 GiB]" time=2024-07-21T07:39:07.768Z level=INFO source=sched.go:701 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-85bbeb31e9a57b841db2386003d8b057acbe0dce01e1939711cd533ccbc69bca gpu=GPU-4101ce7d-41d5-c2ee-2fe9-927eb4440974 parallel=4 available=84527415296 required="67.3 GiB" time=2024-07-21T07:39:07.769Z level=DEBUG source=server.go:100 msg="system memory" total="216.3 GiB" free="212.5 GiB" free_swap="0 B" time=2024-07-21T07:39:07.769Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[78.7 GiB]" time=2024-07-21T07:39:07.769Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=57 layers.offload=57 layers.split="" memory.available="[78.7 GiB]" memory.required.full="67.3 GiB" memory.required.partial="67.3 GiB" memory.required.kv="1.8 GiB" memory.required.allocations="[67.3 GiB]" memory.weights.total="64.7 GiB" memory.weights.repeating="64.5 GiB" memory.weights.nonrepeating="157.5 MiB" memory.graph.full="832.0 MiB" memory.graph.partial="1.1 GiB" time=2024-07-21T07:39:07.769Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu/ollama_llama_server time=2024-07-21T07:39:07.769Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu_avx/ollama_llama_server time=2024-07-21T07:39:07.769Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu_avx2/ollama_llama_server time=2024-07-21T07:39:07.769Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cuda_v11/ollama_llama_server time=2024-07-21T07:39:07.769Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/rocm_v60102/ollama_llama_server time=2024-07-21T07:39:07.770Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu/ollama_llama_server time=2024-07-21T07:39:07.770Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu_avx/ollama_llama_server time=2024-07-21T07:39:07.770Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cpu_avx2/ollama_llama_server time=2024-07-21T07:39:07.770Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/cuda_v11/ollama_llama_server time=2024-07-21T07:39:07.770Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama232821427/runners/rocm_v60102/ollama_llama_server time=2024-07-21T07:39:07.770Z level=INFO source=server.go:383 msg="starting llama server" cmd="/tmp/ollama232821427/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-85bbeb31e9a57b841db2386003d8b057acbe0dce01e1939711cd533ccbc69bca --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 57 --verbose --parallel 4 --port 37297" time=2024-07-21T07:39:07.770Z level=DEBUG source=server.go:398 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/tmp/ollama232821427/runners/cuda_v11:/tmp/ollama232821427/runners:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_VISIBLE_DEVICES=GPU-4101ce7d-41d5-c2ee-2fe9-927eb4440974]" time=2024-07-21T07:39:07.770Z level=INFO source=sched.go:437 msg="loaded runners" count=1 time=2024-07-21T07:39:07.770Z level=INFO source=server.go:571 msg="waiting for llama runner to start responding" time=2024-07-21T07:39:07.770Z level=INFO source=server.go:612 msg="waiting for server to become available" status="llm server error" INFO [main] build info | build=1 commit="a8db2a9" tid="139752639410176" timestamp=1721547547 INFO [main] system info | n_threads=24 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="139752639410176" timestamp=1721547547 total_threads=24 INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="23" port="37297" tid="139752639410176" timestamp=1721547547 llama_model_loader: loaded meta data with 28 key-value pairs and 563 tensors from /root/.ollama/models/blobs/sha256-85bbeb31e9a57b841db2386003d8b057acbe0dce01e1939711cd533ccbc69bca (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = Mixtral-8x22B-Instruct-v0.1 llama_model_loader: - kv 2: llama.block_count u32 = 56 llama_model_loader: - kv 3: llama.context_length u32 = 65536 llama_model_loader: - kv 4: llama.embedding_length u32 = 6144 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 16384 llama_model_loader: - kv 6: llama.attention.head_count u32 = 48 llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 8: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: llama.expert_count u32 = 8 llama_model_loader: - kv 11: llama.expert_used_count u32 = 2 llama_model_loader: - kv 12: general.file_type u32 = 12 llama_model_loader: - kv 13: llama.vocab_size u32 = 32768 llama_model_loader: - kv 14: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 15: tokenizer.ggml.model str = llama llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,32768] = ["<unk>", "<s>", "</s>", "[INST]", "[... llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,32768] = [-1000.000000, -1000.000000, -1000.00... llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,32768] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 24: tokenizer.chat_template.tool_use str = {{bos_token}}{% set user_messages = m... llama_model_loader: - kv 25: tokenizer.chat_templates arr[str,1] = ["tool_use"] llama_model_loader: - kv 26: tokenizer.chat_template str = {{bos_token}}{% for message in messag... llama_model_loader: - kv 27: general.quantization_version u32 = 2 llama_model_loader: - type f32: 113 tensors llama_model_loader: - type f16: 56 tensors llama_model_loader: - type q8_0: 112 tensors llama_model_loader: - type q3_K: 169 tensors llama_model_loader: - type q4_K: 53 tensors llama_model_loader: - type q5_K: 59 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens cache size = 259 llm_load_vocab: token to piece cache size = 0.1732 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32768 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 65536 llm_load_print_meta: n_embd = 6144 llm_load_print_meta: n_layer = 56 llm_load_print_meta: n_head = 48 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 6 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 16384 llm_load_print_meta: n_expert = 8 llm_load_print_meta: n_expert_used = 2 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 65536 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 8x22B llm_load_print_meta: model ftype = Q3_K - Medium llm_load_print_meta: model params = 140.63 B llm_load_print_meta: model size = 63.14 GiB (3.86 BPW) llm_load_print_meta: general.name = Mixtral-8x22B-Instruct-v0.1 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 781 '<0x0A>' llm_load_print_meta: max token length = 48 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA A100 80GB PCIe, compute capability 8.0, VMM: yes time=2024-07-21T07:39:08.021Z level=INFO source=server.go:612 msg="waiting for server to become available" status="llm server loading model" llm_load_tensors: ggml ctx size = 0.51 MiB ` ### OS Linux ### GPU Nvidia ### CPU _No response_ ### Ollama version 2.6 (docker image 2.7)
GiteaMirror added the question label 2026-05-03 22:09:21 -05:00
GiteaMirror commented 2026-05-03 22:09:30 -05:00
Author
Owner
Copy Link

@juangon commented on GitHub (Jul 21, 2024):

ollama_mixtra_debug_logs.txt

<!-- gh-comment-id:2241514324 --> @juangon commented on GitHub (Jul 21, 2024): [ollama_mixtra_debug_logs.txt](https://github.com/user-attachments/files/16323430/ollama_mixtra_debug_logs.txt)
GiteaMirror commented 2026-05-03 22:09:31 -05:00
Author
Owner
Copy Link

@juangon commented on GitHub (Jul 21, 2024):

Some tests after using "use_mmap":false, and setting OLLAMA_NUM_PARALLEL=1 and OLLAMA_MAX_LOADED_MODELS=1:

llm_load_tensors: offloading 56 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 57/57 layers to GPU llm_load_tensors: CUDA_Host buffer size = 82.50 MiB llm_load_tensors: CUDA0 buffer size = 64571.65 MiB time=2024-07-21T09:37:04.768Z level=DEBUG source=server.go:623 msg="model load progress 0.01" time=2024-07-21T09:37:13.306Z level=DEBUG source=server.go:623 msg="model load progress 0.02" time=2024-07-21T09:37:23.095Z level=DEBUG source=server.go:623 msg="model load progress 0.03" time=2024-07-21T09:37:31.133Z level=DEBUG source=server.go:623 msg="model load progress 0.04" time=2024-07-21T09:37:40.673Z level=DEBUG source=server.go:623 msg="model load progress 0.05" time=2024-07-21T09:37:47.954Z level=DEBUG source=server.go:623 msg="model load progress 0.06" time=2024-07-21T09:37:57.998Z level=DEBUG source=server.go:623 msg="model load progress 0.07" time=2024-07-21T09:38:11.808Z level=DEBUG source=server.go:623 msg="model load progress 0.08" time=2024-07-21T09:38:22.854Z level=DEBUG source=server.go:623 msg="model load progress 0.09" time=2024-07-21T09:38:27.123Z level=DEBUG source=server.go:623 msg="model load progress 0.10" time=2024-07-21T09:38:37.668Z level=DEBUG source=server.go:623 msg="model load progress 0.11" time=2024-07-21T09:38:43.695Z level=DEBUG source=server.go:623 msg="model load progress 0.12" time=2024-07-21T09:38:53.237Z level=DEBUG source=server.go:623 msg="model load progress 0.13" time=2024-07-21T09:39:02.527Z level=DEBUG source=server.go:623 msg="model load progress 0.14" time=2024-07-21T09:39:12.570Z level=DEBUG source=server.go:623 msg="model load progress 0.15" time=2024-07-21T09:39:19.097Z level=DEBUG source=server.go:623 msg="model load progress 0.16" time=2024-07-21T09:39:31.651Z level=DEBUG source=server.go:623 msg="model load progress 0.17" time=2024-07-21T09:39:43.452Z level=DEBUG source=server.go:623 msg="model load progress 0.18" time=2024-07-21T09:39:47.721Z level=DEBUG source=server.go:623 msg="model load progress 0.19" time=2024-07-21T09:39:58.514Z level=DEBUG source=server.go:623 msg="model load progress 0.20" time=2024-07-21T09:40:08.054Z level=DEBUG source=server.go:623 msg="model load progress 0.21" time=2024-07-21T09:40:13.077Z level=DEBUG source=server.go:623 msg="model load progress 0.22" time=2024-07-21T09:40:22.870Z level=DEBUG source=server.go:623 msg="model load progress 0.23" time=2024-07-21T09:40:35.676Z level=DEBUG source=server.go:623 msg="model load progress 0.24" time=2024-07-21T09:40:48.232Z level=DEBUG source=server.go:623 msg="model load progress 0.25" time=2024-07-21T09:40:53.505Z level=DEBUG source=server.go:623 msg="model load progress 0.26" time=2024-07-21T09:41:07.065Z level=DEBUG source=server.go:623 msg="model load progress 0.27" time=2024-07-21T09:41:18.362Z level=DEBUG source=server.go:623 msg="model load progress 0.28" time=2024-07-21T09:41:24.388Z level=DEBUG source=server.go:623 msg="model load progress 0.29" time=2024-07-21T09:41:35.181Z level=DEBUG source=server.go:623 msg="model load progress 0.30" time=2024-07-21T09:41:47.233Z level=DEBUG source=server.go:623 msg="model load progress 0.31"

So, 4 minutes to load only 31% of the model. Additionally, I saw in my GPU metrics that 65GB are being used, it seems model is loaded in GPU but somehow Ollama thinks is only loaded at 31%? How is that possible?

<!-- gh-comment-id:2241549094 --> @juangon commented on GitHub (Jul 21, 2024): Some tests after using "use_mmap":false, and setting OLLAMA_NUM_PARALLEL=1 and OLLAMA_MAX_LOADED_MODELS=1: `llm_load_tensors: offloading 56 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 57/57 layers to GPU llm_load_tensors: CUDA_Host buffer size = 82.50 MiB llm_load_tensors: CUDA0 buffer size = 64571.65 MiB time=2024-07-21T09:37:04.768Z level=DEBUG source=server.go:623 msg="model load progress 0.01" time=2024-07-21T09:37:13.306Z level=DEBUG source=server.go:623 msg="model load progress 0.02" time=2024-07-21T09:37:23.095Z level=DEBUG source=server.go:623 msg="model load progress 0.03" time=2024-07-21T09:37:31.133Z level=DEBUG source=server.go:623 msg="model load progress 0.04" time=2024-07-21T09:37:40.673Z level=DEBUG source=server.go:623 msg="model load progress 0.05" time=2024-07-21T09:37:47.954Z level=DEBUG source=server.go:623 msg="model load progress 0.06" time=2024-07-21T09:37:57.998Z level=DEBUG source=server.go:623 msg="model load progress 0.07" time=2024-07-21T09:38:11.808Z level=DEBUG source=server.go:623 msg="model load progress 0.08" time=2024-07-21T09:38:22.854Z level=DEBUG source=server.go:623 msg="model load progress 0.09" time=2024-07-21T09:38:27.123Z level=DEBUG source=server.go:623 msg="model load progress 0.10" time=2024-07-21T09:38:37.668Z level=DEBUG source=server.go:623 msg="model load progress 0.11" time=2024-07-21T09:38:43.695Z level=DEBUG source=server.go:623 msg="model load progress 0.12" time=2024-07-21T09:38:53.237Z level=DEBUG source=server.go:623 msg="model load progress 0.13" time=2024-07-21T09:39:02.527Z level=DEBUG source=server.go:623 msg="model load progress 0.14" time=2024-07-21T09:39:12.570Z level=DEBUG source=server.go:623 msg="model load progress 0.15" time=2024-07-21T09:39:19.097Z level=DEBUG source=server.go:623 msg="model load progress 0.16" time=2024-07-21T09:39:31.651Z level=DEBUG source=server.go:623 msg="model load progress 0.17" time=2024-07-21T09:39:43.452Z level=DEBUG source=server.go:623 msg="model load progress 0.18" time=2024-07-21T09:39:47.721Z level=DEBUG source=server.go:623 msg="model load progress 0.19" time=2024-07-21T09:39:58.514Z level=DEBUG source=server.go:623 msg="model load progress 0.20" time=2024-07-21T09:40:08.054Z level=DEBUG source=server.go:623 msg="model load progress 0.21" time=2024-07-21T09:40:13.077Z level=DEBUG source=server.go:623 msg="model load progress 0.22" time=2024-07-21T09:40:22.870Z level=DEBUG source=server.go:623 msg="model load progress 0.23" time=2024-07-21T09:40:35.676Z level=DEBUG source=server.go:623 msg="model load progress 0.24" time=2024-07-21T09:40:48.232Z level=DEBUG source=server.go:623 msg="model load progress 0.25" time=2024-07-21T09:40:53.505Z level=DEBUG source=server.go:623 msg="model load progress 0.26" time=2024-07-21T09:41:07.065Z level=DEBUG source=server.go:623 msg="model load progress 0.27" time=2024-07-21T09:41:18.362Z level=DEBUG source=server.go:623 msg="model load progress 0.28" time=2024-07-21T09:41:24.388Z level=DEBUG source=server.go:623 msg="model load progress 0.29" time=2024-07-21T09:41:35.181Z level=DEBUG source=server.go:623 msg="model load progress 0.30" time=2024-07-21T09:41:47.233Z level=DEBUG source=server.go:623 msg="model load progress 0.31"` So, 4 minutes to load only 31% of the model. Additionally, I saw in my GPU metrics that 65GB are being used, it seems model is loaded in GPU but somehow Ollama thinks is only loaded at 31%? How is that possible?
GiteaMirror commented 2026-05-03 22:09:32 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 21, 2024):

I think that means the VRAM is fully allocated at the start of the load, but it takes time to transfer the data in to the allocated memory. Is the model on local disk or network disk?

<!-- gh-comment-id:2241552317 --> @rick-github commented on GitHub (Jul 21, 2024): I think that means the VRAM is fully allocated at the start of the load, but it takes time to transfer the data in to the allocated memory. Is the model on local disk or network disk?
GiteaMirror commented 2026-05-03 22:09:33 -05:00
Author
Owner
Copy Link

@juangon commented on GitHub (Jul 21, 2024):

Thanks for answering! It's a Network disk. Type azurefile from Azure Kubernetes Service. I am doing another test using azure-csi-premium as it has more performance.

<!-- gh-comment-id:2241557870 --> @juangon commented on GitHub (Jul 21, 2024): Thanks for answering! It's a Network disk. Type azurefile from Azure Kubernetes Service. I am doing another test using azure-csi-premium as it has more performance.
GiteaMirror commented 2026-05-03 22:09:34 -05:00
Author
Owner
Copy Link

@juangon commented on GitHub (Jul 21, 2024):

No way @rick-github , it seems that after 6 minutes only loaded 51% of the model with use_mmap=True . Any ideas on how to improve loading performance? Additionally, when enabling OLLAMA_KEEP_ALIVE, where is this model stored? In GPU or RAM? As we are sharing the GPU, we can't just simply let all model loaded there so other instances can't use that GPU.

<!-- gh-comment-id:2241583433 --> @juangon commented on GitHub (Jul 21, 2024): No way @rick-github , it seems that after 6 minutes only loaded 51% of the model with `use_mmap=True` . Any ideas on how to improve loading performance? Additionally, when enabling `OLLAMA_KEEP_ALIVE`, where is this model stored? In GPU or RAM? As we are sharing the GPU, we can't just simply let all model loaded there so other instances can't use that GPU.
GiteaMirror commented 2026-05-03 22:09:35 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 21, 2024):

I don't have an A100 and I'm not running in the cloud so the following experiment might not be relevant.

I downloaded mixtral:8x22b-instruct-v0.1-q3_K_M to a NFS mounted drive, flushed the page cache (echo 1 | sudo tee /proc/sys/vm/drop_caches), and then loaded the model (time curl localhost:11434/api/generate -d '{"model":"mixtral:8x22b-instruct-v0.1-q3_K_M"}'). This first attempt timed out after 5 minutes. However, while it was loading, I could see the page cache usage going up (vmstat 5), pausing at 37G when the load failed. Second attempt succeeded after 4m41s, with the page cache reaching 65G. I then unloaded the model (curl localhost:11434/api/generate -d '{"model":"mixtral:8x22b-instruct-v0.1-q3_K_M", "keep_alive":0}') and re-loaded, which this time took 4.2s.

I then copied the model to local disk, flushed the page cache, restarted ollama to use the local disk, and loaded the model in 11s.

So in my case, the page cache is instrumental in getting good load times. When a load fails, a large portion of the data is in the cache so subsequent attempts succeed. From your logs, your A100 machine has ~256G so it should have plenty of space for caching model data. However, if you're running ollama in an environment that restricts memory (eg, in a cgroup with a small memory.limit_in_bytes, or some other Azure feature) then it could be that it prevents ollama from using the page cache effectively. In that case, copying the model to local disk may be a workaround.

As for OLLAMA_KEEP_ALIVE, yes, the model is kept in GPU VRAM. If the model is unloaded (either because the keep alive time expired or because another model is being loaded) and then reloaded, the page cache would speed up the load time.

<!-- gh-comment-id:2241626200 --> @rick-github commented on GitHub (Jul 21, 2024): I don't have an A100 and I'm not running in the cloud so the following experiment might not be relevant. I downloaded mixtral:8x22b-instruct-v0.1-q3_K_M to a NFS mounted drive, flushed the page cache (`echo 1 | sudo tee /proc/sys/vm/drop_caches`), and then loaded the model (`time curl localhost:11434/api/generate -d '{"model":"mixtral:8x22b-instruct-v0.1-q3_K_M"}'`). This first attempt timed out after 5 minutes. However, while it was loading, I could see the page cache usage going up (`vmstat 5`), pausing at 37G when the load failed. Second attempt succeeded after 4m41s, with the page cache reaching 65G. I then unloaded the model (`curl localhost:11434/api/generate -d '{"model":"mixtral:8x22b-instruct-v0.1-q3_K_M", "keep_alive":0}'`) and re-loaded, which this time took 4.2s. I then copied the model to local disk, flushed the page cache, restarted ollama to use the local disk, and loaded the model in 11s. So in my case, the page cache is instrumental in getting good load times. When a load fails, a large portion of the data is in the cache so subsequent attempts succeed. From your logs, your A100 machine has ~256G so it should have plenty of space for caching model data. However, if you're running ollama in an environment that restricts memory (eg, in a cgroup with a small memory.limit_in_bytes, or some other Azure feature) then it could be that it prevents ollama from using the page cache effectively. In that case, copying the model to local disk may be a workaround. As for `OLLAMA_KEEP_ALIVE`, yes, the model is kept in GPU VRAM. If the model is unloaded (either because the keep alive time expired or because another model is being loaded) and then reloaded, the page cache would speed up the load time.
GiteaMirror commented 2026-05-03 22:09:36 -05:00
Author
Owner
Copy Link

@juangon commented on GitHub (Jul 21, 2024):

Thanks for that @rick-github . I don't know if this is related but due to resource saving, we have a shared File system. We first run a container downloading the model in the shared mounted path (/root/.ollama/models). We execute ollama pull for this. Then we have another serverless instance that scales up by demand and scales down after some idle time. This serverless instance executes ollama container mounting shared File system where the model was previously pulled. The problem is that as we want to scale down the GPU node and save cost, we can't count on a local filesystem, at least without first copying the model from the shared filesystem

<!-- gh-comment-id:2241638982 --> @juangon commented on GitHub (Jul 21, 2024): Thanks for that @rick-github . I don't know if this is related but due to resource saving, we have a shared File system. We first run a container downloading the model in the shared mounted path (/root/.ollama/models). We execute ollama pull for this. Then we have another serverless instance that scales up by demand and scales down after some idle time. This serverless instance executes ollama container mounting shared File system where the model was previously pulled. The problem is that as we want to scale down the GPU node and save cost, we can't count on a local filesystem, at least without first copying the model from the shared filesystem
GiteaMirror commented 2026-05-03 22:09:38 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 21, 2024):

Hmm, that means that the container that is being scaled will lose its page cache when downsized, so the model would have to be pulled in its entirety from the shared file system when the container is scaled up. If you can't use local disk then you have to rely on fast networking from the shared filesystem, which is hopefully located close to your serving container. I don't know Azure, maybe there's some way to have a persistent sidecar on your serving container that is just a lot of RAM that can provide a virtual filesystem proxying the shared file system and acting as a page cache. Assuming that this is actually the problem, of course.

<!-- gh-comment-id:2241651943 --> @rick-github commented on GitHub (Jul 21, 2024): Hmm, that means that the container that is being scaled will lose its page cache when downsized, so the model would have to be pulled in its entirety from the shared file system when the container is scaled up. If you can't use local disk then you have to rely on fast networking from the shared filesystem, which is hopefully located close to your serving container. I don't know Azure, maybe there's some way to have a persistent sidecar on your serving container that is just a lot of RAM that can provide a virtual filesystem proxying the shared file system and acting as a page cache. Assuming that this is actually the problem, of course.
GiteaMirror commented 2026-05-03 22:09:39 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jul 22, 2024):

In my experience, block-based storage is typically going to perform better than file-based, and it sounds like your slow load time is largely based on I/O performance to transfer the 67.3 GiB.

Does the model eventually load with or without setting use_mmap? If there's a scenario where we timeout even though the system was still loading, we should fix that, but if the model does eventually load, I think we can close the issue.

<!-- gh-comment-id:2243943579 --> @dhiltgen commented on GitHub (Jul 22, 2024): In my experience, block-based storage is typically going to perform better than file-based, and it sounds like your slow load time is largely based on I/O performance to transfer the 67.3 GiB. Does the model eventually load with or without setting use_mmap? If there's a scenario where we timeout even though the system was still loading, we should fix that, but if the model does eventually load, I think we can close the issue.
GiteaMirror commented 2026-05-03 22:09:40 -05:00
Author
Owner
Copy Link

@juangon commented on GitHub (Jul 23, 2024):

Thanks @dhiltgen. Yes I've made a test and azurefile (Network) seems to perform much worse comparing to default storage class (which is block based I think). Gonna close the issue. Thanks very much!

<!-- gh-comment-id:2244457652 --> @juangon commented on GitHub (Jul 23, 2024): Thanks @dhiltgen. Yes I've made a test and azurefile (Network) seems to perform much worse comparing to default storage class (which is block based I think). Gonna close the issue. Thanks very much!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#65667
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?