github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-08 08:59:13 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #9170] ollama miscalulates CUDA memory to allocate (deepseek-r1:671b) #5970

New Issue
Open
opened 2026-04-12 17:19:24 -05:00 by GiteaMirror · 12 comments
GiteaMirror commented 2026-04-12 17:19:24 -05:00
Owner
Copy Link

Originally created by @axkibe on GitHub (Feb 17, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9170

What is the issue?

I tried running deepseek-r1:671b on a Dual A100 machine with 512GB RAM (maybe thats a bit out of reach anyway, 70b model runs)

Anyway when I try to run it, ollama keeps trying to allocate more CUDA memory than there is, or fails to take into account the two GPU are not connected via NVLink and thus cannot allocate the sum of ther memory into one buffer.

| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-PCIE-40GB          Off | 00000000:01:00.0 Off |                  Off |
| N/A   35C    P0              39W / 250W |      7MiB / 40960MiB |	0%	Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-PCIE-40GB          Off | 00000000:81:00.0 Off |                  Off |
| N/A   32C    P0              35W / 250W |      7MiB / 40960MiB |	0%	Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

In the Modelfile file I now wrote:

PARAMETER num_gpu 20
PARAMETER num_thread 64

If I reduce num_gpu to 4 the model loads, but with reduced context then only uses one GPU and runs with about 2 tokens per second..

Thus I believe with all experiments, it always assume the GPUs are connected, and thus tries to allocate 70G into one CUDA buffer, but it cannot. I could allocate 2 times 35G.

PS: In case it isn't obvious, I have no real idea what I'm doing, this is all very new to me.

Relevant log output

CUDA_VISIBLE_DEVICES=0,1 OLLAMA_MODELS=/models/ OLLAMA_LOAD_TIMEOUT=60m OLLAMA_GPU_OVERHEAD=268435456 ollama/bin/ollama serve
2025/02/17 12:12:00 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,1 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:268435456 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:1h0m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/models/ OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-02-17T12:12:00.163+01:00 level=INFO source=images.go:432 msg="total blobs: 23"
time=2025-02-17T12:12:00.163+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-02-17T12:12:00.163+01:00 level=INFO source=routes.go:1237 msg="Listening on 127.0.0.1:11434 (version 0.5.10)"
time=2025-02-17T12:12:00.164+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-02-17T12:12:03.723+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-f11c6588-5f6c-7f62-3912-efd7ba9f2580 library=cuda variant=v12 compute=8.0 driver=12.3 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB"
time=2025-02-17T12:12:03.723+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-f90d439c-7334-230e-9dda-ff50f09e8bee library=cuda variant=v12 compute=8.0 driver=12.3 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB"
[GIN] 2025/02/17 - 12:12:42 | 200 |     145.613µs |       127.0.0.1 | HEAD     "/"
[GIN] 2025/02/17 - 12:12:42 | 200 |   14.716052ms |       127.0.0.1 | POST     "/api/show"
time=2025-02-17T12:12:43.656+01:00 level=INFO source=server.go:100 msg="system memory" total="503.7 GiB" free="489.2 GiB" free_swap="0 B"
time=2025-02-17T12:12:43.658+01:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=20 layers.model=62 layers.offload=13 layers.split=7,6 memory.available="[39.0 GiB 39.0 GiB]" memory.gpu_overhead="256.0 MiB" memory.required.full="414.0 GiB" memory.required.partial="72.7 GiB" memory.required.kv="9.5 GiB" memory.required.allocations="[36.6 GiB 36.1 GiB]" memory.weights.total="385.0 GiB" memory.weights.repeating="384.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="1019.5 MiB" memory.graph.partial="1019.5 MiB"
time=2025-02-17T12:12:43.659+01:00 level=INFO source=server.go:381 msg="starting llama server" cmd="ollama/bin/ollama runner --model /models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 --ctx-size 2048 --batch-size 512 --n-gpu-layers 20 --threads 64 --parallel 1 --tensor-split 7,6 --port 39623"
time=2025-02-17T12:12:43.660+01:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-02-17T12:12:43.660+01:00 level=INFO source=server.go:558 msg="waiting for llama runner to start responding"
time=2025-02-17T12:12:43.661+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error"
time=2025-02-17T12:12:43.691+01:00 level=INFO source=runner.go:936 msg="starting go runner"
time=2025-02-17T12:12:43.691+01:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=64
time=2025-02-17T12:12:43.692+01:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:39623"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes
  Device 1: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes
load_backend: loaded CUDA backend from ollama/lib/ollama/cuda_v12/libggml-cuda.so
time=2025-02-17T12:12:43.913+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model"
load_backend: loaded CPU backend from ollama/lib/ollama/libggml-cpu-haswell.so
llama_load_model_from_file: using device CUDA0 (NVIDIA A100-PCIE-40GB) - 39910 MiB free
llama_load_model_from_file: using device CUDA1 (NVIDIA A100-PCIE-40GB) - 39910 MiB free
llama_model_loader: loaded meta data with 42 key-value pairs and 1025 tensors from /models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                         general.size_label str              = 256x20B
llama_model_loader: - kv   3:                      deepseek2.block_count u32              = 61
llama_model_loader: - kv   4:                   deepseek2.context_length u32              = 163840
llama_model_loader: - kv   5:                 deepseek2.embedding_length u32              = 7168
llama_model_loader: - kv   6:              deepseek2.feed_forward_length u32              = 18432
llama_model_loader: - kv   7:             deepseek2.attention.head_count u32              = 128
llama_model_loader: - kv   8:          deepseek2.attention.head_count_kv u32              = 128
llama_model_loader: - kv   9:                   deepseek2.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  10: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  11:                deepseek2.expert_used_count u32              = 8
llama_model_loader: - kv  12:        deepseek2.leading_dense_block_count u32              = 3
llama_model_loader: - kv  13:                       deepseek2.vocab_size u32              = 129280
llama_model_loader: - kv  14:            deepseek2.attention.q_lora_rank u32              = 1536
llama_model_loader: - kv  15:           deepseek2.attention.kv_lora_rank u32              = 512
llama_model_loader: - kv  16:             deepseek2.attention.key_length u32              = 192
llama_model_loader: - kv  17:           deepseek2.attention.value_length u32              = 128
llama_model_loader: - kv  18:       deepseek2.expert_feed_forward_length u32              = 2048
llama_model_loader: - kv  19:                     deepseek2.expert_count u32              = 256
llama_model_loader: - kv  20:              deepseek2.expert_shared_count u32              = 1
llama_model_loader: - kv  21:             deepseek2.expert_weights_scale f32              = 2.500000
llama_model_loader: - kv  22:              deepseek2.expert_weights_norm bool             = true
llama_model_loader: - kv  23:               deepseek2.expert_gating_func u32              = 2
llama_model_loader: - kv  24:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  25:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  26:              deepseek2.rope.scaling.factor f32              = 40.000000
llama_model_loader: - kv  27: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  28: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
llama_model_loader: - kv  29:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  30:                         tokenizer.ggml.pre str              = deepseek-v3
llama_model_loader: - kv  31:                      tokenizer.ggml.tokens arr[str,129280]  = ["<｜begin▁of▁sentence｜>", "<�...
llama_model_loader: - kv  32:                  tokenizer.ggml.token_type arr[i32,129280]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  33:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
llama_model_loader: - kv  34:                tokenizer.ggml.bos_token_id u32              = 0
llama_model_loader: - kv  35:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  36:            tokenizer.ggml.padding_token_id u32              = 1
llama_model_loader: - kv  37:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  38:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  39:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  40:               general.quantization_version u32              = 2
llama_model_loader: - kv  41:                          general.file_type u32              = 15
llama_model_loader: - type  f32:  361 tensors
llama_model_loader: - type q4_K:  606 tensors
llama_model_loader: - type q6_K:   58 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 818
llm_load_vocab: token to piece cache size = 0.8223 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = deepseek2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 129280
llm_load_print_meta: n_merges         = 127741
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 163840
llm_load_print_meta: n_embd           = 7168
llm_load_print_meta: n_layer          = 61
llm_load_print_meta: n_head           = 128
llm_load_print_meta: n_head_kv        = 128
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 192
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 24576
llm_load_print_meta: n_embd_v_gqa     = 16384
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 18432
llm_load_print_meta: n_expert         = 256
llm_load_print_meta: n_expert_used    = 8
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = yarn
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 0.025
llm_load_print_meta: n_ctx_orig_yarn  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 671B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 671.03 B
llm_load_print_meta: model size       = 376.65 GiB (4.82 BPW) 
llm_load_print_meta: general.name     = n/a
llm_load_print_meta: BOS token        = 0 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token        = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOT token        = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token        = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token         = 131 'Ä'
llm_load_print_meta: FIM PRE token    = 128801 '<｜fim▁begin｜>'
llm_load_print_meta: FIM SUF token    = 128800 '<｜fim▁hole｜>'
llm_load_print_meta: FIM MID token    = 128802 '<｜fim▁end｜>'
llm_load_print_meta: EOG token        = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: max token length = 256
llm_load_print_meta: n_layer_dense_lead   = 3
llm_load_print_meta: n_lora_q             = 1536
llm_load_print_meta: n_lora_kv            = 512
llm_load_print_meta: n_ff_exp             = 2048
llm_load_print_meta: n_expert_shared      = 1
llm_load_print_meta: expert_weights_scale = 2.5
llm_load_print_meta: expert_weights_norm  = 1
llm_load_print_meta: expert_gating_func   = sigmoid
llm_load_print_meta: rope_yarn_log_mul    = 0.1000
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 71680.09 MiB on device 0: cudaMalloc failed: out of memory
llama_model_load: error loading model: unable to allocate CUDA0 buffer

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.5.10

Originally created by @axkibe on GitHub (Feb 17, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9170 ### What is the issue? I tried running deepseek-r1:671b on a Dual A100 machine with 512GB RAM (maybe thats a bit out of reach anyway, 70b model runs) Anyway when I try to run it, ollama keeps trying to allocate more CUDA memory than there is, or fails to take into account the two GPU are not connected via NVLink and thus cannot allocate the sum of ther memory into one buffer. ```+---------------------------------------------------------------------------------------+ | NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA A100-PCIE-40GB Off | 00000000:01:00.0 Off | Off | | N/A 35C P0 39W / 250W | 7MiB / 40960MiB | 0% Default | | | | Disabled | +-----------------------------------------+----------------------+----------------------+ | 1 NVIDIA A100-PCIE-40GB Off | 00000000:81:00.0 Off | Off | | N/A 32C P0 35W / 250W | 7MiB / 40960MiB | 0% Default | | | | Disabled | +-----------------------------------------+----------------------+----------------------+ ``` In the Modelfile file I now wrote: ``` PARAMETER num_gpu 20 PARAMETER num_thread 64 ``` If I reduce num_gpu to 4 the model loads, but with reduced context then only uses one GPU and runs with about 2 tokens per second.. Thus I believe with all experiments, it always assume the GPUs are connected, and thus tries to allocate 70G into one CUDA buffer, but it cannot. I could allocate 2 times 35G. PS: In case it isn't obvious, I have no real idea what I'm doing, this is all very new to me. ### Relevant log output ```shell CUDA_VISIBLE_DEVICES=0,1 OLLAMA_MODELS=/models/ OLLAMA_LOAD_TIMEOUT=60m OLLAMA_GPU_OVERHEAD=268435456 ollama/bin/ollama serve 2025/02/17 12:12:00 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,1 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:268435456 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:1h0m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/models/ OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-02-17T12:12:00.163+01:00 level=INFO source=images.go:432 msg="total blobs: 23" time=2025-02-17T12:12:00.163+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" time=2025-02-17T12:12:00.163+01:00 level=INFO source=routes.go:1237 msg="Listening on 127.0.0.1:11434 (version 0.5.10)" time=2025-02-17T12:12:00.164+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-02-17T12:12:03.723+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-f11c6588-5f6c-7f62-3912-efd7ba9f2580 library=cuda variant=v12 compute=8.0 driver=12.3 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" time=2025-02-17T12:12:03.723+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-f90d439c-7334-230e-9dda-ff50f09e8bee library=cuda variant=v12 compute=8.0 driver=12.3 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" [GIN] 2025/02/17 - 12:12:42 | 200 | 145.613µs | 127.0.0.1 | HEAD "/" [GIN] 2025/02/17 - 12:12:42 | 200 | 14.716052ms | 127.0.0.1 | POST "/api/show" time=2025-02-17T12:12:43.656+01:00 level=INFO source=server.go:100 msg="system memory" total="503.7 GiB" free="489.2 GiB" free_swap="0 B" time=2025-02-17T12:12:43.658+01:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=20 layers.model=62 layers.offload=13 layers.split=7,6 memory.available="[39.0 GiB 39.0 GiB]" memory.gpu_overhead="256.0 MiB" memory.required.full="414.0 GiB" memory.required.partial="72.7 GiB" memory.required.kv="9.5 GiB" memory.required.allocations="[36.6 GiB 36.1 GiB]" memory.weights.total="385.0 GiB" memory.weights.repeating="384.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="1019.5 MiB" memory.graph.partial="1019.5 MiB" time=2025-02-17T12:12:43.659+01:00 level=INFO source=server.go:381 msg="starting llama server" cmd="ollama/bin/ollama runner --model /models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 --ctx-size 2048 --batch-size 512 --n-gpu-layers 20 --threads 64 --parallel 1 --tensor-split 7,6 --port 39623" time=2025-02-17T12:12:43.660+01:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2025-02-17T12:12:43.660+01:00 level=INFO source=server.go:558 msg="waiting for llama runner to start responding" time=2025-02-17T12:12:43.661+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error" time=2025-02-17T12:12:43.691+01:00 level=INFO source=runner.go:936 msg="starting go runner" time=2025-02-17T12:12:43.691+01:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=64 time=2025-02-17T12:12:43.692+01:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:39623" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 2 CUDA devices: Device 0: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes Device 1: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes load_backend: loaded CUDA backend from ollama/lib/ollama/cuda_v12/libggml-cuda.so time=2025-02-17T12:12:43.913+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model" load_backend: loaded CPU backend from ollama/lib/ollama/libggml-cpu-haswell.so llama_load_model_from_file: using device CUDA0 (NVIDIA A100-PCIE-40GB) - 39910 MiB free llama_load_model_from_file: using device CUDA1 (NVIDIA A100-PCIE-40GB) - 39910 MiB free llama_model_loader: loaded meta data with 42 key-value pairs and 1025 tensors from /models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = deepseek2 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.size_label str = 256x20B llama_model_loader: - kv 3: deepseek2.block_count u32 = 61 llama_model_loader: - kv 4: deepseek2.context_length u32 = 163840 llama_model_loader: - kv 5: deepseek2.embedding_length u32 = 7168 llama_model_loader: - kv 6: deepseek2.feed_forward_length u32 = 18432 llama_model_loader: - kv 7: deepseek2.attention.head_count u32 = 128 llama_model_loader: - kv 8: deepseek2.attention.head_count_kv u32 = 128 llama_model_loader: - kv 9: deepseek2.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 10: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 11: deepseek2.expert_used_count u32 = 8 llama_model_loader: - kv 12: deepseek2.leading_dense_block_count u32 = 3 llama_model_loader: - kv 13: deepseek2.vocab_size u32 = 129280 llama_model_loader: - kv 14: deepseek2.attention.q_lora_rank u32 = 1536 llama_model_loader: - kv 15: deepseek2.attention.kv_lora_rank u32 = 512 llama_model_loader: - kv 16: deepseek2.attention.key_length u32 = 192 llama_model_loader: - kv 17: deepseek2.attention.value_length u32 = 128 llama_model_loader: - kv 18: deepseek2.expert_feed_forward_length u32 = 2048 llama_model_loader: - kv 19: deepseek2.expert_count u32 = 256 llama_model_loader: - kv 20: deepseek2.expert_shared_count u32 = 1 llama_model_loader: - kv 21: deepseek2.expert_weights_scale f32 = 2.500000 llama_model_loader: - kv 22: deepseek2.expert_weights_norm bool = true llama_model_loader: - kv 23: deepseek2.expert_gating_func u32 = 2 llama_model_loader: - kv 24: deepseek2.rope.dimension_count u32 = 64 llama_model_loader: - kv 25: deepseek2.rope.scaling.type str = yarn llama_model_loader: - kv 26: deepseek2.rope.scaling.factor f32 = 40.000000 llama_model_loader: - kv 27: deepseek2.rope.scaling.original_context_length u32 = 4096 llama_model_loader: - kv 28: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 30: tokenizer.ggml.pre str = deepseek-v3 llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,129280] = ["<｜begin▁of▁sentence｜>", "<�... llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e... llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0 llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 1 llama_model_loader: - kv 36: tokenizer.ggml.padding_token_id u32 = 1 llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 38: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 39: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 40: general.quantization_version u32 = 2 llama_model_loader: - kv 41: general.file_type u32 = 15 llama_model_loader: - type f32: 361 tensors llama_model_loader: - type q4_K: 606 tensors llama_model_loader: - type q6_K: 58 tensors llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect llm_load_vocab: special tokens cache size = 818 llm_load_vocab: token to piece cache size = 0.8223 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = deepseek2 llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 129280 llm_load_print_meta: n_merges = 127741 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 163840 llm_load_print_meta: n_embd = 7168 llm_load_print_meta: n_layer = 61 llm_load_print_meta: n_head = 128 llm_load_print_meta: n_head_kv = 128 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 192 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 24576 llm_load_print_meta: n_embd_v_gqa = 16384 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 18432 llm_load_print_meta: n_expert = 256 llm_load_print_meta: n_expert_used = 8 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = yarn llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 0.025 llm_load_print_meta: n_ctx_orig_yarn = 4096 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 671B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 671.03 B llm_load_print_meta: model size = 376.65 GiB (4.82 BPW) llm_load_print_meta: general.name = n/a llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>' llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOT token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: PAD token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: LF token = 131 'Ä' llm_load_print_meta: FIM PRE token = 128801 '<｜fim▁begin｜>' llm_load_print_meta: FIM SUF token = 128800 '<｜fim▁hole｜>' llm_load_print_meta: FIM MID token = 128802 '<｜fim▁end｜>' llm_load_print_meta: EOG token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: max token length = 256 llm_load_print_meta: n_layer_dense_lead = 3 llm_load_print_meta: n_lora_q = 1536 llm_load_print_meta: n_lora_kv = 512 llm_load_print_meta: n_ff_exp = 2048 llm_load_print_meta: n_expert_shared = 1 llm_load_print_meta: expert_weights_scale = 2.5 llm_load_print_meta: expert_weights_norm = 1 llm_load_print_meta: expert_gating_func = sigmoid llm_load_print_meta: rope_yarn_log_mul = 0.1000 ggml_backend_cuda_buffer_type_alloc_buffer: allocating 71680.09 MiB on device 0: cudaMalloc failed: out of memory llama_model_load: error loading model: unable to allocate CUDA0 buffer ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.10
GiteaMirror added the bug label 2026-04-12 17:19:24 -05:00
GiteaMirror commented 2026-04-12 17:19:25 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 17, 2025):

ollama has calculated that it can only fit 13 layers on the GPUs (layers.offload=13) which would take 36G of 39G on each GPU. You have overridden that by setting num_gpu 20. The GPU loader tries to follow your instructions but it can't fit the extra 7 layers in the remaining 6G of GPU VRAM. If you remove PARAMETER num_gpu 20 from the Modelfile and allow ollama to choose how many layers to offload it might work better. There is still the possibility that ollama will mis-calculate - deepseek architecture has some problems with accurate memory prediction - but that can be solved by reducing num_gpu, say to 11 or 12.

Note that deepseek-r1:671b has a current issue where too many tokens in the context buffer (the sum of input and output tokens) can cause the model to crash: https://github.com/ollama/ollama/issues/5975

<!-- gh-comment-id:2662870237 --> @rick-github commented on GitHub (Feb 17, 2025): ollama has calculated that it can only fit 13 layers on the GPUs (`layers.offload=13`) which would take 36G of 39G on each GPU. You have overridden that by setting `num_gpu 20`. The GPU loader tries to follow your instructions but it can't fit the extra 7 layers in the remaining 6G of GPU VRAM. If you remove `PARAMETER num_gpu 20` from the Modelfile and allow ollama to choose how many layers to offload it might work better. There is still the possibility that ollama will mis-calculate - deepseek architecture has some problems with accurate memory prediction - but that can be solved by reducing `num_gpu`, say to 11 or 12. Note that deepseek-r1:671b has a current issue where too many tokens in the context buffer (the sum of input and output tokens) can cause the model to crash: https://github.com/ollama/ollama/issues/5975
GiteaMirror commented 2026-04-12 17:19:26 -05:00
Author
Owner
Copy Link

@axkibe commented on GitHub (Feb 17, 2025):

Thanks I'll try as soon I get that node again, and will come back, as far I recall it would misallocate for everything above num_gpu 4.
(my basic question right now is simply if this machine can run deepseek-r1:671b with a reasonable tokens/s performance)

<!-- gh-comment-id:2662925861 --> @axkibe commented on GitHub (Feb 17, 2025): Thanks I'll try as soon I get that node again, and will come back, as far I recall it would misallocate for everything above num_gpu 4. (my basic question right now is simply if this machine can run deepseek-r1:671b with a reasonable tokens/s performance)
GiteaMirror commented 2026-04-12 17:19:27 -05:00
Author
Owner
Copy Link

@axkibe commented on GitHub (Feb 17, 2025):

Okay so here: without forcing num_gpu at all:

layers.offload=13

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 46963.85 MiB on device 0: cudaMalloc failed: out of memory
time=2025-02-17T18:50:35.845+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model

The device just has 40960MiB (according to nvidia-smi) and 7 MiB seem to be used by overhead.

Forcing num_gpu to 12:
layers.offload=12

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 42640.01 MiB on device 1: cudaMalloc failed: out of memory
llama_model_load: error loading model: unable to allocate CUDA1 buffer

Again it still overshoots.

Forcing num_gpu to 11:
layers.offload=11

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 40784.79 MiB on device 0: cudaMalloc failed: out of memory
llama_model_load: error loading model: unable to allocate CUDA0 buffe

num_gpu 10: first time it gets a different output:

llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 0
llama_kv_cache_init:      CUDA0 KV buffer size =  3200.00 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =  3200.00 MiB
llama_kv_cache_init:        CPU KV buffer size = 32640.00 MiB
llama_new_context_with_model: KV self size  = 39040.00 MiB, K (f16): 23424.00 MiB, V (f16): 15616.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     2.08 MiB
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 5039.50 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 5284300800
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2950.77 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 3094106112
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 7774.04 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 8151670784
llama_new_context_with_model: failed to allocate compute buffers
panic: unable to create llama contex

I guess the big alloc succeeded, but it wants to allocate more and keeps failing.

This stays the same down to num_gpus=5

num_gpus=4 is the max config that sucessfully loads, but utilizes only one GPU, with 64 CPU threads runs about just as fast num_gpu=0.

PS: Dunno why num_gpu=0 takes ages to load, while with num_gpu=4 its online in a few minutes.

<!-- gh-comment-id:2663869320 --> @axkibe commented on GitHub (Feb 17, 2025): Okay so here: without forcing num_gpu at all: layers.offload=13 ``` ggml_backend_cuda_buffer_type_alloc_buffer: allocating 46963.85 MiB on device 0: cudaMalloc failed: out of memory time=2025-02-17T18:50:35.845+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model ``` The device just has 40960MiB (according to nvidia-smi) and 7 MiB seem to be used by overhead. Forcing num_gpu to 12: layers.offload=12 ``` ggml_backend_cuda_buffer_type_alloc_buffer: allocating 42640.01 MiB on device 1: cudaMalloc failed: out of memory llama_model_load: error loading model: unable to allocate CUDA1 buffer ``` Again it still overshoots. Forcing num_gpu to 11: layers.offload=11 ``` ggml_backend_cuda_buffer_type_alloc_buffer: allocating 40784.79 MiB on device 0: cudaMalloc failed: out of memory llama_model_load: error loading model: unable to allocate CUDA0 buffe ``` num_gpu 10: first time it gets a different output: ``` llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 0 llama_kv_cache_init: CUDA0 KV buffer size = 3200.00 MiB llama_kv_cache_init: CUDA1 KV buffer size = 3200.00 MiB llama_kv_cache_init: CPU KV buffer size = 32640.00 MiB llama_new_context_with_model: KV self size = 39040.00 MiB, K (f16): 23424.00 MiB, V (f16): 15616.00 MiB llama_new_context_with_model: CPU output buffer size = 2.08 MiB ggml_backend_cuda_buffer_type_alloc_buffer: allocating 5039.50 MiB on device 0: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 5284300800 ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2950.77 MiB on device 0: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 3094106112 ggml_backend_cuda_buffer_type_alloc_buffer: allocating 7774.04 MiB on device 0: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 8151670784 llama_new_context_with_model: failed to allocate compute buffers panic: unable to create llama contex ``` I guess the big alloc succeeded, but it wants to allocate more and keeps failing. This stays the same down to num_gpus=5 num_gpus=4 is the max config that sucessfully loads, but utilizes only one GPU, with 64 CPU threads runs about just as fast num_gpu=0. PS: Dunno why num_gpu=0 takes ages to load, while with num_gpu=4 its online in a few minutes.
GiteaMirror commented 2026-04-12 17:19:27 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 17, 2025):

Please provide the full log for unset num_gpu.

<!-- gh-comment-id:2663877218 --> @rick-github commented on GitHub (Feb 17, 2025): Please provide the full log for unset `num_gpu`.
GiteaMirror commented 2026-04-12 17:19:28 -05:00
Author
Owner
Copy Link

@axkibe commented on GitHub (Feb 17, 2025):

Sure, with unmodified model as pulled:

CUDA_VISIBLE_DEVICES=0,1 OLLAMA_MODELS=/models/ OLLAMA_LOAD_TIMEOUT=60m OLLAMA_GPU_OVERHEAD=268435456 ollama/bin/ollama serve
2025/02/17 19:48:25 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,1 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:268435456 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:1h0m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/models/ OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-02-17T19:48:25.836+01:00 level=INFO source=images.go:432 msg="total blobs: 39"
time=2025-02-17T19:48:25.837+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-02-17T19:48:25.838+01:00 level=INFO source=routes.go:1237 msg="Listening on 127.0.0.1:11434 (version 0.5.10)"
time=2025-02-17T19:48:25.838+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-02-17T19:48:29.400+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-e705b73c-c68e-be4b-1775-e487f85472be library=cuda variant=v12 compute=8.0 driver=12.3 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB"
time=2025-02-17T19:48:29.401+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-63903996-8074-d0a2-7b37-673dc5753484 library=cuda variant=v12 compute=8.0 driver=12.3 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB"
[GIN] 2025/02/17 - 19:48:37 | 200 |     194.374µs |       127.0.0.1 | HEAD     "/"
[GIN] 2025/02/17 - 19:48:37 | 200 |   41.419691ms |       127.0.0.1 | POST     "/api/show"
time=2025-02-17T19:48:38.294+01:00 level=INFO source=server.go:100 msg="system memory" total="503.7 GiB" free="488.9 GiB" free_swap="0 B"
time=2025-02-17T19:48:38.295+01:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=62 layers.offload=13 layers.split=7,6 memory.available="[39.0 GiB 39.0 GiB]" memory.gpu_overhead="256.0 MiB" memory.required.full="414.0 GiB" memory.required.partial="72.7 GiB" memory.required.kv="9.5 GiB" memory.required.allocations="[36.6 GiB 36.1 GiB]" memory.weights.total="385.0 GiB" memory.weights.repeating="384.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="1019.5 MiB" memory.graph.partial="1019.5 MiB"
time=2025-02-17T19:48:38.296+01:00 level=INFO source=server.go:381 msg="starting llama server" cmd="ollama/bin/ollama runner --model /models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 --ctx-size 2048 --batch-size 512 --n-gpu-layers 13 --threads 128 --parallel 1 --tensor-split 7,6 --port 38403"
time=2025-02-17T19:48:38.296+01:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2025-02-17T19:48:38.296+01:00 level=INFO source=server.go:558 msg="waiting for llama runner to start responding"
time=2025-02-17T19:48:38.297+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error"
time=2025-02-17T19:48:38.328+01:00 level=INFO source=runner.go:936 msg="starting go runner"
time=2025-02-17T19:48:38.328+01:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=128
time=2025-02-17T19:48:38.328+01:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:38403"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes
  Device 1: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes
time=2025-02-17T19:48:38.549+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model"
load_backend: loaded CUDA backend from ollama/lib/ollama/cuda_v12/libggml-cuda.so
load_backend: loaded CPU backend from ollama/lib/ollama/libggml-cpu-haswell.so
llama_load_model_from_file: using device CUDA0 (NVIDIA A100-PCIE-40GB) - 39910 MiB free
llama_load_model_from_file: using device CUDA1 (NVIDIA A100-PCIE-40GB) - 39910 MiB free
llama_model_loader: loaded meta data with 42 key-value pairs and 1025 tensors from /models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                         general.size_label str              = 256x20B
llama_model_loader: - kv   3:                      deepseek2.block_count u32              = 61
llama_model_loader: - kv   4:                   deepseek2.context_length u32              = 163840
llama_model_loader: - kv   5:                 deepseek2.embedding_length u32              = 7168
llama_model_loader: - kv   6:              deepseek2.feed_forward_length u32              = 18432
llama_model_loader: - kv   7:             deepseek2.attention.head_count u32              = 128
llama_model_loader: - kv   8:          deepseek2.attention.head_count_kv u32              = 128
llama_model_loader: - kv   9:                   deepseek2.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  10: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  11:                deepseek2.expert_used_count u32              = 8
llama_model_loader: - kv  12:        deepseek2.leading_dense_block_count u32              = 3
llama_model_loader: - kv  13:                       deepseek2.vocab_size u32              = 129280
llama_model_loader: - kv  14:            deepseek2.attention.q_lora_rank u32              = 1536
llama_model_loader: - kv  15:           deepseek2.attention.kv_lora_rank u32              = 512
llama_model_loader: - kv  16:             deepseek2.attention.key_length u32              = 192
llama_model_loader: - kv  17:           deepseek2.attention.value_length u32              = 128
llama_model_loader: - kv  18:       deepseek2.expert_feed_forward_length u32              = 2048
llama_model_loader: - kv  19:                     deepseek2.expert_count u32              = 256
llama_model_loader: - kv  20:              deepseek2.expert_shared_count u32              = 1
llama_model_loader: - kv  21:             deepseek2.expert_weights_scale f32              = 2.500000
llama_model_loader: - kv  22:              deepseek2.expert_weights_norm bool             = true
llama_model_loader: - kv  23:               deepseek2.expert_gating_func u32              = 2
llama_model_loader: - kv  24:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  25:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  26:              deepseek2.rope.scaling.factor f32              = 40.000000
llama_model_loader: - kv  27: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  28: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
llama_model_loader: - kv  29:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  30:                         tokenizer.ggml.pre str              = deepseek-v3
llama_model_loader: - kv  31:                      tokenizer.ggml.tokens arr[str,129280]  = ["<｜begin▁of▁sentence｜>", "<�...
llama_model_loader: - kv  32:                  tokenizer.ggml.token_type arr[i32,129280]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  33:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
llama_model_loader: - kv  34:                tokenizer.ggml.bos_token_id u32              = 0
llama_model_loader: - kv  35:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  36:            tokenizer.ggml.padding_token_id u32              = 1
llama_model_loader: - kv  37:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  38:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  39:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  40:               general.quantization_version u32              = 2
llama_model_loader: - kv  41:                          general.file_type u32              = 15
llama_model_loader: - type  f32:  361 tensors
llama_model_loader: - type q4_K:  606 tensors
llama_model_loader: - type q6_K:   58 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 818
llm_load_vocab: token to piece cache size = 0.8223 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = deepseek2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 129280
llm_load_print_meta: n_merges         = 127741
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 163840
llm_load_print_meta: n_embd           = 7168
llm_load_print_meta: n_layer          = 61
llm_load_print_meta: n_head           = 128
llm_load_print_meta: n_head_kv        = 128
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 192
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 24576
llm_load_print_meta: n_embd_v_gqa     = 16384
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 18432
llm_load_print_meta: n_expert         = 256
llm_load_print_meta: n_expert_used    = 8
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = yarn
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 0.025
llm_load_print_meta: n_ctx_orig_yarn  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 671B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 671.03 B
llm_load_print_meta: model size       = 376.65 GiB (4.82 BPW) 
llm_load_print_meta: general.name     = n/a
llm_load_print_meta: BOS token        = 0 '<｜begin▁of▁sentence｜>'
llm_load_print_meta: EOS token        = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: EOT token        = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: PAD token        = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: LF token         = 131 'Ä'
llm_load_print_meta: FIM PRE token    = 128801 '<｜fim▁begin｜>'
llm_load_print_meta: FIM SUF token    = 128800 '<｜fim▁hole｜>'
llm_load_print_meta: FIM MID token    = 128802 '<｜fim▁end｜>'
llm_load_print_meta: EOG token        = 1 '<｜end▁of▁sentence｜>'
llm_load_print_meta: max token length = 256
llm_load_print_meta: n_layer_dense_lead   = 3
llm_load_print_meta: n_lora_q             = 1536
llm_load_print_meta: n_lora_kv            = 512
llm_load_print_meta: n_ff_exp             = 2048
llm_load_print_meta: n_expert_shared      = 1
llm_load_print_meta: expert_weights_scale = 2.5
llm_load_print_meta: expert_weights_norm  = 1
llm_load_print_meta: expert_gating_func   = sigmoid
llm_load_print_meta: rope_yarn_log_mul    = 0.1000
time=2025-02-17T19:50:38.927+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server not responding"
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 46963.85 MiB on device 0: cudaMalloc failed: out of memory
time=2025-02-17T19:52:46.405+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model"
llama_model_load: error loading model: unable to allocate CUDA0 buffer
llama_load_model_from_file: failed to load model
panic: unable to load model: /models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9

goroutine 28 [running]:
github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0003e9b90, {0xd, 0x0, 0x1, 0x0, {0xc0007e3638, 0x2, 0x2}, 0xc0007fa9a0, 0x0}, ...)
	github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad
created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
	github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d
time=2025-02-17T19:52:51.424+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error"
time=2025-02-17T19:52:51.484+01:00 level=ERROR source=server.go:422 msg="llama runner terminated" error="exit status 2"
time=2025-02-17T19:52:51.675+01:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer"
[GIN] 2025/02/17 - 19:52:51 | 500 |         4m14s |       127.0.0.1 | POST     "/api/generate"
time=2025-02-17T19:52:56.843+01:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.167863752 model=/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
time=2025-02-17T19:52:57.292+01:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.616928677 model=/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
time=2025-02-17T19:52:57.726+01:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.050270064 model=/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
<!-- gh-comment-id:2663892818 --> @axkibe commented on GitHub (Feb 17, 2025): Sure, with unmodified model as pulled: ``` CUDA_VISIBLE_DEVICES=0,1 OLLAMA_MODELS=/models/ OLLAMA_LOAD_TIMEOUT=60m OLLAMA_GPU_OVERHEAD=268435456 ollama/bin/ollama serve 2025/02/17 19:48:25 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,1 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:268435456 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:1h0m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/models/ OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-02-17T19:48:25.836+01:00 level=INFO source=images.go:432 msg="total blobs: 39" time=2025-02-17T19:48:25.837+01:00 level=INFO source=images.go:439 msg="total unused blobs removed: 0" time=2025-02-17T19:48:25.838+01:00 level=INFO source=routes.go:1237 msg="Listening on 127.0.0.1:11434 (version 0.5.10)" time=2025-02-17T19:48:25.838+01:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-02-17T19:48:29.400+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-e705b73c-c68e-be4b-1775-e487f85472be library=cuda variant=v12 compute=8.0 driver=12.3 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" time=2025-02-17T19:48:29.401+01:00 level=INFO source=types.go:130 msg="inference compute" id=GPU-63903996-8074-d0a2-7b37-673dc5753484 library=cuda variant=v12 compute=8.0 driver=12.3 name="NVIDIA A100-PCIE-40GB" total="39.4 GiB" available="39.0 GiB" [GIN] 2025/02/17 - 19:48:37 | 200 | 194.374µs | 127.0.0.1 | HEAD "/" [GIN] 2025/02/17 - 19:48:37 | 200 | 41.419691ms | 127.0.0.1 | POST "/api/show" time=2025-02-17T19:48:38.294+01:00 level=INFO source=server.go:100 msg="system memory" total="503.7 GiB" free="488.9 GiB" free_swap="0 B" time=2025-02-17T19:48:38.295+01:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=62 layers.offload=13 layers.split=7,6 memory.available="[39.0 GiB 39.0 GiB]" memory.gpu_overhead="256.0 MiB" memory.required.full="414.0 GiB" memory.required.partial="72.7 GiB" memory.required.kv="9.5 GiB" memory.required.allocations="[36.6 GiB 36.1 GiB]" memory.weights.total="385.0 GiB" memory.weights.repeating="384.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="1019.5 MiB" memory.graph.partial="1019.5 MiB" time=2025-02-17T19:48:38.296+01:00 level=INFO source=server.go:381 msg="starting llama server" cmd="ollama/bin/ollama runner --model /models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 --ctx-size 2048 --batch-size 512 --n-gpu-layers 13 --threads 128 --parallel 1 --tensor-split 7,6 --port 38403" time=2025-02-17T19:48:38.296+01:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2025-02-17T19:48:38.296+01:00 level=INFO source=server.go:558 msg="waiting for llama runner to start responding" time=2025-02-17T19:48:38.297+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error" time=2025-02-17T19:48:38.328+01:00 level=INFO source=runner.go:936 msg="starting go runner" time=2025-02-17T19:48:38.328+01:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=128 time=2025-02-17T19:48:38.328+01:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:38403" ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 2 CUDA devices: Device 0: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes Device 1: NVIDIA A100-PCIE-40GB, compute capability 8.0, VMM: yes time=2025-02-17T19:48:38.549+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model" load_backend: loaded CUDA backend from ollama/lib/ollama/cuda_v12/libggml-cuda.so load_backend: loaded CPU backend from ollama/lib/ollama/libggml-cpu-haswell.so llama_load_model_from_file: using device CUDA0 (NVIDIA A100-PCIE-40GB) - 39910 MiB free llama_load_model_from_file: using device CUDA1 (NVIDIA A100-PCIE-40GB) - 39910 MiB free llama_model_loader: loaded meta data with 42 key-value pairs and 1025 tensors from /models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = deepseek2 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.size_label str = 256x20B llama_model_loader: - kv 3: deepseek2.block_count u32 = 61 llama_model_loader: - kv 4: deepseek2.context_length u32 = 163840 llama_model_loader: - kv 5: deepseek2.embedding_length u32 = 7168 llama_model_loader: - kv 6: deepseek2.feed_forward_length u32 = 18432 llama_model_loader: - kv 7: deepseek2.attention.head_count u32 = 128 llama_model_loader: - kv 8: deepseek2.attention.head_count_kv u32 = 128 llama_model_loader: - kv 9: deepseek2.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 10: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 llama_model_loader: - kv 11: deepseek2.expert_used_count u32 = 8 llama_model_loader: - kv 12: deepseek2.leading_dense_block_count u32 = 3 llama_model_loader: - kv 13: deepseek2.vocab_size u32 = 129280 llama_model_loader: - kv 14: deepseek2.attention.q_lora_rank u32 = 1536 llama_model_loader: - kv 15: deepseek2.attention.kv_lora_rank u32 = 512 llama_model_loader: - kv 16: deepseek2.attention.key_length u32 = 192 llama_model_loader: - kv 17: deepseek2.attention.value_length u32 = 128 llama_model_loader: - kv 18: deepseek2.expert_feed_forward_length u32 = 2048 llama_model_loader: - kv 19: deepseek2.expert_count u32 = 256 llama_model_loader: - kv 20: deepseek2.expert_shared_count u32 = 1 llama_model_loader: - kv 21: deepseek2.expert_weights_scale f32 = 2.500000 llama_model_loader: - kv 22: deepseek2.expert_weights_norm bool = true llama_model_loader: - kv 23: deepseek2.expert_gating_func u32 = 2 llama_model_loader: - kv 24: deepseek2.rope.dimension_count u32 = 64 llama_model_loader: - kv 25: deepseek2.rope.scaling.type str = yarn llama_model_loader: - kv 26: deepseek2.rope.scaling.factor f32 = 40.000000 llama_model_loader: - kv 27: deepseek2.rope.scaling.original_context_length u32 = 4096 llama_model_loader: - kv 28: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 30: tokenizer.ggml.pre str = deepseek-v3 llama_model_loader: - kv 31: tokenizer.ggml.tokens arr[str,129280] = ["<｜begin▁of▁sentence｜>", "<�... llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e... llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0 llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 1 llama_model_loader: - kv 36: tokenizer.ggml.padding_token_id u32 = 1 llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 38: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 39: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 40: general.quantization_version u32 = 2 llama_model_loader: - kv 41: general.file_type u32 = 15 llama_model_loader: - type f32: 361 tensors llama_model_loader: - type q4_K: 606 tensors llama_model_loader: - type q6_K: 58 tensors llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect llm_load_vocab: special tokens cache size = 818 llm_load_vocab: token to piece cache size = 0.8223 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = deepseek2 llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 129280 llm_load_print_meta: n_merges = 127741 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 163840 llm_load_print_meta: n_embd = 7168 llm_load_print_meta: n_layer = 61 llm_load_print_meta: n_head = 128 llm_load_print_meta: n_head_kv = 128 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 192 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 24576 llm_load_print_meta: n_embd_v_gqa = 16384 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 18432 llm_load_print_meta: n_expert = 256 llm_load_print_meta: n_expert_used = 8 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = yarn llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 0.025 llm_load_print_meta: n_ctx_orig_yarn = 4096 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 671B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 671.03 B llm_load_print_meta: model size = 376.65 GiB (4.82 BPW) llm_load_print_meta: general.name = n/a llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>' llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: EOT token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: PAD token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: LF token = 131 'Ä' llm_load_print_meta: FIM PRE token = 128801 '<｜fim▁begin｜>' llm_load_print_meta: FIM SUF token = 128800 '<｜fim▁hole｜>' llm_load_print_meta: FIM MID token = 128802 '<｜fim▁end｜>' llm_load_print_meta: EOG token = 1 '<｜end▁of▁sentence｜>' llm_load_print_meta: max token length = 256 llm_load_print_meta: n_layer_dense_lead = 3 llm_load_print_meta: n_lora_q = 1536 llm_load_print_meta: n_lora_kv = 512 llm_load_print_meta: n_ff_exp = 2048 llm_load_print_meta: n_expert_shared = 1 llm_load_print_meta: expert_weights_scale = 2.5 llm_load_print_meta: expert_weights_norm = 1 llm_load_print_meta: expert_gating_func = sigmoid llm_load_print_meta: rope_yarn_log_mul = 0.1000 time=2025-02-17T19:50:38.927+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server not responding" ggml_backend_cuda_buffer_type_alloc_buffer: allocating 46963.85 MiB on device 0: cudaMalloc failed: out of memory time=2025-02-17T19:52:46.405+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server loading model" llama_model_load: error loading model: unable to allocate CUDA0 buffer llama_load_model_from_file: failed to load model panic: unable to load model: /models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 goroutine 28 [running]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0003e9b90, {0xd, 0x0, 0x1, 0x0, {0xc0007e3638, 0x2, 0x2}, 0xc0007fa9a0, 0x0}, ...) github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d time=2025-02-17T19:52:51.424+01:00 level=INFO source=server.go:592 msg="waiting for server to become available" status="llm server error" time=2025-02-17T19:52:51.484+01:00 level=ERROR source=server.go:422 msg="llama runner terminated" error="exit status 2" time=2025-02-17T19:52:51.675+01:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer" [GIN] 2025/02/17 - 19:52:51 | 500 | 4m14s | 127.0.0.1 | POST "/api/generate" time=2025-02-17T19:52:56.843+01:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.167863752 model=/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 time=2025-02-17T19:52:57.292+01:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.616928677 model=/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 time=2025-02-17T19:52:57.726+01:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.050270064 model=/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 ```
GiteaMirror commented 2026-04-12 17:19:30 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 17, 2025):

That's passing strange.

ollama calculates 2*36GiB (memory.required.allocations="[36.6 GiB 36.1 GiB]") for 13 layers with a split of 6,7, and the runner tries to allocate 45.9GiB. Probably coincidence that the difference is around the same as the KV cache (memory.required.kv="9.5 GiB").

I've seen ollama get allocations wrong many times but not so far off as here. Does it work any better if you roll back to a version of ollama from before the runner split, say 0.5.4?

<!-- gh-comment-id:2663911975 --> @rick-github commented on GitHub (Feb 17, 2025): That's passing strange. ollama calculates 2*36GiB (`memory.required.allocations="[36.6 GiB 36.1 GiB]"`) for 13 layers with a split of 6,7, and the runner tries to allocate 45.9GiB. Probably coincidence that the difference is around the same as the KV cache (`memory.required.kv="9.5 GiB"`). I've seen ollama get allocations wrong many times but not so far off as here. Does it work any better if you roll back to a version of ollama from before the runner split, say 0.5.4?
GiteaMirror commented 2026-04-12 17:19:30 -05:00
Author
Owner
Copy Link

@axkibe commented on GitHub (Feb 18, 2025):

0.5.4 crashes immediatly on startup, I pulled the model extra another time, in case 0.5.4 pulls something differently than 0.5.10

I guess there is no parameter/environment way to disable layer splitting?

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x0000555dca08beae in ?? ()
No symbol "frame" in current context.
[Inferior 1 (process 68338) detached]
SIGABRT: abort
PC=0x1501192e3acf m=4 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 34 gp=0xc0004ae000 m=4 mp=0xc0002b5508 [syscall]:
runtime.cgocall(0x555dca2c6970, 0xc000504b78)
	runtime/cgocall.go:167 +0x4b fp=0xc000504b50 sp=0xc000504b18 pc=0x555dca07ab2b
github.com/ollama/ollama/llama._Cfunc_llama_load_model_from_file(0x1500bc000c20, {0x0, 0xd, 0x1, 0x0, 0xc000480048, 0x0, 0x555dca2c6380, 0xc00050a000, 0x0, ...})
	_cgo_gotypes.go:700 +0x50 fp=0xc000504b78 sp=0xc000504b50 pc=0x555dca125410
github.com/ollama/ollama/llama.LoadModelFromFile.func1({0x7ffc117bc31b?, 0xc0004b4530?}, {0x0, 0xd, 0x1, 0x0, 0xc000480048, 0x0, 0x555dca2c6380, 0xc00050a000, ...})
	github.com/ollama/ollama/llama/llama.go:311 +0x127 fp=0xc000504c78 sp=0xc000504b78 pc=0x555dca128027
github.com/ollama/ollama/llama.LoadModelFromFile({0x7ffc117bc31b, 0x72}, {0xd, 0x0, 0x1, 0x0, {0xc000480048, 0x2, 0x2}, 0xc000486030, ...})
	github.com/ollama/ollama/llama/llama.go:311 +0x2d6 fp=0xc000504dc8 sp=0xc000504c78 pc=0x555dca127d16
github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc000490000, {0xd, 0x0, 0x1, 0x0, {0xc000480048, 0x2, 0x2}, 0xc000486030, 0x0}, ...)
	github.com/ollama/ollama/llama/runner/runner.go:859 +0xc5 fp=0xc000504f10 sp=0xc000504dc8 pc=0x555dca2c3de5
github.com/ollama/ollama/llama/runner.Execute.gowrap1()
	github.com/ollama/ollama/llama/runner/runner.go:979 +0xda fp=0xc000504fe0 sp=0xc000504f10 pc=0x555dca2c573a
runtime.goexit({})
	runtime/asm_amd64.s:1700 +0x1 fp=0xc000504fe8 sp=0xc000504fe0 pc=0x555dca088561
created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
	github.com/ollama/ollama/llama/runner/runner.go:979 +0xd0d
<!-- gh-comment-id:2665458876 --> @axkibe commented on GitHub (Feb 18, 2025): 0.5.4 crashes immediatly on startup, I pulled the model extra another time, in case 0.5.4 pulls something differently than 0.5.10 I guess there is no parameter/environment way to disable layer splitting? ``` [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". 0x0000555dca08beae in ?? () No symbol "frame" in current context. [Inferior 1 (process 68338) detached] SIGABRT: abort PC=0x1501192e3acf m=4 sigcode=18446744073709551610 signal arrived during cgo execution goroutine 34 gp=0xc0004ae000 m=4 mp=0xc0002b5508 [syscall]: runtime.cgocall(0x555dca2c6970, 0xc000504b78) runtime/cgocall.go:167 +0x4b fp=0xc000504b50 sp=0xc000504b18 pc=0x555dca07ab2b github.com/ollama/ollama/llama._Cfunc_llama_load_model_from_file(0x1500bc000c20, {0x0, 0xd, 0x1, 0x0, 0xc000480048, 0x0, 0x555dca2c6380, 0xc00050a000, 0x0, ...}) _cgo_gotypes.go:700 +0x50 fp=0xc000504b78 sp=0xc000504b50 pc=0x555dca125410 github.com/ollama/ollama/llama.LoadModelFromFile.func1({0x7ffc117bc31b?, 0xc0004b4530?}, {0x0, 0xd, 0x1, 0x0, 0xc000480048, 0x0, 0x555dca2c6380, 0xc00050a000, ...}) github.com/ollama/ollama/llama/llama.go:311 +0x127 fp=0xc000504c78 sp=0xc000504b78 pc=0x555dca128027 github.com/ollama/ollama/llama.LoadModelFromFile({0x7ffc117bc31b, 0x72}, {0xd, 0x0, 0x1, 0x0, {0xc000480048, 0x2, 0x2}, 0xc000486030, ...}) github.com/ollama/ollama/llama/llama.go:311 +0x2d6 fp=0xc000504dc8 sp=0xc000504c78 pc=0x555dca127d16 github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc000490000, {0xd, 0x0, 0x1, 0x0, {0xc000480048, 0x2, 0x2}, 0xc000486030, 0x0}, ...) github.com/ollama/ollama/llama/runner/runner.go:859 +0xc5 fp=0xc000504f10 sp=0xc000504dc8 pc=0x555dca2c3de5 github.com/ollama/ollama/llama/runner.Execute.gowrap1() github.com/ollama/ollama/llama/runner/runner.go:979 +0xda fp=0xc000504fe0 sp=0xc000504f10 pc=0x555dca2c573a runtime.goexit({}) runtime/asm_amd64.s:1700 +0x1 fp=0xc000504fe8 sp=0xc000504fe0 pc=0x555dca088561 created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 github.com/ollama/ollama/llama/runner/runner.go:979 +0xd0d ```
GiteaMirror commented 2026-04-12 17:19:30 -05:00
Author
Owner
Copy Link

@charliboy commented on GitHub (Feb 20, 2025):

I also have the same problem. My machine has three NVIDIA RTX A6000 Ada graphics cards, with 114GB of video memory and 512GB of memory. ubuntu 22.04, ollama 0. 5.11,The Deepseek 671b model was downloaded from the ollama website and has not been modified. I saw in the logs that it can calculate correctly, but it still reports errors when running：

ollama run deepseek-r1:671b --verbose
Error: llama runner process has terminated: error loading model: unable to allocate CUDA2 buffer
$ nvidia-smi
Thu Feb 20 12:41:00 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.15 Driver Version: 570.86.15 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX 6000 Ada Gene... Off | 00000000:41:00.0 Off | Off |
| 30% 42C P8 27W / 300W | 4MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA RTX 6000 Ada Gene... Off | 00000000:A1:00.0 Off | Off |
| 30% 34C P8 22W / 300W | 4MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA RTX 6000 Ada Gene... Off | 00000000:E1:00.0 Off | Off |
| 0% 41C P8 24W / 300W | 4MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Feb 20 12:26:57 suma systemd[1]: Started Ollama Service.
Feb 20 12:26:57 suma ollama[169846]: 2025/02/20 12:26:57 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,1,2 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY:http://192.168.11.115:18989 HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:30m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Feb 20 12:26:57 suma ollama[169846]: time=2025-02-20T12:26:57.740Z level=INFO source=images.go:432 msg="total blobs: 24"
Feb 20 12:26:57 suma ollama[169846]: time=2025-02-20T12:26:57.741Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"
Feb 20 12:26:57 suma ollama[169846]: time=2025-02-20T12:26:57.741Z level=INFO source=routes.go:1237 msg="Listening on [::]:11434 (version 0.5.11)"
Feb 20 12:26:57 suma ollama[169846]: time=2025-02-20T12:26:57.741Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
Feb 20 12:27:02 suma ollama[169846]: time=2025-02-20T12:27:02.614Z level=INFO source=types.go:130 msg="inference compute" id=GPU-b12b24ac-85de-8cf6-0d32-8d91d448e6d4 library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB"
Feb 20 12:27:02 suma ollama[169846]: time=2025-02-20T12:27:02.614Z level=INFO source=types.go:130 msg="inference compute" id=GPU-b7d261cb-d515-ff0a-fb8a-9da91967097b library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB"
Feb 20 12:27:02 suma ollama[169846]: time=2025-02-20T12:27:02.614Z level=INFO source=types.go:130 msg="inference compute" id=GPU-72fc8a27-617d-162d-2c0d-18ba84cc7678 library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB"
Feb 20 12:27:06 suma ollama[169846]: [GIN] 2025/02/20 - 12:27:06 | 200 | 176.14µs | 127.0.0.1 | GET "/api/version"
Feb 20 12:27:21 suma ollama[169846]: [GIN] 2025/02/20 - 12:27:21 | 200 | 43.56µs | 127.0.0.1 | HEAD "/"
Feb 20 12:27:21 suma ollama[169846]: [GIN] 2025/02/20 - 12:27:21 | 200 | 6.46148ms | 127.0.0.1 | GET "/api/tags"
Feb 20 12:27:38 suma ollama[169846]: [GIN] 2025/02/20 - 12:27:38 | 200 | 61.25µs | 127.0.0.1 | HEAD "/"
Feb 20 12:27:38 suma ollama[169846]: [GIN] 2025/02/20 - 12:27:38 | 200 | 2.36149ms | 127.0.0.1 | POST "/api/show"
Feb 20 12:27:39 suma ollama[169846]: time=2025-02-20T12:27:39.222Z level=WARN source=memory.go:123 msg="model missing blk.0 layer size"
Feb 20 12:27:39 suma ollama[169846]: panic: interface conversion: interface {} is nil, not llm.array
Feb 20 12:27:39 suma ollama[169846]: goroutine 138 [running]:
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm.GGML.GraphSize({{0x561cec056120?, 0xc0006700a0?}, {0x561cec0560a8?, 0xc000630008?}}, 0x2000, 0x200, {0x0, 0x0})
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm/ggml.go:367 +0x10bf
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm.EstimateGPULayers({_, _, _}, , {, _, _}, {{0x2000, 0x200, 0xffffffffffffffff, ...}, ...})
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm/memory.go:138 +0x5fa
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm.PredictServerFit({0xc0004cdba8?, 0x561ceae8b4ff?, 0xc0004cd8c0?}, 0xc0008a2040, {0xc0004cd920?, _, _}, {0x0, 0x0, 0x0}, ...)
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm/memory.go:22 +0xbd
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server.pickBestFullFitByLibrary(0xc000660000, 0xc0008a2040, {0xc0005a4308?, 0x3?, 0x4?}, 0xc00064dcf8)
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server/sched.go:713 +0x6f3
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server.(Scheduler).processPending(0xc0002170e0, {0x561cec059f40, 0xc000570410})
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server/sched.go:225 +0xe6c
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server.(Scheduler).Run.func1()
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server/sched.go:107 +0x1f
Feb 20 12:27:39 suma ollama[169846]: created by github.com/ollama/ollama/server.(Scheduler).Run in goroutine 1
Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server/sched.go:106 +0xb4
Feb 20 12:27:39 suma systemd[1]: ollama.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 20 12:27:39 suma systemd[1]: ollama.service: Failed with result 'exit-code'.
Feb 20 12:27:39 suma systemd[1]: ollama.service: Consumed 6.293s CPU time.
Feb 20 12:27:42 suma systemd[1]: ollama.service: Scheduled restart job, restart counter is at 1.
Feb 20 12:27:42 suma systemd[1]: Stopped Ollama Service.
Feb 20 12:27:42 suma systemd[1]: ollama.service: Consumed 6.293s CPU time.
Feb 20 12:27:42 suma systemd[1]: Started Ollama Service.
Feb 20 12:27:43 suma ollama[170277]: 2025/02/20 12:27:43 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,1,2 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY:http://192.168.11.115:18989 HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:30m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost: https://localhost: http://127.0.0.1 https://127.0.0.1 http://127.0.0.1: https://127.0.0.1: http://0.0.0.0 https://0.0.0.0 http://0.0.0.0: https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Feb 20 12:27:43 suma ollama[170277]: time=2025-02-20T12:27:43.004Z level=INFO source=images.go:432 msg="total blobs: 24"
Feb 20 12:27:43 suma ollama[170277]: time=2025-02-20T12:27:43.005Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"
Feb 20 12:27:43 suma ollama[170277]: time=2025-02-20T12:27:43.005Z level=INFO source=routes.go:1237 msg="Listening on [::]:11434 (version 0.5.11)"
Feb 20 12:27:43 suma ollama[170277]: time=2025-02-20T12:27:43.005Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
Feb 20 12:27:47 suma ollama[170277]: time=2025-02-20T12:27:47.862Z level=INFO source=types.go:130 msg="inference compute" id=GPU-b12b24ac-85de-8cf6-0d32-8d91d448e6d4 library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB"
Feb 20 12:27:47 suma ollama[170277]: time=2025-02-20T12:27:47.863Z level=INFO source=types.go:130 msg="inference compute" id=GPU-b7d261cb-d515-ff0a-fb8a-9da91967097b library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB"
Feb 20 12:27:47 suma ollama[170277]: time=2025-02-20T12:27:47.863Z level=INFO source=types.go:130 msg="inference compute" id=GPU-72fc8a27-617d-162d-2c0d-18ba84cc7678 library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB"
Feb 20 12:27:56 suma ollama[170277]: [GIN] 2025/02/20 - 12:27:56 | 200 | 101.56µs | 127.0.0.1 | HEAD "/"
Feb 20 12:27:56 suma ollama[170277]: [GIN] 2025/02/20 - 12:27:56 | 200 | 34.185839ms | 127.0.0.1 | POST "/api/show"
Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.453Z level=INFO source=server.go:100 msg="system memory" total="503.7 GiB" free="492.5 GiB" free_swap="8.0 GiB"
Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.454Z level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=62 layers.offload=21 layers.split=7,7,7 memory.available="[47.0 GiB 47.0 GiB 47.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="410.5 GiB" memory.required.partial="126.0 GiB" memory.required.kv="9.5 GiB" memory.required.allocations="[45.0 GiB 40.5 GiB 40.5 GiB]" memory.weights.total="385.0 GiB" memory.weights.repeating="384.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="1019.5 MiB" memory.graph.partial="1019.5 MiB"
Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.455Z level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 --ctx-size 2048 --batch-size 512 --n-gpu-layers 21 --threads 64 --parallel 1 --tensor-split 7,7,7 --port 44637"
Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.456Z level=INFO source=sched.go:449 msg="loaded runners" count=1
Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.456Z level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.456Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.500Z level=INFO source=runner.go:936 msg="starting go runner"
Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.500Z level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=64
Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.500Z level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:44637"
Feb 20 12:27:57 suma ollama[170277]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
Feb 20 12:27:57 suma ollama[170277]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Feb 20 12:27:57 suma ollama[170277]: ggml_cuda_init: found 3 CUDA devices:
Feb 20 12:27:57 suma ollama[170277]: Device 0: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes
Feb 20 12:27:57 suma ollama[170277]: Device 1: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes
Feb 20 12:27:57 suma ollama[170277]: Device 2: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes
Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.709Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
Feb 20 12:27:57 suma ollama[170277]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so
Feb 20 12:27:57 suma ollama[170277]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so
Feb 20 12:27:58 suma ollama[170277]: llama_load_model_from_file: using device CUDA0 (NVIDIA RTX 6000 Ada Generation) - 48087 MiB free
Feb 20 12:27:58 suma ollama[170277]: llama_load_model_from_file: using device CUDA1 (NVIDIA RTX 6000 Ada Generation) - 48087 MiB free
Feb 20 12:27:58 suma ollama[170277]: llama_load_model_from_file: using device CUDA2 (NVIDIA RTX 6000 Ada Generation) - 48087 MiB free
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: loaded meta data with 42 key-value pairs and 1025 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 (version GGUF V3 (latest))
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 0: general.architecture str = deepseek2
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 1: general.type str = model
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 2: general.size_label str = 256x20B
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 3: deepseek2.block_count u32 = 61
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 4: deepseek2.context_length u32 = 163840
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 5: deepseek2.embedding_length u32 = 7168
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 6: deepseek2.feed_forward_length u32 = 18432
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 7: deepseek2.attention.head_count u32 = 128
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 8: deepseek2.attention.head_count_kv u32 = 128
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 9: deepseek2.rope.freq_base f32 = 10000.000000
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 10: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 11: deepseek2.expert_used_count u32 = 8
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 12: deepseek2.leading_dense_block_count u32 = 3
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 13: deepseek2.vocab_size u32 = 129280
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 14: deepseek2.attention.q_lora_rank u32 = 1536
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 15: deepseek2.attention.kv_lora_rank u32 = 512
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 16: deepseek2.attention.key_length u32 = 192
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 17: deepseek2.attention.value_length u32 = 128
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 18: deepseek2.expert_feed_forward_length u32 = 2048
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 19: deepseek2.expert_count u32 = 256
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 20: deepseek2.expert_shared_count u32 = 1
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 21: deepseek2.expert_weights_scale f32 = 2.500000
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 22: deepseek2.expert_weights_norm bool = true
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 23: deepseek2.expert_gating_func u32 = 2
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 24: deepseek2.rope.dimension_count u32 = 64
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 25: deepseek2.rope.scaling.type str = yarn
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 26: deepseek2.rope.scaling.factor f32 = 40.000000
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 27: deepseek2.rope.scaling.original_context_length u32 = 4096
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 28: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 30: tokenizer.ggml.pre str = deepseek-v3
Feb 20 12:27:58 suma ollama[170277]: [132B blob data]
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 1
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 36: tokenizer.ggml.padding_token_id u32 = 1
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = true
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 38: tokenizer.ggml.add_eos_token bool = false
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 39: tokenizer.chat_template str = {% if not add_generation_prompt is de...
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 40: general.quantization_version u32 = 2
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 41: general.file_type u32 = 15
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - type f32: 361 tensors
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - type q4_K: 606 tensors
Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - type q6_K: 58 tensors
Feb 20 12:27:58 suma ollama[170277]: llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
Feb 20 12:27:58 suma ollama[170277]: llm_load_vocab: special tokens cache size = 818
Feb 20 12:27:58 suma ollama[170277]: llm_load_vocab: token to piece cache size = 0.8223 MB
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: format = GGUF V3 (latest)
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: arch = deepseek2
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: vocab type = BPE
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_vocab = 129280
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_merges = 127741
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: vocab_only = 0
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_ctx_train = 163840
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_embd = 7168
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_layer = 61
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_head = 128
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_head_kv = 128
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_rot = 64
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_swa = 0
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_embd_head_k = 192
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_embd_head_v = 128
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_gqa = 1
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_embd_k_gqa = 24576
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_embd_v_gqa = 16384
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: f_norm_eps = 0.0e+00
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: f_clamp_kqv = 0.0e+00
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: f_logit_scale = 0.0e+00
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_ff = 18432
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_expert = 256
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_expert_used = 8
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: causal attn = 1
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: pooling type = 0
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: rope type = 0
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: rope scaling = yarn
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: freq_base_train = 10000.0
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: freq_scale_train = 0.025
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_ctx_orig_yarn = 4096
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: rope_finetuned = unknown
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: ssm_d_conv = 0
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: ssm_d_inner = 0
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: ssm_d_state = 0
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: ssm_dt_rank = 0
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: ssm_dt_b_c_rms = 0
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: model type = 671B
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: model ftype = Q4_K - Medium
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: model params = 671.03 B
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: model size = 376.65 GiB (4.82 BPW)
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: general.name = n/a
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>'
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>'
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: EOT token = 1 '<｜end▁of▁sentence｜>'
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: PAD token = 1 '<｜end▁of▁sentence｜>'
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: LF token = 131 'Ä'
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: FIM PRE token = 128801 '<｜fim▁begin｜>'
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: FIM SUF token = 128800 '<｜fim▁hole｜>'
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: FIM MID token = 128802 '<｜fim▁end｜>'
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: EOG token = 1 '<｜end▁of▁sentence｜>'
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: max token length = 256
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_layer_dense_lead = 3
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_lora_q = 1536
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_lora_kv = 512
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_ff_exp = 2048
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_expert_shared = 1
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: expert_weights_scale = 2.5
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: expert_weights_norm = 1
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: expert_gating_func = sigmoid
Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: rope_yarn_log_mul = 0.1000
Feb 20 12:28:09 suma ollama[170277]: time=2025-02-20T12:28:09.944Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding"
Feb 20 12:28:35 suma ollama[170277]: time=2025-02-20T12:28:35.895Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
Feb 20 12:28:49 suma ollama[170277]: time=2025-02-20T12:28:49.131Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding"
Feb 20 12:28:49 suma ollama[170277]: time=2025-02-20T12:28:49.388Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
Feb 20 12:28:50 suma ollama[170277]: time=2025-02-20T12:28:50.340Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding"
Feb 20 12:28:50 suma ollama[170277]: time=2025-02-20T12:28:50.591Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
Feb 20 12:28:52 suma ollama[170277]: time=2025-02-20T12:28:52.796Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding"
Feb 20 12:29:03 suma ollama[170277]: time=2025-02-20T12:29:03.011Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model"
Feb 20 12:29:03 suma ollama[170277]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 49746.68 MiB on device 2: cudaMalloc failed: out of memory
Feb 20 12:29:32 suma ollama[170277]: llama_model_load: error loading model: unable to allocate CUDA2 buffer
Feb 20 12:29:32 suma ollama[170277]: llama_load_model_from_file: failed to load model
Feb 20 12:29:33 suma ollama[170277]: panic: unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Feb 20 12:29:33 suma ollama[170277]: goroutine 5 [running]:
Feb 20 12:29:33 suma ollama[170277]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0001f2480, {0x15, 0x0, 0x1, 0x0, {0xc0005effa0, 0x3, 0x3}, 0xc00059dd10, 0x0}, ...)
Feb 20 12:29:33 suma ollama[170277]: github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad
Feb 20 12:29:33 suma ollama[170277]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
Feb 20 12:29:33 suma ollama[170277]: github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d
Feb 20 12:29:33 suma ollama[170277]: time=2025-02-20T12:29:33.296Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding"
Feb 20 12:29:33 suma ollama[170277]: time=2025-02-20T12:29:33.546Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
Feb 20 12:29:33 suma ollama[170277]: time=2025-02-20T12:29:33.562Z level=ERROR source=server.go:421 msg="llama runner terminated" error="exit status 2"
Feb 20 12:29:33 suma ollama[170277]: time=2025-02-20T12:29:33.797Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA2 buffer"
Feb 20 12:29:33 suma ollama[170277]: [GIN] 2025/02/20 - 12:29:33 | 500 | 1m37s | 127.0.0.1 | POST "/api/generate"
Feb 20 12:29:39 suma ollama[170277]: time=2025-02-20T12:29:39.122Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.32472629 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Feb 20 12:29:39 suma ollama[170277]: time=2025-02-20T12:29:39.742Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.945308388 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Feb 20 12:29:40 suma ollama[170277]: time=2025-02-20T12:29:40.362Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.565192755 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Feb 20 12:44:03 suma ollama[170277]: [GIN] 2025/02/20 - 12:44:03 | 200 | 92.48µs | 127.0.0.1 | GET "/api/version"

<!-- gh-comment-id:2671394426 --> @charliboy commented on GitHub (Feb 20, 2025): I also have the same problem. My machine has three NVIDIA RTX A6000 Ada graphics cards, with 114GB of video memory and 512GB of memory. ubuntu 22.04, ollama 0. 5.11,The Deepseek 671b model was downloaded from the ollama website and has not been modified. I saw in the logs that it can calculate correctly, but it still reports errors when running： ollama run deepseek-r1:671b --verbose Error: llama runner process has terminated: error loading model: unable to allocate CUDA2 buffer $ nvidia-smi Thu Feb 20 12:41:00 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.86.15 Driver Version: 570.86.15 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX 6000 Ada Gene... Off | 00000000:41:00.0 Off | Off | | 30% 42C P8 27W / 300W | 4MiB / 49140MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX 6000 Ada Gene... Off | 00000000:A1:00.0 Off | Off | | 30% 34C P8 22W / 300W | 4MiB / 49140MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA RTX 6000 Ada Gene... Off | 00000000:E1:00.0 Off | Off | | 0% 41C P8 24W / 300W | 4MiB / 49140MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ Feb 20 12:26:57 suma systemd[1]: Started Ollama Service. Feb 20 12:26:57 suma ollama[169846]: 2025/02/20 12:26:57 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,1,2 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY:http://192.168.11.115:18989 HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:30m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" Feb 20 12:26:57 suma ollama[169846]: time=2025-02-20T12:26:57.740Z level=INFO source=images.go:432 msg="total blobs: 24" Feb 20 12:26:57 suma ollama[169846]: time=2025-02-20T12:26:57.741Z level=INFO source=images.go:439 msg="total unused blobs removed: 0" Feb 20 12:26:57 suma ollama[169846]: time=2025-02-20T12:26:57.741Z level=INFO source=routes.go:1237 msg="Listening on [::]:11434 (version 0.5.11)" Feb 20 12:26:57 suma ollama[169846]: time=2025-02-20T12:26:57.741Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" Feb 20 12:27:02 suma ollama[169846]: time=2025-02-20T12:27:02.614Z level=INFO source=types.go:130 msg="inference compute" id=GPU-b12b24ac-85de-8cf6-0d32-8d91d448e6d4 library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB" Feb 20 12:27:02 suma ollama[169846]: time=2025-02-20T12:27:02.614Z level=INFO source=types.go:130 msg="inference compute" id=GPU-b7d261cb-d515-ff0a-fb8a-9da91967097b library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB" Feb 20 12:27:02 suma ollama[169846]: time=2025-02-20T12:27:02.614Z level=INFO source=types.go:130 msg="inference compute" id=GPU-72fc8a27-617d-162d-2c0d-18ba84cc7678 library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB" Feb 20 12:27:06 suma ollama[169846]: [GIN] 2025/02/20 - 12:27:06 | 200 | 176.14µs | 127.0.0.1 | GET "/api/version" Feb 20 12:27:21 suma ollama[169846]: [GIN] 2025/02/20 - 12:27:21 | 200 | 43.56µs | 127.0.0.1 | HEAD "/" Feb 20 12:27:21 suma ollama[169846]: [GIN] 2025/02/20 - 12:27:21 | 200 | 6.46148ms | 127.0.0.1 | GET "/api/tags" Feb 20 12:27:38 suma ollama[169846]: [GIN] 2025/02/20 - 12:27:38 | 200 | 61.25µs | 127.0.0.1 | HEAD "/" Feb 20 12:27:38 suma ollama[169846]: [GIN] 2025/02/20 - 12:27:38 | 200 | 2.36149ms | 127.0.0.1 | POST "/api/show" Feb 20 12:27:39 suma ollama[169846]: time=2025-02-20T12:27:39.222Z level=WARN source=memory.go:123 msg="model missing blk.0 layer size" Feb 20 12:27:39 suma ollama[169846]: panic: interface conversion: interface {} is nil, not *llm.array Feb 20 12:27:39 suma ollama[169846]: goroutine 138 [running]: Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm.GGML.GraphSize({{0x561cec056120?, 0xc0006700a0?}, {0x561cec0560a8?, 0xc000630008?}}, 0x2000, 0x200, {0x0, 0x0}) Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm/ggml.go:367 +0x10bf Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm.EstimateGPULayers({_, _, _}, _, {_, _, _}, {{0x2000, 0x200, 0xffffffffffffffff, ...}, ...}) Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm/memory.go:138 +0x5fa Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm.PredictServerFit({0xc0004cdba8?, 0x561ceae8b4ff?, 0xc0004cd8c0?}, 0xc0008a2040, {0xc0004cd920?, _, _}, {0x0, 0x0, 0x0}, ...) Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/llm/memory.go:22 +0xbd Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server.pickBestFullFitByLibrary(0xc000660000, 0xc0008a2040, {0xc0005a4308?, 0x3?, 0x4?}, 0xc00064dcf8) Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server/sched.go:713 +0x6f3 Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server.(*Scheduler).processPending(0xc0002170e0, {0x561cec059f40, 0xc000570410}) Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server/sched.go:225 +0xe6c Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server.(*Scheduler).Run.func1() Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server/sched.go:107 +0x1f Feb 20 12:27:39 suma ollama[169846]: created by github.com/ollama/ollama/server.(*Scheduler).Run in goroutine 1 Feb 20 12:27:39 suma ollama[169846]: github.com/ollama/ollama/server/sched.go:106 +0xb4 Feb 20 12:27:39 suma systemd[1]: ollama.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Feb 20 12:27:39 suma systemd[1]: ollama.service: Failed with result 'exit-code'. Feb 20 12:27:39 suma systemd[1]: ollama.service: Consumed 6.293s CPU time. Feb 20 12:27:42 suma systemd[1]: ollama.service: Scheduled restart job, restart counter is at 1. Feb 20 12:27:42 suma systemd[1]: Stopped Ollama Service. Feb 20 12:27:42 suma systemd[1]: ollama.service: Consumed 6.293s CPU time. Feb 20 12:27:42 suma systemd[1]: Started Ollama Service. Feb 20 12:27:43 suma ollama[170277]: 2025/02/20 12:27:43 routes.go:1186: INFO server config env="map[CUDA_VISIBLE_DEVICES:0,1,2 GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY:http://192.168.11.115:18989 HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:2562047h47m16.854775807s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:30m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" Feb 20 12:27:43 suma ollama[170277]: time=2025-02-20T12:27:43.004Z level=INFO source=images.go:432 msg="total blobs: 24" Feb 20 12:27:43 suma ollama[170277]: time=2025-02-20T12:27:43.005Z level=INFO source=images.go:439 msg="total unused blobs removed: 0" Feb 20 12:27:43 suma ollama[170277]: time=2025-02-20T12:27:43.005Z level=INFO source=routes.go:1237 msg="Listening on [::]:11434 (version 0.5.11)" Feb 20 12:27:43 suma ollama[170277]: time=2025-02-20T12:27:43.005Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" Feb 20 12:27:47 suma ollama[170277]: time=2025-02-20T12:27:47.862Z level=INFO source=types.go:130 msg="inference compute" id=GPU-b12b24ac-85de-8cf6-0d32-8d91d448e6d4 library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB" Feb 20 12:27:47 suma ollama[170277]: time=2025-02-20T12:27:47.863Z level=INFO source=types.go:130 msg="inference compute" id=GPU-b7d261cb-d515-ff0a-fb8a-9da91967097b library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB" Feb 20 12:27:47 suma ollama[170277]: time=2025-02-20T12:27:47.863Z level=INFO source=types.go:130 msg="inference compute" id=GPU-72fc8a27-617d-162d-2c0d-18ba84cc7678 library=cuda variant=v12 compute=8.9 driver=12.8 name="NVIDIA RTX 6000 Ada Generation" total="47.4 GiB" available="47.0 GiB" Feb 20 12:27:56 suma ollama[170277]: [GIN] 2025/02/20 - 12:27:56 | 200 | 101.56µs | 127.0.0.1 | HEAD "/" Feb 20 12:27:56 suma ollama[170277]: [GIN] 2025/02/20 - 12:27:56 | 200 | 34.185839ms | 127.0.0.1 | POST "/api/show" Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.453Z level=INFO source=server.go:100 msg="system memory" total="503.7 GiB" free="492.5 GiB" free_swap="8.0 GiB" Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.454Z level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=62 layers.offload=21 layers.split=7,7,7 memory.available="[47.0 GiB 47.0 GiB 47.0 GiB]" memory.gpu_overhead="0 B" memory.required.full="410.5 GiB" memory.required.partial="126.0 GiB" memory.required.kv="9.5 GiB" memory.required.allocations="[45.0 GiB 40.5 GiB 40.5 GiB]" memory.weights.total="385.0 GiB" memory.weights.repeating="384.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="1019.5 MiB" memory.graph.partial="1019.5 MiB" Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.455Z level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 --ctx-size 2048 --batch-size 512 --n-gpu-layers 21 --threads 64 --parallel 1 --tensor-split 7,7,7 --port 44637" Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.456Z level=INFO source=sched.go:449 msg="loaded runners" count=1 Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.456Z level=INFO source=server.go:557 msg="waiting for llama runner to start responding" Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.456Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.500Z level=INFO source=runner.go:936 msg="starting go runner" Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.500Z level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" threads=64 Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.500Z level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:44637" Feb 20 12:27:57 suma ollama[170277]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Feb 20 12:27:57 suma ollama[170277]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Feb 20 12:27:57 suma ollama[170277]: ggml_cuda_init: found 3 CUDA devices: Feb 20 12:27:57 suma ollama[170277]: Device 0: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes Feb 20 12:27:57 suma ollama[170277]: Device 1: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes Feb 20 12:27:57 suma ollama[170277]: Device 2: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes Feb 20 12:27:57 suma ollama[170277]: time=2025-02-20T12:27:57.709Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" Feb 20 12:27:57 suma ollama[170277]: load_backend: loaded CUDA backend from /usr/local/lib/ollama/cuda_v12/libggml-cuda.so Feb 20 12:27:57 suma ollama[170277]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-haswell.so Feb 20 12:27:58 suma ollama[170277]: llama_load_model_from_file: using device CUDA0 (NVIDIA RTX 6000 Ada Generation) - 48087 MiB free Feb 20 12:27:58 suma ollama[170277]: llama_load_model_from_file: using device CUDA1 (NVIDIA RTX 6000 Ada Generation) - 48087 MiB free Feb 20 12:27:58 suma ollama[170277]: llama_load_model_from_file: using device CUDA2 (NVIDIA RTX 6000 Ada Generation) - 48087 MiB free Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: loaded meta data with 42 key-value pairs and 1025 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 (version GGUF V3 (latest)) Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 0: general.architecture str = deepseek2 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 1: general.type str = model Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 2: general.size_label str = 256x20B Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 3: deepseek2.block_count u32 = 61 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 4: deepseek2.context_length u32 = 163840 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 5: deepseek2.embedding_length u32 = 7168 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 6: deepseek2.feed_forward_length u32 = 18432 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 7: deepseek2.attention.head_count u32 = 128 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 8: deepseek2.attention.head_count_kv u32 = 128 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 9: deepseek2.rope.freq_base f32 = 10000.000000 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 10: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 11: deepseek2.expert_used_count u32 = 8 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 12: deepseek2.leading_dense_block_count u32 = 3 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 13: deepseek2.vocab_size u32 = 129280 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 14: deepseek2.attention.q_lora_rank u32 = 1536 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 15: deepseek2.attention.kv_lora_rank u32 = 512 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 16: deepseek2.attention.key_length u32 = 192 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 17: deepseek2.attention.value_length u32 = 128 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 18: deepseek2.expert_feed_forward_length u32 = 2048 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 19: deepseek2.expert_count u32 = 256 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 20: deepseek2.expert_shared_count u32 = 1 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 21: deepseek2.expert_weights_scale f32 = 2.500000 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 22: deepseek2.expert_weights_norm bool = true Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 23: deepseek2.expert_gating_func u32 = 2 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 24: deepseek2.rope.dimension_count u32 = 64 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 25: deepseek2.rope.scaling.type str = yarn Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 26: deepseek2.rope.scaling.factor f32 = 40.000000 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 27: deepseek2.rope.scaling.original_context_length u32 = 4096 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 28: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 30: tokenizer.ggml.pre str = deepseek-v3 Feb 20 12:27:58 suma ollama[170277]: [132B blob data] Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e... Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 1 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 36: tokenizer.ggml.padding_token_id u32 = 1 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = true Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 38: tokenizer.ggml.add_eos_token bool = false Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 39: tokenizer.chat_template str = {% if not add_generation_prompt is de... Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 40: general.quantization_version u32 = 2 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - kv 41: general.file_type u32 = 15 Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - type f32: 361 tensors Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - type q4_K: 606 tensors Feb 20 12:27:58 suma ollama[170277]: llama_model_loader: - type q6_K: 58 tensors Feb 20 12:27:58 suma ollama[170277]: llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect Feb 20 12:27:58 suma ollama[170277]: llm_load_vocab: special tokens cache size = 818 Feb 20 12:27:58 suma ollama[170277]: llm_load_vocab: token to piece cache size = 0.8223 MB Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: format = GGUF V3 (latest) Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: arch = deepseek2 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: vocab type = BPE Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_vocab = 129280 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_merges = 127741 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: vocab_only = 0 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_ctx_train = 163840 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_embd = 7168 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_layer = 61 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_head = 128 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_head_kv = 128 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_rot = 64 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_swa = 0 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_embd_head_k = 192 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_embd_head_v = 128 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_gqa = 1 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_embd_k_gqa = 24576 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_embd_v_gqa = 16384 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: f_norm_eps = 0.0e+00 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: f_logit_scale = 0.0e+00 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_ff = 18432 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_expert = 256 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_expert_used = 8 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: causal attn = 1 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: pooling type = 0 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: rope type = 0 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: rope scaling = yarn Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: freq_base_train = 10000.0 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: freq_scale_train = 0.025 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_ctx_orig_yarn = 4096 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: rope_finetuned = unknown Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: ssm_d_conv = 0 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: ssm_d_inner = 0 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: ssm_d_state = 0 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: ssm_dt_rank = 0 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: ssm_dt_b_c_rms = 0 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: model type = 671B Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: model ftype = Q4_K - Medium Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: model params = 671.03 B Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: model size = 376.65 GiB (4.82 BPW) Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: general.name = n/a Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>' Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>' Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: EOT token = 1 '<｜end▁of▁sentence｜>' Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: PAD token = 1 '<｜end▁of▁sentence｜>' Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: LF token = 131 'Ä' Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: FIM PRE token = 128801 '<｜fim▁begin｜>' Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: FIM SUF token = 128800 '<｜fim▁hole｜>' Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: FIM MID token = 128802 '<｜fim▁end｜>' Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: EOG token = 1 '<｜end▁of▁sentence｜>' Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: max token length = 256 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_layer_dense_lead = 3 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_lora_q = 1536 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_lora_kv = 512 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_ff_exp = 2048 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: n_expert_shared = 1 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: expert_weights_scale = 2.5 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: expert_weights_norm = 1 Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: expert_gating_func = sigmoid Feb 20 12:27:58 suma ollama[170277]: llm_load_print_meta: rope_yarn_log_mul = 0.1000 Feb 20 12:28:09 suma ollama[170277]: time=2025-02-20T12:28:09.944Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding" Feb 20 12:28:35 suma ollama[170277]: time=2025-02-20T12:28:35.895Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" Feb 20 12:28:49 suma ollama[170277]: time=2025-02-20T12:28:49.131Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding" Feb 20 12:28:49 suma ollama[170277]: time=2025-02-20T12:28:49.388Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" Feb 20 12:28:50 suma ollama[170277]: time=2025-02-20T12:28:50.340Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding" Feb 20 12:28:50 suma ollama[170277]: time=2025-02-20T12:28:50.591Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" Feb 20 12:28:52 suma ollama[170277]: time=2025-02-20T12:28:52.796Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding" Feb 20 12:29:03 suma ollama[170277]: time=2025-02-20T12:29:03.011Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server loading model" Feb 20 12:29:03 suma ollama[170277]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 49746.68 MiB on device 2: cudaMalloc failed: out of memory Feb 20 12:29:32 suma ollama[170277]: llama_model_load: error loading model: unable to allocate CUDA2 buffer Feb 20 12:29:32 suma ollama[170277]: llama_load_model_from_file: failed to load model Feb 20 12:29:33 suma ollama[170277]: panic: unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 Feb 20 12:29:33 suma ollama[170277]: goroutine 5 [running]: Feb 20 12:29:33 suma ollama[170277]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0001f2480, {0x15, 0x0, 0x1, 0x0, {0xc0005effa0, 0x3, 0x3}, 0xc00059dd10, 0x0}, ...) Feb 20 12:29:33 suma ollama[170277]: github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad Feb 20 12:29:33 suma ollama[170277]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 Feb 20 12:29:33 suma ollama[170277]: github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d Feb 20 12:29:33 suma ollama[170277]: time=2025-02-20T12:29:33.296Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server not responding" Feb 20 12:29:33 suma ollama[170277]: time=2025-02-20T12:29:33.546Z level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" Feb 20 12:29:33 suma ollama[170277]: time=2025-02-20T12:29:33.562Z level=ERROR source=server.go:421 msg="llama runner terminated" error="exit status 2" Feb 20 12:29:33 suma ollama[170277]: time=2025-02-20T12:29:33.797Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA2 buffer" Feb 20 12:29:33 suma ollama[170277]: [GIN] 2025/02/20 - 12:29:33 | 500 | 1m37s | 127.0.0.1 | POST "/api/generate" Feb 20 12:29:39 suma ollama[170277]: time=2025-02-20T12:29:39.122Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.32472629 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 Feb 20 12:29:39 suma ollama[170277]: time=2025-02-20T12:29:39.742Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.945308388 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 Feb 20 12:29:40 suma ollama[170277]: time=2025-02-20T12:29:40.362Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.565192755 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 Feb 20 12:44:03 suma ollama[170277]: [GIN] 2025/02/20 - 12:44:03 | 200 | 92.48µs | 127.0.0.1 | GET "/api/version"
GiteaMirror commented 2026-04-12 17:19:31 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 20, 2025):

Possibly related: https://github.com/ollama/ollama/pull/9243

<!-- gh-comment-id:2672295528 --> @rick-github commented on GitHub (Feb 20, 2025): Possibly related: https://github.com/ollama/ollama/pull/9243
GiteaMirror commented 2026-04-12 17:19:32 -05:00
Author
Owner
Copy Link

@itej89 commented on GitHub (Feb 21, 2025):

Possibly related: #9243

Yes, I've created the PR to address this same issue noticed on MI210.

<!-- gh-comment-id:2675753156 --> @itej89 commented on GitHub (Feb 21, 2025): > Possibly related: [#9243](https://github.com/ollama/ollama/pull/9243) Yes, I've created the PR to address this same issue noticed on MI210.
GiteaMirror commented 2026-04-12 17:19:32 -05:00
Author
Owner
Copy Link

@axkibe commented on GitHub (Feb 25, 2025):

Possibly related: #9243

Yes, I've created the PR to address this same issue noticed on MI210.

Yes it works with the patch:

layers.model=62 layers.offload=8 layers.split=4,4

It might still undershoot somewhat tough (used 34/40 and 30/40 GB of both A100)

| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-PCIE-40GB          Off | 00000000:01:00.0 Off |                  Off |
| N/A   32C    P0              61W / 250W |  34547MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-PCIE-40GB          Off | 00000000:81:00.0 Off |                  Off |
| N/A   30C    P0              55W / 250W |  30187MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

It runs with ~2 token per second, so I guess a dual A100 (without NVLink) is still massively underequipped to run deepsek:671b in a satisfactory way. And it messages me about reduced context too.

<!-- gh-comment-id:2681681585 --> @axkibe commented on GitHub (Feb 25, 2025): > > Possibly related: [#9243](https://github.com/ollama/ollama/pull/9243) > > Yes, I've created the PR to address this same issue noticed on MI210. Yes it works with the patch: ```layers.model=62 layers.offload=8 layers.split=4,4``` It might still undershoot somewhat tough (used 34/40 and 30/40 GB of both A100) ```+---------------------------------------------------------------------------------------+ | NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA A100-PCIE-40GB Off | 00000000:01:00.0 Off | Off | | N/A 32C P0 61W / 250W | 34547MiB / 40960MiB | 0% Default | | | | Disabled | +-----------------------------------------+----------------------+----------------------+ | 1 NVIDIA A100-PCIE-40GB Off | 00000000:81:00.0 Off | Off | | N/A 30C P0 55W / 250W | 30187MiB / 40960MiB | 0% Default | | | | Disabled | +-----------------------------------------+----------------------+----------------------+ ``` It runs with ~2 token per second, so I guess a dual A100 (without NVLink) is still massively underequipped to run deepsek:671b in a satisfactory way. And it messages me about reduced context too.
GiteaMirror commented 2026-04-12 17:19:33 -05:00
Author
Owner
Copy Link

@itej89 commented on GitHub (Feb 25, 2025):

Possibly related: #9243

Yes, I've created the PR to address this same issue noticed on MI210.

Yes it works with the patch:

layers.model=62 layers.offload=8 layers.split=4,4

It might still undershoot somewhat tough (used 34/40 and 30/40 GB of both A100)

| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-PCIE-40GB          Off | 00000000:01:00.0 Off |                  Off |
| N/A   32C    P0              61W / 250W |  34547MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-PCIE-40GB          Off | 00000000:81:00.0 Off |                  Off |
| N/A   30C    P0              55W / 250W |  30187MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

It runs with ~2 token per second, so I guess a dual A100 (without NVLink) is still massively underequipped to run deepsek:671b in a satisfactory way. And it messages me about reduced context too.

Thanks for sharing the results. Yes the observed undershoot is because of the large final layer sizes (each one is ~6-7GB) and also the initial buffer allocation require further tuning for optimal memory usage.

<!-- gh-comment-id:2682489565 --> @itej89 commented on GitHub (Feb 25, 2025): > > > Possibly related: [#9243](https://github.com/ollama/ollama/pull/9243) > > > > > > Yes, I've created the PR to address this same issue noticed on MI210. > > Yes it works with the patch: > > `layers.model=62 layers.offload=8 layers.split=4,4` > > It might still undershoot somewhat tough (used 34/40 and 30/40 GB of both A100) > > ``` > | NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 | > |-----------------------------------------+----------------------+----------------------+ > | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | > | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | > | | | MIG M. | > |=========================================+======================+======================| > | 0 NVIDIA A100-PCIE-40GB Off | 00000000:01:00.0 Off | Off | > | N/A 32C P0 61W / 250W | 34547MiB / 40960MiB | 0% Default | > | | | Disabled | > +-----------------------------------------+----------------------+----------------------+ > | 1 NVIDIA A100-PCIE-40GB Off | 00000000:81:00.0 Off | Off | > | N/A 30C P0 55W / 250W | 30187MiB / 40960MiB | 0% Default | > | | | Disabled | > +-----------------------------------------+----------------------+----------------------+ > ``` > > It runs with ~2 token per second, so I guess a dual A100 (without NVLink) is still massively underequipped to run deepsek:671b in a satisfactory way. And it messages me about reduced context too. Thanks for sharing the results. Yes the observed undershoot is because of the large final layer sizes (each one is ~6-7GB) and also the initial buffer allocation require further tuning for optimal memory usage.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5970
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?