[GH-ISSUE #7072] Deepseek-v2.5 fails to load on a system with 24GB VRAM (RTX 3090) and 128GB RAM #51000

New Issue

GiteaMirror · 2026-04-28T17:46:41-05:00

GiteaMirror commented

2026-04-28 17:46:41 -05:00

Originally created by @LeonidShamis on GitHub (Oct 2, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7072

What is the issue?

I'm unable to load the deepseek-v2.5 model on a system with 24GB VRAM (RTX 3090) and 128GB RAM:

$ ollama --version
ollama version is 0.3.11
$

$ ollama list | grep -e ID -e deepseek-v2.5
NAME                             ID              SIZE      MODIFIED
deepseek-v2.5:latest             409b2dd8a3c4    132 GB    9 hours ago

$ ollama show deepseek-v2.5
  Model
    architecture        deepseek2
    parameters          235.7B
    context length      163840
    embedding length    5120
    quantization        Q4_0

  Parameters
    stop    "<|begin?of?sentence|>"
    stop    "<|end?of?sentence|>"
    stop    "<|User|>"
    stop    "<|Assistant|>"
    stop    "<|fim?begin|>"
    stop    "<|fim?hole|>"
    stop    "<|fim?end|>"

  License
    DEEPSEEK LICENSE AGREEMENT
    Version 1.0, 23 October 2023

$ ollama run deepseek-v2.5
Error: llama runner process has terminated: error:failed to create context with model '/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c'
$

Journatctl output:

$ sudo journalctl -f
Oct 02 09:20:41 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:20:41 | 200 |       14.89µs |       127.0.0.1 | HEAD     "/"
Oct 02 09:20:41 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:20:41 | 200 |   22.898622ms |       127.0.0.1 | GET      "/api/tags"
Oct 02 09:20:56 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost
Oct 02 09:21:04 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost
Oct 02 09:21:08 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:08 | 200 |       18.64µs |       127.0.0.1 | HEAD     "/"
Oct 02 09:21:08 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:08 | 200 |    3.682586ms |       127.0.0.1 | GET      "/api/tags"
Oct 02 09:21:23 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:23 | 200 |       29.73µs |       127.0.0.1 | HEAD     "/"
Oct 02 09:21:23 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:23 | 200 |    3.880099ms |       127.0.0.1 | GET      "/api/tags"
Oct 02 09:21:28 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:28 | 200 |       15.64µs |       127.0.0.1 | HEAD     "/"
Oct 02 09:21:28 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:28 | 200 |    1.227682ms |       127.0.0.1 | GET      "/api/tags"
Oct 02 09:21:35 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:35 | 200 |        17.3µs |       127.0.0.1 | HEAD     "/"
Oct 02 09:21:35 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:35 | 200 |   29.058041ms |       127.0.0.1 | POST     "/api/show"
Oct 02 09:22:02 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:22:02 | 200 |       15.01µs |       127.0.0.1 | HEAD     "/"
Oct 02 09:22:02 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:22:02 | 200 |   13.210104ms |       127.0.0.1 | POST     "/api/show"
Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.094+10:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-cebf1f58-0f47-0efa-a683-ceeb3c1755cc library=cuda total="23.7 GiB" available="8.8 GiB"
Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.909+10:00 level=INFO source=server.go:103 msg="system memory" total="125.7 GiB" free="118.6 GiB" free_swap="7.2 GiB"
Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.910+10:00 level=INFO source=memory.go:326 msg="offload to cuda" layers.requested=-1 layers.model=61 layers.offload=10 layers.split="" memory.available="[23.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="134.5 GiB" memory.required.partial="22.1 GiB" memory.required.kv="9.4 GiB" memory.required.allocations="[22.1 GiB]" memory.weights.total="132.5 GiB" memory.weights.repeating="132.1 GiB" memory.weights.nonrepeating="410.2 MiB" memory.graph.full="642.0 MiB" memory.graph.partial="891.5 MiB"
Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.911+10:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/tmp/ollama1470999858/runners/cuda_v12/ollama_llama_server --model /mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 10 --no-mmap --parallel 1 --port 43133"
Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.911+10:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.911+10:00 level=INFO source=server.go:587 msg="waiting for llama runner to start responding"
Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.912+10:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error"
Oct 02 09:22:03 xlr2 ollama[3244519]: INFO [main] build info | build=10 commit="9225b05" tid="139874777862144" timestamp=1727824923
Oct 02 09:22:03 xlr2 ollama[3244519]: INFO [main] system info | n_threads=8 n_threads_batch=8 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="139874777862144" timestamp=1727824923 total_threads=16
Oct 02 09:22:03 xlr2 ollama[3244519]: INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="15" port="43133" tid="139874777862144" timestamp=1727824923
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: loaded meta data with 46 key-value pairs and 959 tensors from /mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c (version GGUF V3 (latest))
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv   1:                               general.type str              = model
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv   2:                               general.name str              = DeepSeek V2.5
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv   3:                            general.version str              = V2.5
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv   4:                           general.basename str              = DeepSeek
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv   5:                         general.size_label str              = 160x14B
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv   6:                            general.license str              = other
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv   7:                       general.license.name str              = deepseek
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv   8:                       general.license.link str              = https://github.com/deepseek-ai/DeepSe...
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv   9:                      deepseek2.block_count u32              = 60
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  10:                   deepseek2.context_length u32              = 163840
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  11:                 deepseek2.embedding_length u32              = 5120
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  12:              deepseek2.feed_forward_length u32              = 12288
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  13:             deepseek2.attention.head_count u32              = 128
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  14:          deepseek2.attention.head_count_kv u32              = 128
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  15:                   deepseek2.rope.freq_base f32              = 10000.000000
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  16: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  17:                deepseek2.expert_used_count u32              = 6
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  18:                          general.file_type u32              = 2
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  19:        deepseek2.leading_dense_block_count u32              = 1
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  20:                       deepseek2.vocab_size u32              = 102400
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  21:            deepseek2.attention.q_lora_rank u32              = 1536
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  22:           deepseek2.attention.kv_lora_rank u32              = 512
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  23:             deepseek2.attention.key_length u32              = 192
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  24:           deepseek2.attention.value_length u32              = 128
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  25:       deepseek2.expert_feed_forward_length u32              = 1536
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  26:                     deepseek2.expert_count u32              = 160
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  27:              deepseek2.expert_shared_count u32              = 2
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  28:             deepseek2.expert_weights_scale f32              = 16.000000
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  29:             deepseek2.rope.dimension_count u32              = 64
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  30:                deepseek2.rope.scaling.type str              = yarn
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  31:              deepseek2.rope.scaling.factor f32              = 40.000000
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  32: deepseek2.rope.scaling.original_context_length u32              = 4096
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  33: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  34:                       tokenizer.ggml.model str              = gpt2
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  35:                         tokenizer.ggml.pre str              = deepseek-llm
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  36:                      tokenizer.ggml.tokens arr[str,102400]  = ["!", "\"", "#", "$", "%", "&", "'", ...
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  37:                  tokenizer.ggml.token_type arr[i32,102400]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  38:                      tokenizer.ggml.merges arr[str,99757]   = ["G G", "G t", "G a", "i n", "h e...
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  39:                tokenizer.ggml.bos_token_id u32              = 100000
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  40:                tokenizer.ggml.eos_token_id u32              = 100001
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  41:            tokenizer.ggml.padding_token_id u32              = 100001
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  42:               tokenizer.ggml.add_bos_token bool             = true
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  43:               tokenizer.ggml.add_eos_token bool             = false
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  44:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv  45:               general.quantization_version u32              = 2
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - type  f32:  300 tensors
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - type q4_0:  658 tensors
Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - type q6_K:    1 tensors
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_vocab: special tokens cache size = 18
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_vocab: token to piece cache size = 0.6411 MB
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: format           = GGUF V3 (latest)
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: arch             = deepseek2
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: vocab type       = BPE
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_vocab          = 102400
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_merges         = 99757
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: vocab_only       = 0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_ctx_train      = 163840
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_embd           = 5120
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_layer          = 60
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_head           = 128
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_head_kv        = 128
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_rot            = 64
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_swa            = 0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_embd_head_k    = 192
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_embd_head_v    = 128
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_gqa            = 1
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_embd_k_gqa     = 24576
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_embd_v_gqa     = 16384
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: f_logit_scale    = 0.0e+00
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_ff             = 12288
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_expert         = 160
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_expert_used    = 6
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: causal attn      = 1
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: pooling type     = 0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: rope type        = 0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: rope scaling     = yarn
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: freq_base_train  = 10000.0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: freq_scale_train = 0.025
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_ctx_orig_yarn  = 4096
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: rope_finetuned   = unknown
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: ssm_d_conv       = 0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: ssm_d_inner      = 0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: ssm_d_state      = 0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: ssm_dt_rank      = 0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: ssm_dt_b_c_rms   = 0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: model type       = 236B
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: model ftype      = Q4_0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: model params     = 235.74 B
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: model size       = 123.78 GiB (4.51 BPW)
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: general.name     = DeepSeek V2.5
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: BOS token        = 100000 '<|begin?of?sentence|>'
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: EOS token        = 100001 '<|end?of?sentence|>'
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: PAD token        = 100001 '<|end?of?sentence|>'
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: LF token         = 126 'Ä'
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: max token length = 256
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_layer_dense_lead   = 1
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_lora_q             = 1536
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_lora_kv            = 512
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_ff_exp             = 1536
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_expert_shared      = 2
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: expert_weights_scale = 16.0
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: rope_yarn_log_mul    = 0.1000
Oct 02 09:22:04 xlr2 ollama[2576620]: time=2024-10-02T09:22:04.163+10:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model"
Oct 02 09:22:04 xlr2 ollama[2576620]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Oct 02 09:22:04 xlr2 ollama[2576620]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Oct 02 09:22:04 xlr2 ollama[2576620]: ggml_cuda_init: found 1 CUDA devices:
Oct 02 09:22:04 xlr2 ollama[2576620]:   Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_tensors: ggml ctx size =    0.80 MiB
Oct 02 09:22:05 xlr2 ollama[2576620]: time=2024-10-02T09:22:05.618+10:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server not responding"
Oct 02 09:22:33 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost
Oct 02 09:22:37 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost
Oct 02 09:22:47 xlr2 ollama[2576620]: time=2024-10-02T09:22:47.399+10:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model"
Oct 02 09:22:51 xlr2 ollama[2576620]: llm_load_tensors: offloading 10 repeating layers to GPU
Oct 02 09:22:51 xlr2 ollama[2576620]: llm_load_tensors: offloaded 10/61 layers to GPU
Oct 02 09:22:51 xlr2 ollama[2576620]: llm_load_tensors:  CUDA_Host buffer size = 105416.00 MiB
Oct 02 09:22:51 xlr2 ollama[2576620]: llm_load_tensors:      CUDA0 buffer size = 21335.35 MiB
Oct 02 09:23:10 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost
Oct 02 09:23:17 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost
Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: n_ctx      = 2048
Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: n_batch    = 512
Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: n_ubatch   = 512
Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: flash_attn = 0
Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: freq_base  = 10000.0
Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: freq_scale = 0.025
Oct 02 09:23:48 xlr2 ollama[2576620]: llama_kv_cache_init:  CUDA_Host KV buffer size =  8000.00 MiB
Oct 02 09:23:48 xlr2 ollama[2576620]: llama_kv_cache_init:      CUDA0 KV buffer size =  1600.00 MiB
Oct 02 09:23:48 xlr2 ollama[2576620]: llama_new_context_with_model: KV self size  = 9600.00 MiB, K (f16): 5760.00 MiB, V (f16): 3840.00 MiB
Oct 02 09:23:48 xlr2 ollama[2576620]: llama_new_context_with_model:  CUDA_Host  output buffer size =     0.41 MiB
Oct 02 09:23:48 xlr2 ollama[2576620]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 842.00 MiB on device 0: cudaMalloc failed: out of memory
Oct 02 09:23:48 xlr2 ollama[2576620]: ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 882903040
Oct 02 09:23:48 xlr2 ollama[2576620]: llama_new_context_with_model: failed to allocate compute buffers
Oct 02 09:23:50 xlr2 ollama[2576620]: llama_init_from_gpt_params: error: failed to create context with model '/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c'
Oct 02 09:23:52 xlr2 kernel: clocksource: Long readout interval, skipping watchdog check: cs_nsec: 1586308203 wd_nsec: 1586308525
Oct 02 09:24:05 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost
Oct 02 09:24:09 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost
Oct 02 09:24:10 xlr2 ollama[3244519]: ERROR [load_model] unable to load model | model="/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c" tid="139874777862144" timestamp=1727825050
Oct 02 09:24:10 xlr2 ollama[2576620]: terminate called without an active exception
Oct 02 09:24:10 xlr2 ollama[2576620]: time=2024-10-02T09:24:10.958+10:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error"
Oct 02 09:24:11 xlr2 ollama[2576620]: time=2024-10-02T09:24:11.213+10:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:failed to create context with model '/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c'"
Oct 02 09:24:11 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:24:11 | 500 |          2m8s |       127.0.0.1 | POST     "/api/generate"
Oct 02 09:24:16 xlr2 ollama[2576620]: time=2024-10-02T09:24:16.336+10:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.122496579 model=/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c
Oct 02 09:24:16 xlr2 ollama[2576620]: time=2024-10-02T09:24:16.586+10:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.372482202 model=/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c
Oct 02 09:24:16 xlr2 ollama[2576620]: time=2024-10-02T09:24:16.837+10:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.623037141 model=/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c

System info:

$ uname -a
Linux xlr2 5.15.0-112-generic #122-Ubuntu SMP Thu May 23 07:48:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
$ nproc
16
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           125Gi       5.3Gi       113Gi        33Mi       6.5Gi       119Gi
Swap:          8.0Gi       887Mi       7.1Gi
$ nvidia-smi
Wed Oct  2 10:04:06 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        On  | 00000000:0C:00.0 Off |                  N/A |
|  0%   33C    P8              38W / 390W |      3MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
$

I tried with both 0.3.11 and the latest 0.3.12 versions of Ollama.

Here is a snapshot of htop and nvtop outputs:

Here is the recording showing the entire process:

https://github.com/user-attachments/assets/e306990a-a340-4a8f-b7de-b52d9b3de5e1

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.3.11

Originally created by @LeonidShamis on GitHub (Oct 2, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7072 ### What is the issue? I'm unable to load the [deepseek-v2.5](https://ollama.com/library/deepseek-v2.5) model on a system with 24GB VRAM (RTX 3090) and 128GB RAM: ``` $ ollama --version ollama version is 0.3.11 $ $ ollama list | grep -e ID -e deepseek-v2.5 NAME ID SIZE MODIFIED deepseek-v2.5:latest 409b2dd8a3c4 132 GB 9 hours ago $ ollama show deepseek-v2.5 Model architecture deepseek2 parameters 235.7B context length 163840 embedding length 5120 quantization Q4_0 Parameters stop "<|begin?of?sentence|>" stop "<|end?of?sentence|>" stop "<|User|>" stop "<|Assistant|>" stop "<|fim?begin|>" stop "<|fim?hole|>" stop "<|fim?end|>" License DEEPSEEK LICENSE AGREEMENT Version 1.0, 23 October 2023 $ ollama run deepseek-v2.5 Error: llama runner process has terminated: error:failed to create context with model '/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c' $ ``` Journatctl output: ``` $ sudo journalctl -f Oct 02 09:20:41 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:20:41 | 200 | 14.89µs | 127.0.0.1 | HEAD "/" Oct 02 09:20:41 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:20:41 | 200 | 22.898622ms | 127.0.0.1 | GET "/api/tags" Oct 02 09:20:56 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost Oct 02 09:21:04 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost Oct 02 09:21:08 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:08 | 200 | 18.64µs | 127.0.0.1 | HEAD "/" Oct 02 09:21:08 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:08 | 200 | 3.682586ms | 127.0.0.1 | GET "/api/tags" Oct 02 09:21:23 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:23 | 200 | 29.73µs | 127.0.0.1 | HEAD "/" Oct 02 09:21:23 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:23 | 200 | 3.880099ms | 127.0.0.1 | GET "/api/tags" Oct 02 09:21:28 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:28 | 200 | 15.64µs | 127.0.0.1 | HEAD "/" Oct 02 09:21:28 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:28 | 200 | 1.227682ms | 127.0.0.1 | GET "/api/tags" Oct 02 09:21:35 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:35 | 200 | 17.3µs | 127.0.0.1 | HEAD "/" Oct 02 09:21:35 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:21:35 | 200 | 29.058041ms | 127.0.0.1 | POST "/api/show" Oct 02 09:22:02 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:22:02 | 200 | 15.01µs | 127.0.0.1 | HEAD "/" Oct 02 09:22:02 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:22:02 | 200 | 13.210104ms | 127.0.0.1 | POST "/api/show" Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.094+10:00 level=INFO source=sched.go:507 msg="updated VRAM based on existing loaded models" gpu=GPU-cebf1f58-0f47-0efa-a683-ceeb3c1755cc library=cuda total="23.7 GiB" available="8.8 GiB" Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.909+10:00 level=INFO source=server.go:103 msg="system memory" total="125.7 GiB" free="118.6 GiB" free_swap="7.2 GiB" Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.910+10:00 level=INFO source=memory.go:326 msg="offload to cuda" layers.requested=-1 layers.model=61 layers.offload=10 layers.split="" memory.available="[23.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="134.5 GiB" memory.required.partial="22.1 GiB" memory.required.kv="9.4 GiB" memory.required.allocations="[22.1 GiB]" memory.weights.total="132.5 GiB" memory.weights.repeating="132.1 GiB" memory.weights.nonrepeating="410.2 MiB" memory.graph.full="642.0 MiB" memory.graph.partial="891.5 MiB" Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.911+10:00 level=INFO source=server.go:388 msg="starting llama server" cmd="/tmp/ollama1470999858/runners/cuda_v12/ollama_llama_server --model /mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 10 --no-mmap --parallel 1 --port 43133" Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.911+10:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.911+10:00 level=INFO source=server.go:587 msg="waiting for llama runner to start responding" Oct 02 09:22:03 xlr2 ollama[2576620]: time=2024-10-02T09:22:03.912+10:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error" Oct 02 09:22:03 xlr2 ollama[3244519]: INFO [main] build info | build=10 commit="9225b05" tid="139874777862144" timestamp=1727824923 Oct 02 09:22:03 xlr2 ollama[3244519]: INFO [main] system info | n_threads=8 n_threads_batch=8 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="139874777862144" timestamp=1727824923 total_threads=16 Oct 02 09:22:03 xlr2 ollama[3244519]: INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="15" port="43133" tid="139874777862144" timestamp=1727824923 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: loaded meta data with 46 key-value pairs and 959 tensors from /mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c (version GGUF V3 (latest)) Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 0: general.architecture str = deepseek2 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 1: general.type str = model Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 2: general.name str = DeepSeek V2.5 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 3: general.version str = V2.5 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 4: general.basename str = DeepSeek Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 5: general.size_label str = 160x14B Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 6: general.license str = other Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 7: general.license.name str = deepseek Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 8: general.license.link str = https://github.com/deepseek-ai/DeepSe... Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 9: deepseek2.block_count u32 = 60 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 10: deepseek2.context_length u32 = 163840 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 11: deepseek2.embedding_length u32 = 5120 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 12: deepseek2.feed_forward_length u32 = 12288 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 13: deepseek2.attention.head_count u32 = 128 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 14: deepseek2.attention.head_count_kv u32 = 128 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 15: deepseek2.rope.freq_base f32 = 10000.000000 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 16: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 17: deepseek2.expert_used_count u32 = 6 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 18: general.file_type u32 = 2 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 19: deepseek2.leading_dense_block_count u32 = 1 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 20: deepseek2.vocab_size u32 = 102400 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 21: deepseek2.attention.q_lora_rank u32 = 1536 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 22: deepseek2.attention.kv_lora_rank u32 = 512 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 23: deepseek2.attention.key_length u32 = 192 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 24: deepseek2.attention.value_length u32 = 128 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 25: deepseek2.expert_feed_forward_length u32 = 1536 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 26: deepseek2.expert_count u32 = 160 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 27: deepseek2.expert_shared_count u32 = 2 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 28: deepseek2.expert_weights_scale f32 = 16.000000 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 29: deepseek2.rope.dimension_count u32 = 64 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 30: deepseek2.rope.scaling.type str = yarn Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 31: deepseek2.rope.scaling.factor f32 = 40.000000 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 32: deepseek2.rope.scaling.original_context_length u32 = 4096 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 33: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 34: tokenizer.ggml.model str = gpt2 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 35: tokenizer.ggml.pre str = deepseek-llm Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 36: tokenizer.ggml.tokens arr[str,102400] = ["!", "\"", "#", "$", "%", "&", "'", ... Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 37: tokenizer.ggml.token_type arr[i32,102400] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 38: tokenizer.ggml.merges arr[str,99757] = ["G G", "G t", "G a", "i n", "h e... Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 39: tokenizer.ggml.bos_token_id u32 = 100000 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 40: tokenizer.ggml.eos_token_id u32 = 100001 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 41: tokenizer.ggml.padding_token_id u32 = 100001 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 42: tokenizer.ggml.add_bos_token bool = true Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 43: tokenizer.ggml.add_eos_token bool = false Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 44: tokenizer.chat_template str = {% if not add_generation_prompt is de... Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - kv 45: general.quantization_version u32 = 2 Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - type f32: 300 tensors Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - type q4_0: 658 tensors Oct 02 09:22:03 xlr2 ollama[2576620]: llama_model_loader: - type q6_K: 1 tensors Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_vocab: special tokens cache size = 18 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_vocab: token to piece cache size = 0.6411 MB Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: format = GGUF V3 (latest) Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: arch = deepseek2 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: vocab type = BPE Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_vocab = 102400 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_merges = 99757 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: vocab_only = 0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_ctx_train = 163840 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_embd = 5120 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_layer = 60 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_head = 128 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_head_kv = 128 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_rot = 64 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_swa = 0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_embd_head_k = 192 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_embd_head_v = 128 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_gqa = 1 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_embd_k_gqa = 24576 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_embd_v_gqa = 16384 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: f_norm_eps = 0.0e+00 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: f_logit_scale = 0.0e+00 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_ff = 12288 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_expert = 160 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_expert_used = 6 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: causal attn = 1 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: pooling type = 0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: rope type = 0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: rope scaling = yarn Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: freq_base_train = 10000.0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: freq_scale_train = 0.025 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_ctx_orig_yarn = 4096 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: rope_finetuned = unknown Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: ssm_d_conv = 0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: ssm_d_inner = 0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: ssm_d_state = 0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: ssm_dt_rank = 0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: ssm_dt_b_c_rms = 0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: model type = 236B Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: model ftype = Q4_0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: model params = 235.74 B Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: model size = 123.78 GiB (4.51 BPW) Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: general.name = DeepSeek V2.5 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: BOS token = 100000 '<|begin?of?sentence|>' Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: EOS token = 100001 '<|end?of?sentence|>' Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: PAD token = 100001 '<|end?of?sentence|>' Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: LF token = 126 'Ä' Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: max token length = 256 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_layer_dense_lead = 1 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_lora_q = 1536 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_lora_kv = 512 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_ff_exp = 1536 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: n_expert_shared = 2 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: expert_weights_scale = 16.0 Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_print_meta: rope_yarn_log_mul = 0.1000 Oct 02 09:22:04 xlr2 ollama[2576620]: time=2024-10-02T09:22:04.163+10:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model" Oct 02 09:22:04 xlr2 ollama[2576620]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Oct 02 09:22:04 xlr2 ollama[2576620]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Oct 02 09:22:04 xlr2 ollama[2576620]: ggml_cuda_init: found 1 CUDA devices: Oct 02 09:22:04 xlr2 ollama[2576620]: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Oct 02 09:22:04 xlr2 ollama[2576620]: llm_load_tensors: ggml ctx size = 0.80 MiB Oct 02 09:22:05 xlr2 ollama[2576620]: time=2024-10-02T09:22:05.618+10:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server not responding" Oct 02 09:22:33 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost Oct 02 09:22:37 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost Oct 02 09:22:47 xlr2 ollama[2576620]: time=2024-10-02T09:22:47.399+10:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server loading model" Oct 02 09:22:51 xlr2 ollama[2576620]: llm_load_tensors: offloading 10 repeating layers to GPU Oct 02 09:22:51 xlr2 ollama[2576620]: llm_load_tensors: offloaded 10/61 layers to GPU Oct 02 09:22:51 xlr2 ollama[2576620]: llm_load_tensors: CUDA_Host buffer size = 105416.00 MiB Oct 02 09:22:51 xlr2 ollama[2576620]: llm_load_tensors: CUDA0 buffer size = 21335.35 MiB Oct 02 09:23:10 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost Oct 02 09:23:17 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: n_ctx = 2048 Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: n_batch = 512 Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: n_ubatch = 512 Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: flash_attn = 0 Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: freq_base = 10000.0 Oct 02 09:23:44 xlr2 ollama[2576620]: llama_new_context_with_model: freq_scale = 0.025 Oct 02 09:23:48 xlr2 ollama[2576620]: llama_kv_cache_init: CUDA_Host KV buffer size = 8000.00 MiB Oct 02 09:23:48 xlr2 ollama[2576620]: llama_kv_cache_init: CUDA0 KV buffer size = 1600.00 MiB Oct 02 09:23:48 xlr2 ollama[2576620]: llama_new_context_with_model: KV self size = 9600.00 MiB, K (f16): 5760.00 MiB, V (f16): 3840.00 MiB Oct 02 09:23:48 xlr2 ollama[2576620]: llama_new_context_with_model: CUDA_Host output buffer size = 0.41 MiB Oct 02 09:23:48 xlr2 ollama[2576620]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 842.00 MiB on device 0: cudaMalloc failed: out of memory Oct 02 09:23:48 xlr2 ollama[2576620]: ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 882903040 Oct 02 09:23:48 xlr2 ollama[2576620]: llama_new_context_with_model: failed to allocate compute buffers Oct 02 09:23:50 xlr2 ollama[2576620]: llama_init_from_gpt_params: error: failed to create context with model '/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c' Oct 02 09:23:52 xlr2 kernel: clocksource: Long readout interval, skipping watchdog check: cs_nsec: 1586308203 wd_nsec: 1586308525 Oct 02 09:24:05 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost Oct 02 09:24:09 xlr2 systemd-networkd[3394145]: enp7s0: DHCPv6 lease lost Oct 02 09:24:10 xlr2 ollama[3244519]: ERROR [load_model] unable to load model | model="/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c" tid="139874777862144" timestamp=1727825050 Oct 02 09:24:10 xlr2 ollama[2576620]: terminate called without an active exception Oct 02 09:24:10 xlr2 ollama[2576620]: time=2024-10-02T09:24:10.958+10:00 level=INFO source=server.go:621 msg="waiting for server to become available" status="llm server error" Oct 02 09:24:11 xlr2 ollama[2576620]: time=2024-10-02T09:24:11.213+10:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error:failed to create context with model '/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c'" Oct 02 09:24:11 xlr2 ollama[2576620]: [GIN] 2024/10/02 - 09:24:11 | 500 | 2m8s | 127.0.0.1 | POST "/api/generate" Oct 02 09:24:16 xlr2 ollama[2576620]: time=2024-10-02T09:24:16.336+10:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.122496579 model=/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c Oct 02 09:24:16 xlr2 ollama[2576620]: time=2024-10-02T09:24:16.586+10:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.372482202 model=/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c Oct 02 09:24:16 xlr2 ollama[2576620]: time=2024-10-02T09:24:16.837+10:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.623037141 model=/mnt/ssd/ai/ollama/.ollama/models/blobs/sha256-799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c ``` System info: ``` $ uname -a Linux xlr2 5.15.0-112-generic #122-Ubuntu SMP Thu May 23 07:48:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux $ nproc 16 $ free -h total used free shared buff/cache available Mem: 125Gi 5.3Gi 113Gi 33Mi 6.5Gi 119Gi Swap: 8.0Gi 887Mi 7.1Gi $ nvidia-smi Wed Oct 2 10:04:06 2024 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.161.08 Driver Version: 535.161.08 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce RTX 3090 On | 00000000:0C:00.0 Off | N/A | | 0% 33C P8 38W / 390W | 3MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | No running processes found | +---------------------------------------------------------------------------------------+ $ ``` I tried with both `0.3.11` and the latest `0.3.12` versions of Ollama. Here is a snapshot of `htop` and `nvtop` outputs: <img width="820" alt="htop_nvtop" src="https://github.com/user-attachments/assets/e21266eb-1d3b-4ab6-9324-5604fc99efb4"> Here is the recording showing the entire process: https://github.com/user-attachments/assets/e306990a-a340-4a8f-b7de-b52d9b3de5e1 ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.3.11

GiteaMirror added the bug label 2026-04-28 17:46:41 -05:00

GiteaMirror closed this issue

2026-04-28 17:46:43 -05:00

GiteaMirror commented

2026-04-28 17:46:45 -05:00

@rick-github commented on GitHub (Oct 2, 2024):

Oct 02 09:23:48 xlr2 ollama[2576620]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 842.00 MiB on device 0: cudaMalloc failed: out of memory
Oct 02 09:23:48 xlr2 ollama[2576620]: ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 882903040
Oct 02 09:23:48 xlr2 ollama[2576620]: llama_new_context_with_model: failed to allocate compute buffers

As background: in managing models, ollama takes a best-guess at figuring out how much of the model will reside in GPU VRAM and how much will reside in system RAM. Based on that, it tells the backend, llama.cpp, how many layers to load in VRAM. However, it's up to llama.cpp to actually allocate the memory. The deepseek class of models is a slightly non-standard architecture that sometimes results in disagreements between ollama and llama.cpp about how memory is actually needed. I think that is what is happening here.

You can mitigate this by telling ollama to load fewer layers into VRAM. ollama is currently trying for 10 layers:

Oct 02 09:22:51 xlr2 ollama[2576620]: llm_load_tensors: offloading 10 repeating layers to GPU

so you could try lowering that to 9 to see if you get more success. See here for ways to reduce num_gpu, just replace 0 with 9 (or less if 9 doesn't help).

As mentioned, deepseek's alternate architecture has seen it be a bit flaky and a workaround has been developed. I'm not sure if deepseek-v2.5 needs something similar but thought I'd mention it just in case.

@rick-github commented on GitHub (Oct 2, 2024): ``` Oct 02 09:23:48 xlr2 ollama[2576620]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 842.00 MiB on device 0: cudaMalloc failed: out of memory Oct 02 09:23:48 xlr2 ollama[2576620]: ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 882903040 Oct 02 09:23:48 xlr2 ollama[2576620]: llama_new_context_with_model: failed to allocate compute buffers ``` As background: in managing models, ollama takes a best-guess at figuring out how much of the model will reside in GPU VRAM and how much will reside in system RAM. Based on that, it tells the backend, llama.cpp, how many layers to load in VRAM. However, it's up to llama.cpp to actually allocate the memory. The deepseek class of models is a slightly non-standard architecture that sometimes results in disagreements between ollama and llama.cpp about how memory is actually needed. I think that is what is happening here. You can mitigate this by telling ollama to load fewer layers into VRAM. ollama is currently trying for 10 layers: ``` Oct 02 09:22:51 xlr2 ollama[2576620]: llm_load_tensors: offloading 10 repeating layers to GPU ``` so you could try lowering that to 9 to see if you get more success. See [here](https://github.com/ollama/ollama/issues/6950#issuecomment-2373663650) for ways to reduce `num_gpu`, just replace 0 with 9 (or less if 9 doesn't help). As mentioned, deepseek's alternate architecture has seen it be a bit flaky and a [workaround](https://github.com/ollama/ollama/issues/6199#issuecomment-2295952982) has been developed. I'm not sure if deepseek-v2.5 needs something similar but thought I'd mention it just in case.

GiteaMirror commented

2026-04-28 17:46:47 -05:00

@LeonidShamis commented on GitHub (Oct 2, 2024):

@rick-github Thank you very much for your prompt response and advice - it helped me resolve the issue!

I followed your advice to modify the model through a Modelfile and I can successfully run it at 2.7 - 3.6 tokens/s.

Here is how I did it if someone else wants to try it as well:

/mnt/ssd/ai/ollama$ ollama show --modelfile deepseek-v2.5:latest > deepseek-v2.5:latest.Modelfile
/mnt/ssd/ai/ollama$ cp deepseek-v2.5\:latest.Modelfile deepseek-v2.5\:modified.Modelfile
/mnt/ssd/ai/ollama$ nano deepseek-v2.5\:modified.Modelfile
/mnt/ssd/ai/ollama$ diff -y --suppress-common-lines deepseek-v2.5\:latest.Modelfile deepseek-v2.5\:modified.Modelfile
                                                              > PARAMETER num_gpu 9
/mnt/ssd/ai/ollama$
/mnt/ssd/ai/ollama$ ollama create deepseek-v2.5:modified -f deepseek-v2.5\:modified.Modelfile
transferring model data 100%
using existing layer sha256:799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c
using existing layer sha256:8aa4c0321ccdc9d51ffd07ae148be00bc344371869685a11d7724a90761de423
using existing layer sha256:ccfee4895df06bcab524151c278e8dde88bbe76165a24ecbcbcf9fafd71fd2b3
creating new layer sha256:7ad969b93b0426904196c773d59e7a4cabdbe8411099c8d06f1966846286823e
creating new layer sha256:b204ce5f22655b608215fec041bdb7490fd4a7e00bf01a2810ddf11eaa971f1b
writing manifest
success
/mnt/ssd/ai/ollama$ ollama list | grep -e ID -e deepseek-v2.5
NAME                             ID              SIZE      MODIFIED
deepseek-v2.5:modified           1d0a40127905    132 GB    10 minutes ago
deepseek-v2.5:latest             409b2dd8a3c4    132 GB    15 hours ago
/mnt/ssd/ai/ollama$

Run the modified model:

$ ollama run deepseek-v2.5:modified
>>> /set verbose
Set 'verbose' mode.
>>> who created you?
I am an intelligent assistant developed by DeepSeek, a company in China.

total duration:       4.744183093s
load duration:        10.956836ms
prompt eval count:    7 token(s)
prompt eval duration: 344.168ms
prompt eval rate:     20.34 tokens/s
eval count:           16 token(s)
eval duration:        4.344258s
eval rate:            3.68 tokens/s
>>> Write 10 sentences that all end with the word 'apple'
1. She packed her lunch and included a crunchy apple.
2. After school, he stopped by the market to buy a fresh apple.
3. The teacher rewarded him for good behavior with an extra sticker on his chart next to the picture of an apple.
4. In the orchard, they picked the ripest fruit, including a juicy red apple.
5. On her way home, she decided to grab a quick snack and chose an organic apple.
6. The recipe called for one peeled and diced apple.
7. During their picnic, everyone enjoyed their sandwiches and shared slices of tart green apple.
8. He reached into the fruit bowl and picked out his favorite golden delicious apple.
9. She added cinnamon to her oatmeal and topped it with a sweet honeycrisp apple.
10. After dinner, they sat on the porch and each had a crisp Granny Smith apple.

total duration:       1m12.902265539s
load duration:        10.773516ms
prompt eval count:    40 token(s)
prompt eval duration: 3.056361s
prompt eval rate:     13.09 tokens/s
eval count:           188 token(s)
eval duration:        1m9.70291s
eval rate:            2.70 tokens/s
>>>
~$

Now I can take it for a spin with Aider - it should be a great code assistant model, albeit I expect it to be slow compared to a smaller model.

@LeonidShamis commented on GitHub (Oct 2, 2024): @rick-github Thank you very much for your prompt response and advice - it helped me resolve the issue! I followed your advice to modify the model through a _Modelfile_ and I can successfully run it at 2.7 - 3.6 tokens/s. Here is how I did it if someone else wants to try it as well: ``` /mnt/ssd/ai/ollama$ ollama show --modelfile deepseek-v2.5:latest > deepseek-v2.5:latest.Modelfile /mnt/ssd/ai/ollama$ cp deepseek-v2.5\:latest.Modelfile deepseek-v2.5\:modified.Modelfile /mnt/ssd/ai/ollama$ nano deepseek-v2.5\:modified.Modelfile /mnt/ssd/ai/ollama$ diff -y --suppress-common-lines deepseek-v2.5\:latest.Modelfile deepseek-v2.5\:modified.Modelfile > PARAMETER num_gpu 9 /mnt/ssd/ai/ollama$ /mnt/ssd/ai/ollama$ ollama create deepseek-v2.5:modified -f deepseek-v2.5\:modified.Modelfile transferring model data 100% using existing layer sha256:799587243b19fdcc715a4aab927f5700d1b9508bd0b8b0db9dc2bd6fc622979c using existing layer sha256:8aa4c0321ccdc9d51ffd07ae148be00bc344371869685a11d7724a90761de423 using existing layer sha256:ccfee4895df06bcab524151c278e8dde88bbe76165a24ecbcbcf9fafd71fd2b3 creating new layer sha256:7ad969b93b0426904196c773d59e7a4cabdbe8411099c8d06f1966846286823e creating new layer sha256:b204ce5f22655b608215fec041bdb7490fd4a7e00bf01a2810ddf11eaa971f1b writing manifest success /mnt/ssd/ai/ollama$ ollama list | grep -e ID -e deepseek-v2.5 NAME ID SIZE MODIFIED deepseek-v2.5:modified 1d0a40127905 132 GB 10 minutes ago deepseek-v2.5:latest 409b2dd8a3c4 132 GB 15 hours ago /mnt/ssd/ai/ollama$ ``` Run the modified model: ``` $ ollama run deepseek-v2.5:modified >>> /set verbose Set 'verbose' mode. >>> who created you? I am an intelligent assistant developed by DeepSeek, a company in China. total duration: 4.744183093s load duration: 10.956836ms prompt eval count: 7 token(s) prompt eval duration: 344.168ms prompt eval rate: 20.34 tokens/s eval count: 16 token(s) eval duration: 4.344258s eval rate: 3.68 tokens/s >>> Write 10 sentences that all end with the word 'apple' 1. She packed her lunch and included a crunchy apple. 2. After school, he stopped by the market to buy a fresh apple. 3. The teacher rewarded him for good behavior with an extra sticker on his chart next to the picture of an apple. 4. In the orchard, they picked the ripest fruit, including a juicy red apple. 5. On her way home, she decided to grab a quick snack and chose an organic apple. 6. The recipe called for one peeled and diced apple. 7. During their picnic, everyone enjoyed their sandwiches and shared slices of tart green apple. 8. He reached into the fruit bowl and picked out his favorite golden delicious apple. 9. She added cinnamon to her oatmeal and topped it with a sweet honeycrisp apple. 10. After dinner, they sat on the porch and each had a crisp Granny Smith apple. total duration: 1m12.902265539s load duration: 10.773516ms prompt eval count: 40 token(s) prompt eval duration: 3.056361s prompt eval rate: 13.09 tokens/s eval count: 188 token(s) eval duration: 1m9.70291s eval rate: 2.70 tokens/s >>> ~$ ``` Now I can take it for a spin with [Aider](https://aider.chat/docs/leaderboards/) - it should be a great code assistant model, albeit I expect it to be slow compared to a smaller model.

GiteaMirror commented

2026-04-28 17:46:50 -05:00

@tripled-yang commented on GitHub (Oct 22, 2024):

It is cool, bro thanks @LeonidShamis
by the way, Would you be willing to provide the model of the CPU for reference
I am very grateful.
Because I have a dual-path Epyc 7002, the memory bandwidth is approximately 300GB+,
and want to calculate the assistance that CPU+GPU provides for the MOE model.

here is only CPU

who create you
I am an intelligent assistant developed by the Chinese company DeepSeek. My existence is designed to provide information queries, answer questions, and
engage in conversational exchanges to assist users with their needs. If you have any questions or require help, I'm here to serve you! 😊

total duration: 23.582665701s
load duration: 19.9849ms
prompt eval count: 1521 token(s)
prompt eval duration: 1.33116s
prompt eval rate: 1142.61 tokens/s
eval count: 57 token(s)
eval duration: 22.038438s
eval rate: 2.59 tokens/s

Write 10 sentences that all end with the word 'apple'

She bought a shiny red apple.
The teacher asked for an example using an apple.
He reached into his bag and pulled out an apple.
The pie smelled delicious, filled with chunks of apple.
On her way home, she picked up a crisp green apple.
They baked together, their hands covered in flour and apples.
After school, he always looked forward to eating his favorite snack—an apple.
The orchard was full of ripe, ready-to-pick apples.
She carefully sliced the apple into perfect pieces for her guests.
Every morning, she started her day with a fresh, juicy apple.

total duration: 1m3.078166287s
load duration: 20.792885ms
prompt eval count: 1595 token(s)
prompt eval duration: 2.919833s
prompt eval rate: 546.26 tokens/s
eval count: 151 token(s)
eval duration: 59.887639s
eval rate: 2.52 tokens/s

@tripled-yang commented on GitHub (Oct 22, 2024): It is cool, bro thanks @LeonidShamis by the way, Would you be willing to provide the model of the CPU for reference I am very grateful. Because I have a dual-path Epyc 7002, the memory bandwidth is approximately 300GB+, and want to calculate the assistance that CPU+GPU provides for the MOE model. here is only CPU >>> who create you I am an intelligent assistant developed by the Chinese company DeepSeek. My existence is designed to provide information queries, answer questions, and engage in conversational exchanges to assist users with their needs. If you have any questions or require help, I'm here to serve you! 😊 total duration: 23.582665701s load duration: 19.9849ms prompt eval count: 1521 token(s) prompt eval duration: 1.33116s prompt eval rate: 1142.61 tokens/s eval count: 57 token(s) eval duration: 22.038438s eval rate: 2.59 tokens/s >>> Write 10 sentences that all end with the word 'apple' 1. She bought a shiny red apple. 2. The teacher asked for an example using an apple. 3. He reached into his bag and pulled out an apple. 4. The pie smelled delicious, filled with chunks of apple. 5. On her way home, she picked up a crisp green apple. 6. They baked together, their hands covered in flour and apples. 7. After school, he always looked forward to eating his favorite snack—an apple. 8. The orchard was full of ripe, ready-to-pick apples. 9. She carefully sliced the apple into perfect pieces for her guests. 10. Every morning, she started her day with a fresh, juicy apple. total duration: 1m3.078166287s load duration: 20.792885ms prompt eval count: 1595 token(s) prompt eval duration: 2.919833s prompt eval rate: 546.26 tokens/s eval count: 151 token(s) eval duration: 59.887639s eval rate: 2.52 tokens/s

GiteaMirror commented

2026-04-28 17:46:54 -05:00

@LeonidShamis commented on GitHub (Oct 27, 2024):

@zeliang3

model of the CPU

$ cat /proc/cpuinfo | grep 'model name' | uniq
model name      : AMD Ryzen 7 5800X 8-Core Processor
$

@LeonidShamis commented on GitHub (Oct 27, 2024): @zeliang3 > model of the CPU ``` $ cat /proc/cpuinfo | grep 'model name' | uniq model name : AMD Ryzen 7 5800X 8-Core Processor $ ```

Sign in to join this conversation.

Branches Tags

main

hoyyeva/fix-claude-channels-env

parth-update-hermes-launch

hoyyeva/vscode-extension-docs-update

parth-gemma4-chat-template-renderer

parth-api-status-context-length

hoyyeva/wire-up-context-length

hoyyeva/claude-code-context-doc

jmorganca/investigate-issue-17046

hoyyeva/hermes-docs

jmorganca/agent-loop-style

hoyyeva/openclaw

parth-agent-loop

hoyyeva/ollama-vscode-extension

brucemacd/cache-metrics

brucemacd/hermes-desktop

hoyyeva/docs-vscode

parth-input-style-experiment

brucemacd/docs-glm52

hoyyeva/poc-docs

Parth/mlx-launch-recommendations

parth-first-time-app-cli-experience

test/darwin-xcode-pin

improve-cloud-model-recommendations

hoyyeva/goose-docs

jmorganca/context-limit-fixes

hoyyeva/qwen-doc

hoyyeva/vscode-docs

jmorganca/remove-mlx-imagegen-code

parth-copilot-token-length-defaults

hoyyeva/poolside-windows

laguna-support

jmorganca/harden-markdown-rendering

laguna-renderer-parser

laguna-llamacpp

codex/make-integration-hidden-and-lunchable

brucemacd/omp-docs

pdevine/gguf-mtp-oldstyle

hoyyeva/migrate-pi

hoyyeva/anthropic-local-image-path

parth-launch-codex-app

hoyyeva/anthropic-reference-images-path

parth-anthropic-reference-images-path

brucemacd/download-before-remove

hoyyeva/editor-config-repair

parth-mlx-decode-checkpoints

parth/hide-claude-desktop-till-release

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

parth/update-claude-docs

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#51000