github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #11073] Out of memory on AMD multi GPU instance despite having sufficient VRAM #69364

New Issue
Open
opened 2026-05-04 17:54:59 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-05-04 17:54:59 -05:00
Owner
Copy Link

Originally created by @digiperfect-tech on GitHub (Jun 14, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11073

Originally assigned to: @jessegross on GitHub.

What is the issue?

Ollama throws out of memory error despite there being sufficient VRAM. Model size is approx 160GB so in principle should fit even on a single GPU.

amd-smi  monitor
GPU  POWER   GPU_T   MEM_T   GFX_CLK   GFX%   MEM%   ENC%   DEC%      VRAM_USAGE
  0  151 W   39 °C   32 °C   170 MHz    0 %    0 %    N/A    0 %    0.3/191.7 GB
  1  152 W   35 °C   30 °C   199 MHz    0 %    0 %    N/A    0 %    0.3/191.7 GB
  2  151 W   37 °C   30 °C   161 MHz    0 %    0 %    N/A    0 %    0.3/191.7 GB
  3  151 W   37 °C   32 °C   175 MHz    0 %    0 %    N/A    0 %    0.3/191.7 GB
  4  154 W   39 °C   32 °C   163 MHz    0 %    0 %    N/A    0 %    0.3/191.7 GB
  5  153 W   36 °C   30 °C   171 MHz    0 %    0 %    N/A    0 %    0.3/191.7 GB
  6  152 W   40 °C   32 °C   188 MHz    0 %    0 %    N/A    0 %    0.3/191.7 GB
  7  156 W   36 °C   32 °C   191 MHz    0 %    0 %    N/A    0 %    0.3/191.7 GB

Even tried with OLLAMA_SCHED_SPREAD=1; ollama run ....

Relevant log output

Error: llama runner process has terminated: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 35041316864

OS

Linux

GPU

AMD

CPU

No response

Ollama version

0.9.0

Originally created by @digiperfect-tech on GitHub (Jun 14, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11073 Originally assigned to: @jessegross on GitHub. ### What is the issue? Ollama throws out of memory error despite there being sufficient VRAM. Model size is approx 160GB so in principle should fit even on a single GPU. ``` amd-smi monitor GPU POWER GPU_T MEM_T GFX_CLK GFX% MEM% ENC% DEC% VRAM_USAGE 0 151 W 39 °C 32 °C 170 MHz 0 % 0 % N/A 0 % 0.3/191.7 GB 1 152 W 35 °C 30 °C 199 MHz 0 % 0 % N/A 0 % 0.3/191.7 GB 2 151 W 37 °C 30 °C 161 MHz 0 % 0 % N/A 0 % 0.3/191.7 GB 3 151 W 37 °C 32 °C 175 MHz 0 % 0 % N/A 0 % 0.3/191.7 GB 4 154 W 39 °C 32 °C 163 MHz 0 % 0 % N/A 0 % 0.3/191.7 GB 5 153 W 36 °C 30 °C 171 MHz 0 % 0 % N/A 0 % 0.3/191.7 GB 6 152 W 40 °C 32 °C 188 MHz 0 % 0 % N/A 0 % 0.3/191.7 GB 7 156 W 36 °C 32 °C 191 MHz 0 % 0 % N/A 0 % 0.3/191.7 GB ``` Even tried with `OLLAMA_SCHED_SPREAD=1; ollama run ....` ### Relevant log output ```shell Error: llama runner process has terminated: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 35041316864 ``` ### OS Linux ### GPU AMD ### CPU _No response_ ### Ollama version 0.9.0
GiteaMirror added the bug label 2026-05-04 17:54:59 -05:00
GiteaMirror commented 2026-05-04 17:55:03 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 14, 2025):

Server logs will aid in debugging.

<!-- gh-comment-id:2972476550 --> @rick-github commented on GitHub (Jun 14, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will aid in debugging.
GiteaMirror commented 2026-05-04 17:55:07 -05:00
Author
Owner
Copy Link

@digiperfect-tech commented on GitHub (Jun 15, 2025):

For simplicitly - I am trying a small model on single GPU instance - ecountering same error.

Model size: 161 GB
Model: hf.co/unsloth/DeepSeek-R1-0528-GGUF:latest


GPU VRAM: 192 GB
System RAM: 235 GB

Incidentally - following MOE model works fine, the total model size is slightly smaller than the model that fails. But IMO - above model should work seamlessly given the sizes and extra system RAM available?

Model that works:
qwen3:235b-a22b 142GB

Server logs (the log file was not present inside ~/.ollama.... but journalctl gave the logs).

Jun 15 08:09:58 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 systemd[1]: ollama.service: Deactivated successfully.
Jun 15 08:09:58 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 systemd[1]: ollama.service: Consumed 14.296s CPU time.
Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 systemd[1]: ollama.service: Scheduled restart job, restart counter is at 2.
Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 systemd[1]: Started ollama.service - Ollama Service.
Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.110Z level=INFO source=routes.go:1234 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.110Z level=INFO source=images.go:479 msg="total blobs: 21"
Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.110Z level=INFO source=images.go:486 msg="total unused blobs removed: 0"
Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.111Z level=INFO source=routes.go:1287 msg="Listening on 127.0.0.1:11434 (version 0.9.0)"
Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.111Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.117Z level=INFO source=amd_linux.go:386 msg="amdgpu is supported" gpu=GPU-7524e56f4b0facdf gpu_type=gfx942
Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.119Z level=INFO source=types.go:130 msg="inference compute" id=GPU-7524e56f4b0facdf library=rocm variant="" compute=gfx942 driver=6.12 name=1002:74b5 total="191.7 GiB" available="191.4 GiB"
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: [GIN] 2025/06/15 - 08:10:25 | 200 |      36.657µs |       127.0.0.1 | HEAD     "/"
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: [GIN] 2025/06/15 - 08:10:25 | 200 |   19.514005ms |       127.0.0.1 | POST     "/api/show"
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.797Z level=INFO source=sched.go:788 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44 gpu=GPU-7524e56f4b0facdf parallel=2 available=205523197952 required="167.8 GiB"
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.797Z level=INFO source=server.go:135 msg="system memory" total="235.9 GiB" free="230.2 GiB" free_swap="0 B"
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.798Z level=INFO source=server.go:168 msg=offload library=rocm layers.requested=-1 layers.model=62 layers.offload=62 layers.split="" memory.available="[191.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="167.8 GiB" memory.required.partial="167.8 GiB" memory.required.kv="16.2 GiB" memory.required.allocations="[167.8 GiB]" memory.weights.total="150.0 GiB" memory.weights.repeating="149.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="556.3 MiB" memory.graph.partial="1019.5 MiB"
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: loaded meta data with 60 key-value pairs and 1086 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44 (version GGUF V3 (latest))
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   1:                               general.type str              = model
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   2:                               general.name str              = Deepseek-R1-0528
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   3:                           general.basename str              = Deepseek-R1-0528
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   4:                       general.quantized_by str              = Unsloth
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   5:                         general.size_label str              = 256x20B
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   6:                            general.license str              = mit
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   7:                           general.repo_url str              = https://huggingface.co/unsloth
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   9:                  general.base_model.0.name str              = DeepSeek R1 0528
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  10:               general.base_model.0.version str              = 0528
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  11:          general.base_model.0.organization str              = Deepseek Ai
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  12:              general.base_model.0.repo_url str              = https://huggingface.co/deepseek-ai/De...
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  13:                               general.tags arr[str,3]       = ["deepseek", "unsloth", "transformers"]
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  14:                          general.languages arr[str,1]       = ["en"]
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  15:                      deepseek2.block_count u32              = 61
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  16:                   deepseek2.context_length u32              = 163840
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  17:                 deepseek2.embedding_length u32              = 7168
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  18:              deepseek2.feed_forward_length u32              = 18432
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  19:             deepseek2.attention.head_count u32              = 128
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  20:          deepseek2.attention.head_count_kv u32              = 1
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  21:                   deepseek2.rope.freq_base f32              = 10000.000000
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  22: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  23:                deepseek2.expert_used_count u32              = 8
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  24:        deepseek2.leading_dense_block_count u32              = 3
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  25:                       deepseek2.vocab_size u32              = 129280
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  26:            deepseek2.attention.q_lora_rank u32              = 1536
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  27:           deepseek2.attention.kv_lora_rank u32              = 512
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  28:             deepseek2.attention.key_length u32              = 576
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  29:           deepseek2.attention.value_length u32              = 512
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  30:         deepseek2.attention.key_length_mla u32              = 192
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  31:       deepseek2.attention.value_length_mla u32              = 128
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  32:       deepseek2.expert_feed_forward_length u32              = 2048
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  33:                     deepseek2.expert_count u32              = 256
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  34:              deepseek2.expert_shared_count u32              = 1
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  35:             deepseek2.expert_weights_scale f32              = 2.500000
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  36:              deepseek2.expert_weights_norm bool             = true
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  37:               deepseek2.expert_gating_func u32              = 2
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  38:             deepseek2.rope.dimension_count u32              = 64
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  39:                deepseek2.rope.scaling.type str              = yarn
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  40:              deepseek2.rope.scaling.factor f32              = 40.000000
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  41: deepseek2.rope.scaling.original_context_length u32              = 4096
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  42: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  43:                       tokenizer.ggml.model str              = gpt2
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  44:                         tokenizer.ggml.pre str              = deepseek-v3
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: [132B blob data]
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  46:                  tokenizer.ggml.token_type arr[i32,129280]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  47:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  48:                tokenizer.ggml.bos_token_id u32              = 0
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  49:                tokenizer.ggml.eos_token_id u32              = 1
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  50:            tokenizer.ggml.padding_token_id u32              = 2
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  51:               tokenizer.ggml.add_bos_token bool             = true
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  52:               tokenizer.ggml.add_eos_token bool             = false
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  53:                    tokenizer.chat_template str              = {%- if not add_generation_prompt is d...
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  54:               general.quantization_version u32              = 2
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  55:                          general.file_type u32              = 24
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  56:                      quantize.imatrix.file str              = DeepSeek-R1-0528-GGUF/imatrix_unsloth...
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  57:                   quantize.imatrix.dataset str              = unsloth_calibration_DeepSeek-R1-0528-...
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  58:             quantize.imatrix.entries_count i32              = 659
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  59:              quantize.imatrix.chunks_count i32              = 720
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type  f32:  361 tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q8_0:  122 tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q4_K:   56 tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q5_K:   36 tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q6_K:   17 tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq2_xxs:   30 tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq3_xxs:   54 tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq1_s:  126 tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq3_s:  148 tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq4_xs:  136 tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file format = GGUF V3 (latest)
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file type   = IQ1_S - 1.5625 bpw
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file size   = 150.51 GiB (1.93 BPW)
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: special tokens cache size = 818
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: token to piece cache size = 0.8223 MB
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: arch             = deepseek2
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: vocab_only       = 1
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: model type       = ?B
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: model params     = 671.03 B
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: general.name     = Deepseek-R1-0528
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_layer_dense_lead   = 0
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_lora_q             = 0
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_lora_kv            = 0
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_k_mla    = 0
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_v_mla    = 0
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_ff_exp             = 0
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_expert_shared      = 0
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_weights_scale = 0.0
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_weights_norm  = 0
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_gating_func   = unknown
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: rope_yarn_log_mul    = 0.0000
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: vocab type       = BPE
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_vocab          = 129280
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_merges         = 127741
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: BOS token        = 0 '<｜begin▁of▁sentence｜>'
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOS token        = 1 '<｜end▁of▁sentence｜>'
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOT token        = 1 '<｜end▁of▁sentence｜>'
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: PAD token        = 2 '<｜▁pad▁｜>'
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: LF token         = 201 'Ċ'
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM PRE token    = 128801 '<｜fim▁begin｜>'
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM SUF token    = 128800 '<｜fim▁hole｜>'
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM MID token    = 128802 '<｜fim▁end｜>'
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOG token        = 1 '<｜end▁of▁sentence｜>'
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: max token length = 256
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_load: vocab only - skipping tensors
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.967Z level=INFO source=server.go:431 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44 --ctx-size 131072 --batch-size 512 --n-gpu-layers 62 --threads 20 --parallel 2 --port 40669"
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.968Z level=INFO source=sched.go:483 msg="loaded runners" count=1
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.968Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.968Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding"
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.976Z level=INFO source=runner.go:815 msg="starting go runner"
Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-icelake.so
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_cuda_init: found 1 ROCm devices:
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]:   Device 0: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_backend: loaded ROCm backend from /usr/local/lib/ollama/rocm/libggml-hip.so
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:27.049Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 ROCm.0.NO_VMM=1 ROCm.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc)
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_load_from_file_impl: using device ROCm0 (AMD Instinct MI300X VF) - 195958 MiB free
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:27.049Z level=INFO source=runner.go:874 msg="Server listening on 127.0.0.1:40669"
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: loaded meta data with 60 key-value pairs and 1086 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44 (version GGUF V3 (latest))
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   1:                               general.type str              = model
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   2:                               general.name str              = Deepseek-R1-0528
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   3:                           general.basename str              = Deepseek-R1-0528
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   4:                       general.quantized_by str              = Unsloth
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   5:                         general.size_label str              = 256x20B
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   6:                            general.license str              = mit
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   7:                           general.repo_url str              = https://huggingface.co/unsloth
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv   9:                  general.base_model.0.name str              = DeepSeek R1 0528
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  10:               general.base_model.0.version str              = 0528
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  11:          general.base_model.0.organization str              = Deepseek Ai
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  12:              general.base_model.0.repo_url str              = https://huggingface.co/deepseek-ai/De...
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  13:                               general.tags arr[str,3]       = ["deepseek", "unsloth", "transformers"]
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  14:                          general.languages arr[str,1]       = ["en"]
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  15:                      deepseek2.block_count u32              = 61
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  16:                   deepseek2.context_length u32              = 163840
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  17:                 deepseek2.embedding_length u32              = 7168
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  18:              deepseek2.feed_forward_length u32              = 18432
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  19:             deepseek2.attention.head_count u32              = 128
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  20:          deepseek2.attention.head_count_kv u32              = 1
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  21:                   deepseek2.rope.freq_base f32              = 10000.000000
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  22: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  23:                deepseek2.expert_used_count u32              = 8
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  24:        deepseek2.leading_dense_block_count u32              = 3
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  25:                       deepseek2.vocab_size u32              = 129280
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  26:            deepseek2.attention.q_lora_rank u32              = 1536
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  27:           deepseek2.attention.kv_lora_rank u32              = 512
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  28:             deepseek2.attention.key_length u32              = 576
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  29:           deepseek2.attention.value_length u32              = 512
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  30:         deepseek2.attention.key_length_mla u32              = 192
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  31:       deepseek2.attention.value_length_mla u32              = 128
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  32:       deepseek2.expert_feed_forward_length u32              = 2048
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  33:                     deepseek2.expert_count u32              = 256
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  34:              deepseek2.expert_shared_count u32              = 1
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  35:             deepseek2.expert_weights_scale f32              = 2.500000
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  36:              deepseek2.expert_weights_norm bool             = true
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  37:               deepseek2.expert_gating_func u32              = 2
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  38:             deepseek2.rope.dimension_count u32              = 64
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  39:                deepseek2.rope.scaling.type str              = yarn
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  40:              deepseek2.rope.scaling.factor f32              = 40.000000
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  41: deepseek2.rope.scaling.original_context_length u32              = 4096
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  42: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  43:                       tokenizer.ggml.model str              = gpt2
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  44:                         tokenizer.ggml.pre str              = deepseek-v3
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: [132B blob data]
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  46:                  tokenizer.ggml.token_type arr[i32,129280]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  47:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  48:                tokenizer.ggml.bos_token_id u32              = 0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  49:                tokenizer.ggml.eos_token_id u32              = 1
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  50:            tokenizer.ggml.padding_token_id u32              = 2
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  51:               tokenizer.ggml.add_bos_token bool             = true
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  52:               tokenizer.ggml.add_eos_token bool             = false
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  53:                    tokenizer.chat_template str              = {%- if not add_generation_prompt is d...
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  54:               general.quantization_version u32              = 2
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  55:                          general.file_type u32              = 24
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  56:                      quantize.imatrix.file str              = DeepSeek-R1-0528-GGUF/imatrix_unsloth...
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  57:                   quantize.imatrix.dataset str              = unsloth_calibration_DeepSeek-R1-0528-...
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  58:             quantize.imatrix.entries_count i32              = 659
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv  59:              quantize.imatrix.chunks_count i32              = 720
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type  f32:  361 tensors
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q8_0:  122 tensors
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q4_K:   56 tensors
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q5_K:   36 tensors
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q6_K:   17 tensors
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq2_xxs:   30 tensors
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq3_xxs:   54 tensors
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq1_s:  126 tensors
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq3_s:  148 tensors
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq4_xs:  136 tensors
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file format = GGUF V3 (latest)
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file type   = IQ1_S - 1.5625 bpw
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file size   = 150.51 GiB (1.93 BPW)
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: special tokens cache size = 818
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: token to piece cache size = 0.8223 MB
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: arch             = deepseek2
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: vocab_only       = 0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_ctx_train      = 163840
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd           = 7168
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_layer          = 61
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_head           = 128
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_head_kv        = 1
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_rot            = 64
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_swa            = 0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_swa_pattern    = 1
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_k    = 576
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_v    = 512
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_gqa            = 128
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_k_gqa     = 576
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_v_gqa     = 512
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_norm_eps       = 0.0e+00
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_norm_rms_eps   = 1.0e-06
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_clamp_kqv      = 0.0e+00
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_max_alibi_bias = 0.0e+00
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_logit_scale    = 0.0e+00
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_attn_scale     = 0.0e+00
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_ff             = 18432
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_expert         = 256
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_expert_used    = 8
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: causal attn      = 1
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: pooling type     = 0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: rope type        = 0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: rope scaling     = yarn
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: freq_base_train  = 10000.0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: freq_scale_train = 0.025
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_ctx_orig_yarn  = 4096
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: rope_finetuned   = unknown
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: ssm_d_conv       = 0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: ssm_d_inner      = 0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: ssm_d_state      = 0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: ssm_dt_rank      = 0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: ssm_dt_b_c_rms   = 0
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: model type       = 671B
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: model params     = 671.03 B
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: general.name     = Deepseek-R1-0528
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_layer_dense_lead   = 3
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_lora_q             = 1536
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_lora_kv            = 512
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_k_mla    = 192
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_v_mla    = 128
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_ff_exp             = 2048
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_expert_shared      = 1
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_weights_scale = 2.5
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_weights_norm  = 1
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_gating_func   = sigmoid
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: rope_yarn_log_mul    = 0.1000
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: vocab type       = BPE
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_vocab          = 129280
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_merges         = 127741
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: BOS token        = 0 '<｜begin▁of▁sentence｜>'
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOS token        = 1 '<｜end▁of▁sentence｜>'
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOT token        = 1 '<｜end▁of▁sentence｜>'
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: PAD token        = 2 '<｜▁pad▁｜>'
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: LF token         = 201 'Ċ'
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM PRE token    = 128801 '<｜fim▁begin｜>'
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM SUF token    = 128800 '<｜fim▁hole｜>'
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM MID token    = 128802 '<｜fim▁end｜>'
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOG token        = 1 '<｜end▁of▁sentence｜>'
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: max token length = 256
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors: loading model tensors, this can take a while... (mmap = true)
Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:27.222Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
Jun 15 08:10:31 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors: offloading 61 repeating layers to GPU
Jun 15 08:10:31 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors: offloading output layer to GPU
Jun 15 08:10:31 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors: offloaded 62/62 layers to GPU
Jun 15 08:10:31 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors:        ROCm0 model buffer size = 153627.73 MiB
Jun 15 08:10:31 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors:   CPU_Mapped model buffer size =   497.11 MiB
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: constructing llama_context
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_seq_max     = 2
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_ctx         = 131072
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_ctx_per_seq = 65536
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_batch       = 1024
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_ubatch      = 512
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: causal_attn   = 1
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: flash_attn    = 0
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: freq_base     = 10000.0
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: freq_scale    = 0.025
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_ctx_per_seq (65536) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context:  ROCm_Host  output buffer size =     1.04 MiB
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_kv_cache_unified: kv_size = 131072, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 1, padding = 32
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_kv_cache_unified:      ROCm0 KV buffer size = 16592.00 MiB
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_kv_cache_unified: KV self size  = 16592.00 MiB, K (f16): 8784.00 MiB, V (f16): 7808.00 MiB
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 33418.00 MiB on device 0: cudaMalloc failed: out of memory
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 35041316864
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: panic: unable to create llama context
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: goroutine 50 [running]:
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: github.com/ollama/ollama/runner/llamarunner.(*Server).loadModel(0xc0004e2360, {0x3e, 0x0, 0x1, {0x0, 0x0, 0x0}, 0xc0005036d0, 0x0}, {0x7ffcddb8bc94, ...}, ...)
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]:         github.com/ollama/ollama/runner/llamarunner/runner.go:757 +0x389
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]:         github.com/ollama/ollama/runner/llamarunner/runner.go:848 +0xb57
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:39.374Z level=ERROR source=server.go:457 msg="llama runner terminated" error="exit status 2"
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:39.510Z level=ERROR source=sched.go:489 msg="error loading llama server" error="llama runner process has terminated: cudaMalloc failed: out of memory\nggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 35041316864"
Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: [GIN] 2025/06/15 - 08:10:39 | 500 | 13.740989178s |       127.0.0.1 | POST     "/api/generate"
Jun 15 08:10:44 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:44.511Z level=WARN source=sched.go:687 msg="gpu VRAM usage didn't recover within timeout" seconds=5.001273245 runner.size="167.8 GiB" runner.vram="167.8 GiB" runner.parallel=2 runner.pid=5741 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44
Jun 15 08:10:44 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:44.760Z level=WARN source=sched.go:687 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250772528 runner.size="167.8 GiB" runner.vram="167.8 GiB" runner.parallel=2 runner.pid=5741 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44
<!-- gh-comment-id:2973581927 --> @digiperfect-tech commented on GitHub (Jun 15, 2025): For simplicitly - I am trying a small model on single GPU instance - ecountering same error. ``` Model size: 161 GB Model: hf.co/unsloth/DeepSeek-R1-0528-GGUF:latest ``` ``` GPU VRAM: 192 GB System RAM: 235 GB ``` Incidentally - following MOE model works fine, the total model size is slightly smaller than the model that fails. But IMO - above model should work seamlessly given the sizes and extra system RAM available? Model that works: `qwen3:235b-a22b 142GB` Server logs (the log file was not present inside ~/.ollama.... but journalctl gave the logs). ``` Jun 15 08:09:58 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 systemd[1]: ollama.service: Deactivated successfully. Jun 15 08:09:58 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 systemd[1]: ollama.service: Consumed 14.296s CPU time. Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 systemd[1]: ollama.service: Scheduled restart job, restart counter is at 2. Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 systemd[1]: Started ollama.service - Ollama Service. Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.110Z level=INFO source=routes.go:1234 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/usr/share/ollama/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.110Z level=INFO source=images.go:479 msg="total blobs: 21" Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.110Z level=INFO source=images.go:486 msg="total unused blobs removed: 0" Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.111Z level=INFO source=routes.go:1287 msg="Listening on 127.0.0.1:11434 (version 0.9.0)" Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.111Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.117Z level=INFO source=amd_linux.go:386 msg="amdgpu is supported" gpu=GPU-7524e56f4b0facdf gpu_type=gfx942 Jun 15 08:10:02 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:02.119Z level=INFO source=types.go:130 msg="inference compute" id=GPU-7524e56f4b0facdf library=rocm variant="" compute=gfx942 driver=6.12 name=1002:74b5 total="191.7 GiB" available="191.4 GiB" Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: [GIN] 2025/06/15 - 08:10:25 | 200 | 36.657µs | 127.0.0.1 | HEAD "/" Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: [GIN] 2025/06/15 - 08:10:25 | 200 | 19.514005ms | 127.0.0.1 | POST "/api/show" Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.797Z level=INFO source=sched.go:788 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44 gpu=GPU-7524e56f4b0facdf parallel=2 available=205523197952 required="167.8 GiB" Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.797Z level=INFO source=server.go:135 msg="system memory" total="235.9 GiB" free="230.2 GiB" free_swap="0 B" Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.798Z level=INFO source=server.go:168 msg=offload library=rocm layers.requested=-1 layers.model=62 layers.offload=62 layers.split="" memory.available="[191.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="167.8 GiB" memory.required.partial="167.8 GiB" memory.required.kv="16.2 GiB" memory.required.allocations="[167.8 GiB]" memory.weights.total="150.0 GiB" memory.weights.repeating="149.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="556.3 MiB" memory.graph.partial="1019.5 MiB" Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: loaded meta data with 60 key-value pairs and 1086 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44 (version GGUF V3 (latest)) Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 0: general.architecture str = deepseek2 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 1: general.type str = model Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 2: general.name str = Deepseek-R1-0528 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 3: general.basename str = Deepseek-R1-0528 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 4: general.quantized_by str = Unsloth Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 5: general.size_label str = 256x20B Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 6: general.license str = mit Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 7: general.repo_url str = https://huggingface.co/unsloth Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 8: general.base_model.count u32 = 1 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 9: general.base_model.0.name str = DeepSeek R1 0528 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 10: general.base_model.0.version str = 0528 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 11: general.base_model.0.organization str = Deepseek Ai Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/deepseek-ai/De... Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 13: general.tags arr[str,3] = ["deepseek", "unsloth", "transformers"] Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 14: general.languages arr[str,1] = ["en"] Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 15: deepseek2.block_count u32 = 61 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 16: deepseek2.context_length u32 = 163840 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 17: deepseek2.embedding_length u32 = 7168 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 18: deepseek2.feed_forward_length u32 = 18432 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 19: deepseek2.attention.head_count u32 = 128 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 20: deepseek2.attention.head_count_kv u32 = 1 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 21: deepseek2.rope.freq_base f32 = 10000.000000 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 22: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 23: deepseek2.expert_used_count u32 = 8 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 24: deepseek2.leading_dense_block_count u32 = 3 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 25: deepseek2.vocab_size u32 = 129280 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 26: deepseek2.attention.q_lora_rank u32 = 1536 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 27: deepseek2.attention.kv_lora_rank u32 = 512 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 28: deepseek2.attention.key_length u32 = 576 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 29: deepseek2.attention.value_length u32 = 512 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 30: deepseek2.attention.key_length_mla u32 = 192 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 31: deepseek2.attention.value_length_mla u32 = 128 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 32: deepseek2.expert_feed_forward_length u32 = 2048 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 33: deepseek2.expert_count u32 = 256 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 34: deepseek2.expert_shared_count u32 = 1 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 35: deepseek2.expert_weights_scale f32 = 2.500000 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 36: deepseek2.expert_weights_norm bool = true Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 37: deepseek2.expert_gating_func u32 = 2 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 38: deepseek2.rope.dimension_count u32 = 64 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 39: deepseek2.rope.scaling.type str = yarn Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 40: deepseek2.rope.scaling.factor f32 = 40.000000 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 41: deepseek2.rope.scaling.original_context_length u32 = 4096 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 42: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 43: tokenizer.ggml.model str = gpt2 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 44: tokenizer.ggml.pre str = deepseek-v3 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: [132B blob data] Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 46: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 47: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e... Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 48: tokenizer.ggml.bos_token_id u32 = 0 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 49: tokenizer.ggml.eos_token_id u32 = 1 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 50: tokenizer.ggml.padding_token_id u32 = 2 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 51: tokenizer.ggml.add_bos_token bool = true Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 52: tokenizer.ggml.add_eos_token bool = false Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 53: tokenizer.chat_template str = {%- if not add_generation_prompt is d... Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 54: general.quantization_version u32 = 2 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 55: general.file_type u32 = 24 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 56: quantize.imatrix.file str = DeepSeek-R1-0528-GGUF/imatrix_unsloth... Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 57: quantize.imatrix.dataset str = unsloth_calibration_DeepSeek-R1-0528-... Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 58: quantize.imatrix.entries_count i32 = 659 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 59: quantize.imatrix.chunks_count i32 = 720 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type f32: 361 tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q8_0: 122 tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q4_K: 56 tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q5_K: 36 tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q6_K: 17 tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq2_xxs: 30 tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq3_xxs: 54 tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq1_s: 126 tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq3_s: 148 tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq4_xs: 136 tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file format = GGUF V3 (latest) Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file type = IQ1_S - 1.5625 bpw Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file size = 150.51 GiB (1.93 BPW) Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: special tokens cache size = 818 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: token to piece cache size = 0.8223 MB Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: arch = deepseek2 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: vocab_only = 1 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: model type = ?B Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: model params = 671.03 B Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: general.name = Deepseek-R1-0528 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_layer_dense_lead = 0 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_lora_q = 0 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_lora_kv = 0 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_k_mla = 0 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_v_mla = 0 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_ff_exp = 0 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_expert_shared = 0 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_weights_scale = 0.0 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_weights_norm = 0 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_gating_func = unknown Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: rope_yarn_log_mul = 0.0000 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: vocab type = BPE Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_vocab = 129280 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_merges = 127741 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: BOS token = 0 '<｜begin▁of▁sentence｜>' Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOS token = 1 '<｜end▁of▁sentence｜>' Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOT token = 1 '<｜end▁of▁sentence｜>' Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: PAD token = 2 '<｜▁pad▁｜>' Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: LF token = 201 'Ċ' Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM PRE token = 128801 '<｜fim▁begin｜>' Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM SUF token = 128800 '<｜fim▁hole｜>' Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM MID token = 128802 '<｜fim▁end｜>' Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOG token = 1 '<｜end▁of▁sentence｜>' Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: max token length = 256 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_load: vocab only - skipping tensors Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.967Z level=INFO source=server.go:431 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44 --ctx-size 131072 --batch-size 512 --n-gpu-layers 62 --threads 20 --parallel 2 --port 40669" Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.968Z level=INFO source=sched.go:483 msg="loaded runners" count=1 Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.968Z level=INFO source=server.go:591 msg="waiting for llama runner to start responding" Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.968Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server not responding" Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.976Z level=INFO source=runner.go:815 msg="starting go runner" Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_backend: loaded CPU backend from /usr/local/lib/ollama/libggml-cpu-icelake.so Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_cuda_init: found 1 ROCm devices: Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: Device 0: AMD Instinct MI300X VF, gfx942:sramecc+:xnack- (0x942), VMM: no, Wave Size: 64 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_backend: loaded ROCm backend from /usr/local/lib/ollama/rocm/libggml-hip.so Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:27.049Z level=INFO source=ggml.go:104 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.AVX512=1 CPU.0.AVX512_VBMI=1 CPU.0.AVX512_VNNI=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 ROCm.0.NO_VMM=1 ROCm.0.PEER_MAX_BATCH_SIZE=128 compiler=cgo(gcc) Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_load_from_file_impl: using device ROCm0 (AMD Instinct MI300X VF) - 195958 MiB free Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:27.049Z level=INFO source=runner.go:874 msg="Server listening on 127.0.0.1:40669" Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: loaded meta data with 60 key-value pairs and 1086 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44 (version GGUF V3 (latest)) Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 0: general.architecture str = deepseek2 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 1: general.type str = model Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 2: general.name str = Deepseek-R1-0528 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 3: general.basename str = Deepseek-R1-0528 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 4: general.quantized_by str = Unsloth Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 5: general.size_label str = 256x20B Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 6: general.license str = mit Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 7: general.repo_url str = https://huggingface.co/unsloth Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 8: general.base_model.count u32 = 1 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 9: general.base_model.0.name str = DeepSeek R1 0528 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 10: general.base_model.0.version str = 0528 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 11: general.base_model.0.organization str = Deepseek Ai Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/deepseek-ai/De... Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 13: general.tags arr[str,3] = ["deepseek", "unsloth", "transformers"] Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 14: general.languages arr[str,1] = ["en"] Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 15: deepseek2.block_count u32 = 61 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 16: deepseek2.context_length u32 = 163840 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 17: deepseek2.embedding_length u32 = 7168 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 18: deepseek2.feed_forward_length u32 = 18432 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 19: deepseek2.attention.head_count u32 = 128 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 20: deepseek2.attention.head_count_kv u32 = 1 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 21: deepseek2.rope.freq_base f32 = 10000.000000 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 22: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 23: deepseek2.expert_used_count u32 = 8 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 24: deepseek2.leading_dense_block_count u32 = 3 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 25: deepseek2.vocab_size u32 = 129280 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 26: deepseek2.attention.q_lora_rank u32 = 1536 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 27: deepseek2.attention.kv_lora_rank u32 = 512 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 28: deepseek2.attention.key_length u32 = 576 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 29: deepseek2.attention.value_length u32 = 512 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 30: deepseek2.attention.key_length_mla u32 = 192 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 31: deepseek2.attention.value_length_mla u32 = 128 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 32: deepseek2.expert_feed_forward_length u32 = 2048 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 33: deepseek2.expert_count u32 = 256 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 34: deepseek2.expert_shared_count u32 = 1 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 35: deepseek2.expert_weights_scale f32 = 2.500000 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 36: deepseek2.expert_weights_norm bool = true Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 37: deepseek2.expert_gating_func u32 = 2 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 38: deepseek2.rope.dimension_count u32 = 64 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 39: deepseek2.rope.scaling.type str = yarn Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 40: deepseek2.rope.scaling.factor f32 = 40.000000 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 41: deepseek2.rope.scaling.original_context_length u32 = 4096 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 42: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 43: tokenizer.ggml.model str = gpt2 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 44: tokenizer.ggml.pre str = deepseek-v3 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: [132B blob data] Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 46: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 47: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e... Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 48: tokenizer.ggml.bos_token_id u32 = 0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 49: tokenizer.ggml.eos_token_id u32 = 1 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 50: tokenizer.ggml.padding_token_id u32 = 2 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 51: tokenizer.ggml.add_bos_token bool = true Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 52: tokenizer.ggml.add_eos_token bool = false Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 53: tokenizer.chat_template str = {%- if not add_generation_prompt is d... Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 54: general.quantization_version u32 = 2 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 55: general.file_type u32 = 24 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 56: quantize.imatrix.file str = DeepSeek-R1-0528-GGUF/imatrix_unsloth... Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 57: quantize.imatrix.dataset str = unsloth_calibration_DeepSeek-R1-0528-... Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 58: quantize.imatrix.entries_count i32 = 659 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - kv 59: quantize.imatrix.chunks_count i32 = 720 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type f32: 361 tensors Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q8_0: 122 tensors Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q4_K: 56 tensors Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q5_K: 36 tensors Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type q6_K: 17 tensors Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq2_xxs: 30 tensors Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq3_xxs: 54 tensors Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq1_s: 126 tensors Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq3_s: 148 tensors Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_model_loader: - type iq4_xs: 136 tensors Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file format = GGUF V3 (latest) Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file type = IQ1_S - 1.5625 bpw Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: file size = 150.51 GiB (1.93 BPW) Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: special tokens cache size = 818 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load: token to piece cache size = 0.8223 MB Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: arch = deepseek2 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: vocab_only = 0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_ctx_train = 163840 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd = 7168 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_layer = 61 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_head = 128 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_head_kv = 1 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_rot = 64 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_swa = 0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_swa_pattern = 1 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_k = 576 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_v = 512 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_gqa = 128 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_k_gqa = 576 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_v_gqa = 512 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_norm_eps = 0.0e+00 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_norm_rms_eps = 1.0e-06 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_clamp_kqv = 0.0e+00 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_max_alibi_bias = 0.0e+00 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_logit_scale = 0.0e+00 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: f_attn_scale = 0.0e+00 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_ff = 18432 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_expert = 256 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_expert_used = 8 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: causal attn = 1 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: pooling type = 0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: rope type = 0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: rope scaling = yarn Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: freq_base_train = 10000.0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: freq_scale_train = 0.025 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_ctx_orig_yarn = 4096 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: rope_finetuned = unknown Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: ssm_d_conv = 0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: ssm_d_inner = 0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: ssm_d_state = 0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: ssm_dt_rank = 0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: ssm_dt_b_c_rms = 0 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: model type = 671B Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: model params = 671.03 B Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: general.name = Deepseek-R1-0528 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_layer_dense_lead = 3 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_lora_q = 1536 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_lora_kv = 512 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_k_mla = 192 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_embd_head_v_mla = 128 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_ff_exp = 2048 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_expert_shared = 1 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_weights_scale = 2.5 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_weights_norm = 1 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: expert_gating_func = sigmoid Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: rope_yarn_log_mul = 0.1000 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: vocab type = BPE Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_vocab = 129280 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: n_merges = 127741 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: BOS token = 0 '<｜begin▁of▁sentence｜>' Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOS token = 1 '<｜end▁of▁sentence｜>' Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOT token = 1 '<｜end▁of▁sentence｜>' Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: PAD token = 2 '<｜▁pad▁｜>' Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: LF token = 201 'Ċ' Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM PRE token = 128801 '<｜fim▁begin｜>' Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM SUF token = 128800 '<｜fim▁hole｜>' Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: FIM MID token = 128802 '<｜fim▁end｜>' Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: EOG token = 1 '<｜end▁of▁sentence｜>' Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: print_info: max token length = 256 Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors: loading model tensors, this can take a while... (mmap = true) Jun 15 08:10:27 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:27.222Z level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" Jun 15 08:10:31 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors: offloading 61 repeating layers to GPU Jun 15 08:10:31 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors: offloading output layer to GPU Jun 15 08:10:31 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors: offloaded 62/62 layers to GPU Jun 15 08:10:31 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors: ROCm0 model buffer size = 153627.73 MiB Jun 15 08:10:31 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: load_tensors: CPU_Mapped model buffer size = 497.11 MiB Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: constructing llama_context Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_seq_max = 2 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_ctx = 131072 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_ctx_per_seq = 65536 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_batch = 1024 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_ubatch = 512 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: causal_attn = 1 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: flash_attn = 0 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: freq_base = 10000.0 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: freq_scale = 0.025 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: n_ctx_per_seq (65536) < n_ctx_train (163840) -- the full capacity of the model will not be utilized Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_context: ROCm_Host output buffer size = 1.04 MiB Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_kv_cache_unified: kv_size = 131072, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 1, padding = 32 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_kv_cache_unified: ROCm0 KV buffer size = 16592.00 MiB Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_kv_cache_unified: KV self size = 16592.00 MiB, K (f16): 8784.00 MiB, V (f16): 7808.00 MiB Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 33418.00 MiB on device 0: cudaMalloc failed: out of memory Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 35041316864 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: panic: unable to create llama context Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: goroutine 50 [running]: Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: github.com/ollama/ollama/runner/llamarunner.(*Server).loadModel(0xc0004e2360, {0x3e, 0x0, 0x1, {0x0, 0x0, 0x0}, 0xc0005036d0, 0x0}, {0x7ffcddb8bc94, ...}, ...) Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: github.com/ollama/ollama/runner/llamarunner/runner.go:757 +0x389 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: github.com/ollama/ollama/runner/llamarunner/runner.go:848 +0xb57 Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:39.374Z level=ERROR source=server.go:457 msg="llama runner terminated" error="exit status 2" Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:39.510Z level=ERROR source=sched.go:489 msg="error loading llama server" error="llama runner process has terminated: cudaMalloc failed: out of memory\nggml_gallocr_reserve_n: failed to allocate ROCm0 buffer of size 35041316864" Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: [GIN] 2025/06/15 - 08:10:39 | 500 | 13.740989178s | 127.0.0.1 | POST "/api/generate" Jun 15 08:10:44 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:44.511Z level=WARN source=sched.go:687 msg="gpu VRAM usage didn't recover within timeout" seconds=5.001273245 runner.size="167.8 GiB" runner.vram="167.8 GiB" runner.parallel=2 runner.pid=5741 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44 Jun 15 08:10:44 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:44.760Z level=WARN source=sched.go:687 msg="gpu VRAM usage didn't recover within timeout" seconds=5.250772528 runner.size="167.8 GiB" runner.vram="167.8 GiB" runner.parallel=2 runner.pid=5741 runner.model=/usr/share/ollama/.ollama/models/blobs/sha256-fe40cbf872192b5cbb7af80d9c33bcb7d52d0d224be09e919249678209ec0c44 ```
GiteaMirror commented 2026-05-04 17:55:08 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 16, 2025):

Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.798Z level=INFO
 source=server.go:168 msg=offload library=rocm layers.requested=-1 layers.model=62 layers.offload=62 layers.split=""
 memory.available="[191.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="167.8 GiB"
 memory.required.partial="167.8 GiB" memory.required.kv="16.2 GiB" memory.required.allocations="[167.8 GiB]"
 memory.weights.total="150.0 GiB" memory.weights.repeating="149.3 GiB" memory.weights.nonrepeating="725.0 MiB"
 memory.graph.full="556.3 MiB" memory.graph.partial="1019.5 MiB"

ollama server estimates need 167.8G of 191.4G to offload all 62 layers.

Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 33418.00 MiB on device 0: cudaMalloc failed: out of memory

ollama runner dies with OOM allocating 33G during the model load.

The estimation logic is sometimes a little inaccurate, but there should be 23G of free VRAM so even if the estimation is a little off, there should be free VRAM to absorb the error.

Can you show the logs of the failure when OLLAMA_SCHED_SPREAD=1?

<!-- gh-comment-id:2976875846 --> @rick-github commented on GitHub (Jun 16, 2025): ``` Jun 15 08:10:25 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: time=2025-06-15T08:10:25.798Z level=INFO source=server.go:168 msg=offload library=rocm layers.requested=-1 layers.model=62 layers.offload=62 layers.split="" memory.available="[191.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="167.8 GiB" memory.required.partial="167.8 GiB" memory.required.kv="16.2 GiB" memory.required.allocations="[167.8 GiB]" memory.weights.total="150.0 GiB" memory.weights.repeating="149.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="556.3 MiB" memory.graph.partial="1019.5 MiB" ``` ollama server estimates need 167.8G of 191.4G to offload all 62 layers. ``` Jun 15 08:10:39 ml-ai-ubuntu-gpu-mi300x1-192gb-atl1 ollama[5704]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 33418.00 MiB on device 0: cudaMalloc failed: out of memory ``` ollama runner dies with OOM allocating 33G during the model load. The estimation logic is sometimes a little inaccurate, but there should be 23G of free VRAM so even if the estimation is a little off, there should be free VRAM to absorb the error. Can you show the logs of the failure when `OLLAMA_SCHED_SPREAD=1`?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#69364
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?