github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8597] Error: llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer (4x L40S, 384GB system RAM, Deepseek-R1) #5560

New Issue
Closed
opened 2026-04-12 16:49:04 -05:00 by GiteaMirror · 18 comments
GiteaMirror commented 2026-04-12 16:49:04 -05:00
Owner
Copy Link

Originally created by @orlyandico on GitHub (Jan 26, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8597

What is the issue?

I am aware that 4 x L40S only has 192GB of VRAM, but the machine also has 384GB of system RAM. The error seems to indicate that 45108 MiB of RAM is being allocated with cudaMalloc and this is failing. This is very close to the GPU limit (46068 MiB). On my home setup (2x P40, admittedly not trying such a huge model) the GPU never gets close to its VRAM limit (typically only 22GB out of 24GB).

Jan 26 17:48:20 ip-172-31-3-6 ollama[1418]: llm_load_print_meta: rope_yarn_log_mul    = 0.1000
Jan 26 17:50:35 ip-172-31-3-6 ollama[1418]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 45108.64 MiB on device 0: cudaMalloc failed: out of memory
Jan 26 17:50:35 ip-172-31-3-6 ollama[1418]: llama_model_load: error loading model: unable to allocate CUDA0 buffer
Jan 26 17:50:35 ip-172-31-3-6 ollama[1418]: llama_load_model_from_file: failed to load model
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: panic: unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: goroutine 34 [running]:
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0001a0000, {0x1b, 0x0, 0x0, 0x0, {0xc000194090, 0x4, 0x4}, 0xc00018a060, 0x0}, ...)
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]:         github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]:         github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d
Jan 26 17:51:24 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:24.077Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
Jan 26 17:51:24 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:24.578Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer\nllama_load_model_from_file: failed to load model"
Jan 26 17:51:24 ip-172-31-3-6 ollama[1418]: [GIN] 2025/01/26 - 17:51:24 | 500 |          3m6s |       127.0.0.1 | POST     "/api/generate"
Jan 26 17:51:30 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:30.190Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.6116838510000004 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1>
Jan 26 17:51:30 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:30.947Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.369278328 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Jan 26 17:51:31 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:31.707Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=7.128851916 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Jan 26 17:51:40 ip-172-31-3-6 ollama[1418]: [GIN] 2025/01/26 - 17:51:40 | 200 |      65.972µs |       127.0.0.1 | GET      "/api/version"


ubuntu@ip-172-31-3-6:~$ ollama --version
ollama version is 0.5.7
ubuntu@ip-172-31-3-6:~$ free
               total        used        free      shared  buff/cache   available
Mem:       390837004     4152012   385439624        3248     1245368   383763992
Swap:              0           0           0


ubuntu@ip-172-31-3-6:~$ nvidia-smi
Sun Jan 26 17:48:08 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    On  |   00000000:38:00.0 Off |                    0 |
| N/A   22C    P8             22W /  350W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L40S                    On  |   00000000:3A:00.0 Off |                    0 |
| N/A   22C    P8             21W /  350W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L40S                    On  |   00000000:3C:00.0 Off |                    0 |
| N/A   22C    P8             44W /  350W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L40S                    On  |   00000000:3E:00.0 Off |                    0 |
| N/A   23C    P8             22W /  350W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.5.7

Originally created by @orlyandico on GitHub (Jan 26, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8597 ### What is the issue? I am aware that 4 x L40S only has 192GB of VRAM, but the machine also has 384GB of system RAM. The error seems to indicate that 45108 MiB of RAM is being allocated with cudaMalloc and this is failing. This is very close to the GPU limit (46068 MiB). On my home setup (2x P40, admittedly not trying such a huge model) the GPU never gets close to its VRAM limit (typically only 22GB out of 24GB). ``` Jan 26 17:48:20 ip-172-31-3-6 ollama[1418]: llm_load_print_meta: rope_yarn_log_mul = 0.1000 Jan 26 17:50:35 ip-172-31-3-6 ollama[1418]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 45108.64 MiB on device 0: cudaMalloc failed: out of memory Jan 26 17:50:35 ip-172-31-3-6 ollama[1418]: llama_model_load: error loading model: unable to allocate CUDA0 buffer Jan 26 17:50:35 ip-172-31-3-6 ollama[1418]: llama_load_model_from_file: failed to load model Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: panic: unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: goroutine 34 [running]: Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0001a0000, {0x1b, 0x0, 0x0, 0x0, {0xc000194090, 0x4, 0x4}, 0xc00018a060, 0x0}, ...) Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 Jan 26 17:51:23 ip-172-31-3-6 ollama[1418]: github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d Jan 26 17:51:24 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:24.077Z level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error" Jan 26 17:51:24 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:24.578Z level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer\nllama_load_model_from_file: failed to load model" Jan 26 17:51:24 ip-172-31-3-6 ollama[1418]: [GIN] 2025/01/26 - 17:51:24 | 500 | 3m6s | 127.0.0.1 | POST "/api/generate" Jan 26 17:51:30 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:30.190Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.6116838510000004 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1> Jan 26 17:51:30 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:30.947Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.369278328 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 Jan 26 17:51:31 ip-172-31-3-6 ollama[1418]: time=2025-01-26T17:51:31.707Z level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=7.128851916 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 Jan 26 17:51:40 ip-172-31-3-6 ollama[1418]: [GIN] 2025/01/26 - 17:51:40 | 200 | 65.972µs | 127.0.0.1 | GET "/api/version" ubuntu@ip-172-31-3-6:~$ ollama --version ollama version is 0.5.7 ubuntu@ip-172-31-3-6:~$ free total used free shared buff/cache available Mem: 390837004 4152012 385439624 3248 1245368 383763992 Swap: 0 0 0 ubuntu@ip-172-31-3-6:~$ nvidia-smi Sun Jan 26 17:48:08 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA L40S On | 00000000:38:00.0 Off | 0 | | N/A 22C P8 22W / 350W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA L40S On | 00000000:3A:00.0 Off | 0 | | N/A 22C P8 21W / 350W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA L40S On | 00000000:3C:00.0 Off | 0 | | N/A 22C P8 44W / 350W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA L40S On | 00000000:3E:00.0 Off | 0 | | N/A 23C P8 22W / 350W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.5.7
GiteaMirror added the bug label 2026-04-12 16:49:04 -05:00
GiteaMirror commented 2026-04-12 16:49:05 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 26, 2025):

Earlier log lines would show the memory calculations, but there are some standard OOM mitigations:

  1. Set OLLAMA_GPU_OVERHEAD to give the runner a buffer to grow in to (eg, OLLAMA_GPU_OVERHEAD=536870912 to reserve 512M)
  2. Enable flash attention by setting OLLAMA_FLASH_ATTENTION=1 in the server environment. Flash attention is a more efficient use of memory and may reduce memory pressure (note FA is not supported on all model architectures or GPUs, check the logs for flash to verify it's active).
  3. If flash attention is enabled, further gains can be achieved with KV quantization.
  4. Reduce the number layers that ollama thinks it can offload to the GPU by setting num_gpu, see here.
  5. In Linux with Nvidia devices, set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1. This will allow the GPU to offload to CPU memory if VRAM is exhausted. This is only useful for small amounts of memory as there is a performance penalty. However, in the case where the goal is to reduce OOMs, the amount offloaded will be small and the impact minimal.
  6. Set OLLAMA_NUM_PARALLEL to 1. This is the default, so this only helps if it has previously been set to something larger. This reduces the size of the KV cache.
  7. Reduce the size of the KV cache by lowering the value of num_ctx, either in a Modelfile or an API call, or by setting OLLAMA_CONTEXT_LENGTH.
  8. The ollama engine has a better allocation strategy, try using it by setting OLLAMA_NEW_ENGINE=1 in the server environment. Note this only works for model architectures supported by the ollama engine, see here for the currently supported families.
<!-- gh-comment-id:2614533288 --> @rick-github commented on GitHub (Jan 26, 2025): Earlier log lines would show the memory calculations, but there are some standard OOM mitigations: 1. Set [`OLLAMA_GPU_OVERHEAD`](https://github.com/ollama/ollama/blob/5f8051180e3b9aeafc153f6b5056e7358a939c88/envconfig/config.go#L237) to give the runner a buffer to grow in to (eg, `OLLAMA_GPU_OVERHEAD=536870912` to reserve 512M) 2. Enable flash attention by setting [`OLLAMA_FLASH_ATTENTION=1`](https://github.com/ollama/ollama/blob/5f8051180e3b9aeafc153f6b5056e7358a939c88/envconfig/config.go#L236) in the server environment. Flash attention is a more efficient use of memory and may reduce memory pressure (note FA is not supported on all model architectures or GPUs, check the logs for `flash` to verify it's active). 3. If flash attention is enabled, further gains can be achieved with [KV quantization](https://github.com/ollama/ollama/blob/main/docs/faq.mdx#how-can-i-set-the-quantization-type-for-the-kv-cache). 4. Reduce the number layers that ollama thinks it can offload to the GPU by setting `num_gpu`, see [here](https://github.com/ollama/ollama/issues/6950#issuecomment-2373663650). 5. In Linux with Nvidia devices, set `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1`. This will allow the GPU to offload to CPU memory if VRAM is exhausted. This is only useful for small amounts of memory as there is a [performance penalty](https://github.com/ollama/ollama/issues/7584#issuecomment-2466715900). However, in the case where the goal is to reduce OOMs, the amount offloaded will be small and the impact minimal. 6. Set [`OLLAMA_NUM_PARALLEL`](https://github.com/ollama/ollama/blob/7cfd4aee4d9956b89dbbb103ee4877194abfe670/envconfig/config.go#L251) to 1. This is the default, so this only helps if it has previously been set to something larger. This reduces the size of the KV cache. 7. Reduce the size of the KV cache by lowering the value of `num_ctx`, either in a [Modelfile](https://github.com/ollama/ollama/blob/main/docs/openai.mdx#setting-the-context-size) or an [API call](https://github.com/ollama/ollama/blob/main/docs/faq.mdx#how-can-i-specify-the-context-window-size), or by setting [`OLLAMA_CONTEXT_LENGTH`](https://github.com/ollama/ollama/blob/0f3f9e353df96d4cfc40ac19114c782a57fe30f5/envconfig/config.go#L258). 8. The ollama engine has a better allocation strategy, try using it by setting [`OLLAMA_NEW_ENGINE=1`](https://github.com/ollama/ollama/blob/main/envconfig/config.go#L294) in the server environment. Note this only works for model architectures supported by the ollama engine, see [here](https://github.com/ollama/ollama/tree/main/model/models) for the currently supported families.
GiteaMirror commented 2026-04-12 16:49:05 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Jan 26, 2025):

in my case, the Deepseek-R1 model is over 400GB. There's only 180GB of VRAM. Basically half of the layers would have to be loaded into system memory, so inference over those layers would be very slow. Is it even worthwhile to try running Deepseek-R1 with only 4 GPUs?

<!-- gh-comment-id:2614542991 --> @orlyandico commented on GitHub (Jan 26, 2025): in my case, the Deepseek-R1 model is over 400GB. There's only 180GB of VRAM. Basically half of the layers would have to be loaded into system memory, so inference over those layers would be very slow. Is it even worthwhile to try running Deepseek-R1 with only 4 GPUs?
GiteaMirror commented 2026-04-12 16:49:06 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 26, 2025):

Is it even worthwhile to try running Deepseek-R1 with only 4 GPUs?

Only for coolness points. Try the disttill versions, the 70b-llama-distill-fp16 quant will fit and should be pretty capable.

<!-- gh-comment-id:2614544260 --> @rick-github commented on GitHub (Jan 26, 2025): > Is it even worthwhile to try running Deepseek-R1 with only 4 GPUs? Only for coolness points. Try the `disttill` versions, the [70b-llama-distill-fp16](https://ollama.com/library/deepseek-r1:70b-llama-distill-fp16) quant will fit and should be pretty capable.
GiteaMirror commented 2026-04-12 16:49:06 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Jan 26, 2025):

I've run the Ollama 4-bpw quant of the 70B Llama3 distill on my 2x P40, and... on my zero-shot code generation benchmark it's slightly better than Athene-V2 but not at Gemini 1206 or Sonnet 3.5v2 level (granted, the Deepseek-R1 70B Llama distill 4-bpw appears to be the best 70B class model I've tried). Hence my desire to test the full Deepseek-R1 model.

<!-- gh-comment-id:2614548936 --> @orlyandico commented on GitHub (Jan 26, 2025): I've run the Ollama 4-bpw quant of the 70B Llama3 distill on my 2x P40, and... on my zero-shot code generation benchmark it's slightly better than Athene-V2 but not at Gemini 1206 or Sonnet 3.5v2 level (granted, the Deepseek-R1 70B Llama distill 4-bpw appears to be the best 70B class model I've tried). Hence my desire to test the full Deepseek-R1 model.
GiteaMirror commented 2026-04-12 16:49:07 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 26, 2025):

You will need more hardware or a smaller quant. I ran deepseek-v3 q3 on 8x40G A100s a few weeks back and got 6-7 tokens/s. Currently the price/performance ratio doesn't work for running these models on consumer hardware, except where the quant is so small it compromises the quality of the response.

<!-- gh-comment-id:2614554468 --> @rick-github commented on GitHub (Jan 26, 2025): You will need more hardware or a smaller quant. I ran deepseek-v3 q3 on 8x40G A100s a few weeks back and got 6-7 tokens/s. Currently the price/performance ratio doesn't work for running these models on consumer hardware, except where the quant is so small it compromises the quality of the response.
GiteaMirror commented 2026-04-12 16:49:07 -05:00
Author
Owner
Copy Link

@js-2024 commented on GitHub (Jan 27, 2025):

I'm getting the same error trying to run Deepseek-r1:671b. I think it's trying to allocate too many layers to GPU?

System specs -- 10x3090, 512GB System RAM.

$ ollama run deepseek-r1:671b
Error: llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer
llama_load_model_from_file: failed to load model

Logs:

Jan 26 20:53:11 zeus-desktop ollama[88558]: [GIN] 2025/01/26 - 20:53:11 | 200 |       49.67µs |       127.0.0.1 | HEAD     "/"
Jan 26 20:53:12 zeus-desktop ollama[88558]: [GIN] 2025/01/26 - 20:53:12 | 200 |   26.427758ms |       127.0.0.1 | POST     "/api/show"
Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.619-05:00 level=INFO source=server.go:104 msg="system memory" total="503.6 GiB" free="494.1 GiB" free_swap="2.0 GiB"
Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.620-05:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=62 layers.offload=33 layers.split=4,4,4,3,3,3,3,3,3,3 memory.available="[23.1 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="416.8 GiB" memory.required.partial="217.4 GiB" memory.required.kv="9.5 GiB" memory.required.allocations="[21.9 GiB 21.9 GiB 21.9 GiB 21.4 GiB 22.3 GiB 22.3 GiB 21.4 GiB 21.4 GiB 21.4 GiB 21.4 GiB]" memory.weights.total="385.0 GiB" memory.weights.repeating="384.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="1019.5 MiB" memory.graph.partial="1019.5 MiB"
Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.621-05:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 --ctx-size 2048 --batch-size 512 --n-gpu-layers 33 --threads 16 --parallel 1 --tensor-split 4,4,4,3,3,3,3,3,3,3 --port 34603"
Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.621-05:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.621-05:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding"
Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.622-05:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.672-05:00 level=INFO source=runner.go:936 msg="starting go runner"
Jan 26 20:53:15 zeus-desktop ollama[88558]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Jan 26 20:53:15 zeus-desktop ollama[88558]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Jan 26 20:53:15 zeus-desktop ollama[88558]: ggml_cuda_init: found 10 CUDA devices:
Jan 26 20:53:15 zeus-desktop ollama[88558]:   Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Jan 26 20:53:15 zeus-desktop ollama[88558]:   Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Jan 26 20:53:15 zeus-desktop ollama[88558]:   Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Jan 26 20:53:15 zeus-desktop ollama[88558]:   Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Jan 26 20:53:15 zeus-desktop ollama[88558]:   Device 4: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Jan 26 20:53:15 zeus-desktop ollama[88558]:   Device 5: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Jan 26 20:53:15 zeus-desktop ollama[88558]:   Device 6: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Jan 26 20:53:15 zeus-desktop ollama[88558]:   Device 7: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Jan 26 20:53:15 zeus-desktop ollama[88558]:   Device 8: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Jan 26 20:53:15 zeus-desktop ollama[88558]:   Device 9: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
Jan 26 20:53:16 zeus-desktop ollama[88558]: time=2025-01-26T20:53:16.909-05:00 level=INFO source=runner.go:937 msg=system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=16
Jan 26 20:53:16 zeus-desktop ollama[88558]: time=2025-01-26T20:53:16.909-05:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:34603"
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23685 MiB free
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA1 (NVIDIA GeForce RTX 3090) - 23982 MiB free
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA2 (NVIDIA GeForce RTX 3090) - 23982 MiB free
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA3 (NVIDIA GeForce RTX 3090) - 23982 MiB free
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA4 (NVIDIA GeForce RTX 3090) - 23982 MiB free
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA5 (NVIDIA GeForce RTX 3090) - 23982 MiB free
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA6 (NVIDIA GeForce RTX 3090) - 23982 MiB free
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA7 (NVIDIA GeForce RTX 3090) - 23982 MiB free
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA8 (NVIDIA GeForce RTX 3090) - 23982 MiB free
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA9 (NVIDIA GeForce RTX 3090) - 23982 MiB free
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: loaded meta data with 42 key-value pairs and 1025 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 (version GGUF V3 (latest))
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv   1:                               general.type str              = model
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv   2:                         general.size_label str              = 256x20B
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv   3:                      deepseek2.block_count u32              = 61
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv   4:                   deepseek2.context_length u32              = 163840
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv   5:                 deepseek2.embedding_length u32              = 7168
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv   6:              deepseek2.feed_forward_length u32              = 18432
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv   7:             deepseek2.attention.head_count u32              = 128
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv   8:          deepseek2.attention.head_count_kv u32              = 128
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv   9:                   deepseek2.rope.freq_base f32              = 10000.000000
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  10: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000001
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  11:                deepseek2.expert_used_count u32              = 8
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  12:        deepseek2.leading_dense_block_count u32              = 3
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  13:                       deepseek2.vocab_size u32              = 129280
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  14:            deepseek2.attention.q_lora_rank u32              = 1536
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  15:           deepseek2.attention.kv_lora_rank u32              = 512
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  16:             deepseek2.attention.key_length u32              = 192
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  17:           deepseek2.attention.value_length u32              = 128
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  18:       deepseek2.expert_feed_forward_length u32              = 2048
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  19:                     deepseek2.expert_count u32              = 256
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  20:              deepseek2.expert_shared_count u32              = 1
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  21:             deepseek2.expert_weights_scale f32              = 2.500000
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  22:              deepseek2.expert_weights_norm bool             = true
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  23:               deepseek2.expert_gating_func u32              = 2
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  24:             deepseek2.rope.dimension_count u32              = 64
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  25:                deepseek2.rope.scaling.type str              = yarn
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  26:              deepseek2.rope.scaling.factor f32              = 40.000000
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  27: deepseek2.rope.scaling.original_context_length u32              = 4096
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  28: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  29:                       tokenizer.ggml.model str              = gpt2
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  30:                         tokenizer.ggml.pre str              = deepseek-v3
Jan 26 20:53:17 zeus-desktop ollama[88558]: time=2025-01-26T20:53:17.127-05:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model"
Jan 26 20:53:17 zeus-desktop ollama[88558]: [132B blob data]
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  32:                  tokenizer.ggml.token_type arr[i32,129280]  = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  33:                      tokenizer.ggml.merges arr[str,127741]  = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  34:                tokenizer.ggml.bos_token_id u32              = 0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  35:                tokenizer.ggml.eos_token_id u32              = 1
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  36:            tokenizer.ggml.padding_token_id u32              = 1
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  37:               tokenizer.ggml.add_bos_token bool             = true
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  38:               tokenizer.ggml.add_eos_token bool             = false
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  39:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  40:               general.quantization_version u32              = 2
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv  41:                          general.file_type u32              = 15
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - type  f32:  361 tensors
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - type q4_K:  606 tensors
Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - type q6_K:   58 tensors
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_vocab: special tokens cache size = 818
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_vocab: token to piece cache size = 0.8223 MB
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: format           = GGUF V3 (latest)
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: arch             = deepseek2
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: vocab type       = BPE
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_vocab          = 129280
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_merges         = 127741
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: vocab_only       = 0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_ctx_train      = 163840
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_embd           = 7168
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_layer          = 61
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_head           = 128
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_head_kv        = 128
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_rot            = 64
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_swa            = 0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_embd_head_k    = 192
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_embd_head_v    = 128
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_gqa            = 1
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_embd_k_gqa     = 24576
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_embd_v_gqa     = 16384
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: f_logit_scale    = 0.0e+00
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_ff             = 18432
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_expert         = 256
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_expert_used    = 8
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: causal attn      = 1
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: pooling type     = 0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: rope type        = 0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: rope scaling     = yarn
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: freq_base_train  = 10000.0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: freq_scale_train = 0.025
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_ctx_orig_yarn  = 4096
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: rope_finetuned   = unknown
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: ssm_d_conv       = 0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: ssm_d_inner      = 0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: ssm_d_state      = 0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: ssm_dt_rank      = 0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: ssm_dt_b_c_rms   = 0
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: model type       = 671B
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: model ftype      = Q4_K - Medium
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: model params     = 671.03 B
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: model size       = 376.65 GiB (4.82 BPW)
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: general.name     = n/a
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: BOS token        = 0 '<｜begin▁of▁sentence｜>'
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: EOS token        = 1 '<｜end▁of▁sentence｜>'
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: EOT token        = 1 '<｜end▁of▁sentence｜>'
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: PAD token        = 1 '<｜end▁of▁sentence｜>'
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: LF token         = 131 'Ä'
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: FIM PRE token    = 128801 '<｜fim▁begin｜>'
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: FIM SUF token    = 128800 '<｜fim▁hole｜>'
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: FIM MID token    = 128802 '<｜fim▁end｜>'
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: EOG token        = 1 '<｜end▁of▁sentence｜>'
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: max token length = 256
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_layer_dense_lead   = 3
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_lora_q             = 1536
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_lora_kv            = 512
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_ff_exp             = 2048
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_expert_shared      = 1
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: expert_weights_scale = 2.5
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: expert_weights_norm  = 1
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: expert_gating_func   = sigmoid
Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: rope_yarn_log_mul    = 0.1000
Jan 26 20:54:01 zeus-desktop ollama[88558]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 25643.85 MiB on device 0: cudaMalloc failed: out of memory
Jan 26 20:54:21 zeus-desktop ollama[88558]: llama_model_load: error loading model: unable to allocate CUDA0 buffer
Jan 26 20:54:21 zeus-desktop ollama[88558]: llama_load_model_from_file: failed to load model
Jan 26 20:54:21 zeus-desktop ollama[88558]: panic: unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Jan 26 20:54:21 zeus-desktop ollama[88558]: goroutine 22 [running]:
Jan 26 20:54:21 zeus-desktop ollama[88558]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0001461b0, {0x21, 0x0, 0x1, 0x0, {0xc000124210, 0xa, 0xa}, 0xc000112270, 0x0}, ...)
Jan 26 20:54:21 zeus-desktop ollama[88558]:         github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad
Jan 26 20:54:21 zeus-desktop ollama[88558]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1
Jan 26 20:54:21 zeus-desktop ollama[88558]:         github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d
Jan 26 20:54:21 zeus-desktop ollama[88558]: time=2025-01-26T20:54:21.986-05:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error"
Jan 26 20:54:22 zeus-desktop ollama[88558]: time=2025-01-26T20:54:22.738-05:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer\nllama_load_model_from_file: failed to load model"
Jan 26 20:54:22 zeus-desktop ollama[88558]: [GIN] 2025/01/26 - 20:54:22 | 500 |         1m10s |       127.0.0.1 | POST     "/api/generate"
Jan 26 20:54:28 zeus-desktop ollama[88558]: time=2025-01-26T20:54:28.040-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.301107748 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Jan 26 20:54:29 zeus-desktop ollama[88558]: time=2025-01-26T20:54:29.670-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.931209137 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
Jan 26 20:54:31 zeus-desktop ollama[88558]: time=2025-01-26T20:54:31.305-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=8.566103296 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9
<!-- gh-comment-id:2614724887 --> @js-2024 commented on GitHub (Jan 27, 2025): I'm getting the same error trying to run Deepseek-r1:671b. I think it's trying to allocate too many layers to GPU? System specs -- 10x3090, 512GB System RAM. $ ollama run deepseek-r1:671b Error: llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer llama_load_model_from_file: failed to load model Logs: ``` Jan 26 20:53:11 zeus-desktop ollama[88558]: [GIN] 2025/01/26 - 20:53:11 | 200 | 49.67µs | 127.0.0.1 | HEAD "/" Jan 26 20:53:12 zeus-desktop ollama[88558]: [GIN] 2025/01/26 - 20:53:12 | 200 | 26.427758ms | 127.0.0.1 | POST "/api/show" Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.619-05:00 level=INFO source=server.go:104 msg="system memory" total="503.6 GiB" free="494.1 GiB" free_swap="2.0 GiB" Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.620-05:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=62 layers.offload=33 layers.split=4,4,4,3,3,3,3,3,3,3 memory.available="[23.1 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB 23.4 GiB]" memory.gpu_overhead="0 B" memory.required.full="416.8 GiB" memory.required.partial="217.4 GiB" memory.required.kv="9.5 GiB" memory.required.allocations="[21.9 GiB 21.9 GiB 21.9 GiB 21.4 GiB 22.3 GiB 22.3 GiB 21.4 GiB 21.4 GiB 21.4 GiB 21.4 GiB]" memory.weights.total="385.0 GiB" memory.weights.repeating="384.3 GiB" memory.weights.nonrepeating="725.0 MiB" memory.graph.full="1019.5 MiB" memory.graph.partial="1019.5 MiB" Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.621-05:00 level=INFO source=server.go:376 msg="starting llama server" cmd="/usr/local/lib/ollama/runners/cuda_v12_avx/ollama_llama_server runner --model /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 --ctx-size 2048 --batch-size 512 --n-gpu-layers 33 --threads 16 --parallel 1 --tensor-split 4,4,4,3,3,3,3,3,3,3 --port 34603" Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.621-05:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.621-05:00 level=INFO source=server.go:555 msg="waiting for llama runner to start responding" Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.622-05:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error" Jan 26 20:53:15 zeus-desktop ollama[88558]: time=2025-01-26T20:53:15.672-05:00 level=INFO source=runner.go:936 msg="starting go runner" Jan 26 20:53:15 zeus-desktop ollama[88558]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Jan 26 20:53:15 zeus-desktop ollama[88558]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Jan 26 20:53:15 zeus-desktop ollama[88558]: ggml_cuda_init: found 10 CUDA devices: Jan 26 20:53:15 zeus-desktop ollama[88558]: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Jan 26 20:53:15 zeus-desktop ollama[88558]: Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Jan 26 20:53:15 zeus-desktop ollama[88558]: Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Jan 26 20:53:15 zeus-desktop ollama[88558]: Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Jan 26 20:53:15 zeus-desktop ollama[88558]: Device 4: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Jan 26 20:53:15 zeus-desktop ollama[88558]: Device 5: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Jan 26 20:53:15 zeus-desktop ollama[88558]: Device 6: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Jan 26 20:53:15 zeus-desktop ollama[88558]: Device 7: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Jan 26 20:53:15 zeus-desktop ollama[88558]: Device 8: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Jan 26 20:53:15 zeus-desktop ollama[88558]: Device 9: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Jan 26 20:53:16 zeus-desktop ollama[88558]: time=2025-01-26T20:53:16.909-05:00 level=INFO source=runner.go:937 msg=system info="CUDA : ARCHS = 600,610,620,700,720,750,800,860,870,890,900 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 | cgo(gcc)" threads=16 Jan 26 20:53:16 zeus-desktop ollama[88558]: time=2025-01-26T20:53:16.909-05:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:34603" Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23685 MiB free Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA1 (NVIDIA GeForce RTX 3090) - 23982 MiB free Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA2 (NVIDIA GeForce RTX 3090) - 23982 MiB free Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA3 (NVIDIA GeForce RTX 3090) - 23982 MiB free Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA4 (NVIDIA GeForce RTX 3090) - 23982 MiB free Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA5 (NVIDIA GeForce RTX 3090) - 23982 MiB free Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA6 (NVIDIA GeForce RTX 3090) - 23982 MiB free Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA7 (NVIDIA GeForce RTX 3090) - 23982 MiB free Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA8 (NVIDIA GeForce RTX 3090) - 23982 MiB free Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_load_model_from_file: using device CUDA9 (NVIDIA GeForce RTX 3090) - 23982 MiB free Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: loaded meta data with 42 key-value pairs and 1025 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 (version GGUF V3 (latest)) Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 0: general.architecture str = deepseek2 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 1: general.type str = model Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 2: general.size_label str = 256x20B Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 3: deepseek2.block_count u32 = 61 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 4: deepseek2.context_length u32 = 163840 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 5: deepseek2.embedding_length u32 = 7168 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 6: deepseek2.feed_forward_length u32 = 18432 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 7: deepseek2.attention.head_count u32 = 128 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 8: deepseek2.attention.head_count_kv u32 = 128 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 9: deepseek2.rope.freq_base f32 = 10000.000000 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 10: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 11: deepseek2.expert_used_count u32 = 8 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 12: deepseek2.leading_dense_block_count u32 = 3 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 13: deepseek2.vocab_size u32 = 129280 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 14: deepseek2.attention.q_lora_rank u32 = 1536 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 15: deepseek2.attention.kv_lora_rank u32 = 512 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 16: deepseek2.attention.key_length u32 = 192 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 17: deepseek2.attention.value_length u32 = 128 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 18: deepseek2.expert_feed_forward_length u32 = 2048 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 19: deepseek2.expert_count u32 = 256 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 20: deepseek2.expert_shared_count u32 = 1 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 21: deepseek2.expert_weights_scale f32 = 2.500000 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 22: deepseek2.expert_weights_norm bool = true Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 23: deepseek2.expert_gating_func u32 = 2 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 24: deepseek2.rope.dimension_count u32 = 64 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 25: deepseek2.rope.scaling.type str = yarn Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 26: deepseek2.rope.scaling.factor f32 = 40.000000 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 27: deepseek2.rope.scaling.original_context_length u32 = 4096 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 28: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 29: tokenizer.ggml.model str = gpt2 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 30: tokenizer.ggml.pre str = deepseek-v3 Jan 26 20:53:17 zeus-desktop ollama[88558]: time=2025-01-26T20:53:17.127-05:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server loading model" Jan 26 20:53:17 zeus-desktop ollama[88558]: [132B blob data] Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 32: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 33: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e... Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 34: tokenizer.ggml.bos_token_id u32 = 0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 35: tokenizer.ggml.eos_token_id u32 = 1 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 36: tokenizer.ggml.padding_token_id u32 = 1 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = true Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 38: tokenizer.ggml.add_eos_token bool = false Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 39: tokenizer.chat_template str = {% if not add_generation_prompt is de... Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 40: general.quantization_version u32 = 2 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - kv 41: general.file_type u32 = 15 Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - type f32: 361 tensors Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - type q4_K: 606 tensors Jan 26 20:53:17 zeus-desktop ollama[88558]: llama_model_loader: - type q6_K: 58 tensors Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_vocab: special tokens cache size = 818 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_vocab: token to piece cache size = 0.8223 MB Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: format = GGUF V3 (latest) Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: arch = deepseek2 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: vocab type = BPE Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_vocab = 129280 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_merges = 127741 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: vocab_only = 0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_ctx_train = 163840 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_embd = 7168 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_layer = 61 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_head = 128 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_head_kv = 128 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_rot = 64 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_swa = 0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_embd_head_k = 192 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_embd_head_v = 128 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_gqa = 1 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_embd_k_gqa = 24576 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_embd_v_gqa = 16384 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: f_norm_eps = 0.0e+00 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: f_logit_scale = 0.0e+00 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_ff = 18432 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_expert = 256 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_expert_used = 8 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: causal attn = 1 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: pooling type = 0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: rope type = 0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: rope scaling = yarn Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: freq_base_train = 10000.0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: freq_scale_train = 0.025 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_ctx_orig_yarn = 4096 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: rope_finetuned = unknown Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: ssm_d_conv = 0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: ssm_d_inner = 0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: ssm_d_state = 0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: ssm_dt_rank = 0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: ssm_dt_b_c_rms = 0 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: model type = 671B Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: model ftype = Q4_K - Medium Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: model params = 671.03 B Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: model size = 376.65 GiB (4.82 BPW) Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: general.name = n/a Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>' Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>' Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: EOT token = 1 '<｜end▁of▁sentence｜>' Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: PAD token = 1 '<｜end▁of▁sentence｜>' Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: LF token = 131 'Ä' Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: FIM PRE token = 128801 '<｜fim▁begin｜>' Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: FIM SUF token = 128800 '<｜fim▁hole｜>' Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: FIM MID token = 128802 '<｜fim▁end｜>' Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: EOG token = 1 '<｜end▁of▁sentence｜>' Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: max token length = 256 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_layer_dense_lead = 3 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_lora_q = 1536 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_lora_kv = 512 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_ff_exp = 2048 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: n_expert_shared = 1 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: expert_weights_scale = 2.5 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: expert_weights_norm = 1 Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: expert_gating_func = sigmoid Jan 26 20:53:17 zeus-desktop ollama[88558]: llm_load_print_meta: rope_yarn_log_mul = 0.1000 Jan 26 20:54:01 zeus-desktop ollama[88558]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 25643.85 MiB on device 0: cudaMalloc failed: out of memory Jan 26 20:54:21 zeus-desktop ollama[88558]: llama_model_load: error loading model: unable to allocate CUDA0 buffer Jan 26 20:54:21 zeus-desktop ollama[88558]: llama_load_model_from_file: failed to load model Jan 26 20:54:21 zeus-desktop ollama[88558]: panic: unable to load model: /usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 Jan 26 20:54:21 zeus-desktop ollama[88558]: goroutine 22 [running]: Jan 26 20:54:21 zeus-desktop ollama[88558]: github.com/ollama/ollama/llama/runner.(*Server).loadModel(0xc0001461b0, {0x21, 0x0, 0x1, 0x0, {0xc000124210, 0xa, 0xa}, 0xc000112270, 0x0}, ...) Jan 26 20:54:21 zeus-desktop ollama[88558]: github.com/ollama/ollama/llama/runner/runner.go:852 +0x3ad Jan 26 20:54:21 zeus-desktop ollama[88558]: created by github.com/ollama/ollama/llama/runner.Execute in goroutine 1 Jan 26 20:54:21 zeus-desktop ollama[88558]: github.com/ollama/ollama/llama/runner/runner.go:970 +0xd0d Jan 26 20:54:21 zeus-desktop ollama[88558]: time=2025-01-26T20:54:21.986-05:00 level=INFO source=server.go:589 msg="waiting for server to become available" status="llm server error" Jan 26 20:54:22 zeus-desktop ollama[88558]: time=2025-01-26T20:54:22.738-05:00 level=ERROR source=sched.go:455 msg="error loading llama server" error="llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer\nllama_load_model_from_file: failed to load model" Jan 26 20:54:22 zeus-desktop ollama[88558]: [GIN] 2025/01/26 - 20:54:22 | 500 | 1m10s | 127.0.0.1 | POST "/api/generate" Jan 26 20:54:28 zeus-desktop ollama[88558]: time=2025-01-26T20:54:28.040-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.301107748 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 Jan 26 20:54:29 zeus-desktop ollama[88558]: time=2025-01-26T20:54:29.670-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=6.931209137 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 Jan 26 20:54:31 zeus-desktop ollama[88558]: time=2025-01-26T20:54:31.305-05:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=8.566103296 model=/usr/share/ollama/.ollama/models/blobs/sha256-9801e7fce27dbf3d0bfb468b7b21f1d132131a546dfc43e50518631b8b1800a9 ```
GiteaMirror commented 2026-04-12 16:49:08 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 27, 2025):

https://github.com/ollama/ollama/issues/8597#issuecomment-2614533288

<!-- gh-comment-id:2614795820 --> @rick-github commented on GitHub (Jan 27, 2025): https://github.com/ollama/ollama/issues/8597#issuecomment-2614533288
GiteaMirror commented 2026-04-12 16:49:08 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Jan 27, 2025):

It looks that even 8 x L40S (360GB VRAM) would not be sufficient to load the model.. so there would be spillover into system RAM regardless.

<!-- gh-comment-id:2615300055 --> @orlyandico commented on GitHub (Jan 27, 2025): It looks that even 8 x L40S (360GB VRAM) would not be sufficient to load the model.. so there would be spillover into system RAM regardless.
GiteaMirror commented 2026-04-12 16:49:09 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 27, 2025):

Correct. If you wanted to run deepseek-r1 on 8 x L40S you would need to use a smaller quant.

<!-- gh-comment-id:2615309141 --> @rick-github commented on GitHub (Jan 27, 2025): Correct. If you wanted to run deepseek-r1 on 8 x L40S you would need to use a smaller quant.
GiteaMirror commented 2026-04-12 16:49:09 -05:00
Author
Owner
Copy Link

@js-2024 commented on GitHub (Jan 27, 2025):

Quick update. Following the guidance rick-github provided, I'm able to get it running on my local server.

Specifically, I used:

$ ollama show --modelfile deepseek-r1:671b | sed -e 's/^FROM.*/FROM deepseek-r1:671b/' > Modelfile
Edit Modelfile with a text editor
Added PARAMETER num_gpu 20 to Modelfile to force number of GPU offload layers to 20 (2 per 24GB 3090)
Saved Modelfile
$ ollama create deepseek.r1:20gpu
$ ollama run deepseek.r1:20gpu

This runs the model using ~18GB of each GPU's 24GB (on the default 2048 context).

<!-- gh-comment-id:2615684379 --> @js-2024 commented on GitHub (Jan 27, 2025): Quick update. Following the guidance rick-github provided, I'm able to get it running on my local server. Specifically, I used: $ ollama show --modelfile deepseek-r1:671b | sed -e 's/^FROM.*/FROM deepseek-r1:671b/' > Modelfile Edit Modelfile with a text editor Added PARAMETER num_gpu 20 to Modelfile to force number of GPU offload layers to 20 (2 per 24GB 3090) Saved Modelfile $ ollama create deepseek.r1:20gpu $ ollama run deepseek.r1:20gpu This runs the model using ~18GB of each GPU's 24GB (on the default 2048 context).
GiteaMirror commented 2026-04-12 16:49:10 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Jan 27, 2025):

What sort of tokens/second are you getting with 180GB on the GPU and the rest on the CPU?

<!-- gh-comment-id:2616079064 --> @orlyandico commented on GitHub (Jan 27, 2025): What sort of tokens/second are you getting with 180GB on the GPU and the rest on the CPU?
GiteaMirror commented 2026-04-12 16:49:10 -05:00
Author
Owner
Copy Link

@js-2024 commented on GitHub (Jan 27, 2025):

What sort of tokens/second are you getting with 180GB on the GPU and the rest on the CPU?

I didn't measure it at the time, and am in my work day so can't run it again right now to get a real measurement, but it felt like it was in the 3-5 t/s range based on the test generations. Please note that this was with basically no context (and the max context set to 2048).

<!-- gh-comment-id:2616089997 --> @js-2024 commented on GitHub (Jan 27, 2025): > What sort of tokens/second are you getting with 180GB on the GPU and the rest on the CPU? I didn't measure it at the time, and am in my work day so can't run it again right now to get a real measurement, but it felt like it was in the 3-5 t/s range based on the test generations. Please note that this was with basically no context (and the max context set to 2048).
GiteaMirror commented 2026-04-12 16:49:11 -05:00
Author
Owner
Copy Link

@SeekPoint commented on GitHub (Jan 28, 2025):

Quick update. Following the guidance rick-github provided, I'm able to get it running on my local server.

Specifically, I used:

ollama show --modelfile deepseek-r1:671b | sed -e 's/^FROM.*/FROM deepseek-r1:671b/' > Modelfile Edit Modelfile with a text editor Added PARAMETER num_gpu 20 to Modelfile to force number of GPU offload layers to 20 (2 per 24GB 3090) Saved Modelfile ollama create deepseek.r1:20gpu $ ollama run deepseek.r1:20gpu

This runs the model using ~18GB of each GPU's 24GB (on the default 2048 context).

I doesn't work on my server with 22080ti22GB+512GBmem

<!-- gh-comment-id:2617626797 --> @SeekPoint commented on GitHub (Jan 28, 2025): > Quick update. Following the guidance rick-github provided, I'm able to get it running on my local server. > > Specifically, I used: > > $ ollama show --modelfile deepseek-r1:671b | sed -e 's/^FROM.*/FROM deepseek-r1:671b/' > Modelfile Edit Modelfile with a text editor Added PARAMETER num_gpu 20 to Modelfile to force number of GPU offload layers to 20 (2 per 24GB 3090) Saved Modelfile $ ollama create deepseek.r1:20gpu $ ollama run deepseek.r1:20gpu > > This runs the model using ~18GB of each GPU's 24GB (on the default 2048 context). I doesn't work on my server with 2*2080ti*22GB+512GBmem
GiteaMirror commented 2026-04-12 16:49:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 28, 2025):

Server logs will aid in debugging.

<!-- gh-comment-id:2617639683 --> @rick-github commented on GitHub (Jan 28, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will aid in debugging.
GiteaMirror commented 2026-04-12 16:49:13 -05:00
Author
Owner
Copy Link

@vvidovic commented on GitHub (Jan 29, 2025):

4. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1

None of those worked for me, only changing the num_gpu approach worked:

$ OLLAMA_GPU_OVERHEAD=536870912 ollama run command-r7b:7b
Error: llama runner process has terminated: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 1531936768
llama_new_context_with_model: failed to allocate compute buffers

$ OLLAMA_FLASH_ATTENTION=1 ollama run command-r7b:7b
Error: llama runner process has terminated: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 1531936768
llama_new_context_with_model: failed to allocate compute buffers

$ GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ollama run command-r7b:7b
Error: llama runner process has terminated: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 1531936768
llama_new_context_with_model: failed to allocate compute buffers
<!-- gh-comment-id:2621937604 --> @vvidovic commented on GitHub (Jan 29, 2025): > 4\. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 None of those worked for me, only changing the `num_gpu` approach worked: ``` $ OLLAMA_GPU_OVERHEAD=536870912 ollama run command-r7b:7b Error: llama runner process has terminated: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 1531936768 llama_new_context_with_model: failed to allocate compute buffers $ OLLAMA_FLASH_ATTENTION=1 ollama run command-r7b:7b Error: llama runner process has terminated: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 1531936768 llama_new_context_with_model: failed to allocate compute buffers $ GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ollama run command-r7b:7b Error: llama runner process has terminated: cudaMalloc failed: out of memory ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 1531936768 llama_new_context_with_model: failed to allocate compute buffers ```
GiteaMirror commented 2026-04-12 16:49:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 29, 2025):

These variables need to be set in the server environment. OLLAMA_GPU_OVERHEAD=536870912 also just a suggestion, it needs to be adjusted per GPU/model. For example, command-7b:7b-12-2024-fp16 needs more: https://github.com/ollama/ollama/issues/8471#issuecomment-2604624681

<!-- gh-comment-id:2621949703 --> @rick-github commented on GitHub (Jan 29, 2025): These variables need to be set in the [server environment](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server). `OLLAMA_GPU_OVERHEAD=536870912` also just a suggestion, it needs to be adjusted per GPU/model. For example, command-7b:7b-12-2024-fp16 needs more: https://github.com/ollama/ollama/issues/8471#issuecomment-2604624681
GiteaMirror commented 2026-04-12 16:49:13 -05:00
Author
Owner
Copy Link

@vvidovic commented on GitHub (Jan 30, 2025):

These variables need to be set in the server environment. OLLAMA_GPU_OVERHEAD=536870912 also just a suggestion, it needs to be adjusted per GPU/model. For example, command-7b:7b-12-2024-fp16 needs more: #8471 (comment)

Thanks a lot for your help, it makes sense that environment variables for a client don't make any difference.

I did some testing and here are results for my machine.

  • the GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 seems to work best
  • reserving fixed size of memory (OLLAMA_GPU_OVERHEAD) works too but it doesn't seems as a good choice for my case because that approach would cause models that can fit in a GPU to be split between CPU and GPU
  • I didn't notice any difference when using OLLAMA_FLASH_ATTENTION=1

The only "downside" of GGML_CUDA_ENABLE_UNIFIED_MEMORY is that it seems that nvidia-smi reports "wrong" (much smaller) GPU usage by ollama, not sure how can that be. I did quite a few measured experiments and I didn't notice that this settings affect the speed of model inference. Here is the nvidia-smi output for comparison when running the ollama run granite3.1-moe:3b "Write 200 words about who you are." command:

# Using GGML_CUDA_ENABLE_UNIFIED_MEMORY
$ nvidia-smi
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      4937      G   /usr/lib/xorg/Xorg                          143MiB |
|    0   N/A  N/A    469496      C   ...rs/cuda_v12_avx/ollama_llama_server       96MiB |
+---------------------------------------------------------------------------------------+

# Without GGML_CUDA_ENABLE_UNIFIED_MEMORY
$ nvidia-smi
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      4937      G   /usr/lib/xorg/Xorg                          143MiB |
|    0   N/A  N/A    469601      C   ...rs/cuda_v12_avx/ollama_llama_server     2944MiB |
+---------------------------------------------------------------------------------------+

In both cases, ollama ps reports that everything is executed on GPU:

$ ollama ps
NAME                 ID              SIZE      PROCESSOR    UNTIL              
granite3.1-moe:3b    df6f6578dba8    3.4 GB    100% GPU     4 minutes from now

By the way, setting GGML_CUDA_ENABLE_UNIFIED_MEMORY to 1 or 0 results in the same behaviour.

<!-- gh-comment-id:2623767185 --> @vvidovic commented on GitHub (Jan 30, 2025): > These variables need to be set in the [server environment](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server). `OLLAMA_GPU_OVERHEAD=536870912` also just a suggestion, it needs to be adjusted per GPU/model. For example, command-7b:7b-12-2024-fp16 needs more: [#8471 (comment)](https://github.com/ollama/ollama/issues/8471#issuecomment-2604624681) Thanks a lot for your help, it makes sense that environment variables for a client don't make any difference. I did some testing and here are results for my machine. - the `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1` seems to work best - reserving fixed size of memory (`OLLAMA_GPU_OVERHEAD`) works too but it doesn't seems as a good choice for my case because that approach would cause models that can fit in a GPU to be split between CPU and GPU - I didn't notice any difference when using `OLLAMA_FLASH_ATTENTION=1` The only "downside" of `GGML_CUDA_ENABLE_UNIFIED_MEMORY` is that it seems that `nvidia-smi` reports "wrong" (much smaller) GPU usage by `ollama`, not sure how can that be. I did quite a few measured experiments and I didn't notice that this settings affect the speed of model inference. Here is the `nvidia-smi` output for comparison when running the `ollama run granite3.1-moe:3b "Write 200 words about who you are."` command: ``` # Using GGML_CUDA_ENABLE_UNIFIED_MEMORY $ nvidia-smi +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 4937 G /usr/lib/xorg/Xorg 143MiB | | 0 N/A N/A 469496 C ...rs/cuda_v12_avx/ollama_llama_server 96MiB | +---------------------------------------------------------------------------------------+ # Without GGML_CUDA_ENABLE_UNIFIED_MEMORY $ nvidia-smi +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 4937 G /usr/lib/xorg/Xorg 143MiB | | 0 N/A N/A 469601 C ...rs/cuda_v12_avx/ollama_llama_server 2944MiB | +---------------------------------------------------------------------------------------+ ``` In both cases, `ollama ps` reports that everything is executed on GPU: ``` $ ollama ps NAME ID SIZE PROCESSOR UNTIL granite3.1-moe:3b df6f6578dba8 3.4 GB 100% GPU 4 minutes from now ``` By the way, setting `GGML_CUDA_ENABLE_UNIFIED_MEMORY` to `1` or `0` results in the same behaviour.
GiteaMirror commented 2026-04-12 16:49:13 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Jan 30, 2025):

Setting server parameters cleared the error.

<!-- gh-comment-id:2625216217 --> @orlyandico commented on GitHub (Jan 30, 2025): Setting server parameters cleared the error.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5560
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?