github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #3078] Ollama is not using the 100% of RTX4000 VRAM (18 of 20GB) #1894

New Issue
Open
opened 2026-04-12 11:59:32 -05:00 by GiteaMirror · 29 comments
GiteaMirror commented 2026-04-12 11:59:32 -05:00
Owner
Copy Link

Originally created by @nfsecurity on GitHub (Mar 12, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3078

Originally assigned to: @mxyng on GitHub.

Hi, thank you for the wonderful ollama project and the amazing community!

Screenshot 2024-03-12 at 8 32 31 AM

I am testing the Mixtral 3Bit Quantized model under a RTX400 with 20GB of VRAM. The model is 20GB of size and as you can see in the screenshot of nvidia-smi, ollama is using only 18GB and the rest of the model was loaded to the system RAM.

Is this normal? or is it an issue?, Can I force ollama to use the 100% of VRAM?, thank you!!!

Originally created by @nfsecurity on GitHub (Mar 12, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3078 Originally assigned to: @mxyng on GitHub. Hi, thank you for the wonderful ollama project and the amazing community! <img width="742" alt="Screenshot 2024-03-12 at 8 32 31 AM" src="https://github.com/ollama/ollama/assets/16274031/a47d6ad9-3602-4ffe-984d-0ec858f95b6f"> I am testing the Mixtral 3Bit Quantized model under a RTX400 with 20GB of VRAM. The model is 20GB of size and as you can see in the screenshot of nvidia-smi, ollama is using only 18GB and the rest of the model was loaded to the system RAM. Is this normal? or is it an issue?, Can I force ollama to use the 100% of VRAM?, thank you!!!
GiteaMirror added the performancenvidia labels 2026-04-12 11:59:33 -05:00
GiteaMirror commented 2026-04-12 11:59:34 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Mar 12, 2024):

Adding my report here, seems to be a similar issue.

I'm getting less than 1 token per second with 2x P40 and Smaug-72B-v0.1-q4_k_m.gguf quantised model from HuggingFace (4.84 bpw). CPU is at 400%, GPU's hover at 20-40% CPU utilisation, log says only 65 of 81 layers are offloaded to the GPU; the model is 40GB in size, 16GB on each GPU is used for the model and 2GB for the KV cache, total of 18GB VRAM per GPU verified by nvidia-smi. Total of 36GB, but I have 48GB in total. I figure if all of the 81 layers were on the GPU, this would use 20GB of VRAM, leaving 4GB for KV cache.

Logs:

Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: format           = GGUF V3 (latest)
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: arch             = llama
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: vocab type       = BPE
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_vocab          = 152064
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_merges         = 151387
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_ctx_train      = 32768
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_embd           = 8192
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_head           = 64
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_head_kv        = 64
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_layer          = 80
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_rot            = 128
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_embd_head_k    = 128
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_embd_head_v    = 128
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_gqa            = 1
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_embd_k_gqa     = 8192
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_embd_v_gqa     = 8192
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_ff             = 24576
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_expert         = 0
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_expert_used    = 0
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: pooling type     = 0
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: rope type        = 0
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: rope scaling     = linear
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: freq_base_train  = 1000000.0
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: freq_scale_train = 1
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_yarn_orig_ctx  = 32768
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: rope_finetuned   = unknown
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: model type       = 65B
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: model ftype      = Q4_K - Medium
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: model params     = 72.29 B
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: model size       = 40.76 GiB (4.84 BPW)
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: general.name     = snapshots
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: EOS token        = 151643 '<|endoftext|>'
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: UNK token        = 151643 '<|endoftext|>'
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: LF token         = 148848 'ÄĬ'
Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_tensors: ggml ctx size =    1.19 MiB
Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: offloading 65 repeating layers to GPU
Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: offloaded 65/81 layers to GPU
Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors:        CPU buffer size = 41737.81 MiB
Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors:      CUDA0 buffer size = 16176.19 MiB
Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors:      CUDA1 buffer size = 16170.00 MiB

nvidia-smi output:

Every 1.0s: nvidia-smi                         ThinkStation-S30: Tue Mar 12 17:19:48 2024

Tue Mar 12 17:19:48 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla P40                      On  | 00000000:02:00.0 Off |                  Off |
| N/A   63C    P0              54W / 170W |  18900MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla P40                      On  | 00000000:06:00.0 Off |                  Off |
| N/A   63C    P0              52W / 170W |  18828MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A     16976      C   /usr/local/bin/ollama                     18896MiB |
|    1   N/A  N/A     16976      C   /usr/local/bin/ollama                     18824MiB |
+---------------------------------------------------------------------------------------+
<!-- gh-comment-id:1992178758 --> @orlyandico commented on GitHub (Mar 12, 2024): Adding my report here, seems to be a similar issue. I'm getting less than 1 token per second with 2x P40 and `Smaug-72B-v0.1-q4_k_m.gguf` quantised model from HuggingFace (4.84 bpw). CPU is at 400%, GPU's hover at 20-40% CPU utilisation, log says only 65 of 81 layers are offloaded to the GPU; the model is 40GB in size, 16GB on each GPU is used for the model and 2GB for the KV cache, total of 18GB VRAM per GPU verified by nvidia-smi. Total of 36GB, but I have 48GB in total. I figure if all of the 81 layers were on the GPU, this would use 20GB of VRAM, leaving 4GB for KV cache. Logs: ``` Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: format = GGUF V3 (latest) Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: arch = llama Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: vocab type = BPE Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_vocab = 152064 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_merges = 151387 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_ctx_train = 32768 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_embd = 8192 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_head = 64 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_head_kv = 64 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_layer = 80 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_rot = 128 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_embd_head_k = 128 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_embd_head_v = 128 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_gqa = 1 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_embd_k_gqa = 8192 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_embd_v_gqa = 8192 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: f_norm_eps = 0.0e+00 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_ff = 24576 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_expert = 0 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_expert_used = 0 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: pooling type = 0 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: rope type = 0 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: rope scaling = linear Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: freq_base_train = 1000000.0 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: freq_scale_train = 1 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: n_yarn_orig_ctx = 32768 Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: rope_finetuned = unknown Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: model type = 65B Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: model ftype = Q4_K - Medium Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: model params = 72.29 B Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: model size = 40.76 GiB (4.84 BPW) Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: general.name = snapshots Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: BOS token = 151643 '<|endoftext|>' Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: EOS token = 151643 '<|endoftext|>' Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: UNK token = 151643 '<|endoftext|>' Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: PAD token = 151643 '<|endoftext|>' Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_print_meta: LF token = 148848 'ÄĬ' Mar 12 17:18:09 ThinkStation-S30 ollama[16976]: llm_load_tensors: ggml ctx size = 1.19 MiB Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: offloading 65 repeating layers to GPU Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: offloaded 65/81 layers to GPU Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: CPU buffer size = 41737.81 MiB Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: CUDA0 buffer size = 16176.19 MiB Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: CUDA1 buffer size = 16170.00 MiB ``` nvidia-smi output: ``` Every 1.0s: nvidia-smi ThinkStation-S30: Tue Mar 12 17:19:48 2024 Tue Mar 12 17:19:48 2024 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 Tesla P40 On | 00000000:02:00.0 Off | Off | | N/A 63C P0 54W / 170W | 18900MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ | 1 Tesla P40 On | 00000000:06:00.0 Off | Off | | N/A 63C P0 52W / 170W | 18828MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 16976 C /usr/local/bin/ollama 18896MiB | | 1 N/A N/A 16976 C /usr/local/bin/ollama 18824MiB | +---------------------------------------------------------------------------------------+ ```
GiteaMirror commented 2026-04-12 11:59:36 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Mar 12, 2024):

The more-aggressively quantised version Smaug-72B-v0.1-q2_k.gguf loads entirely on the GPU, runs at 100% GPU and 40-60% GPU compute on each of the two P40's.

It seems that the rounding logic for determining how many layers to put on the GPU is excessively conservative, but I can't find it in the source code...

Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: format           = GGUF V3 (latest)
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: arch             = llama
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: vocab type       = BPE
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_vocab          = 152064
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_merges         = 151387
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_ctx_train      = 32768
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_embd           = 8192
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_head           = 64
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_head_kv        = 64
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_layer          = 80
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_rot            = 128
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_embd_head_k    = 128
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_embd_head_v    = 128
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_gqa            = 1
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_embd_k_gqa     = 8192
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_embd_v_gqa     = 8192
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_ff             = 24576
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_expert         = 0
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_expert_used    = 0
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: pooling type     = 0
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: rope type        = 0
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: rope scaling     = linear
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: freq_base_train  = 1000000.0
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: freq_scale_train = 1
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_yarn_orig_ctx  = 32768
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: rope_finetuned   = unknown
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: model type       = 65B
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: model ftype      = Q2_K - Medium
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: model params     = 72.29 B
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: model size       = 28.26 GiB (3.36 BPW)
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: general.name     = snapshots
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: EOS token        = 151643 '<|endoftext|>'
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: UNK token        = 151643 '<|endoftext|>'
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: LF token         = 148848 'ÄĬ'
Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_tensors: ggml ctx size =    1.19 MiB
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloading 80 repeating layers to GPU
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloading non-repeating layers to GPU
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloaded 81/81 layers to GPU
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors:        CPU buffer size =   389.81 MiB
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors:      CUDA0 buffer size = 14132.19 MiB
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors:      CUDA1 buffer size = 14417.38 MiB
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: ...................................................................>
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: n_ctx      = 2048
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: freq_base  = 1000000.0
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: freq_scale = 1
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   yes
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: ggml_init_cublas: found 2 CUDA devices:
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]:   Device 0: Tesla P40, compute capability 6.1, VMM: yes
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]:   Device 1: Tesla P40, compute capability 6.1, VMM: yes
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_kv_cache_init:      CUDA0 KV buffer size =  2624.00 MiB
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_kv_cache_init:      CUDA1 KV buffer size =  2496.00 MiB
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: KV self size  = 5120.00 MiB, K (f16):>
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model:  CUDA_Host input buffer size   =    2>
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model:      CUDA0 compute buffer size =   32>
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model:      CUDA1 compute buffer size =   32>
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model:  CUDA_Host compute buffer size =    1>
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: graph splits (measure): 3
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: loading library /tmp/ollama506445724/cuda_v11/libext_server.so
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: {"function":"initialize","level":"INFO","line":433,"msg":"initializ>
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: {"function":"initialize","level":"INFO","line":442,"msg":"new slot">
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: time=2024-03-12T20:39:38.488Z level=INFO source=dyn_ext_server.go:1>
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: {"function":"update_slots","level":"INFO","line":1565,"msg":"all sl>
Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: [GIN] 2024/03/12 - 20:39:38 | 200 |         2m27s |       127.0.0.1>
Mar 12 20:39:58 ThinkStation-S30 ollama[43824]: {"function":"launch_slot_with_data","level":"INFO","line":823,"msg">
Mar 12 20:39:58 ThinkStation-S30 ollama[43824]: {"function":"update_slots","level":"INFO","line":1796,"msg":"slot p>
Mar 12 20:39:58 ThinkStation-S30 ollama[43824]: {"function":"update_slots","level":"INFO","line":1821,"msg":"kv cac>
Mar 12 20:42:31 ThinkStation-S30 ollama[43824]: {"function":"print_timings","level":"INFO","line":257,"msg":"prompt>
Mar 12 20:42:31 ThinkStation-S30 ollama[43824]: {"function":"print_timings","level":"INFO","line":271,"msg":"genera>
Mar 12 20:42:31 ThinkStation-S30 ollama[43824]: {"function":"print_timings","level":"INFO","line":281,"msg":"      >
Mar 12 20:42:31 ThinkStation-S30 ollama[43824]: {"function":"update_slots","level":"INFO","line":1627,"msg":"slot r>
Mar 12 20:42:31 ThinkStation-S30 ollama[43824]: [GIN] 2024/03/12 - 20:42:31 | 200 |         2m33s |       127.0.0.1>
<!-- gh-comment-id:1992545769 --> @orlyandico commented on GitHub (Mar 12, 2024): The more-aggressively quantised version `Smaug-72B-v0.1-q2_k.gguf` loads entirely on the GPU, runs at 100% GPU and 40-60% GPU compute on each of the two P40's. It seems that the rounding logic for determining how many layers to put on the GPU is excessively conservative, but I can't find it in the source code... ``` Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: format = GGUF V3 (latest) Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: arch = llama Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: vocab type = BPE Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_vocab = 152064 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_merges = 151387 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_ctx_train = 32768 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_embd = 8192 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_head = 64 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_head_kv = 64 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_layer = 80 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_rot = 128 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_embd_head_k = 128 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_embd_head_v = 128 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_gqa = 1 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_embd_k_gqa = 8192 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_embd_v_gqa = 8192 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: f_norm_eps = 0.0e+00 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_ff = 24576 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_expert = 0 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_expert_used = 0 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: pooling type = 0 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: rope type = 0 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: rope scaling = linear Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: freq_base_train = 1000000.0 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: freq_scale_train = 1 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: n_yarn_orig_ctx = 32768 Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: rope_finetuned = unknown Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: model type = 65B Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: model ftype = Q2_K - Medium Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: model params = 72.29 B Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: model size = 28.26 GiB (3.36 BPW) Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: general.name = snapshots Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: BOS token = 151643 '<|endoftext|>' Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: EOS token = 151643 '<|endoftext|>' Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: UNK token = 151643 '<|endoftext|>' Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: PAD token = 151643 '<|endoftext|>' Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_print_meta: LF token = 148848 'ÄĬ' Mar 12 20:37:13 ThinkStation-S30 ollama[43824]: llm_load_tensors: ggml ctx size = 1.19 MiB Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloading 80 repeating layers to GPU Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloading non-repeating layers to GPU Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloaded 81/81 layers to GPU Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: CPU buffer size = 389.81 MiB Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: CUDA0 buffer size = 14132.19 MiB Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: CUDA1 buffer size = 14417.38 MiB Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: ...................................................................> Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: n_ctx = 2048 Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: freq_base = 1000000.0 Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: freq_scale = 1 Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: ggml_init_cublas: CUDA_USE_TENSOR_CORES: no Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: ggml_init_cublas: found 2 CUDA devices: Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: Device 0: Tesla P40, compute capability 6.1, VMM: yes Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: Device 1: Tesla P40, compute capability 6.1, VMM: yes Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_kv_cache_init: CUDA0 KV buffer size = 2624.00 MiB Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_kv_cache_init: CUDA1 KV buffer size = 2496.00 MiB Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: KV self size = 5120.00 MiB, K (f16):> Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: CUDA_Host input buffer size = 2> Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: CUDA0 compute buffer size = 32> Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: CUDA1 compute buffer size = 32> Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: CUDA_Host compute buffer size = 1> Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: llama_new_context_with_model: graph splits (measure): 3 Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: loading library /tmp/ollama506445724/cuda_v11/libext_server.so Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: {"function":"initialize","level":"INFO","line":433,"msg":"initializ> Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: {"function":"initialize","level":"INFO","line":442,"msg":"new slot"> Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: time=2024-03-12T20:39:38.488Z level=INFO source=dyn_ext_server.go:1> Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: {"function":"update_slots","level":"INFO","line":1565,"msg":"all sl> Mar 12 20:39:38 ThinkStation-S30 ollama[43824]: [GIN] 2024/03/12 - 20:39:38 | 200 | 2m27s | 127.0.0.1> Mar 12 20:39:58 ThinkStation-S30 ollama[43824]: {"function":"launch_slot_with_data","level":"INFO","line":823,"msg"> Mar 12 20:39:58 ThinkStation-S30 ollama[43824]: {"function":"update_slots","level":"INFO","line":1796,"msg":"slot p> Mar 12 20:39:58 ThinkStation-S30 ollama[43824]: {"function":"update_slots","level":"INFO","line":1821,"msg":"kv cac> Mar 12 20:42:31 ThinkStation-S30 ollama[43824]: {"function":"print_timings","level":"INFO","line":257,"msg":"prompt> Mar 12 20:42:31 ThinkStation-S30 ollama[43824]: {"function":"print_timings","level":"INFO","line":271,"msg":"genera> Mar 12 20:42:31 ThinkStation-S30 ollama[43824]: {"function":"print_timings","level":"INFO","line":281,"msg":" > Mar 12 20:42:31 ThinkStation-S30 ollama[43824]: {"function":"update_slots","level":"INFO","line":1627,"msg":"slot r> Mar 12 20:42:31 ThinkStation-S30 ollama[43824]: [GIN] 2024/03/12 - 20:42:31 | 200 | 2m33s | 127.0.0.1> ```
GiteaMirror commented 2026-04-12 11:59:37 -05:00
Author
Owner
Copy Link

@AdaptiveStep commented on GitHub (Mar 13, 2024):

I wish if it could say "Using only CPU" or "Using only GPU" . Or "Slow inference detected" .. or stuff like that. We are not supposed to discover these things by accident.

<!-- gh-comment-id:1994059128 --> @AdaptiveStep commented on GitHub (Mar 13, 2024): I wish if it could say "Using only CPU" or "Using only GPU" . Or "Slow inference detected" .. or stuff like that. We are not supposed to discover these things by accident.
GiteaMirror commented 2026-04-12 11:59:38 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Mar 13, 2024):

This part tells us the model is running entirely (or almost entirely) on the GPU:

Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloading 80 repeating layers to GPU
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloading non-repeating layers to GPU
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloaded 81/81 layers to GPU
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors:        CPU buffer size =   389.81 MiB
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors:      CUDA0 buffer size = 14132.19 MiB
Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors:      CUDA1 buffer size = 14417.38 MiB

And this disaster is telling us that at least some of the layers are evaluating on the CPU. Interestingly even though 32GB of the 40GB model is loaded on the GPU's, the entire model is also loaded into host CPU RAM...

Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: offloading 65 repeating layers to GPU
Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: offloaded 65/81 layers to GPU
Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors:        CPU buffer size = 41737.81 MiB
Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors:      CUDA0 buffer size = 16176.19 MiB
Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors:      CUDA1 buffer size = 16170.00 MiB
<!-- gh-comment-id:1994097497 --> @orlyandico commented on GitHub (Mar 13, 2024): This part tells us the model is running entirely (or almost entirely) on the GPU: ``` Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloading 80 repeating layers to GPU Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloading non-repeating layers to GPU Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: offloaded 81/81 layers to GPU Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: CPU buffer size = 389.81 MiB Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: CUDA0 buffer size = 14132.19 MiB Mar 12 20:39:31 ThinkStation-S30 ollama[43824]: llm_load_tensors: CUDA1 buffer size = 14417.38 MiB ``` And this disaster is telling us that at least some of the layers are evaluating on the CPU. Interestingly even though 32GB of the 40GB model is loaded on the GPU's, the **entire** model is also loaded into host CPU RAM... ``` Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: offloading 65 repeating layers to GPU Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: offloaded 65/81 layers to GPU Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: CPU buffer size = 41737.81 MiB Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: CUDA0 buffer size = 16176.19 MiB Mar 12 17:18:11 ThinkStation-S30 ollama[16976]: llm_load_tensors: CUDA1 buffer size = 16170.00 MiB ```
GiteaMirror commented 2026-04-12 11:59:38 -05:00
Author
Owner
Copy Link

@nfsecurity commented on GitHub (Mar 18, 2024):

So, is it a bug in the VRAM memory allocation or is it the expected behavior ?

<!-- gh-comment-id:2002874618 --> @nfsecurity commented on GitHub (Mar 18, 2024): So, is it a bug in the VRAM memory allocation or is it the expected behavior ?
GiteaMirror commented 2026-04-12 11:59:39 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Mar 18, 2024):

I was able to fix this by RTFM.

You can set the num_gpu parameter to force a certain number of layers onto the GPU (s).

For Smaug-72B 4-bit quantised:

orly@ThinkStation-S30:~$ ollama run smaug-72b:latest
>>> /set parameter num_gpu 81
Set parameter 'num_gpu' to '81'

If you have the GGUF file you can also add the number of GPU layers to the modelfile before importing it into Ollama, e.g.

FROM ./Smaug-72B-v0.1-q4_k_m.gguf 
PARAMETER num_gpu 81
TEMPLATE "[INST] {{ .Prompt }} [/INST]"

(I knew from the Ollama logs that this particular GGUF had 81 layers)

What I noticed is that with Smaug-72b 4-bit quantised (4.85bpw) and all of the 81 layers on the GPU, each of my P40's has less than 1GB of free RAM when inferencing. So things (barely) fit. I am guessing the Ollama default estimation algorithm is overly conservative because you will get a crash if you OOM the GPU(s). You can try setting the parameter manually on your setup/model and see if it works.

<!-- gh-comment-id:2004682607 --> @orlyandico commented on GitHub (Mar 18, 2024): I was able to fix this by RTFM. You can set the num_gpu parameter to force a certain number of layers onto the GPU (s). For Smaug-72B 4-bit quantised: ``` orly@ThinkStation-S30:~$ ollama run smaug-72b:latest >>> /set parameter num_gpu 81 Set parameter 'num_gpu' to '81' ``` If you have the GGUF file you can also add the number of GPU layers to the modelfile before importing it into Ollama, e.g. ``` FROM ./Smaug-72B-v0.1-q4_k_m.gguf PARAMETER num_gpu 81 TEMPLATE "[INST] {{ .Prompt }} [/INST]" ``` (I knew from the Ollama logs that this particular GGUF had 81 layers) What I noticed is that with Smaug-72b 4-bit quantised (4.85bpw) and all of the 81 layers on the GPU, each of my P40's has less than 1GB of free RAM when inferencing. So things (barely) fit. I am guessing the Ollama default estimation algorithm is overly conservative because you will get a crash if you OOM the GPU(s). You can try setting the parameter manually on your setup/model and see if it works.
GiteaMirror commented 2026-04-12 11:59:41 -05:00
Author
Owner
Copy Link

@nfsecurity commented on GitHub (Mar 19, 2024):

Thank you @orlyandico, I was able to increase by 2 the number of layers deployed in the VRAM GPU using Mixtral 3BitQ (from 29/33 to 31/33). I gained some performance in tokens per second. Now, ollama process is using more memory:

Screenshot 2024-03-18 at 7 25 22 PM

Your advice about OOM is very important, we have to monitor carefully the memory usage.

<!-- gh-comment-id:2005460669 --> @nfsecurity commented on GitHub (Mar 19, 2024): Thank you @orlyandico, I was able to increase by 2 the number of layers deployed in the VRAM GPU using Mixtral 3BitQ (from 29/33 to 31/33). I gained some performance in tokens per second. Now, ollama process is using more memory: <img width="741" alt="Screenshot 2024-03-18 at 7 25 22 PM" src="https://github.com/ollama/ollama/assets/16274031/408f26fd-848c-4cbc-b497-71a644285c2f"> Your advice about OOM is very important, we have to monitor carefully the memory usage.
GiteaMirror commented 2026-04-12 11:59:42 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Mar 19, 2024):

You won't get the full benefit of GPU unless all the layers are on the GPU. You might be better off using a slightly more quantized model e.g. 3bpw instead of 4bpw, so everything can fit on the GPU. But since you're already using a 3bpw model... probably not a great idea.

<!-- gh-comment-id:2005522386 --> @orlyandico commented on GitHub (Mar 19, 2024): You won't get the full benefit of GPU unless **all** the layers are on the GPU. You might be better off using a slightly more quantized model e.g. 3bpw instead of 4bpw, so everything can fit on the GPU. But since you're already using a 3bpw model... probably not a great idea.
GiteaMirror commented 2026-04-12 11:59:43 -05:00
Author
Owner
Copy Link

@strikeoncmputrz commented on GitHub (Apr 12, 2024):

@orlyandico did you just guess on the number of layers to offload after seeing 81 total in Ollama logs, or is there some algorithm one can apply to estimate? I've stuck with exl2 and smaller models because I don't want to futz around with guessing num_layers / num_gpu and cranking my CPUs up to 100% repeatedly while testing but Mixtral8x22 has me reconsidering offloading some layers to CPU :-)

<!-- gh-comment-id:2050813641 --> @strikeoncmputrz commented on GitHub (Apr 12, 2024): @orlyandico did you just guess on the number of layers to offload after seeing 81 total in Ollama logs, or is there some algorithm one can apply to estimate? I've stuck with exl2 and smaller models because I don't want to futz around with guessing num_layers / num_gpu and cranking my CPUs up to 100% repeatedly while testing but Mixtral8x22 has me reconsidering offloading some layers to CPU :-)
GiteaMirror commented 2026-04-12 11:59:44 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Apr 12, 2024):

the logs display the number of layers.

it was just a guess on my part that all the layers fit.

incidentally i tried to do the same thing on an A6000-ada (rented).
smaug-72b doesn’t fit. the A6000-ada 48GB has 45GB of memory reported by
Nvidia-SMI. I guess because it has a video output so some VRAM is needed by
the framebuffer.

On Fri, 12 Apr 2024 at 02:38, strikeoncmputrz @.***>
wrote:

@orlyandico https://github.com/orlyandico did you just guess on the
number of layers or is there some algorithm one can apply to estimate? I've
stuck with exl2 and smaller models because I don't want to futz around with
guessing num_layers / num_gpu and cranking my CPUs up to 100% repeatedly
while testing but Mixtral8x22 has me reconsidering offloading some layers
to CPU :-)

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/3078#issuecomment-2050813641,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKDS3EWAHCWDOPKEKP3NYLY443JLAVCNFSM6AAAAABESINW5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJQHAYTGNRUGE
.
You are receiving this because you were mentioned.Message ID:
@.***>

<!-- gh-comment-id:2051130190 --> @orlyandico commented on GitHub (Apr 12, 2024): the logs display the number of layers. it was just a guess on my part that all the layers fit. incidentally i tried to do the same thing on an A6000-ada (rented). smaug-72b doesn’t fit. the A6000-ada 48GB has *45GB* of memory reported by Nvidia-SMI. I guess because it has a video output so some VRAM is needed by the framebuffer. On Fri, 12 Apr 2024 at 02:38, strikeoncmputrz ***@***.***> wrote: > @orlyandico <https://github.com/orlyandico> did you just guess on the > number of layers or is there some algorithm one can apply to estimate? I've > stuck with exl2 and smaller models because I don't want to futz around with > guessing num_layers / num_gpu and cranking my CPUs up to 100% repeatedly > while testing but Mixtral8x22 has me reconsidering offloading some layers > to CPU :-) > > — > Reply to this email directly, view it on GitHub > <https://github.com/ollama/ollama/issues/3078#issuecomment-2050813641>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAKDS3EWAHCWDOPKEKP3NYLY443JLAVCNFSM6AAAAABESINW5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJQHAYTGNRUGE> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >
GiteaMirror commented 2026-04-12 11:59:46 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Apr 12, 2024):

The logs showed the number of layers loaded on the GPUs, and nvidia-smi
displayed the VRAM consumption. The logs also displayed how much memory was
used for KV cache.

So, take VRAM, subtract KV cache, and what’s left is what the model took.
By looking at the # of layers loaded and VRAM used, I extrapolated that all
81 would still fit.

On the A6000-Ada, all layers actually fit in the 45GB VRAM…. but there’s
nothing left for KV cache! so ironically Smaug-72b, Qwen-72b 4bpw fit on
2x24GB P40, but do NOT fit on 1x 48GB A6000-Ada….

On Fri, 12 Apr 2024 at 02:38, strikeoncmputrz @.***>
wrote:

@orlyandico https://github.com/orlyandico did you just guess on the
number of layers or is there some algorithm one can apply to estimate? I've
stuck with exl2 and smaller models because I don't want to futz around with
guessing num_layers / num_gpu and cranking my CPUs up to 100% repeatedly
while testing but Mixtral8x22 has me reconsidering offloading some layers
to CPU :-)

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/3078#issuecomment-2050813641,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKDS3EWAHCWDOPKEKP3NYLY443JLAVCNFSM6AAAAABESINW5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJQHAYTGNRUGE
.
You are receiving this because you were mentioned.Message ID:
@.***>

<!-- gh-comment-id:2051402477 --> @orlyandico commented on GitHub (Apr 12, 2024): The logs showed the number of layers loaded on the GPUs, and nvidia-smi displayed the VRAM consumption. The logs also displayed how much memory was used for KV cache. So, take VRAM, subtract KV cache, and what’s left is what the model took. By looking at the # of layers loaded and VRAM used, I extrapolated that all 81 would still fit. On the A6000-Ada, all layers actually fit in the 45GB VRAM…. but there’s nothing left for KV cache! so ironically Smaug-72b, Qwen-72b 4bpw fit on 2x24GB P40, but do NOT fit on 1x 48GB A6000-Ada…. On Fri, 12 Apr 2024 at 02:38, strikeoncmputrz ***@***.***> wrote: > @orlyandico <https://github.com/orlyandico> did you just guess on the > number of layers or is there some algorithm one can apply to estimate? I've > stuck with exl2 and smaller models because I don't want to futz around with > guessing num_layers / num_gpu and cranking my CPUs up to 100% repeatedly > while testing but Mixtral8x22 has me reconsidering offloading some layers > to CPU :-) > > — > Reply to this email directly, view it on GitHub > <https://github.com/ollama/ollama/issues/3078#issuecomment-2050813641>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAKDS3EWAHCWDOPKEKP3NYLY443JLAVCNFSM6AAAAABESINW5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJQHAYTGNRUGE> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >
GiteaMirror commented 2026-04-12 11:59:48 -05:00
Author
Owner
Copy Link

@JKratto commented on GitHub (Aug 21, 2024):

Hi there,

I've also noticed some changes in performance after a previous upgrade of Ollama (it's been 6-10 weeks), and I'd like to share my experience. I hope it can help diagnose and solve the issue.

My system: 5x 8GB Quadro RTX 4000

Previously, with mixtral 8x7B q4 on all the GPUs (no layers offloaded to CPU), I achieved a performance of approximately 23 t/s. However, after one day of upgrading Ollama, I observed that the same configuration resulted in mitral 8x7B q4 not loading on the GPUs only. Instead, 3 out of 33 layers were offloaded to the CPU, causing a significant performance drop to about 10 t/s.
If I manually set /set parameter num_gpu 33, the model loads to the VRAM only again, and the performance returns to 23 t/s. This behavior is consistent even when using only 4 GPUs; the model still fits with a manually set num_gpu value of 33 and no performance drop. Just as an interesting data point I observe GPU compute utilization rates of around 15-20% (5 GPU inference) and 20-25% (4 GPU inference).

Based on my calculations, Mixtral should fit into about 23 GB of VRAM; however, considering the memory required for cache, etc., and the VRAM utilization rates with 5 GPUs peaking around 74% (most utilized GPU) and about 94% with 4 GPUs (about 84% average), I believe that 4 GPUs represent the minimum viable configuration.
I recently set up a 6 GPU system, where Ollama loads all layers into VRAM by default. Almost 50 % of the VRAM is free causing significant inefficiency.

As a side line, I am using Ollama with the Open WebUI, and this setup makes loading the default model with 33/33 layers offloaded to GPU challenging (the num_gpu option was added recently but is per chat only and while I can change the setting, I can't really ask this of other users who certainly are not power-users). It appears that Olama is too cautious when loading models into VRAM. Is there a way around this, or should Olama be more eager to load models into VRAM?

<!-- gh-comment-id:2301506031 --> @JKratto commented on GitHub (Aug 21, 2024): Hi there, I've also noticed some changes in performance after a previous upgrade of Ollama (it's been 6-10 weeks), and I'd like to share my experience. I hope it can help diagnose and solve the issue. My system: 5x 8GB Quadro RTX 4000 Previously, with mixtral 8x7B q4 on all the GPUs (no layers offloaded to CPU), I achieved a performance of approximately 23 t/s. However, after one day of upgrading Ollama, I observed that the same configuration resulted in mitral 8x7B q4 not loading on the GPUs only. Instead, 3 out of 33 layers were offloaded to the CPU, causing a significant performance drop to about 10 t/s. If I manually set `/set parameter num_gpu 33`, the model loads to the VRAM only again, and the performance returns to 23 t/s. This behavior is consistent even when using only 4 GPUs; the model still fits with a manually set `num_gpu` value of 33 and no performance drop. Just as an interesting data point I observe GPU compute utilization rates of around 15-20% (5 GPU inference) and 20-25% (4 GPU inference). Based on my calculations, Mixtral should fit into about 23 GB of VRAM; however, considering the memory required for cache, etc., and the VRAM utilization rates with 5 GPUs peaking around 74% (most utilized GPU) and about 94% with 4 GPUs (about 84% average), I believe that 4 GPUs represent the minimum viable configuration. I recently set up a 6 GPU system, where Ollama loads all layers into VRAM by default. Almost 50 % of the VRAM is free causing significant inefficiency. As a side line, I am using Ollama with the Open WebUI, and this setup makes loading the default model with 33/33 layers offloaded to GPU challenging (the num_gpu option was added recently but is per chat only and while I can change the setting, I can't really ask this of other users who certainly are not power-users). It appears that Olama is too cautious when loading models into VRAM. Is there a way around this, or should Olama be more eager to load models into VRAM?
GiteaMirror commented 2026-04-12 11:59:49 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Aug 21, 2024):

I recently tried the (bundled) Llama3-70B from the Ollama repo and it fits without issue on 48GB VRAM, no offloading to CPU. I do think that Ollama is too cautious/conservative for stability reasons, but that the estimation algorithm may (or does) change between versions. I am guessing that setting num_gpu manually is the only foolproof mechanism (you can create a new model with the setting embedded in the Modelfile). This is at the risk of crashing Ollama if something else changes (e.g. the KV cache algorithm wants more memory).

<!-- gh-comment-id:2301535334 --> @orlyandico commented on GitHub (Aug 21, 2024): I recently tried the (bundled) Llama3-70B from the Ollama repo and it fits without issue on 48GB VRAM, no offloading to CPU. I do think that Ollama is too cautious/conservative for stability reasons, but that the estimation algorithm may (or does) change between versions. I am guessing that setting num_gpu manually is the only foolproof mechanism (you can create a new model with the setting embedded in the Modelfile). This is at the risk of crashing Ollama if something else changes (e.g. the KV cache algorithm wants more memory).
GiteaMirror commented 2026-04-12 11:59:50 -05:00
Author
Owner
Copy Link

@hdnh2006 commented on GitHub (Oct 18, 2024):

So is not there a clear solution for this? I am running a 14B parameters and I can see using ollama psthe following:
image

But I still can see I have 2GB of memory free:
image

I think ollama is limiting the amount of RAM that can be used.

is not some parameter like OLLAMA_GPU_USAGE=0.9,0.99,1,... that we can set?

<!-- gh-comment-id:2421960239 --> @hdnh2006 commented on GitHub (Oct 18, 2024): So is not there a clear solution for this? I am running a 14B parameters and I can see using `ollama ps`the following: ![image](https://github.com/user-attachments/assets/23c67574-4454-4142-8f15-fdf15b096cda) But I still can see I have 2GB of memory free: ![image](https://github.com/user-attachments/assets/550f4422-04de-46d0-9bb5-b2384be30859) I think ollama is limiting the amount of RAM that can be used. is not some parameter like `OLLAMA_GPU_USAGE=0.9,0.99,1,...` that we can set?
GiteaMirror commented 2026-04-12 11:59:51 -05:00
Author
Owner
Copy Link

@JKratto commented on GitHub (Oct 19, 2024):

@hdnh2006 It is not. The behavior clearly changed (as I state) during one of the upgrades. Since then I always have to manually allocate all layers to GPU.
Also, not knowing how the algorithm behind allocation works, the "allow 100 % allocation" parameter would be great. I run a headless system and it's only purpose is ollama. I can guarantee, that the VRAM is not utilised in any other way, so please use all of it - "yes I know what I am doing".

This problem must be a problem of thousands of people. How come nobody else seems to mind?

<!-- gh-comment-id:2423711080 --> @JKratto commented on GitHub (Oct 19, 2024): @hdnh2006 It is not. The behavior clearly changed (as I state) during one of the upgrades. Since then I always have to manually allocate all layers to GPU. Also, not knowing how the algorithm behind allocation works, the "allow 100 % allocation" parameter would be great. I run a headless system and it's only purpose is ollama. I can guarantee, that the VRAM is not utilised in any other way, so please use all of it - "yes I know what I am doing". This problem must be a problem of thousands of people. How come nobody else seems to mind?
GiteaMirror commented 2026-04-12 11:59:53 -05:00
Author
Owner
Copy Link

@Kira-PH commented on GitHub (Nov 21, 2024):

There are numerous threads on this. I'm trying to load Llama 3.1 70b on two A6000s, and with Flash Attention enabled, nvidia-smi reports only about 32GB used on each GPU, meaning I have another 32GB (across both GPUs) unutilised. ollama ps tells me I have a 16%/84% CPU/GPU split.

<!-- gh-comment-id:2490886002 --> @Kira-PH commented on GitHub (Nov 21, 2024): There are numerous threads on this. I'm trying to load Llama 3.1 70b on two A6000s, and with Flash Attention enabled, nvidia-smi reports only about 32GB used on each GPU, meaning I have another 32GB (across both GPUs) unutilised. `ollama ps` tells me I have a 16%/84% CPU/GPU split.
GiteaMirror commented 2026-04-12 11:59:54 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Nov 21, 2024):

that’s odd, I can load Llama-3.1-70b on two P40’s and it loads completely
into VRAM.

On Thu, 21 Nov 2024 at 11:37, Hugh Phoenix-Hulme @.***>
wrote:

There are numerous threads on this. I'm trying to load Llama 3.1 70b on
two A6000s, and with Flash Attention enabled, nvidia-smi reports only about
32GB used on each GPU, meaning I have another 32GB (across both GPUs)
unutilised. ollama ps tells me I have a 16%/84% CPU/GPU split.

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/3078#issuecomment-2490886002,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKDS3BUMBEFEQT3QKTS7IL2BXAYRAVCNFSM6AAAAABESINW5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJQHA4DMMBQGI
.
You are receiving this because you were mentioned.Message ID:
@.***>

<!-- gh-comment-id:2491231663 --> @orlyandico commented on GitHub (Nov 21, 2024): that’s odd, I can load Llama-3.1-70b on two P40’s and it loads completely into VRAM. On Thu, 21 Nov 2024 at 11:37, Hugh Phoenix-Hulme ***@***.***> wrote: > There are numerous threads on this. I'm trying to load Llama 3.1 70b on > two A6000s, and with Flash Attention enabled, nvidia-smi reports only about > 32GB used on each GPU, meaning I have another 32GB (across both GPUs) > unutilised. ollama ps tells me I have a 16%/84% CPU/GPU split. > > — > Reply to this email directly, view it on GitHub > <https://github.com/ollama/ollama/issues/3078#issuecomment-2490886002>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAKDS3BUMBEFEQT3QKTS7IL2BXAYRAVCNFSM6AAAAABESINW5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJQHA4DMMBQGI> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >
GiteaMirror commented 2026-04-12 11:59:56 -05:00
Author
Owner
Copy Link

@Kira-PH commented on GitHub (Nov 22, 2024):

that’s odd, I can load Llama-3.1-70b on two P40’s and it loads completely into VRAM.

Yes, that's exactly what I would expect to happen.

<!-- gh-comment-id:2493645043 --> @Kira-PH commented on GitHub (Nov 22, 2024): > that’s odd, I can load Llama-3.1-70b on two P40’s and it loads completely into VRAM. Yes, that's exactly what I would expect to happen.
GiteaMirror commented 2026-04-12 11:59:57 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Nov 22, 2024):

Are you on the latest version? Many moons ago Ollama had this issue when I tried to load Smaug-72b onto the two P40's, it was not using all of the VRAM. I had to manually force all the layers to load into VRAM with a custom Modelfile.

FROM ./Smaug-72B-v0.1-q4_k_m.gguf
PARAMETER num_gpu 81

But more recent versions of Ollama now "do the right thing"

<!-- gh-comment-id:2493672544 --> @orlyandico commented on GitHub (Nov 22, 2024): Are you on the latest version? Many moons ago Ollama had this issue when I tried to load Smaug-72b onto the two P40's, it was not using all of the VRAM. I had to manually force all the layers to load into VRAM with a custom Modelfile. FROM ./Smaug-72B-v0.1-q4_k_m.gguf PARAMETER num_gpu 81 But more recent versions of Ollama now "do the right thing"
GiteaMirror commented 2026-04-12 11:59:58 -05:00
Author
Owner
Copy Link

@Kira-PH commented on GitHub (Nov 22, 2024):

Thanks for taking the time. I was using 0.4.1 but recently upgraded to 0.4.2 from the website.
I did try creating a modelfile with num_gpu 80 for Llama 3.1 70b, but that made no difference.

btw you mention above that nvidia-smi reported your A6000 had 45GB available. That's probably because of ECC being enabled, the top right value in the 3rd column should read "Off". sudo nvidia-smi -e 0 to disable it, if you haven't already figured that out.

<!-- gh-comment-id:2494718792 --> @Kira-PH commented on GitHub (Nov 22, 2024): Thanks for taking the time. I was using 0.4.1 but recently upgraded to 0.4.2 from the website. I did try creating a modelfile with `num_gpu 80` for Llama 3.1 70b, but that made no difference. btw you mention above that nvidia-smi reported your A6000 had 45GB available. That's probably because of ECC being enabled, the top right value in the 3rd column should read "Off". `sudo nvidia-smi -e 0` to disable it, if you haven't already figured that out.
GiteaMirror commented 2026-04-12 11:59:59 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Nov 22, 2024):

I tested an A6000 Ada (48GB) but, it had less memory available to Ollama than the two P40's because the A6000 has a graphics output..

<!-- gh-comment-id:2494727223 --> @orlyandico commented on GitHub (Nov 22, 2024): I tested an A6000 Ada (48GB) but, it had less memory available to Ollama than the two P40's because the A6000 has a graphics output..
GiteaMirror commented 2026-04-12 12:00:00 -05:00
Author
Owner
Copy Link

@Kira-PH commented on GitHub (Nov 22, 2024):

I have all 48GB available on a headless box:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.08             Driver Version: 550.127.08     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A6000               Off |   00000000:03:00.0 Off |                  Off |
| 30%   55C    P8             26W /  300W |   20038MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX A6000               Off |   00000000:07:00.0 Off |                  Off |
| 30%   52C    P8             24W /  300W |   19784MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
<!-- gh-comment-id:2494753923 --> @Kira-PH commented on GitHub (Nov 22, 2024): I have all 48GB available on a headless box: ``` +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.127.08 Driver Version: 550.127.08 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX A6000 Off | 00000000:03:00.0 Off | Off | | 30% 55C P8 26W / 300W | 20038MiB / 49140MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX A6000 Off | 00000000:07:00.0 Off | Off | | 30% 52C P8 24W / 300W | 19784MiB / 49140MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ ```
GiteaMirror commented 2026-04-12 12:00:02 -05:00
Author
Owner
Copy Link

@JKratto commented on GitHub (Nov 25, 2024):

I have recently acquired A4000 GPUs which have 16GB of VRAM each. "Strangely" enough, 2GPUs can hold mixtral 8x7B q4 with all 33 layers offloaded to the GPUs by default. With 4x RTX 4000 (8GB) this did not work. I hope I will be deploying another setup with the leftover RTX 4000s, so a more side-by-side comparison will be possible.
The @HughPH's RTX A6000 case I don't get at all. 😆 I mean, what is the magic behind that memory allocation? 😆

Side note, very much important for me:
Also as I am using Open WebUI, the quality of life improved greatly since a support for per model, per instance "global" settings for the models. I can now set the model parameters as the context length and num_gpu directly in the UI and thus the users are guaranteed good results.
Comment:
I tried to create my own modelfiles before, but wasn't very successful. For some reason I couldn't quickly debug, the models behaved weirdly. The OWUI way works great as it just adds the required "custom" parameters to each request and ollama manages the model accordingly.

<!-- gh-comment-id:2498177860 --> @JKratto commented on GitHub (Nov 25, 2024): I have recently acquired A4000 GPUs which have 16GB of VRAM each. "Strangely" enough, 2GPUs can hold mixtral 8x7B q4 with all 33 layers offloaded to the GPUs **by default**. With 4x RTX 4000 (8GB) this did not work. I hope I will be deploying another setup with the leftover RTX 4000s, so a more side-by-side comparison will be possible. The @HughPH's RTX A6000 case I don't get at all. 😆 I mean, what is the magic behind that memory allocation? 😆 Side note, very much important for me: Also as I am using Open WebUI, the quality of life improved greatly since a support for per model, per instance "global" settings for the models. I can now set the model parameters as the context length and num_gpu directly in the UI and thus the users are guaranteed good results. Comment: I tried to create my own modelfiles before, but wasn't very successful. For some reason I couldn't quickly debug, the models behaved weirdly. The OWUI way works great as it just adds the required "custom" parameters to each request and ollama manages the model accordingly.
GiteaMirror commented 2026-04-12 12:00:04 -05:00
Author
Owner
Copy Link

@Kira-PH commented on GitHub (Nov 25, 2024):

I literally just upgraded to 0.4.4 about 5 minutes ago, and it's now loading Llama 3.1 70b across both A6000s.

<!-- gh-comment-id:2498200285 --> @Kira-PH commented on GitHub (Nov 25, 2024): I literally just upgraded to 0.4.4 about 5 minutes ago, and it's now loading Llama 3.1 70b across both A6000s.
GiteaMirror commented 2026-04-12 12:00:05 -05:00
Author
Owner
Copy Link

@JKratto commented on GitHub (Nov 25, 2024):

TBH, I also upgraded along the way a lot. It might mean the 4x8GB setup is now working as well... Will keep you posted.

<!-- gh-comment-id:2498208859 --> @JKratto commented on GitHub (Nov 25, 2024): TBH, I also upgraded along the way a lot. It might mean the 4x8GB setup is now working as well... Will keep you posted.
GiteaMirror commented 2026-04-12 12:00:06 -05:00
Author
Owner
Copy Link

@Kira-PH commented on GitHub (Nov 25, 2024):

So I have some more information. If I try to use num_ctx=128000 then the size in ollama ps is reported at 119GB and I end up with 16% of the model on the CPU (but about 64GB allocated across the GPUs.)
With num_ctx=32768 it's 61GB in ollama ps, about 48GB allocated across the GPUs.

num_ctx ollama ps size VRAM usage
32768 61GB 48GB
65536 81GB 60GB
98304 101GB 68GB
128000 119GB 64GB (the rest is in RAM)

After loading and using, the 48GB rises to 52GB, so it looks like Ollama is:

  1. Expecting every 32768 tokens of context to require 20GB VRAM when it takes 8GB,
  2. Expecting the model to consume 53GB when it consumes 44GB (with Flash Attention)
<!-- gh-comment-id:2498717674 --> @Kira-PH commented on GitHub (Nov 25, 2024): So I have some more information. If I try to use num_ctx=128000 then the size in `ollama ps` is reported at 119GB and I end up with 16% of the model on the CPU (but about 64GB allocated across the GPUs.) With num_ctx=32768 it's 61GB in `ollama ps`, about 48GB allocated across the GPUs. | num_ctx | `ollama ps` size | VRAM usage | | --- | --- | --- | | 32768 | 61GB | 48GB | | 65536 | 81GB | 60GB | | 98304 | 101GB | 68GB | | 128000 | 119GB | 64GB (the rest is in RAM) | After loading and using, the 48GB rises to 52GB, so it looks like Ollama is: 1. Expecting every 32768 tokens of context to require 20GB VRAM when it takes 8GB, 2. Expecting the model to consume 53GB when it consumes 44GB (with Flash Attention)
GiteaMirror commented 2026-04-12 12:00:09 -05:00
Author
Owner
Copy Link

@hdnh2006 commented on GitHub (Jan 4, 2025):

After 3 months and several versions, I can confirm that nothing has changed and the problem still persists. I am using the latest ollama version:

ollama --version
ollama version is 0.5.4

And this is what I see while trying to run qwen-14b with the following modelfile:

FROM qwen2.5:14b

# sets the temperature to 1 [higher is more creative, lower is more coherent]
PARAMETER temperature 0.5

# sets the context window size to 8192, this controls how many tokens the LLM can use as context to generate the next token
PARAMETER num_ctx 8192

# tokens to generate set to 4096 (max)
PARAMETER num_predict 4096

The model can't be allocated 100% on GPU:
image

Despite I still have 2GB extra of VRAM:
image

No matter what parameter I use for num_gpu, the result is the same, I do it in the modelfile changing this parameter.

PARAMETER num_gpu 200

From my point of view, without any desire to offend, I think it's time to change to another framework. Unfortunately, ollama highly depends on llama.cpp and this is a big limitation. For example, there's still no suppor for parallel request for llama3.2 vision models as the logs indicate:

ene 04 13:33:56 henrymlearning ollama[19895]: time=2025-01-04T13:33:56.951+01:00 level=WARN source=sched.go:137 msg="mllama doesn't support parallel requests yet"

And after several months, the problem with VRAM usage stills persists and we don't know if this problem comes from ollama or from llama.cpp because ollama team doesn't offer any response to this issue.

There are other possibilites to deploy LLMs with ease:

I will check both and let's see what happen. Good luck and thanks ollama for everything.

<!-- gh-comment-id:2571283743 --> @hdnh2006 commented on GitHub (Jan 4, 2025): After 3 months and several versions, I can confirm that nothing has changed and the problem still persists. I am using the latest ollama version: ```bash ollama --version ollama version is 0.5.4 ``` And this is what I see while trying to run qwen-14b with the following modelfile: ```yaml FROM qwen2.5:14b # sets the temperature to 1 [higher is more creative, lower is more coherent] PARAMETER temperature 0.5 # sets the context window size to 8192, this controls how many tokens the LLM can use as context to generate the next token PARAMETER num_ctx 8192 # tokens to generate set to 4096 (max) PARAMETER num_predict 4096 ``` The model can't be allocated 100% on GPU: ![image](https://github.com/user-attachments/assets/f25b369f-96eb-4c4a-96e5-68f23f5980f3) Despite I still have 2GB extra of VRAM: ![image](https://github.com/user-attachments/assets/16182a11-c3b6-4967-8294-66e0a6538d6c) No matter what parameter I use for `num_gpu`, the result is the same, I do it in the modelfile changing this parameter. ``` PARAMETER num_gpu 200 ``` From my point of view, without any desire to offend, I think it's time to change to another framework. Unfortunately, ollama highly depends on llama.cpp and this is a big limitation. For example, there's still no suppor for parallel request for llama3.2 vision models as the logs indicate: ``` ene 04 13:33:56 henrymlearning ollama[19895]: time=2025-01-04T13:33:56.951+01:00 level=WARN source=sched.go:137 msg="mllama doesn't support parallel requests yet" ``` And after several months, the problem with VRAM usage stills persists and we don't know if this problem comes from ollama or from llama.cpp because ollama team doesn't offer any response to this issue. There are other possibilites to deploy LLMs with ease: - https://github.com/exo-explore/exo - https://github.com/vllm-project/vllm/ I will check both and let's see what happen. Good luck and thanks ollama for everything.
GiteaMirror commented 2026-04-12 12:00:10 -05:00
Author
Owner
Copy Link

@JKratto commented on GitHub (Jan 5, 2025):

@hdnh2006 I think in this case you might be trying to fit 2x context into your memory. Try setting OLLAMA_NUM_PARALLEL=1 and num_gpu to the correct number of layers for your model. After that the model should be loaded into the VRAM fully.

<!-- gh-comment-id:2571586963 --> @JKratto commented on GitHub (Jan 5, 2025): @hdnh2006 I think in this case you might be trying to fit 2x context into your memory. Try setting OLLAMA_NUM_PARALLEL=1 and num_gpu to the correct number of layers for your model. After that the model should be loaded into the VRAM fully.
GiteaMirror commented 2026-04-12 12:00:11 -05:00
Author
Owner
Copy Link

@hdnh2006 commented on GitHub (Jan 8, 2025):

@hdnh2006 I think in this case you might be trying to fit 2x context into your memory. Try setting OLLAMA_NUM_PARALLEL=1 and num_gpu to the correct number of layers for your model. After that the model should be loaded into the VRAM fully.

Exactly, because I want to deploy the model in production environment, so many users will be calling the model. Other frameworks like vLLM can manage better the VRAM. Actually I was able to deploy the same quantized model using the same GPU with vLLM and I was able to do up to 6 requests in parallel!!
Screenshot from 2025-01-07 14-00-04

I thank ollama for helping me to prototype, it is easy to use and it helps you to create a OpenAI compatible api with ease. But it defnitely manage very bad the VRAM.

<!-- gh-comment-id:2578607147 --> @hdnh2006 commented on GitHub (Jan 8, 2025): > @hdnh2006 I think in this case you might be trying to fit 2x context into your memory. Try setting OLLAMA_NUM_PARALLEL=1 and num_gpu to the correct number of layers for your model. After that the model should be loaded into the VRAM fully. Exactly, because I want to deploy the model in production environment, so many users will be calling the model. Other frameworks like [`vLLM`](https://github.com/vllm-project/vllm) can manage better the VRAM. Actually I was able to deploy the same quantized model using the same GPU with [`vLLM`](https://github.com/vllm-project/vllm) and I was able to do up to 6 requests in parallel!! ![Screenshot from 2025-01-07 14-00-04](https://github.com/user-attachments/assets/8fe5a84f-fe62-42f2-bc85-104f218eb555) I thank ollama for helping me to prototype, it is easy to use and it helps you to create a OpenAI compatible api with ease. But it defnitely manage very bad the VRAM.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label nvidia performance
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1894
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?