github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #7711] Large host RAM allocation when using full gpu offloading #4923

New Issue
Closed
opened 2026-04-12 15:58:48 -05:00 by GiteaMirror · 16 comments
GiteaMirror commented 2026-04-12 15:58:48 -05:00
Owner
Copy Link

Originally created by @CkovMk on GitHub (Nov 17, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7711

What is the issue?

Ollama allocates same size of host (CPU) RAM to what the model requires in VRAM when loading the model. If my understanding is correct, it's not actuallly using these RAM. (Perhaps these are just mapped memory to write VRAM? I'm not expert in these...) And when the model is fully loaded, the buffered RAM usage goes back to normal.

See picture below:

image

I do have enough VRAM, but If host RAM is not big enough, it exits with this error:

Error: llama runner process has terminated: error loading model: unable to allocate backend buffer

Is this expected behavior? Is it possible to optimize host memory requirement when loading model?

I'm asking this because I'm running a passthroughed GPU in a VM, and I want to save some host RAM by only allocating 8GiB RAM to a VM with 24GiB VRAM.

Thanks!

OS

Windows

GPU

Nvidia

CPU

AMD

Ollama version

0.4.2

Originally created by @CkovMk on GitHub (Nov 17, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7711 ### What is the issue? Ollama allocates same size of host (CPU) RAM to what the model requires in VRAM when loading the model. If my understanding is correct, it's not actuallly using these RAM. (Perhaps these are just mapped memory to write VRAM? I'm not expert in these...) And when the model is fully loaded, the buffered RAM usage goes back to normal. See picture below: ![image](https://github.com/user-attachments/assets/01b192aa-0a63-4d25-b018-75a951a4fbad) I do have enough VRAM, but If host RAM is not big enough, it exits with this error: ``` Error: llama runner process has terminated: error loading model: unable to allocate backend buffer ``` Is this expected behavior? Is it possible to optimize host memory requirement when loading model? I'm asking this because I'm running a passthroughed GPU in a VM, and I want to save some host RAM by only allocating 8GiB RAM to a VM with 24GiB VRAM. Thanks! ### OS Windows ### GPU Nvidia ### CPU AMD ### Ollama version 0.4.2
GiteaMirror added the needs more infobug labels 2026-04-12 15:58:48 -05:00
GiteaMirror commented 2026-04-12 15:58:50 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 17, 2024):

Server logs would aid in debugging.

I can't read the legends on the screenshot and Windows is not my field of expertise, but buffered RAM generally means page cache, that is data read from disk is stored in cache in case it is needed again in the future. So it's transient usage and shouldn't affect model loading. unable to allocate backend buffer indicates a problem allocating RAM or VRAM for the model as it's being loaded, which would point to insufficient (V)RAM to host the model. If ollama thinks that a model won't fit into VRAM, it will load part of the model in RAM instead. So to load a model, the system needs RAM + VRAM > size(model_weights) + size(context) + size(graph) + size(adapter). The logs will reveal where the shortfall is, and then a solution can be devised.

<!-- gh-comment-id:2481537539 --> @rick-github commented on GitHub (Nov 17, 2024): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) would aid in debugging. I can't read the legends on the screenshot and Windows is not my field of expertise, but buffered RAM generally means page cache, that is data read from disk is stored in cache in case it is needed again in the future. So it's transient usage and shouldn't affect model loading. `unable to allocate backend buffer` indicates a problem allocating RAM or VRAM for the model as it's being loaded, which would point to insufficient (V)RAM to host the model. If ollama thinks that a model won't fit into VRAM, it will load part of the model in RAM instead. So to load a model, the system needs RAM + VRAM > size(model_weights) + size(context) + size(graph) + size(adapter). The logs will reveal where the shortfall is, and then a solution can be devised.
GiteaMirror commented 2026-04-12 15:58:50 -05:00
Author
Owner
Copy Link

@CkovMk commented on GitHub (Dec 5, 2024):

Hi @rick-github

Sorry for late reply. I upgraded to Ollama 0.4.7 and tested with ollama run qwen2.5:32b. Here are server logs with 32GiB and 8GiB host RAM respectively, the GPU is the same: 1x RTX A5000 24GiB.

server-32GiB-host-RAM.txt

server-8GiB-host-RAM.txt

With 8 GiB host RAM, Ollama failed to load the model, despite VRAM is sufficient.

<!-- gh-comment-id:2520238118 --> @CkovMk commented on GitHub (Dec 5, 2024): Hi @rick-github Sorry for late reply. I upgraded to Ollama 0.4.7 and tested with `ollama run qwen2.5:32b`. Here are server logs with 32GiB and 8GiB host RAM respectively, the GPU is the same: 1x RTX A5000 24GiB. [server-32GiB-host-RAM.txt](https://github.com/user-attachments/files/18023439/server-32GiB-host-RAM.txt) [server-8GiB-host-RAM.txt](https://github.com/user-attachments/files/18023447/server-8GiB-host-RAM.txt) With 8 GiB host RAM, Ollama failed to load the model, despite VRAM is sufficient.
GiteaMirror commented 2026-04-12 15:58:50 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 5, 2024):

The VRAM is only just sufficient - there's 22.7 GiB available and ollama wants to use 21.5 GiB which looks likes it fits, so ollama tells llama.cpp to load the whole model into VRAM. From the alloc failure this would seem to be not accurate. The Windows Nvidia driver uses fallback memory when there's not enough VRAM, so I speculate (again, Windows not my field of expertise) that in the 8GiB case, the driver is trying to use fallback memory and there's not enough to satisfy the alloc requirement. There are some mitigations you can try:

  1. Set OLLAMA_NUM_PARALLEL=1. Currently ollama is using a default of 4, and it's allocating more context buffer than it needs to.
  2. Set OLLAMA_FLASH_ATTENTION=1. FA is a more efficient use of context space and may reduce memory pressure.
  3. Set OLLAMA_GPU_OVERHEAD to give llama.cpp a buffer to grow in to (eg, OLLAMA_GPU_OVERHEAD=536870912 to reserve 512M)
  4. Reduce the number layers that ollama thinks it can offload to the GPU, see here. Ollama is currently offloading 65 layers, try setting num_gpu to 60.
  5. Free up system RAM so that more fallback memory is available. You can force programs into swap by running this script.
<!-- gh-comment-id:2520388042 --> @rick-github commented on GitHub (Dec 5, 2024): The VRAM is only just sufficient - there's 22.7 GiB available and ollama wants to use 21.5 GiB which looks likes it fits, so ollama tells llama.cpp to load the whole model into VRAM. From the alloc failure this would seem to be not accurate. The Windows Nvidia driver uses fallback memory when there's not enough VRAM, so I speculate (again, Windows not my field of expertise) that in the 8GiB case, the driver is trying to use fallback memory and there's not enough to satisfy the alloc requirement. There are some mitigations you can try: 1. Set [`OLLAMA_NUM_PARALLEL=1`](https://github.com/ollama/ollama/blob/5f8051180e3b9aeafc153f6b5056e7358a939c88/envconfig/config.go#L247). Currently ollama is using a default of 4, and it's allocating more context buffer than it needs to. 2. Set [`OLLAMA_FLASH_ATTENTION=1`](https://github.com/ollama/ollama/blob/5f8051180e3b9aeafc153f6b5056e7358a939c88/envconfig/config.go#L236). FA is a more efficient use of context space and may reduce memory pressure. 3. Set [`OLLAMA_GPU_OVERHEAD`](https://github.com/ollama/ollama/blob/5f8051180e3b9aeafc153f6b5056e7358a939c88/envconfig/config.go#L237) to give llama.cpp a buffer to grow in to (eg, `OLLAMA_GPU_OVERHEAD=536870912` to reserve 512M) 4. Reduce the number layers that ollama thinks it can offload to the GPU, see [here](https://github.com/ollama/ollama/issues/6950#issuecomment-2373663650). Ollama is currently offloading 65 layers, try setting `num_gpu` to 60. 5. Free up system RAM so that more fallback memory is available. You can force programs into swap by running [this script](https://github.com/ollama/ollama/issues/6918#issuecomment-2488651203).
GiteaMirror commented 2026-04-12 15:58:51 -05:00
Author
Owner
Copy Link

@CkovMk commented on GitHub (Dec 10, 2024):

I think the problem is not about VRAM - Even if I load qwen2.5:14b, it will report the same failure.

I observed how the RAM usage grows: The allocated host RAM keeps going high when loading the model to VRAM,. Once the load is complete, the host memory usage suddenly drop by almost the size of the model. This may suggests llama.cpp is using host RAM as an staging buffer before sending data to VRAM. If so, it basically means it has to be more host RAM than VRAM to fully utilize the VRAM.

<!-- gh-comment-id:2531652189 --> @CkovMk commented on GitHub (Dec 10, 2024): I think the problem is not about VRAM - Even if I load qwen2.5:14b, it will report the same failure. I observed how the RAM usage grows: The allocated host RAM keeps going high when loading the model to VRAM,. Once the load is complete, the host memory usage suddenly drop by almost the size of the model. This may suggests llama.cpp is using host RAM as an staging buffer before sending data to VRAM. If so, it basically means it has to be more host RAM than VRAM to fully utilize the VRAM.
GiteaMirror commented 2026-04-12 15:58:51 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 10, 2024):

Logs for the qwen2.5:14b failure would be helpful.

<!-- gh-comment-id:2531672744 --> @rick-github commented on GitHub (Dec 10, 2024): Logs for the qwen2.5:14b failure would be helpful.
GiteaMirror commented 2026-04-12 15:58:52 -05:00
Author
Owner
Copy Link

@CkovMk commented on GitHub (Dec 15, 2024):

Logs for the qwen2.5:14b failure:

server.txt

<!-- gh-comment-id:2543921015 --> @CkovMk commented on GitHub (Dec 15, 2024): Logs for the qwen2.5:14b failure: [server.txt](https://github.com/user-attachments/files/18141033/server.txt)
GiteaMirror commented 2026-04-12 15:58:53 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 15, 2024):

The model load failed at the point where llama.cpp was allocating space for the model weights.

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 8148.38 MiB on device 0: cudaMalloc failed: out of memory

ollama thought it had 22.7G available and only needed 10.8G in total:

time=2024-12-15T23:36:37.528+08:00 level=INFO source=memory.go:356
  msg="offload to cuda" layers.requested=-1 layers.model=49 layers.offload=49 layers.split=""
  memory.available="[22.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="10.8 GiB" 
  memory.required.partial="10.8 GiB" memory.required.kv="1.5 GiB" memory.required.allocations="[10.8 GiB]"
  memory.weights.total="8.9 GiB" memory.weights.repeating="8.3 GiB" memo ry.weights.nonrepeating="609.1 MiB"
 memory.graph.full="676.0 MiB" memory.graph.partial="916.1 MiB"

I'm not a CUDA programmer but there a couple of ways this can happen. Another process could have swooped in between the time ollama computed requirements and llama.cpp requested memory, or there's not enough contiguous memory to satisfy the 8G request. What's the output of nvidia-smi? Do you have other processes on the system that allocate/free GPU memory?

<!-- gh-comment-id:2543935430 --> @rick-github commented on GitHub (Dec 15, 2024): The model load failed at the point where llama.cpp was allocating space for the model weights. ``` ggml_backend_cuda_buffer_type_alloc_buffer: allocating 8148.38 MiB on device 0: cudaMalloc failed: out of memory ``` ollama thought it had 22.7G available and only needed 10.8G in total: ``` time=2024-12-15T23:36:37.528+08:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=49 layers.offload=49 layers.split="" memory.available="[22.7 GiB]" memory.gpu_overhead="0 B" memory.required.full="10.8 GiB" memory.required.partial="10.8 GiB" memory.required.kv="1.5 GiB" memory.required.allocations="[10.8 GiB]" memory.weights.total="8.9 GiB" memory.weights.repeating="8.3 GiB" memo ry.weights.nonrepeating="609.1 MiB" memory.graph.full="676.0 MiB" memory.graph.partial="916.1 MiB" ``` I'm not a CUDA programmer but there a couple of ways this can happen. Another process could have swooped in between the time ollama computed requirements and llama.cpp requested memory, or there's [not enough contiguous memory](https://developer.nvidia.com/docs/drive/drive-os/archives/6.0.4/linux/sdk/common/topics/graphics_content/avoiding_memory_fragmentation.html) to satisfy the 8G request. What's the output of `nvidia-smi`? Do you have other processes on the system that allocate/free GPU memory?
GiteaMirror commented 2026-04-12 15:58:53 -05:00
Author
Owner
Copy Link

@CkovMk commented on GitHub (Dec 15, 2024):

no, ollama is the only process using GPU. I'm not even using the GPU for display output, and it shows 0 VRAM usage in task manager.

<!-- gh-comment-id:2543937213 --> @CkovMk commented on GitHub (Dec 15, 2024): no, ollama is the only process using GPU. I'm not even using the GPU for display output, and it shows 0 VRAM usage in task manager.
GiteaMirror commented 2026-04-12 15:58:53 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 15, 2024):

What's the output of nvidia-smi?

<!-- gh-comment-id:2543937992 --> @rick-github commented on GitHub (Dec 15, 2024): What's the output of nvidia-smi?
GiteaMirror commented 2026-04-12 15:58:54 -05:00
Author
Owner
Copy Link

@CkovMk commented on GitHub (Dec 15, 2024):

It should be pretty easy to reproduce this: prepare an Windows machine that has RAM < VRAM, then trying to load an model that fits into VRAM but larger than RAM. It fails in my case.

Here's output of nvidia-smi:

image

<!-- gh-comment-id:2543938855 --> @CkovMk commented on GitHub (Dec 15, 2024): It should be pretty easy to reproduce this: prepare an Windows machine that has RAM < VRAM, then trying to load an model that fits into VRAM but larger than RAM. It fails in my case. Here's output of nvidia-smi: ![image](https://github.com/user-attachments/assets/dd44f70a-f39b-446b-959d-2906f4513c22)
GiteaMirror commented 2026-04-12 15:58:54 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 15, 2024):

So there are processes using the GPU. Try closing them, then running ollama and loading the model.

I don't have access to a Windows machine, but a test with a docker container constrained to 8G loading a 15G model into a 16G GPU worked fine.

<!-- gh-comment-id:2543947439 --> @rick-github commented on GitHub (Dec 15, 2024): So there are processes using the GPU. Try closing them, then running ollama and loading the model. I don't have access to a Windows machine, but a test with a docker container constrained to 8G loading a 15G model into a 16G GPU worked fine.
GiteaMirror commented 2026-04-12 15:58:55 -05:00
Author
Owner
Copy Link

@CkovMk commented on GitHub (Dec 15, 2024):

These are Windows services / core apps that cannot be closed, and as you can see the GPU memory usage is N/A.

<!-- gh-comment-id:2543971086 --> @CkovMk commented on GitHub (Dec 15, 2024): These are Windows services / core apps that cannot be closed, and as you can see the GPU memory usage is N/A.
GiteaMirror commented 2026-04-12 15:58:55 -05:00
Author
Owner
Copy Link

@CkovMk commented on GitHub (Dec 15, 2024):

I don't have access to a Windows machine, but a test with a docker container constrained to 8G loading a 15G model into a 16G GPU worked fine.

Then maybe it's an windows specific issue

<!-- gh-comment-id:2543971337 --> @CkovMk commented on GitHub (Dec 15, 2024): > I don't have access to a Windows machine, but a test with a docker container constrained to 8G loading a 15G model into a 16G GPU worked fine. Then maybe it's an windows specific issue
GiteaMirror commented 2026-04-12 15:58:55 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 15, 2024):

What's the output of nvidia-smi -q?

<!-- gh-comment-id:2543994178 --> @rick-github commented on GitHub (Dec 15, 2024): What's the output of `nvidia-smi -q`?
GiteaMirror commented 2026-04-12 15:58:55 -05:00
Author
Owner
Copy Link

@softnodelabs commented on GitHub (Jan 31, 2025):

I also had this behaviour on Windows 11 and ROCm with an RX 7600 XT. I'm on Linux right now and I don't have this issue anymore. On windows, ollama allocates RAM at the same amount of vram for any model but in my case, the RAM usage did not went down unless I manually unloaded the model or closed ollama server.

<!-- gh-comment-id:2628399095 --> @softnodelabs commented on GitHub (Jan 31, 2025): I also had this behaviour on Windows 11 and ROCm with an RX 7600 XT. I'm on Linux right now and I don't have this issue anymore. On windows, ollama allocates RAM at the same amount of vram for any model but in my case, the RAM usage did not went down unless I manually unloaded the model or closed ollama server.
GiteaMirror commented 2026-04-12 15:58:56 -05:00
Author
Owner
Copy Link

@lainconn commented on GitHub (Feb 19, 2025):

For those of you, who made the same mistake to develop on Windows. If you are using docker-desktop, try to limit RAM usage in .wslconfig. In my case, I limited to 4GB. Now ollama container does not exeed 4GB, even while loading 10GB+ models. Prior to that, I had the same issue, with 10GB+ usage on RAM and 10GB+ on GPU. Also, try updating wsl with wsl --update.

My specs:

Engine:
Version: 27.4.0
API version: 1.47 (minimum version 1.24)
Go version: go1.22.10
Git commit: 92a8393
Built: Sat Dec 7 10:38:57 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.21
GitCommit: 472731909fa34bd7bc9c087e4c27943f9835f111
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0

WSL version: 2.4.11.0
Kernel version: 5.15.167.4-1
WSLg version: 1.0.65
MSRDC version: 1.2.5716
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.19045.5371

<!-- gh-comment-id:2668445158 --> @lainconn commented on GitHub (Feb 19, 2025): For those of you, who made the same mistake to develop on Windows. If you are using docker-desktop, try to limit RAM usage in `.wslconfig`. In my case, I limited to 4GB. Now ollama container does not exeed 4GB, even while loading 10GB+ models. Prior to that, I had the same issue, with 10GB+ usage on RAM and 10GB+ on GPU. Also, try updating wsl with `wsl --update`. My specs: Engine: Version: 27.4.0 API version: 1.47 (minimum version 1.24) Go version: go1.22.10 Git commit: 92a8393 Built: Sat Dec 7 10:38:57 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.21 GitCommit: 472731909fa34bd7bc9c087e4c27943f9835f111 runc: Version: 1.1.13 GitCommit: v1.1.13-0-g58aa920 docker-init: Version: 0.19.0 GitCommit: de40ad0 WSL version: 2.4.11.0 Kernel version: 5.15.167.4-1 WSLg version: 1.0.65 MSRDC version: 1.2.5716 Direct3D version: 1.611.1-81528511 DXCore version: 10.0.26100.1-240331-1435.ge-release Windows version: 10.0.19045.5371
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug needs more info
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#4923
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?