github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8416] OpenWebUI-Ollama does not fully utilize NVIDIA GPU when context length or parallel session icncreases #5405

New Issue
Closed
opened 2026-04-12 16:38:38 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-12 16:38:38 -05:00
Owner
Copy Link

Originally created by @rpaGuyai on GitHub (Jan 14, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8416

What is the issue?

I am hosting OpenWebUI on my server ( specs - AWS G4dn.12xlarge, Memory: 192 GB RAM, GPU: 4 x NVIDIA Tesla T4 GPUs, TITAL 64 gb GPU, 16 GB each).

Issues and scenarios:

I have found a sweet spot to get the optimized result, when the context length is set to 11000 and Environment="OLLAMA_NUM_PARALLEL=10" in ollamma . service file, it works good, utilizing all 4 GPUS and minimum CPU.

However, If I increase either the context length to let say 15000 or num parallel to let say 15, the speed reduces drastically and the load is being shared almost 50- 50 by CPU and GPU, GPU not being utilized fully causing slowness in response when just 5-6 concurrent sessions as there.

If I further increase either od context length to 20K or num parallel to 20, then in such cases and beyond, it stops using GPU and load is fully transferred to CPU which kills the speed totally.

Please can someone help on this, I want to understand what makes the load spill to CPU in such cases and why can't we utilize GPU full after certain extent.

For information - we thought this is the max 12xLarge hardware can support, so we increased to G4dn.metal for testing but the result is same.

Need help from experts please, is it due to some configurations in ollama or openwebui or is it that T4 GPUS in both cases each unit is only 16GB, is it that we need entire GPU memory in just 1GPU to fully utilize it, if the GPU , in my case of 12x Large, it is 64(16*4) split into 4.

Any suggestions or guidance will be very helpful

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.5.5

Originally created by @rpaGuyai on GitHub (Jan 14, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8416 ### What is the issue? I am hosting OpenWebUI on my server ( specs - AWS G4dn.12xlarge, Memory: 192 GB RAM, GPU: 4 x NVIDIA Tesla T4 GPUs, TITAL 64 gb GPU, 16 GB each). Issues and scenarios: I have found a sweet spot to get the optimized result, when the context length is set to 11000 and Environment="OLLAMA_NUM_PARALLEL=10" in ollamma . service file, it works good, utilizing all 4 GPUS and minimum CPU. However, If I increase either the context length to let say 15000 or num parallel to let say 15, the speed reduces drastically and the load is being shared almost 50- 50 by CPU and GPU, GPU not being utilized fully causing slowness in response when just 5-6 concurrent sessions as there. If I further increase either od context length to 20K or num parallel to 20, then in such cases and beyond, it stops using GPU and load is fully transferred to CPU which kills the speed totally. Please can someone help on this, I want to understand what makes the load spill to CPU in such cases and why can't we utilize GPU full after certain extent. For information - we thought this is the max 12xLarge hardware can support, so we increased to G4dn.metal for testing but the result is same. Need help from experts please, is it due to some configurations in ollama or openwebui or is it that T4 GPUS in both cases each unit is only 16GB, is it that we need entire GPU memory in just 1GPU to fully utilize it, if the GPU , in my case of 12x Large, it is 64(16*4) split into 4. Any suggestions or guidance will be very helpful ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.5
GiteaMirror added the bug label 2026-04-12 16:38:38 -05:00
GiteaMirror commented 2026-04-12 16:38:39 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 14, 2025):

ollama will fill up GPUs with model weights and context buffers. Once the combination of weights and context buffers grows more than available VRAM, ollama will start to spill to system RAM. Model layers in VRAM are processed by the GPU, model layers in system RAM are processed by the CPU. The GPU is much faster at performing inference computation than CPU, so the utilization on the GPU decreases as it is doing nothing while waiting for the CPU to finish inference computation on the part of the model in system RAM. Further increasing context allocation by increase context length or parallel computations reaches the point where nothing will fit in VRAM and the entire model is moved to system RAM.

To maximize token generation speed, you need to keep as much as you can in VRAM. The weights are a constant size, but as you increase context length and parallel processing, the VRAM increases by (length * num parallel). If you can't limit either of these factors, you need to increase VRAM with more hardware.

<!-- gh-comment-id:2589140816 --> @rick-github commented on GitHub (Jan 14, 2025): ollama will fill up GPUs with model weights and context buffers. Once the combination of weights and context buffers grows more than available VRAM, ollama will start to spill to system RAM. Model layers in VRAM are processed by the GPU, model layers in system RAM are processed by the CPU. The GPU is much faster at performing inference computation than CPU, so the utilization on the GPU decreases as it is doing nothing while waiting for the CPU to finish inference computation on the part of the model in system RAM. Further increasing context allocation by increase context length or parallel computations reaches the point where nothing will fit in VRAM and the entire model is moved to system RAM. To maximize token generation speed, you need to keep as much as you can in VRAM. The weights are a constant size, but as you increase context length and parallel processing, the VRAM increases by (length * num parallel). If you can't limit either of these factors, you need to increase VRAM with more hardware.
GiteaMirror commented 2026-04-12 16:38:40 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 14, 2025):

Flash attention uses context space more efficiently and can reduce memory pressure.

<!-- gh-comment-id:2589147486 --> @rick-github commented on GitHub (Jan 14, 2025): [Flash attention](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-enable-flash-attention) uses context space more efficiently and can reduce memory pressure.
GiteaMirror commented 2026-04-12 16:38:40 -05:00
Author
Owner
Copy Link

@rpaGuyai commented on GitHub (Jan 14, 2025):

ollama will fill up GPUs with model weights and context buffers. Once the combination of weights and context buffers grows more than available VRAM, ollama will start to spill to system RAM. Model layers in VRAM are processed by the GPU, model layers in system RAM are processed by the CPU. The GPU is much faster at performing inference computation than CPU, so the utilization on the GPU decreases as it is doing nothing while waiting for the CPU to finish inference computation on the part of the model in system RAM. Further increasing context allocation by increase context length or parallel computations reaches the point where nothing will fit in VRAM and the entire model is moved to system RAM.

To maximize token generation speed, you need to keep as much as you can in VRAM. The weights are a constant size, but as you increase context length and parallel processing, the VRAM increases by (length * num parallel). If you can't limit either of these factors, you need to increase VRAM with more hardware.

Is this issue occurring because the GPU memory is split across 4 GPUs, each having 16 GB VRAM, and that 64 GB of total VRAM is not enough to handle larger context lengths and parallelism?
Would a single GPU with higher VRAM (e.g., 40 GB or 80 GB) be more effective in fully utilizing the GPU for such workloads, avoiding the spillover to CPU?

<!-- gh-comment-id:2589516820 --> @rpaGuyai commented on GitHub (Jan 14, 2025): > ollama will fill up GPUs with model weights and context buffers. Once the combination of weights and context buffers grows more than available VRAM, ollama will start to spill to system RAM. Model layers in VRAM are processed by the GPU, model layers in system RAM are processed by the CPU. The GPU is much faster at performing inference computation than CPU, so the utilization on the GPU decreases as it is doing nothing while waiting for the CPU to finish inference computation on the part of the model in system RAM. Further increasing context allocation by increase context length or parallel computations reaches the point where nothing will fit in VRAM and the entire model is moved to system RAM. > > To maximize token generation speed, you need to keep as much as you can in VRAM. The weights are a constant size, but as you increase context length and parallel processing, the VRAM increases by (length * num parallel). If you can't limit either of these factors, you need to increase VRAM with more hardware. Is this issue occurring because the GPU memory is split across 4 GPUs, each having 16 GB VRAM, and that 64 GB of total VRAM is not enough to handle larger context lengths and parallelism? Would a single GPU with higher VRAM (e.g., 40 GB or 80 GB) be more effective in fully utilizing the GPU for such workloads, avoiding the spillover to CPU?
GiteaMirror commented 2026-04-12 16:38:40 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 14, 2025):

There is more overhead with more GPUs, but 64GB is likely not enough to handle the weights, context and parallelism you are configuring, irrespective of how it is divided up.

For an individual completion, a single large GPU is better than many smaller GPUs because of the inefficiency in moving intermediate results between GPUs.

Note that depending on model and workload, a different GPU might work better. I did a quick test a while ago and found that other GPUs worked better than the T4.

<!-- gh-comment-id:2589651319 --> @rick-github commented on GitHub (Jan 14, 2025): There is more overhead with more GPUs, but 64GB is likely not enough to handle the weights, context and parallelism you are configuring, irrespective of how it is divided up. For an individual completion, a single large GPU is better than many smaller GPUs because of the inefficiency in moving intermediate results between GPUs. Note that depending on model and workload, a different GPU might work better. I did a [quick test](https://github.com/ollama/ollama/issues/7534#issuecomment-2461066429) a while ago and found that other GPUs worked better than the T4.
GiteaMirror commented 2026-04-12 16:38:41 -05:00
Author
Owner
Copy Link

@rpaGuyai commented on GitHub (Jan 15, 2025):

There is more overhead with more GPUs, but 64GB is likely not enough to handle the weights, context and parallelism you are configuring, irrespective of how it is divided up.

For an individual completion, a single large GPU is better than many smaller GPUs because of the inefficiency in moving intermediate results between GPUs.

Note that depending on model and workload, a different GPU might work better. I did a quick test a while ago and found that other GPUs worked better than the T4.
Thanks for you insight, we are planning to test on a single large GPU, probably A100, instead of T4.
Also, jut FYI, we upgraded our current G4dn.12xlarge to G4dn.metal which has twice the GPU and RAM and CPU, however still it's T4 and each has 16GB, as a result we literally found no difference in performance, hence our doubt that single high power GPUs like A100 would be better.
Will update after test, meanwhile we are still trying to understand if some issue on software side of Ollama or openWebUI that could be causing this issue.

<!-- gh-comment-id:2591547180 --> @rpaGuyai commented on GitHub (Jan 15, 2025): > There is more overhead with more GPUs, but 64GB is likely not enough to handle the weights, context and parallelism you are configuring, irrespective of how it is divided up. > > For an individual completion, a single large GPU is better than many smaller GPUs because of the inefficiency in moving intermediate results between GPUs. > > Note that depending on model and workload, a different GPU might work better. I did a [quick test](https://github.com/ollama/ollama/issues/7534#issuecomment-2461066429) a while ago and found that other GPUs worked better than the T4. Thanks for you insight, we are planning to test on a single large GPU, probably A100, instead of T4. Also, jut FYI, we upgraded our current G4dn.12xlarge to G4dn.metal which has twice the GPU and RAM and CPU, however still it's T4 and each has 16GB, as a result we literally found no difference in performance, hence our doubt that single high power GPUs like A100 would be better. Will update after test, meanwhile we are still trying to understand if some issue on software side of Ollama or openWebUI that could be causing this issue.
GiteaMirror commented 2026-04-12 16:38:41 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 15, 2025):

It's the same T4 GPU, of course it's going to have the same performance.

https://github.com/ollama/ollama/issues/7648#issuecomment-2473561990

Look at it this way: a woman can create a baby in 9 months. How quickly can 9 women create a baby?

<!-- gh-comment-id:2591555128 --> @rick-github commented on GitHub (Jan 15, 2025): It's the same T4 GPU, of course it's going to have the same performance. https://github.com/ollama/ollama/issues/7648#issuecomment-2473561990 Look at it this way: a woman can create a baby in 9 months. How quickly can 9 women create a baby?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5405
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?