github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9406] ⚠️ CPU limits GPU computation because something is monothreaded (benchmark inside the issue) #6132

New Issue
Closed
opened 2026-04-12 17:28:41 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-04-12 17:28:41 -05:00
Owner
Copy Link

Originally created by @Fade78 on GitHub (Feb 28, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9406

What is the issue?

Since I like ollama, I decided to participate because I think there is a missed opportunity of performance enhancement.

Since I worked in performance benchmarking, I decided to do some tests.

model ctx_length how it's loaded in Ollama Card 1 load Card 2 load ollama process load token per second
qwen2.5:1.5b 2048 qwen2.5:1.5b 65ec06548149 2.0 GB 100% GPU 0% 85% 100% 166
qwen2.5:3b 2048 qwen2.5:3b 357c53fb659c 3.1 GB 100% GPU 0% 90% 100% 98
qwen2.5:7b 32768 qwen2.5:7b 845dbda0ea48 6.0 GB 100% GPU 0% 95% 100% 52
Phi4:14b (large context) 32000 phi4:latest ac896e5b8b34 25 GB 100% GPU 10% 73% 100% 14
qwen2.5:32b 10000 qwen2.5:32b 9f13ba1299af 25 GB 100% GPU 15% 65% 100% 12
  • The processor is a AMD Ryzen 7800X3D, Card 1 is a RTX 4070 12GB and Card2 is a RTX 4060 TI 16GB. Card1 is the primary video adapter.
  • Measures were taken during model inference, not when model is loading nor the model is processing the prompt
  • All PCIe transferts are way below maximum bandwidth (few dozens of MB/s while the slowest card, Card 2, was connected at Gen4 x4 wich allow more than 7GB/s)
  • By ollama process load I mean just this process. 100% means one core out of 8 (16 if you count multithreading). My computer was not loaded by any other task at the time of the test.

The ollama process handling the job for the GPUs is clearly the bottleneck of a full GPU computation. The limit at exactly 100% of one core suggest that there is something to multithread to enhance the performance. To me it seems that the process is already multithread so the fact that it's 100% core every time means that there is a conflict somewhere.

  • Adding more cards will not allow more power to be used, only avoid actual processor and RAM computation, which is already a good thing
  • Removing the bottleneck will increase the performance without changing the hardware

Here is a projection of the performance increase on my rig.

model performance with the CPU thread limitation performance without the bottleneck and same load profile
qwen2.5:1.5b 166 t/s 195 t/s
qwen2.5:3b 98 t/s 109 t/s
qwen2.5:7b 52 t/s 55 t/s
Phi4:14b 14 t/s 19 t/s
qwen2.5:32b 12 t/s 18 t/s

To compute that I just states that the most loaded card will be able to go to 100% of load. This is a low estimation.

If I'm right it means that even with 4x5090 on a 7800X3D computer, the performance will not be significantly faster than my 4060TI for the model I tested.

Bonus: look at this video. A guy is doing LLM benchmark with 4x 4060 TI 16GB. Here is what happens:

  • He gets around 25% of equal compute power on four cards
  • He gats aroung 33% of equal compute power on three cards
  • He gets the same performance for those two configurations
  • In every case, the ollama process is at 100%.
  • It means that if this 100% load could be actually be multithreaded, he could have three (with the three cards setup) to four (with the four cards setup) times the performance.

Next step: I think this must be investigated by people who actually know how to understand what the process is doing.

Relevant log output

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.5.12

Originally created by @Fade78 on GitHub (Feb 28, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9406 ### What is the issue? Since I like ollama, I decided to participate because I think there is a missed opportunity of performance enhancement. Since I worked in performance benchmarking, I decided to do some tests. | model | ctx_length | how it's loaded in Ollama | Card 1 load | Card 2 load | ollama process load | token per second | | --- | --- | --- | --- | --- | --- | --- | | qwen2.5:1.5b | 2048 | qwen2.5:1.5b 65ec06548149 2.0 GB 100% GPU | 0% | 85% | 100% | 166 | | qwen2.5:3b | 2048 | qwen2.5:3b 357c53fb659c 3.1 GB 100% GPU | 0% | 90% | 100% | 98 | | qwen2.5:7b | 32768 | qwen2.5:7b 845dbda0ea48 6.0 GB 100% GPU | 0% | 95% | 100% | 52 | | Phi4:14b (large context) | 32000 |phi4:latest ac896e5b8b34 25 GB 100% GPU | 10% | 73% | 100% | 14 | | qwen2.5:32b | 10000 | qwen2.5:32b 9f13ba1299af 25 GB 100% GPU | 15% | 65% | 100% | 12 | * The processor is a AMD Ryzen 7800X3D, Card 1 is a RTX 4070 12GB and Card2 is a RTX 4060 TI 16GB. Card1 is the primary video adapter. * Measures were taken during model inference, not when model is loading nor the model is processing the prompt * All PCIe transferts are way below maximum bandwidth (few dozens of MB/s while the slowest card, Card 2, was connected at Gen4 x4 wich allow more than 7GB/s) * By *ollama process load* I mean just this process. 100% means one core out of 8 (16 if you count multithreading). My computer was not loaded by any other task at the time of the test. **The ollama process handling the job for the GPUs is clearly the bottleneck of a full GPU computation. The limit at exactly 100% of one core suggest that there is something to multithread to enhance the performance.** To me it seems that the process is already multithread so the fact that it's 100% core every time means that there is a conflict somewhere. * Adding more cards will not allow more power to be used, only avoid actual processor and RAM computation, which is already a good thing * Removing the bottleneck will increase the performance without changing the hardware Here is a projection of the performance increase on my rig. | model | performance with the CPU thread limitation | performance without the bottleneck and same load profile | | --- | --- | --- | | qwen2.5:1.5b | 166 t/s | 195 t/s | | qwen2.5:3b | 98 t/s | 109 t/s | | qwen2.5:7b | 52 t/s | 55 t/s | | Phi4:14b | 14 t/s | 19 t/s | | qwen2.5:32b | 12 t/s | 18 t/s | To compute that I just states that the most loaded card will be able to go to 100% of load. This is a low estimation. If I'm right it means that even with 4x5090 on a 7800X3D computer, the performance will not be significantly faster than my 4060TI for the model I tested. Bonus: look at [this video](https://www.youtube.com/watch?v=OmEiYaPwCF4). A guy is doing LLM benchmark with 4x 4060 TI 16GB. Here is what happens: * He gets around 25% of equal compute power on four cards * He gats aroung 33% of equal compute power on three cards * He gets the **same performance** for those two configurations * In every case, the ollama process is at 100%. * It means that if this 100% load could be actually be multithreaded, he could have three (with the three cards setup) to four (with the four cards setup) times the performance. **Next step:** I think this must be investigated by people who actually know how to understand what the process is doing. ### Relevant log output ```shell ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.5.12
GiteaMirror added the bug label 2026-04-12 17:28:41 -05:00
GiteaMirror commented 2026-04-12 17:28:42 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 28, 2025):

The IPC mechanism between the CPU and GPU is a busy wait, which is why the CPU is at 100%. This has been discussed by people who understand what the process is doing and the conclusion is that multi-threading the inference engine will not improve performance. That's not to say there isn't room for improvement, the busy wait seems wasteful but apparently is a driver level thing.

I scrubbed through the video transcript quickly and didn't see a mention of ollama, but I did hear a mention of localai. It looked like his testing methodology was to ask a question and time the result. For a single completion, multiple GPUs don't improve token generation speed, so 4 cards at 25% each and 3 cards at 33% each is expected. See here for details.

<!-- gh-comment-id:2689445378 --> @rick-github commented on GitHub (Feb 28, 2025): The IPC mechanism between the CPU and GPU is a busy wait, which is why the CPU is at 100%. This has been [discussed](https://github.com/ggml-org/llama.cpp/issues/8684) by people who understand what the process is doing and the conclusion is that multi-threading the inference engine will not improve performance. That's not to say there isn't room for improvement, the busy wait seems wasteful but apparently is a driver level thing. I scrubbed through the video transcript quickly and didn't see a mention of ollama, but I did hear a mention of localai. It looked like his testing methodology was to ask a question and time the result. For a single completion, multiple GPUs don't improve token generation speed, so 4 cards at 25% each and 3 cards at 33% each is expected. See [here](https://github.com/ollama/ollama/issues/7648#issuecomment-2473561990) for details.
GiteaMirror commented 2026-04-12 17:28:43 -05:00
Author
Owner
Copy Link

@Fade78 commented on GitHub (Feb 28, 2025):

% time seconds usecs/call calls errors syscall
98,30 0,792640 2830 280 1 futex
1,70 0,013669 0 16088 sched_yield
0,00 0,000009 1 8 write
0,00 0,000001 1 1 nanosleep
0,00 0,000000 0 8 rt_sigreturn
100,00 0,806319 49 16385 1 total

Indeed, 98% of the time in a futex...

<!-- gh-comment-id:2690035418 --> @Fade78 commented on GitHub (Feb 28, 2025): | % time | seconds | usecs/call | calls | errors | syscall | | ------ | -------:| ---------:| -----:| ------:| ------------- | | 98,30 | 0,792640| 2830 | 280 | 1 | futex | | 1,70 | 0,013669| 0 | 16088 | | sched_yield | | 0,00 | 0,000009| 1 | 8 | | write | | 0,00 | 0,000001| 1 | 1 | | nanosleep | | 0,00 | 0,000000| 0 | 8 | | rt_sigreturn | | 100,00 | 0,806319| 49 | 16385 | 1 | total | Indeed, 98% of the time in a futex...
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6132
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?