github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #6163] GPU Usage Never Exceeds 70% When Using LLaMA 3:8B with Ollama #29611

New Issue
Closed
opened 2026-04-22 08:38:10 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-22 08:38:10 -05:00
Owner
Copy Link

Originally created by @drspam1991 on GitHub (Aug 4, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6163

What is the issue?

When using the Ollama tool with the LLaMA 3:8B model and all 33 offload layers loaded on the GPU, the GPU usage never goes over 70%. This seems suboptimal and may indicate an issue with how resources are being utilized.

Screenshot from 2024-08-04 17-24-02

OS

Linux

GPU

Nvidia

CPU

No response

Ollama version

0.2.8

Originally created by @drspam1991 on GitHub (Aug 4, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6163 ### What is the issue? When using the Ollama tool with the LLaMA 3:8B model and all 33 offload layers loaded on the GPU, the GPU usage never goes over 70%. This seems suboptimal and may indicate an issue with how resources are being utilized. ![Screenshot from 2024-08-04 17-24-02](https://github.com/user-attachments/assets/81d3a8e1-89f0-4d21-9e2c-141db55f585e) ### OS Linux ### GPU Nvidia ### CPU _No response_ ### Ollama version 0.2.8
GiteaMirror added the needs more infobug labels 2026-04-22 08:38:10 -05:00
GiteaMirror commented 2026-04-22 08:38:11 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 4, 2024):

What's the prompt? What's the CPU usage? What's the output of nvidia-smi -q -d POWER,TEMPERATURE? What's the output of nvidia-smi -q | grep -A9 'Clocks Throttle Reasons'?

<!-- gh-comment-id:2267596894 --> @rick-github commented on GitHub (Aug 4, 2024): What's the prompt? What's the CPU usage? What's the output of `nvidia-smi -q -d POWER,TEMPERATURE`? What's the output of `nvidia-smi -q | grep -A9 'Clocks Throttle Reasons'`?
GiteaMirror commented 2026-04-22 08:38:12 -05:00
Author
Owner
Copy Link

@drspam1991 commented on GitHub (Aug 4, 2024):

Thank you for reply.

I use 'hey' load test tool to send parallel request to ollama with this prompt:
"Consider the following Text and list of Topics: \n Text: 'مقایسه آماری ۲ دوره متوالی حراج هنر مدرن و معاصر/ چرا فروش کلی کاهش یافت؟\n' \n Topics: ( Painting|Sculpture|Photography|Drawing|Digital Art|Visual Arts|Theater|Dance|Music|Opera|Performing Arts|Poetry|Fiction|Non-Fiction|Play Script|Literary Arts|Art Movements|Art Styles|Censorship|Art Sales|Book|Events and Festivals|Graphic Design|Interior Design|Industrial Design|Fashion Design|Crafts|Architecture|Applied Arts|International Trade|Unemployment|Learning|Financial Crime|Book|Tax|Inflation|Import/Export|Currency|Banking|Accounting|Blockchain|Real Estate|Valuable Metals|Ministry of Economy|Ministry of Industry|Exchange|Brokerage|Marketing|Labor Market|Labor Migration|Wages and Benefits|Insurance|Investing|Saving|Retirement|Personal Finance|Stock Market|Forex Market|Crypto Currency|Gold|Financial Markets ). \n Choose most related topic(s) to the Text from the list. Your output must be topics name from the list with format [t1, ...] and say nothing else."

This is my CPU usage during test:
Screenshot from 2024-08-04 21-19-24

Output of nvidia-smi -q -d POWER,TEMPERATURE :
==============NVSMI LOG==============

Timestamp : Sun Aug 4 21:13:32 2024
Driver Version : 545.29.06
CUDA Version : 12.3

Attached GPUs : 1
GPU 00000000:0B:00.0
Temperature
GPU Current Temp : 54 C
GPU T.Limit Temp : N/A
GPU Shutdown Temp : 100 C
GPU Slowdown Temp : 97 C
GPU Max Operating Temp : 88 C
GPU Target Temperature : 83 C
Memory Current Temp : N/A
Memory Max Operating Temp : N/A
GPU Power Readings
Power Draw : 2.93 W
Current Power Limit : 225.00 W
Requested Power Limit : 225.00 W
Default Power Limit : 225.00 W
Min Power Limit : 125.00 W
Max Power Limit : 280.00 W
Power Samples
Duration : 52.45 sec
Number of Samples : 119
Max : 42.78 W
Min : 2.88 W
Avg : 4.44 W
GPU Memory Power Readings
Power Draw : N/A
Module Power Readings
Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A

and nvidia-smi -q | grep -A9 'Clocks Throttle Reasons' has not any result

<!-- gh-comment-id:2267619356 --> @drspam1991 commented on GitHub (Aug 4, 2024): Thank you for reply. **I use 'hey' load test tool to send parallel request to ollama with this prompt:** "Consider the following Text and list of Topics: \n Text: 'مقایسه آماری ۲ دوره متوالی حراج هنر مدرن و معاصر/ چرا فروش کلی کاهش یافت؟\n' \n Topics: ( Painting|Sculpture|Photography|Drawing|Digital Art|Visual Arts|Theater|Dance|Music|Opera|Performing Arts|Poetry|Fiction|Non-Fiction|Play Script|Literary Arts|Art Movements|Art Styles|Censorship|Art Sales|Book|Events and Festivals|Graphic Design|Interior Design|Industrial Design|Fashion Design|Crafts|Architecture|Applied Arts|International Trade|Unemployment|Learning|Financial Crime|Book|Tax|Inflation|Import/Export|Currency|Banking|Accounting|Blockchain|Real Estate|Valuable Metals|Ministry of Economy|Ministry of Industry|Exchange|Brokerage|Marketing|Labor Market|Labor Migration|Wages and Benefits|Insurance|Investing|Saving|Retirement|Personal Finance|Stock Market|Forex Market|Crypto Currency|Gold|Financial Markets ). \n Choose most related topic(s) to the Text from the list. Your output must be topics name from the list with format [t1, ...] and say nothing else." **This is my CPU usage during test:** ![Screenshot from 2024-08-04 21-19-24](https://github.com/user-attachments/assets/f26b699e-9f20-43ee-8dde-c23e367e9fdd) Output of `nvidia-smi -q -d POWER,TEMPERATURE` : ==============NVSMI LOG============== Timestamp : Sun Aug 4 21:13:32 2024 Driver Version : 545.29.06 CUDA Version : 12.3 Attached GPUs : 1 GPU 00000000:0B:00.0 Temperature GPU Current Temp : 54 C GPU T.Limit Temp : N/A GPU Shutdown Temp : 100 C GPU Slowdown Temp : 97 C GPU Max Operating Temp : 88 C GPU Target Temperature : 83 C Memory Current Temp : N/A Memory Max Operating Temp : N/A GPU Power Readings Power Draw : 2.93 W Current Power Limit : 225.00 W Requested Power Limit : 225.00 W Default Power Limit : 225.00 W Min Power Limit : 125.00 W Max Power Limit : 280.00 W Power Samples Duration : 52.45 sec Number of Samples : 119 Max : 42.78 W Min : 2.88 W Avg : 4.44 W GPU Memory Power Readings Power Draw : N/A Module Power Readings Power Draw : N/A Current Power Limit : N/A Requested Power Limit : N/A Default Power Limit : N/A Min Power Limit : N/A Max Power Limit : N/A and `nvidia-smi -q | grep -A9 'Clocks Throttle Reasons'` has not any result
GiteaMirror commented 2026-04-22 08:38:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 4, 2024):

Output of nvidia-smi -q -d POWER,TEMPERATURE,PERFORMANCE while the test is running? What is OLLAMA_NUM_PARALLEL set to? Can you include some server logs?

<!-- gh-comment-id:2267624369 --> @rick-github commented on GitHub (Aug 4, 2024): Output of `nvidia-smi -q -d POWER,TEMPERATURE,PERFORMANCE` while the test is running? What is `OLLAMA_NUM_PARALLEL` set to? Can you include some [server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues)?
GiteaMirror commented 2026-04-22 08:38:13 -05:00
Author
Owner
Copy Link

@igorschlum commented on GitHub (Aug 4, 2024):

@drspam1991 did you tried with another model than LLama3:8B? What are the results?

<!-- gh-comment-id:2267628503 --> @igorschlum commented on GitHub (Aug 4, 2024): @drspam1991 did you tried with another model than LLama3:8B? What are the results?
GiteaMirror commented 2026-04-22 08:38:14 -05:00
Author
Owner
Copy Link

@drspam1991 commented on GitHub (Aug 5, 2024):

Output of nvidia-smi -q -d POWER,TEMPERATURE,PERFORMANCE while the test is running? What is OLLAMA_NUM_PARALLEL set to? Can you include some server logs?

This is the result of nvidia-smi -q -d POWER,TEMPERATURE,PERFORMANCE while test is running:

==============NVSMI LOG==============

Timestamp : Mon Aug 5 12:57:09 2024
Driver Version : 545.29.06
CUDA Version : 12.3

Attached GPUs : 1
GPU 00000000:0B:00.0
Performance State : P2
Clocks Event Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Temperature
GPU Current Temp : 77 C
GPU T.Limit Temp : N/A
GPU Shutdown Temp : 100 C
GPU Slowdown Temp : 97 C
GPU Max Operating Temp : 88 C
GPU Target Temperature : 83 C
Memory Current Temp : N/A
Memory Max Operating Temp : N/A
GPU Power Readings
Power Draw : 157.68 W
Current Power Limit : 225.00 W
Requested Power Limit : 225.00 W
Default Power Limit : 225.00 W
Min Power Limit : 125.00 W
Max Power Limit : 280.00 W
Power Samples
Duration : 2.37 sec
Number of Samples : 119
Max : 246.08 W
Min : 60.98 W
Avg : 159.74 W
GPU Memory Power Readings
Power Draw : N/A
Module Power Readings
Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A

The OLLAMA_NUM_PARALLEL is set to 5. When I increase this environment variable from 5, the throughput drops

And this is the ollama log:
ollama.log

<!-- gh-comment-id:2268629974 --> @drspam1991 commented on GitHub (Aug 5, 2024): > Output of `nvidia-smi -q -d POWER,TEMPERATURE,PERFORMANCE` while the test is running? What is `OLLAMA_NUM_PARALLEL` set to? Can you include some [server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues)? **This is the result of** `nvidia-smi -q -d POWER,TEMPERATURE,PERFORMANCE` **while test is running:** ==============NVSMI LOG============== Timestamp : Mon Aug 5 12:57:09 2024 Driver Version : 545.29.06 CUDA Version : 12.3 Attached GPUs : 1 GPU 00000000:0B:00.0 Performance State : P2 Clocks Event Reasons Idle : Not Active Applications Clocks Setting : Not Active SW Power Cap : Active HW Slowdown : Not Active HW Thermal Slowdown : Not Active HW Power Brake Slowdown : Not Active Sync Boost : Not Active SW Thermal Slowdown : Not Active Display Clock Setting : Not Active Temperature GPU Current Temp : 77 C GPU T.Limit Temp : N/A GPU Shutdown Temp : 100 C GPU Slowdown Temp : 97 C GPU Max Operating Temp : 88 C GPU Target Temperature : 83 C Memory Current Temp : N/A Memory Max Operating Temp : N/A GPU Power Readings Power Draw : 157.68 W Current Power Limit : 225.00 W Requested Power Limit : 225.00 W Default Power Limit : 225.00 W Min Power Limit : 125.00 W Max Power Limit : 280.00 W Power Samples Duration : 2.37 sec Number of Samples : 119 Max : 246.08 W Min : 60.98 W Avg : 159.74 W GPU Memory Power Readings Power Draw : N/A Module Power Readings Power Draw : N/A Current Power Limit : N/A Requested Power Limit : N/A Default Power Limit : N/A Min Power Limit : N/A Max Power Limit : N/A The OLLAMA_NUM_PARALLEL is set to 5. When I increase this environment variable from 5, the throughput drops **And this is the ollama log:** [ollama.log](https://github.com/user-attachments/files/16493857/ollama.log)
GiteaMirror commented 2026-04-22 08:38:15 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 5, 2024):

It looks there is some clock throttling happening: SW Power Cap : Active. I'm not sure this fully explains the low GPU usage, but it's worth looking in to.

From the nvidia-smi manual page:

SW Power Cap SW Power Scaling algorithm is reducing the clocks below
 requested clocks because the GPU is consuming too much
 power. E.g. SW power cap limit can be changed with
 nvidia-smi --power-limit=

But you may not be able to do anything about this by adjusting the power-limit:

   -pl, --power-limit=POWER_LIMIT
       Specifies maximum power limit in watts.  Accepts integer and floating point numbers.
       Only on supported devices from Kepler family.  Requires administrator privileges.
       Value  needs  to be between Min and Max Power Limit as reported by nvidia-smi.

From Wikipedia:

Kepler is the codename for a GPU microarchitecture developed by Nvidia, first introduced at retail in April 2012,[1] as the successor to the Fermi microarchitecture. Kepler was Nvidia's first microarchitecture to focus on energy efficiency. Most GeForce 600 series, most GeForce 700 series, and some GeForce 800M series GPUs were based on Kepler, all manufactured in 28 nm. Kepler found use in the GK20A, the GPU component of the Tegra K1 SoC, and in the Quadro Kxxx series, the Quadro NVS 510, and Tesla computing modules.

From GEFORCE_RTX_2080_User_Guide.pdf:

The GeForce® RTX 2080 is powered by the all-new NVIDIA Turing™ architecture to give you incredible new levels of gaming realism, speed, power efficiency, and immersion. This is graphics reinvented.

I also noticed in your first screencap of nvtop that your temperature was at 81°C, and your GPU Target Temperature is 83°C, so you may also be experiencing SW Thermal Slowdown.

You can monitor the GPU temperature, power draw and clock rate in nvtop by adjusting the settings in Setup > Chart > Displayed all GPUs.

<!-- gh-comment-id:2269236072 --> @rick-github commented on GitHub (Aug 5, 2024): It looks there is some clock throttling happening: `SW Power Cap : Active`. I'm not sure this fully explains the low GPU usage, but it's worth looking in to. From the nvidia-smi manual page: ``` SW Power Cap SW Power Scaling algorithm is reducing the clocks below requested clocks because the GPU is consuming too much power. E.g. SW power cap limit can be changed with nvidia-smi --power-limit= ``` But you may not be able to do anything about this by adjusting the power-limit: ``` -pl, --power-limit=POWER_LIMIT Specifies maximum power limit in watts. Accepts integer and floating point numbers. Only on supported devices from Kepler family. Requires administrator privileges. Value needs to be between Min and Max Power Limit as reported by nvidia-smi. ``` From Wikipedia: Kepler is the codename for a [GPU](https://en.wikipedia.org/wiki/GPU) [microarchitecture](https://en.wikipedia.org/wiki/Microarchitecture) developed by [Nvidia](https://en.wikipedia.org/wiki/Nvidia), first introduced at retail in April 2012,[[1]](https://en.wikipedia.org/wiki/Kepler_(microarchitecture)#cite_note-1) as the successor to the [Fermi](https://en.wikipedia.org/wiki/Fermi_(microarchitecture)) microarchitecture. Kepler was Nvidia's first microarchitecture to focus on [energy efficiency](https://en.wikipedia.org/wiki/Efficient_energy_use). Most [GeForce 600 series](https://en.wikipedia.org/wiki/GeForce_600_series), most [GeForce 700 series](https://en.wikipedia.org/wiki/GeForce_700_series), and some [GeForce 800M series](https://en.wikipedia.org/wiki/GeForce_800M_series) GPUs were based on Kepler, all manufactured in 28 nm. Kepler found use in the GK20A, the GPU component of the [Tegra K1](https://en.wikipedia.org/wiki/Tegra_K1) [SoC](https://en.wikipedia.org/wiki/System_on_a_chip), and in the [Quadro](https://en.wikipedia.org/wiki/Nvidia_Quadro) Kxxx series, the Quadro NVS 510, and [Tesla](https://en.wikipedia.org/wiki/Nvidia_Tesla) computing modules. From [GEFORCE_RTX_2080_User_Guide.pdf](https://www.nvidia.com/content/geforce-gtx/GEFORCE_RTX_2080_User_Guide.pdf): The GeForce® RTX 2080 is powered by the all-new NVIDIA Turing™ architecture to give you incredible new levels of gaming realism, speed, power efficiency, and immersion. This is graphics reinvented. I also noticed in your first screencap of nvtop that your temperature was at 81°C, and your `GPU Target Temperature` is 83°C, so you may also be experiencing `SW Thermal Slowdown`. You can monitor the GPU temperature, power draw and clock rate in nvtop by adjusting the settings in Setup > Chart > Displayed all GPUs.
GiteaMirror commented 2026-04-22 08:38:15 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Sep 13, 2024):

@drspam1991 did you end up sorting this out?

<!-- gh-comment-id:2349717961 --> @pdevine commented on GitHub (Sep 13, 2024): @drspam1991 did you end up sorting this out?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug needs more info
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#29611
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?