github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6816] High GPU and CPU usage #30065

New Issue
Closed
opened 2026-04-22 09:30:13 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-22 09:30:13 -05:00
Owner
Copy Link

Originally created by @akseg73 on GitHub (Sep 15, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6816

What is the issue?

So i am running llms with ollama on linux. I downloaded nvtop to check NVIDIA GPU usage. It seems that during inferencing CPU as well GPU usage shoots up. I am not sure why the CPU utilization is so high. Additionally the rate at which tokens are generated doesn't seem to suggest if the GPU is really in use.

Is there some tool to determine the token generation rate? This will help determine if this is the GPU or CPU that is being utilized. Any help is appreciated.

This is what the journalctl -u ollama.service contain (which suggests that the Nvidia GPU is detected by ollama and is being utilized"

Sep 15 06:46:37 fedora ollama[1615]: time=2024-09-15T06:46:37.717-07:00 level=INFO source=gpu.go:200 msg="looking for compatible GPUs"
Sep 15 06:46:38 fedora ollama[1615]: time=2024-09-15T06:46:38.492-07:00 level=INFO source=types.go:107 msg="inference compute" id=GPU-644fab1e-0e88-895f-8468-deb48db36a81 library=cuda variant=v12 compute=8.9 driver=12.6 name="NVIDIA GeForce RTX 4070" total="11.6 GiB" available="11.5 GiB"

OS

No response

GPU

Nvidia

CPU

Intel

Ollama version

No response

Originally created by @akseg73 on GitHub (Sep 15, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6816 ### What is the issue? So i am running llms with ollama on linux. I downloaded nvtop to check NVIDIA GPU usage. It seems that during inferencing CPU as well GPU usage shoots up. I am not sure why the CPU utilization is so high. Additionally the rate at which tokens are generated doesn't seem to suggest if the GPU is really in use. Is there some tool to determine the token generation rate? This will help determine if this is the GPU or CPU that is being utilized. Any help is appreciated. This is what the journalctl -u ollama.service contain (which suggests that the Nvidia GPU is detected by ollama and is being utilized" Sep 15 06:46:37 fedora ollama[1615]: time=2024-09-15T06:46:37.717-07:00 level=INFO source=gpu.go:200 msg="looking for compatible GPUs" Sep 15 06:46:38 fedora ollama[1615]: time=2024-09-15T06:46:38.492-07:00 level=INFO source=types.go:107 msg="inference compute" id=GPU-644fab1e-0e88-895f-8468-deb48db36a81 library=cuda variant=v12 compute=8.9 driver=12.6 name="NVIDIA GeForce RTX 4070" total="11.6 GiB" available="11.5 GiB" ### OS _No response_ ### GPU Nvidia ### CPU Intel ### Ollama version _No response_
GiteaMirror added the bug label 2026-04-22 09:30:13 -05:00
GiteaMirror commented 2026-04-22 09:30:15 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Sep 15, 2024):

It sounds like your GPU has run out of memory and a portion of the model is being loaded into system memory and being run off of the CPU. You can use ollama ps to determine how much of your model is on the GPU or CPU. You'll need to either use a model with a smaller parameter count or a higher quantization level if you want the model to fit onto your GPU.

In the Ollama CLI you can use the command /set verbose to report token generation speeds (or use the --verbose flag when starting it).

I'll go ahead and close out the issue.

<!-- gh-comment-id:2351646543 --> @pdevine commented on GitHub (Sep 15, 2024): It sounds like your GPU has run out of memory and a portion of the model is being loaded into system memory and being run off of the CPU. You can use `ollama ps` to determine how much of your model is on the GPU or CPU. You'll need to either use a model with a smaller parameter count or a higher quantization level if you want the model to fit onto your GPU. In the Ollama CLI you can use the command `/set verbose` to report token generation speeds (or use the --verbose flag when starting it). I'll go ahead and close out the issue.
GiteaMirror commented 2026-04-22 09:30:16 -05:00
Author
Owner
Copy Link

@akseg73 commented on GitHub (Sep 15, 2024):

This is the output of ollama ps ... the model is about 9GB and the space on GPU is about 12GB ... And the video on the monitor should not be utilizing the GPU from the best i can tell

ollama ps
NAME ID SIZE PROCESSOR UNTIL
gemma2_Q8:latest 0e66c6bd7971 12 GB 8%/92% CPU/GPU 4 minutes from now

<!-- gh-comment-id:2351784140 --> @akseg73 commented on GitHub (Sep 15, 2024): This is the output of ollama ps ... the model is about 9GB and the space on GPU is about 12GB ... And the video on the monitor should not be utilizing the GPU from the best i can tell ollama ps NAME ID SIZE PROCESSOR UNTIL gemma2_Q8:latest 0e66c6bd7971 12 GB 8%/92% CPU/GPU 4 minutes from now
GiteaMirror commented 2026-04-22 09:30:17 -05:00
Author
Owner
Copy Link

@akseg73 commented on GitHub (Sep 15, 2024):

Utilizing a different model this is what i see. In this case the model fits into the GPU VRAM, however when utilizing the model i still see a little bit of spike on CPU and integrated graphics ... however the spike on CPU has gone down.

ollama ps
NAME ID SIZE PROCESSOR UNTIL
llama3_Q8:latest 9fe009a18ad9 10 GB 100% GPU 4 minutes from now

<!-- gh-comment-id:2351785970 --> @akseg73 commented on GitHub (Sep 15, 2024): Utilizing a different model this is what i see. In this case the model fits into the GPU VRAM, however when utilizing the model i still see a little bit of spike on CPU and integrated graphics ... however the spike on CPU has gone down. ollama ps NAME ID SIZE PROCESSOR UNTIL llama3_Q8:latest 9fe009a18ad9 10 GB 100% GPU 4 minutes from now
GiteaMirror commented 2026-04-22 09:30:18 -05:00
Author
Owner
Copy Link

@akseg73 commented on GitHub (Sep 15, 2024):

I guess it makes sense. Since some amount of CPU, integrated GPU is needed to display results as well.

<!-- gh-comment-id:2351787013 --> @akseg73 commented on GitHub (Sep 15, 2024): I guess it makes sense. Since some amount of CPU, integrated GPU is needed to display results as well.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#30065
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?