github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #2642] 🚀🔍 GPU Mystery: Unleashing the Power on Small Models but Stuck on Idle with Giants like MiXtral8x7B & Llama 70B on Ubuntu 22 🧩💡 #1564

New Issue
Open
opened 2026-04-12 11:29:56 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-12 11:29:56 -05:00
Owner
Copy Link

Originally created by @jaifar530 on GitHub (Feb 21, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2642

Hi

Using Ubuntu 22.
both commands nvcc --version and nvidia-smi are showing valied outputs.

I've noticed that the GPU is not utilized when running larger models (e.g., MiXtral8x7B, Llama 70B), yet it functions well with smaller models like Mistral and Llama 7B. Is this issue known to others, or is it just me experiencing it? By the way, I tested this on both RTX 3090 and RTX 2080, and both exhibited the same issue with the larger models.

Additionally, with the larger models (Mistral and Llama 70B), the GPU RAM is almost fully utilized, but not the GPU itself (which is very strange), while the CPU is fully utilized.

Here is the summary:

Larger models MiXtral8x7B, Llama 70B
GPU: Not utalised
GPU RAM: utalised
CPU: utalised
RAM: Not utalised

Small models Mistral and Llama 7B
GPU: utalised
GPU RAM: utalised
CPU: not utalised
RAM: not utalised

in summary i can use the power of GPU on small models only unfortuntly.

Originally created by @jaifar530 on GitHub (Feb 21, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2642 Hi Using Ubuntu 22. both commands nvcc --version and nvidia-smi are showing valied outputs. I've noticed that the GPU is not utilized when running larger models (e.g., MiXtral8x7B, Llama 70B), yet it functions well with smaller models like Mistral and Llama 7B. Is this issue known to others, or is it just me experiencing it? By the way, I tested this on both RTX 3090 and RTX 2080, and both exhibited the same issue with the larger models. Additionally, with the larger models (Mistral and Llama 70B), the GPU RAM is almost fully utilized, but not the GPU itself (which is very strange), while the CPU is fully utilized. Here is the summary: Larger models MiXtral8x7B, Llama 70B GPU: Not utalised GPU RAM: utalised CPU: utalised RAM: Not utalised Small models Mistral and Llama 7B GPU: utalised GPU RAM: utalised CPU: not utalised RAM: not utalised in summary i can use the power of GPU on small models only unfortuntly.
GiteaMirror added the bug label 2026-04-12 11:29:56 -05:00
GiteaMirror commented 2026-04-12 11:29:57 -05:00
Author
Owner
Copy Link

@seanmavley commented on GitHub (Feb 21, 2024):

What's your PC specs? Larger models require larger RAMs to work

<!-- gh-comment-id:1957302193 --> @seanmavley commented on GitHub (Feb 21, 2024): What's your PC specs? Larger models require larger RAMs to work
GiteaMirror commented 2026-04-12 11:29:57 -05:00
Author
Owner
Copy Link

@jaifar530 commented on GitHub (Feb 21, 2024):

What's your PC specs? Larger models require larger RAMs to work

The ram is 64 GB

<!-- gh-comment-id:1957773526 --> @jaifar530 commented on GitHub (Feb 21, 2024): > What's your PC specs? Larger models require larger RAMs to work The ram is 64 GB
GiteaMirror commented 2026-04-12 11:29:58 -05:00
Author
Owner
Copy Link

@easp commented on GitHub (Feb 21, 2024):

The models you are using don't fit in VRAM on your cards, so part of them is loaded into RAM and processed on the CPU. What is probably happening is that the GPU ends up spending most of its time waiting for the CPU to process the portion of the model in RAM and as a result GPU utilization is low and generation speeds are low as well.

The logs will contain information about how many of the models layers are loaded onto the GPU.

<!-- gh-comment-id:1958183520 --> @easp commented on GitHub (Feb 21, 2024): The models you are using don't fit in VRAM on your cards, so part of them is loaded into RAM and processed on the CPU. What is probably happening is that the GPU ends up spending most of its time waiting for the CPU to process the portion of the model in RAM and as a result GPU utilization is low and generation speeds are low as well. The [logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will contain information about how many of the models layers are loaded onto the GPU.
GiteaMirror commented 2026-04-12 11:29:58 -05:00
Author
Owner
Copy Link

@farzaaaan commented on GitHub (Feb 23, 2024):

The models you are using don't fit in VRAM on your cards, so part of them is loaded into RAM and processed on the CPU. What is probably happening is that the GPU ends up spending most of its time waiting for the CPU to process the portion of the model in RAM and as a result GPU utilization is low and generation speeds are low as well.

The logs will contain information about how many of the models layers are loaded onto the GPU.

this is very likely what's happening to me too, not sure if this is a valid question, but asking in case: are there ways to resolve this? I just picked an RTX3060 with 12GB of RAM, models up to 13B are running well but for instance codellama:34b is almost entirely being processed in CPU. only about half the layers are offloaded to GPU.

<!-- gh-comment-id:1961902261 --> @farzaaaan commented on GitHub (Feb 23, 2024): > The models you are using don't fit in VRAM on your cards, so part of them is loaded into RAM and processed on the CPU. What is probably happening is that the GPU ends up spending most of its time waiting for the CPU to process the portion of the model in RAM and as a result GPU utilization is low and generation speeds are low as well. > > The [logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will contain information about how many of the models layers are loaded onto the GPU. this is very likely what's happening to me too, not sure if this is a valid question, but asking in case: are there ways to resolve this? I just picked an RTX3060 with 12GB of RAM, models up to 13B are running well but for instance codellama:34b is almost entirely being processed in CPU. only about half the layers are offloaded to GPU.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1564
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?