github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #3742] Slow Performance with Llama2 on a Dual-GPU System - Seeking Advice #48817

New Issue
Closed
opened 2026-04-28 09:31:36 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-04-28 09:31:36 -05:00
Owner
Copy Link

Originally created by @AkiMatsushita on GitHub (Apr 19, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3742

Originally assigned to: @dhiltgen on GitHub.

Hello ollama Community,

I'm encountering extremely slow performance while running ollama on my PC, specifically with models like Llama2 13B. The issue isn't just the slow output speed (around 1 token/min), but I'm also concerned that my GPUs might not be utilized properly. Below are my PC specs:

  • CPU: Intel Core i7 12650H
  • Memory: 32GB
  • GPU0: Intel UHD Graphics
  • GPU1: NVIDIA GeForce RTX 4060 with 8GB VRAM

When running models such as the Llama2 13B, the performance drastically slows down. Interestingly, out of the 8GB VRAM, only about 6.1GB is being used, and the GPU utilization rate is close to 0%.

For comparison, I've also tried running larger models like Llama2 70B on a different PC equipped with a GeForce RTX 4060ti with 16GB VRAM. In this case, almost all of the VRAM is utilized, and the GPU utilization rate reaches about 10%.

I'm wondering if the issue with the first PC might be related to it having two GPUs, which could be causing incorrect GPU utilization, or if it's simply a matter of insufficient VRAM.

Could anyone please advise on whether this is an issue with GPU utilization due to the dual-GPU setup or if the VRAM is indeed insufficient? Any insights or suggestions would be greatly appreciated.

Thank you!

Originally created by @AkiMatsushita on GitHub (Apr 19, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3742 Originally assigned to: @dhiltgen on GitHub. Hello ollama Community, I'm encountering extremely slow performance while running ollama on my PC, specifically with models like Llama2 13B. The issue isn't just the slow output speed (around 1 token/min), but I'm also concerned that my GPUs might not be utilized properly. Below are my PC specs: - CPU: Intel Core i7 12650H - Memory: 32GB - GPU0: Intel UHD Graphics - GPU1: NVIDIA GeForce RTX 4060 with 8GB VRAM When running models such as the Llama2 13B, the performance drastically slows down. Interestingly, out of the 8GB VRAM, only about 6.1GB is being used, and the GPU utilization rate is close to 0%. For comparison, I've also tried running larger models like Llama2 70B on a different PC equipped with a GeForce RTX 4060ti with 16GB VRAM. In this case, almost all of the VRAM is utilized, and the GPU utilization rate reaches about 10%. I'm wondering if the issue with the first PC might be related to it having two GPUs, which could be causing incorrect GPU utilization, or if it's simply a matter of insufficient VRAM. Could anyone please advise on whether this is an issue with GPU utilization due to the dual-GPU setup or if the VRAM is indeed insufficient? Any insights or suggestions would be greatly appreciated. Thank you!
GiteaMirror added the needs more info label 2026-04-28 09:31:36 -05:00
GiteaMirror commented 2026-04-28 09:31:42 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 19, 2024):

Can you share your server log so we can see how much of the model is being loaded in VRAM vs. running on the CPU?

Are you on Windows or Linux?

<!-- gh-comment-id:2067290629 --> @dhiltgen commented on GitHub (Apr 19, 2024): Can you share your server log so we can see how much of the model is being loaded in VRAM vs. running on the CPU? Are you on Windows or Linux?
GiteaMirror commented 2026-04-28 09:31:43 -05:00
Author
Owner
Copy Link

@AkiMatsushita commented on GitHub (Apr 19, 2024):

@dhiltgen thank you for your reply.
Here is my server.log

server.log

<!-- gh-comment-id:2067330980 --> @AkiMatsushita commented on GitHub (Apr 19, 2024): @dhiltgen thank you for your reply. Here is my server.log [server.log](https://github.com/ollama/ollama/files/15045871/server.log)
GiteaMirror commented 2026-04-28 09:31:45 -05:00
Author
Owner
Copy Link

@AkiMatsushita commented on GitHub (Apr 22, 2024):

I use Windows 11 Pro.

<!-- gh-comment-id:2068374240 --> @AkiMatsushita commented on GitHub (Apr 22, 2024): I use Windows 11 Pro.
GiteaMirror commented 2026-04-28 09:31:46 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 22, 2024):

I see a few different scenarios in your logs. gemma seems to be fully loading offloaded 29/29 layers to GPU while the llama model you're loading is about 2/3's in GPU with offloaded 27/41 layers to GPU I would expect the gemma token rate to be good, and the llama rate to be fairly slow.

The Intel GPU isn't supported, so only the 4060 is being used.

<!-- gh-comment-id:2071007232 --> @dhiltgen commented on GitHub (Apr 22, 2024): I see a few different scenarios in your logs. gemma seems to be fully loading `offloaded 29/29 layers to GPU` while the llama model you're loading is about 2/3's in GPU with `offloaded 27/41 layers to GPU` I would expect the gemma token rate to be good, and the llama rate to be fairly slow. The Intel GPU isn't supported, so only the 4060 is being used.
GiteaMirror commented 2026-04-28 09:31:47 -05:00
Author
Owner
Copy Link

@AkiMatsushita commented on GitHub (Apr 22, 2024):

@dhiltgen
Thank you for answering.
I understood that it was simply not enough VRAM for the llama2 13B model.
Decreasing the value of the num_ctx parameter will improve the speed, so I will do that.

<!-- gh-comment-id:2071069721 --> @AkiMatsushita commented on GitHub (Apr 22, 2024): @dhiltgen Thank you for answering. I understood that it was simply not enough VRAM for the llama2 13B model. Decreasing the value of the num_ctx parameter will improve the speed, so I will do that.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label needs more info
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#48817
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?