github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-11-14 21:08:04 -06:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9166] Unable to use GPU at all on windows #5966

New Issue
Closed
opened 2026-04-12 17:19:02 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-12 17:19:02 -05:00
Owner
Copy Link
Copy Source

Originally created by @sancelot on GitHub (Feb 17, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9166

What is the issue?

I have a MSI Pulse 17 B13V computer :

  • 12 cores
  • 1 Iris Intel GPU
  • 1 nvidia RTX 4060

When running inference, the models seem being loaded in Nvidia memory, BUT inference runs on CPUs !!!!

Setting CUDA_VISIBLE_DEVICES has no effect, either by uid or 0,1 .....

This happens either on Windows or WSL.

altough inference works perfectly with Transformers & accelerate !

More informations coming soon

here is my setup
Image

Relevant log output

OS

Windows

GPU

Nvidia

CPU

Intel

Ollama version

0.5.11

Originally created by @sancelot on GitHub (Feb 17, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9166 ### What is the issue? I have a MSI Pulse 17 B13V computer : * 12 cores * 1 Iris Intel GPU * 1 nvidia RTX 4060 When running inference, the models seem being loaded in Nvidia memory, BUT inference runs on CPUs !!!! Setting CUDA_VISIBLE_DEVICES has no effect, either by uid or 0,1 ..... This happens either on Windows or WSL. altough inference works perfectly with Transformers & accelerate ! More informations coming soon here is my setup ![Image](https://github.com/user-attachments/assets/d0ac4af7-d525-4005-8175-9b77723e563a) ### Relevant log output ```shell ``` ### OS Windows ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.11
GiteaMirror added the bug label 2026-04-12 17:19:02 -05:00
GiteaMirror commented 2026-04-12 17:19:03 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Feb 17, 2025):

Server logs will aid in debugging.

<!-- gh-comment-id:2662442113 --> @rick-github commented on GitHub (Feb 17, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will aid in debugging.
GiteaMirror commented 2026-04-12 17:19:04 -05:00
Author
Owner
Copy Link
Copy Source

@sancelot commented on GitHub (Feb 26, 2025):

At first I deactivated Intel Video , to keep only Nvidia GPU.

=> cpu is always used.

I setted up :
OLLAMA_FLASH_ATTENTION=1
OLLAMA_LLM_LIBRARY="cuda_v11"
CUDA_VISIBLE_DEVICES=0

=> Now, I can see that GPU AND CPUs are used.

In the logs, I can see that 33/49 layers are offloaded to GPU. I don't really understand . I could use all layers in the gpu.
Can I change the number of layers offloaded ?

Furthermore, I can notice 1/3 load on cuda => we could achieve 100% load....

I am using qwen2.5:14b
here are some logs

ollamalogs.txt

we can see the model loaded in nvidia ram
Image

<!-- gh-comment-id:2685170115 --> @sancelot commented on GitHub (Feb 26, 2025): At first I deactivated Intel Video , to keep only Nvidia GPU. => cpu is always used. I setted up : OLLAMA_FLASH_ATTENTION=1 OLLAMA_LLM_LIBRARY="cuda_v11" CUDA_VISIBLE_DEVICES=0 => Now, I can see that GPU AND CPUs are used. In the logs, I can see that 33/49 layers are offloaded to GPU. I don't really understand . I could use all layers in the gpu. Can I change the number of layers offloaded ? Furthermore, I can notice 1/3 load on cuda => we could achieve 100% load.... I am using qwen2.5:14b here are some logs [ollamalogs.txt](https://github.com/user-attachments/files/18985478/ollamalogs.txt) we can see the model loaded in nvidia ram ![Image](https://github.com/user-attachments/assets/31ccf7d0-785f-4e5b-86c9-f2ba4df4e0f0)
GiteaMirror commented 2026-04-12 17:19:05 -05:00
Author
Owner
Copy Link
Copy Source

@SynapseScribe commented on GitHub (Feb 26, 2025):

for me on a 4080 super, i kinda realised that small modells < 10GB will work because it can fit in video memory, but when using larger models, it starts filling RAM which in turn bottlenecks on CPU

saw same behavior on LM Studio

<!-- gh-comment-id:2685189542 --> @SynapseScribe commented on GitHub (Feb 26, 2025): for me on a 4080 super, i kinda realised that small modells < 10GB will work because it can fit in video memory, but when using larger models, it starts filling RAM which in turn bottlenecks on CPU saw same behavior on LM Studio
GiteaMirror commented 2026-04-12 17:19:06 -05:00
Author
Owner
Copy Link
Copy Source

@rick-github commented on GitHub (Feb 26, 2025):

time=2025-02-26T15:13:54.580+01:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=49 layers.offload=33 layers.split="" memory.available="[6.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="10.0 GiB" memory.required.partial="6.8 GiB" memory.required.kv="384.0 MiB" memory.required.allocations="[6.8 GiB]" memory.weights.total="7.7 GiB" memory.weights.repeating="7.1 GiB" memory.weights.nonrepeating="609.1 MiB" memory.graph.full="307.0 MiB" memory.graph.partial="916.1 MiB"

The GPU had 6.9G available. ollama calculated that it can fit 33 layers, using 6.8G. 0.1G is not enough to fit the remaining 16 layers.

The reason utilization of the GPU is low and CPU utilization is high is that the GPU processes the layers it has much faster then the CPU can process the layers it has, so the GPU must wait for the CPU to finish.

<!-- gh-comment-id:2685351175 --> @rick-github commented on GitHub (Feb 26, 2025): ``` time=2025-02-26T15:13:54.580+01:00 level=INFO source=server.go:130 msg=offload library=cuda layers.requested=-1 layers.model=49 layers.offload=33 layers.split="" memory.available="[6.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="10.0 GiB" memory.required.partial="6.8 GiB" memory.required.kv="384.0 MiB" memory.required.allocations="[6.8 GiB]" memory.weights.total="7.7 GiB" memory.weights.repeating="7.1 GiB" memory.weights.nonrepeating="609.1 MiB" memory.graph.full="307.0 MiB" memory.graph.partial="916.1 MiB" ``` The GPU had 6.9G available. ollama calculated that it can fit 33 layers, using 6.8G. 0.1G is not enough to fit the remaining 16 layers. The reason utilization of the GPU is low and CPU utilization is high is that the GPU processes the layers it has much faster then the CPU can process the layers it has, so the GPU must wait for the CPU to finish.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No labels bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5966