github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #3201] Ollama is not using my GPU (Windows) #1975

New Issue
Closed
opened 2026-04-12 12:09:29 -05:00 by GiteaMirror · 30 comments
GiteaMirror commented 2026-04-12 12:09:29 -05:00
Owner
Copy Link

Originally created by @davidabou on GitHub (Mar 17, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3201

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

I have restart my PC and I have launched Ollama in the terminal using mistral:7b and a viewer of GPU usage (task manager). I have asked a question, and it replies to me quickly, I see the GPU usage increase around 25%, ok that's seems good. But then I have asked another question and now it take long time and I see the GPU usage oscillate between 0 and 1 percent.

What did you expect to see?

A higher GPU usage with a fast response.

Steps to reproduce

$ ollama run mistral:7b --verbose

Are there any recent changes that introduced the issue?

No response

OS

Windows

Architecture

x86

Platform

No response

Ollama version

0.1.29

GPU

Nvidia

GPU info

| 0 NVIDIA GeForce RTX 3050 ... WDDM | 00000000:01:00.0 Off | N/A |
| N/A 49C P8 5W / 38W | 2998MiB / 4096MiB | 12% Default |

CPU

Intel

Other software

32GB of RAM

Originally created by @davidabou on GitHub (Mar 17, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3201 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? I have restart my PC and I have launched Ollama in the terminal using mistral:7b and a viewer of GPU usage (task manager). I have asked a question, and it replies to me quickly, I see the GPU usage increase around 25%, ok that's seems good. But then I have asked another question and now it take long time and I see the GPU usage oscillate between 0 and 1 percent. ### What did you expect to see? A higher GPU usage with a fast response. ### Steps to reproduce ```bash $ ollama run mistral:7b --verbose ``` ### Are there any recent changes that introduced the issue? _No response_ ### OS Windows ### Architecture x86 ### Platform _No response_ ### Ollama version 0.1.29 ### GPU Nvidia ### GPU info | 0 NVIDIA GeForce RTX 3050 ... WDDM | 00000000:01:00.0 Off | N/A | | N/A 49C P8 5W / 38W | 2998MiB / 4096MiB | 12% Default | ### CPU Intel ### Other software 32GB of RAM
GiteaMirror added the bugnvidiawindows labels 2026-04-12 12:09:29 -05:00
GiteaMirror commented 2026-04-12 12:09:32 -05:00
Author
Owner
Copy Link

@davidabou commented on GitHub (Mar 17, 2024):

I found something that can provide precision. In fact, for the entire loading time, I see the GPU usage increase around 29%, but when the text starts to generate, the GPU usage drops to 0% and it takes forever.

<!-- gh-comment-id:2002624568 --> @davidabou commented on GitHub (Mar 17, 2024): I found something that can provide precision. In fact, for the entire loading time, I see the GPU usage increase around 29%, but when the text starts to generate, the GPU usage drops to 0% and it takes forever.
GiteaMirror commented 2026-04-12 12:09:32 -05:00
Author
Owner
Copy Link

@igorschlum commented on GitHub (Mar 17, 2024):

@nethriis can you please share the prompt if it's a special one. Easier to try to reproduce. Did you try with another LLM like

ollama run tinydolphin

It will help see if it's a memory issue. Have you a way to see CPU usage?

<!-- gh-comment-id:2002652280 --> @igorschlum commented on GitHub (Mar 17, 2024): @nethriis can you please share the prompt if it's a special one. Easier to try to reproduce. Did you try with another LLM like ollama run tinydolphin It will help see if it's a memory issue. Have you a way to see CPU usage?
GiteaMirror commented 2026-04-12 12:09:33 -05:00
Author
Owner
Copy Link

@davidabou commented on GitHub (Mar 18, 2024):

My prompt was

Why is the sky blue?

I tried running with tinyllama and it works fine.
I have Windows Task Manager which allows me to see CPU usage as well.

<!-- gh-comment-id:2003662781 --> @davidabou commented on GitHub (Mar 18, 2024): My prompt was ```plaintext Why is the sky blue? ``` I tried running with `tinyllama` and it works fine. I have Windows Task Manager which allows me to see CPU usage as well.
GiteaMirror commented 2026-04-12 12:09:33 -05:00
Author
Owner
Copy Link

@jerryola1 commented on GitHub (Mar 19, 2024):

have you try used it directly as one of the requred parameter expected by the model, eg device="cuda"

<!-- gh-comment-id:2005462574 --> @jerryola1 commented on GitHub (Mar 19, 2024): have you try used it directly as one of the requred parameter expected by the model, eg device="cuda"
GiteaMirror commented 2026-04-12 12:09:34 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 20, 2024):

We're working on improvements to the memory prediction algorithm which should allow us to more fully utilize the GPUs VRAM without hitting an out-of-memory error. It looks like you're trying to load a 4G model into a 4G GPU which given some overhead, should mostly fit. Any layers we can't fit into VRAM are processed by the CPU. Since the GPU is much faster than CPU, the GPU winds up being idle waiting for the CPU to keep up. If you look in the server log, you'll be able to see a log line that looks something like this:

llm_load_tensors: offloaded 22/33 layers to GPU
<!-- gh-comment-id:2009019508 --> @dhiltgen commented on GitHub (Mar 20, 2024): We're working on improvements to the memory prediction algorithm which should allow us to more fully utilize the GPUs VRAM without hitting an out-of-memory error. It looks like you're trying to load a 4G model into a 4G GPU which given some overhead, should mostly fit. Any layers we can't fit into VRAM are processed by the CPU. Since the GPU is much faster than CPU, the GPU winds up being idle waiting for the CPU to keep up. If you look in the server log, you'll be able to see a log line that looks something like this: ``` llm_load_tensors: offloaded 22/33 layers to GPU ```
GiteaMirror commented 2026-04-12 12:09:35 -05:00
Author
Owner
Copy Link

@davidabou commented on GitHub (Mar 20, 2024):

@dhiltgen Thanks for the answer but where is the server log?

<!-- gh-comment-id:2009918188 --> @davidabou commented on GitHub (Mar 20, 2024): @dhiltgen Thanks for the answer but where is the server log?
GiteaMirror commented 2026-04-12 12:09:35 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 20, 2024):

Check out https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md

<!-- gh-comment-id:2010138431 --> @dhiltgen commented on GitHub (Mar 20, 2024): Check out https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md
GiteaMirror commented 2026-04-12 12:09:36 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 15, 2024):

The closer you get to 33/33 layers, the faster it will be. I'm going to close this issue now, but if you're still having problems, let us know.

<!-- gh-comment-id:2057945153 --> @dhiltgen commented on GitHub (Apr 15, 2024): The closer you get to 33/33 layers, the faster it will be. I'm going to close this issue now, but if you're still having problems, let us know.
GiteaMirror commented 2026-04-12 12:09:37 -05:00
Author
Owner
Copy Link

@cr1cr1 commented on GitHub (May 6, 2024):

Forcing OLLAMA_LLM_LIBRARY=cuda_v11.3 will still use CPU instead of GPU, so only setting the PATH to a directory with cudart64_110.dll, like ollama workdir, seems to do the trick.

https://github.com/ollama/ollama/issues/4008#issuecomment-2096623526

<!-- gh-comment-id:2096639616 --> @cr1cr1 commented on GitHub (May 6, 2024): Forcing `OLLAMA_LLM_LIBRARY=cuda_v11.3` will still use CPU instead of GPU, so only setting the PATH to a directory with `cudart64_110.dll`, like ollama workdir, seems to do the trick. https://github.com/ollama/ollama/issues/4008#issuecomment-2096623526
GiteaMirror commented 2026-04-12 12:09:38 -05:00
Author
Owner
Copy Link

@ACodingfreak commented on GitHub (Aug 9, 2024):

Well in case of Windows11 is there a way I can set ollama to use 100%GPU on my laptop ?

<!-- gh-comment-id:2276987054 --> @ACodingfreak commented on GitHub (Aug 9, 2024): Well in case of Windows11 is there a way I can set ollama to use 100%GPU on my laptop ?
GiteaMirror commented 2026-04-12 12:09:38 -05:00
Author
Owner
Copy Link

@GITHUBWJTANG commented on GitHub (Aug 2, 2025):

Check out your context window size and try from smaller context window.
Looks like ollama falls back to using CPU while the context window saturated the VRAM (i.e. over 90%) and it's obviously visible in perfMon.

Test case: RTX4060/8GVRAM, i7g14/32GRAM, llama | granite
Context window > 8k, 100% CPU observed. Otherwise, GPU is being used.

<!-- gh-comment-id:3146196209 --> @GITHUBWJTANG commented on GitHub (Aug 2, 2025): Check out your context window size and try from smaller context window. Looks like ollama falls back to using CPU while the context window saturated the VRAM (i.e. over 90%) and it's obviously visible in perfMon. Test case: RTX4060/8GVRAM, i7g14/32GRAM, llama | granite Context window > 8k, 100% CPU observed. Otherwise, GPU is being used.
GiteaMirror commented 2026-04-12 12:09:39 -05:00
Author
Owner
Copy Link

@lynrayy commented on GitHub (Aug 5, 2025):

Just tried gpt-oss 20b, it is not using my GPU at all. All the model loaded in RAM, no VRAM affected

<!-- gh-comment-id:3156445818 --> @lynrayy commented on GitHub (Aug 5, 2025): Just tried gpt-oss 20b, it is not using my GPU at all. All the model loaded in RAM, no VRAM affected
GiteaMirror commented 2026-04-12 12:09:40 -05:00
Author
Owner
Copy Link

@lynrayy commented on GitHub (Aug 5, 2025):

Solution?

<!-- gh-comment-id:3156446090 --> @lynrayy commented on GitHub (Aug 5, 2025): Solution?
GiteaMirror commented 2026-04-12 12:09:41 -05:00
Author
Owner
Copy Link

@Drommedhar commented on GitHub (Aug 6, 2025):

I can confirm that gpt-oss 20b is running purely on CPU under windows. Not using a single bit of my 5090 over here

<!-- gh-comment-id:3157378492 --> @Drommedhar commented on GitHub (Aug 6, 2025): I can confirm that gpt-oss 20b is running purely on CPU under windows. Not using a single bit of my 5090 over here
GiteaMirror commented 2026-04-12 12:09:42 -05:00
Author
Owner
Copy Link

@Drommedhar commented on GitHub (Aug 6, 2025):

Just got it to work on GPU. Dont know why but closing down ollama completly (right click in notification bar and exit) and restarting the cmd (and cleaning log directory, so this should not matter) did load it into GPU

Image
<!-- gh-comment-id:3157385145 --> @Drommedhar commented on GitHub (Aug 6, 2025): Just got it to work on GPU. Dont know why but closing down ollama completly (right click in notification bar and exit) and restarting the cmd (and cleaning log directory, so this should not matter) did load it into GPU <img width="768" height="48" alt="Image" src="https://github.com/user-attachments/assets/78c6b6d0-89b7-456b-b629-267eed17672d" />
GiteaMirror commented 2026-04-12 12:09:45 -05:00
Author
Owner
Copy Link

@lynrayy commented on GitHub (Aug 6, 2025):

Not helped for me, it is refusing to use GPU

<!-- gh-comment-id:3157386511 --> @lynrayy commented on GitHub (Aug 6, 2025): Not helped for me, it is refusing to use GPU
GiteaMirror commented 2026-04-12 12:09:46 -05:00
Author
Owner
Copy Link

@lynrayy commented on GitHub (Aug 6, 2025):

Oh it is appears in tray. Will try later

<!-- gh-comment-id:3157387659 --> @lynrayy commented on GitHub (Aug 6, 2025): Oh it is appears in tray. Will try later
GiteaMirror commented 2026-04-12 12:09:47 -05:00
Author
Owner
Copy Link

@moeelbadri commented on GitHub (Aug 6, 2025):

I can confirm that gpt-oss 20b is running purely on CPU under windows. Not using a single bit of my 5090 over here

same, also i tried ur solution didnt work , rtx 3060 12 gb windows 11 ollama windows

in the logs

layers.model=25
layers.offload=13
memory.required.full = 16.9 GiB
memory.required.partial = 10.8 GiB

<!-- gh-comment-id:3160371114 --> @moeelbadri commented on GitHub (Aug 6, 2025): > I can confirm that gpt-oss 20b is running purely on CPU under windows. Not using a single bit of my 5090 over here same, also i tried ur solution didnt work , rtx 3060 12 gb windows 11 ollama windows in the logs layers.model=25 layers.offload=13 memory.required.full = 16.9 GiB memory.required.partial = 10.8 GiB
GiteaMirror commented 2026-04-12 12:09:47 -05:00
Author
Owner
Copy Link

@JoseDv1 commented on GitHub (Aug 6, 2025):

It's happening to me too. I discovered that if you lower the context window, part of the model loads on the GPU. I have an RTX 4070 Laptop 8GB. Ollama on windows 11

<!-- gh-comment-id:3160570925 --> @JoseDv1 commented on GitHub (Aug 6, 2025): It's happening to me too. I discovered that if you lower the context window, part of the model loads on the GPU. I have an RTX 4070 Laptop 8GB. Ollama on windows 11
GiteaMirror commented 2026-04-12 12:09:48 -05:00
Author
Owner
Copy Link

@janhlavsa commented on GitHub (Aug 6, 2025):

Same here - Windows 10, RTX 3090, GPT OSS-20B purely on CPU. Any thoughts how to solve that?

Update:

The context length is a problem:
4K/8K - used VRAM 14,2GB - very fast
16K - used VRAM 16,5GB - very fast
32K - used VRAM 14,1GB - slow (GPU used only partially?)
64K/128K - used VRAM 0GB - very slow (CPU only obviously)

However, it never utilizes all available VRAM. That seems to be an issue.

Update 2:
Just updated to a new version 0.11.3 and the issue seems to be resolved. With 32K context it seems to be purely on GPU (21GB of VRAM used).

<!-- gh-comment-id:3161464911 --> @janhlavsa commented on GitHub (Aug 6, 2025): Same here - Windows 10, RTX 3090, GPT OSS-20B purely on CPU. Any thoughts how to solve that? Update: The context length is a problem: 4K/8K - used VRAM 14,2GB - very fast 16K - used VRAM 16,5GB - very fast 32K - used VRAM 14,1GB - slow (GPU used only partially?) 64K/128K - used VRAM 0GB - very slow (CPU only obviously) However, it never utilizes all available VRAM. That seems to be an issue. Update 2: Just updated to a new version 0.11.3 and the issue seems to be resolved. With 32K context it seems to be purely on GPU (21GB of VRAM used).
GiteaMirror commented 2026-04-12 12:09:49 -05:00
Author
Owner
Copy Link

@juniormayhe commented on GitHub (Aug 7, 2025):

32K context uses zero GPU,
while 4K uses 22% GPU.

So increasing the context length in settings will likely cause a fallback to CPU without the user realizing it.
I would expect this to be bit faster and use 90% of GPU.

Setup:
RAM 128GB
Nvidia 4080 SUPER
Storage SSD nvme

Currently the tool is too slow to be practical—even when using the smallest context window.
This significantly impacts its usefulness.

<!-- gh-comment-id:3163283200 --> @juniormayhe commented on GitHub (Aug 7, 2025): 32K context uses zero GPU, while 4K uses 22% GPU. So increasing the context length in settings will likely cause a fallback to CPU without the user realizing it. I would expect this to be bit faster and use 90% of GPU. Setup: RAM 128GB Nvidia 4080 SUPER Storage SSD nvme Currently the tool is too slow to be practical—even when using the smallest context window. This significantly impacts its usefulness.
GiteaMirror commented 2026-04-12 12:09:50 -05:00
Author
Owner
Copy Link

@lynrayy commented on GitHub (Aug 7, 2025):

Mine context is 4k by default, i can't get lower. I have 6gb VRAM

<!-- gh-comment-id:3163288805 --> @lynrayy commented on GitHub (Aug 7, 2025): Mine context is 4k by default, i can't get lower. I have 6gb VRAM
GiteaMirror commented 2026-04-12 12:09:50 -05:00
Author
Owner
Copy Link

@JoseDv1 commented on GitHub (Aug 7, 2025):

Mine context is 4k by default, i can't get lower. I have 6gb VRAM

Maybe you can set less through environment variables

<!-- gh-comment-id:3164716921 --> @JoseDv1 commented on GitHub (Aug 7, 2025): > Mine context is 4k by default, i can't get lower. I have 6gb VRAM Maybe you can set less through environment variables
GiteaMirror commented 2026-04-12 12:09:51 -05:00
Author
Owner
Copy Link

@JoseDv1 commented on GitHub (Aug 7, 2025):

Mine context is 4k by default, i can't get lower. I have 6gb VRAM

Maybe you can set less through environment variables

<!-- gh-comment-id:3164716961 --> @JoseDv1 commented on GitHub (Aug 7, 2025): > Mine context is 4k by default, i can't get lower. I have 6gb VRAM Maybe you can set less through environment variables
GiteaMirror commented 2026-04-12 12:09:52 -05:00
Author
Owner
Copy Link

@disarticulate commented on GitHub (Oct 24, 2025):

Can we get a comman like rocminfo or nvtop that shows/verifies what GPU is accessible? Seems like a simple abstract. Just listing all the compatibilities doesn't help verify it's picking up the correct GPU.

<!-- gh-comment-id:3444107669 --> @disarticulate commented on GitHub (Oct 24, 2025): Can we get a comman like rocminfo or nvtop that shows/verifies what GPU is accessible? Seems like a simple abstract. Just listing all the compatibilities doesn't help verify it's picking up the correct GPU.
GiteaMirror commented 2026-04-12 12:09:53 -05:00
Author
Owner
Copy Link

@JoewAlabel commented on GitHub (Mar 1, 2026):

still broken. ollama don'tg run on nvidia gpus so its totally sh1t

<!-- gh-comment-id:3979041110 --> @JoewAlabel commented on GitHub (Mar 1, 2026): still broken. ollama don'tg run on nvidia gpus so its totally sh1t
GiteaMirror commented 2026-04-12 12:09:54 -05:00
Author
Owner
Copy Link

@PatParslow commented on GitHub (Mar 3, 2026):

Having had some significant issues with the same behaviour (GTX5070ti) and spending half a day installing different versions (it used to work... so let's try old ones...), including building from source, I discovered it was failing silently because something on WSL2 was occupying the standard port 11434. Might help other people resolve the issue (if they haven't tried checking that). A nice loud error message would be nice...

<!-- gh-comment-id:3990692374 --> @PatParslow commented on GitHub (Mar 3, 2026): Having had some significant issues with the same behaviour (GTX5070ti) and spending half a day installing different versions (it used to work... so let's try old ones...), including building from source, I discovered it was failing silently because something on WSL2 was occupying the standard port 11434. Might help other people resolve the issue (if they haven't tried checking that). A nice loud error message would be nice...
GiteaMirror commented 2026-04-12 12:09:55 -05:00
Author
Owner
Copy Link

@irtiq7 commented on GitHub (Mar 7, 2026):

Is this issue being resolved? I agree with everyone above that ollama is not using GPU.

<!-- gh-comment-id:4016966531 --> @irtiq7 commented on GitHub (Mar 7, 2026): Is this issue being resolved? I agree with everyone above that ollama is not using GPU.
GiteaMirror commented 2026-04-12 12:09:56 -05:00
Author
Owner
Copy Link

@geomorillo commented on GitHub (Mar 23, 2026):

it is not resolved yet

<!-- gh-comment-id:4111514403 --> @geomorillo commented on GitHub (Mar 23, 2026): it is not resolved yet
GiteaMirror commented 2026-04-12 12:09:56 -05:00
Author
Owner
Copy Link

@athuljayaram commented on GitHub (Apr 12, 2026):

not working...same issue

<!-- gh-comment-id:4231485224 --> @athuljayaram commented on GitHub (Apr 12, 2026): not working...same issue
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug nvidia windows
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1975
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?