github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-12 11:15:46 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #4585] Does not run as fast as expected on windows #2876

New Issue
Closed
opened 2026-04-12 13:13:41 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-12 13:13:41 -05:00
Owner
Copy Link

Originally created by @Menghuan1918 on GitHub (May 23, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4585

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

When running llama3 on the same GPU (3060M), it is many times slower on Windows than on Linux.

It seems to be caused by the model not being fully loaded on the GPU on windows, even though the memory is not fully used：

Windows 11：

屏幕截图_20240523_123057

Linux：

图片

OS

Linux, Windows

GPU

Nvidia

CPU

Intel

Ollama version

0.1.38

Originally created by @Menghuan1918 on GitHub (May 23, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4585 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? When running llama3 on the same GPU (3060M), it is many times slower on Windows than on Linux. It seems to be caused by the model not being fully loaded on the GPU on windows, even though the memory is not fully used： Windows 11： ![屏幕截图_20240523_123057](https://github.com/ollama/ollama/assets/122662527/4aa4db13-799d-4268-a753-8d2680808c59) Linux： ![图片](https://github.com/ollama/ollama/assets/122662527/488ede37-a937-4d10-9292-7be91d9d807d) ### OS Linux, Windows ### GPU Nvidia ### CPU Intel ### Ollama version 0.1.38
GiteaMirror added the nvidiabugwindows labels 2026-04-12 13:13:41 -05:00
GiteaMirror commented 2026-04-12 13:13:43 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 23, 2024):

Can you share the server logs for your two scenarios? It's possible more VRAM is being used by apps on Windows, and we try not to trigger VRAM paging by overcommitting VRAM. The logs will help us see what available VRAM is being reported on Linux vs. Windows in your scenario and determine if there might be a bug in our prediction logic.

<!-- gh-comment-id:2127735365 --> @dhiltgen commented on GitHub (May 23, 2024): Can you share the server logs for your two scenarios? It's possible more VRAM is being used by apps on Windows, and we try not to trigger VRAM paging by overcommitting VRAM. The logs will help us see what available VRAM is being reported on Linux vs. Windows in your scenario and determine if there might be a bug in our prediction logic.
GiteaMirror commented 2026-04-12 13:13:43 -05:00
Author
Owner
Copy Link

@Menghuan1918 commented on GitHub (May 24, 2024):

It looks to be the case that the log on Windows shows its only 5G of video memory available, but on linux it has 5.5G.

Winodws：

time=2024-05-22T21:06:50.295+08:00 level=INFO source=types.go:71 msg="inference compute" id=GPU-28ccac44-ceeb-6179-da79-283b3918b2ad library=cuda compute=8.6 driver=12.5 name="NVIDIA GeForce RTX 3060 Laptop GPU" total="6.0 GiB" available="5.0 GiB"
time=2024-05-22T21:06:57.496+08:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=30 memory.available="5.0 GiB" memory.required.full="5.0 GiB" memory.required.partial="4.9 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB"
time=2024-05-22T21:06:57.496+08:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=30 memory.available="5.0 GiB" memory.required.full="5.0 GiB" memory.required.partial="4.9 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB"
time=2024-05-22T21:06:57.497+08:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=30 memory.available="5.0 GiB" memory.required.full="5.0 GiB" memory.required.partial="4.9 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB"

Linux:

May 23 11:52:26 Menghuan ollama[2679]: time=2024-05-23T11:52:26.352+08:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=33 memory.available="5.5 GiB" memory.required.full="5.0 GiB" memory.required.partial="5.0 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB"
May 23 11:52:26 Menghuan ollama[2679]: time=2024-05-23T11:52:26.353+08:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=33 memory.available="5.5 GiB" memory.required.full="5.0 GiB" memory.required.partial="5.0 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB"
May 23 11:52:26 Menghuan ollama[2679]: time=2024-05-23T11:52:26.371+08:00 level=INFO source=server.go:320 msg="starting llama server" cmd="/tmp/ollama4164226115/runners/cuda_v12/ollama_llama_server --model /var/lib/ollama/.ollama/models/blobs/sha256-00e1317cbf74d901080d7100f57580ba8dd8de57203072dc6f668324ba545f29 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --parallel 1 --port 44457"

But if I think there should be enough VRAM to use on Windows... Since I used llama.cpp to fully load the model onto the GPU on Windows, there's still a certain amount of VRAM left over (whether viewed through the task manager or the command line nvidia-smi), and the output is almost as fast as it is under Linux at this point.

20240524_110825

thanks!

ollama_linux.log
llama_cpp_win.log
server_windows.log

<!-- gh-comment-id:2128460408 --> @Menghuan1918 commented on GitHub (May 24, 2024): It looks to be the case that the log on Windows shows its only 5G of video memory available, but on linux it has 5.5G. Winodws： ```log time=2024-05-22T21:06:50.295+08:00 level=INFO source=types.go:71 msg="inference compute" id=GPU-28ccac44-ceeb-6179-da79-283b3918b2ad library=cuda compute=8.6 driver=12.5 name="NVIDIA GeForce RTX 3060 Laptop GPU" total="6.0 GiB" available="5.0 GiB" time=2024-05-22T21:06:57.496+08:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=30 memory.available="5.0 GiB" memory.required.full="5.0 GiB" memory.required.partial="4.9 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB" time=2024-05-22T21:06:57.496+08:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=30 memory.available="5.0 GiB" memory.required.full="5.0 GiB" memory.required.partial="4.9 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB" time=2024-05-22T21:06:57.497+08:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=30 memory.available="5.0 GiB" memory.required.full="5.0 GiB" memory.required.partial="4.9 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB" ``` Linux: ```log May 23 11:52:26 Menghuan ollama[2679]: time=2024-05-23T11:52:26.352+08:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=33 memory.available="5.5 GiB" memory.required.full="5.0 GiB" memory.required.partial="5.0 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB" May 23 11:52:26 Menghuan ollama[2679]: time=2024-05-23T11:52:26.353+08:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=33 memory.available="5.5 GiB" memory.required.full="5.0 GiB" memory.required.partial="5.0 GiB" memory.required.kv="256.0 MiB" memory.weights.total="4.1 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="677.5 MiB" May 23 11:52:26 Menghuan ollama[2679]: time=2024-05-23T11:52:26.371+08:00 level=INFO source=server.go:320 msg="starting llama server" cmd="/tmp/ollama4164226115/runners/cuda_v12/ollama_llama_server --model /var/lib/ollama/.ollama/models/blobs/sha256-00e1317cbf74d901080d7100f57580ba8dd8de57203072dc6f668324ba545f29 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --parallel 1 --port 44457" ``` But if I think there should be enough VRAM to use on Windows... Since I used llama.cpp to fully load the model onto the GPU on Windows, there's still a certain amount of VRAM left over (whether viewed through the task manager or the command line `nvidia-smi`), and the output is almost as fast as it is under Linux at this point. ![20240524_110825](https://github.com/ollama/ollama/assets/122662527/f0d1ac3c-ebcd-4eff-ae58-8914fb46cdd5) thanks! [ollama_linux.log](https://github.com/ollama/ollama/files/15427124/ollama_linux.log) [llama_cpp_win.log](https://github.com/ollama/ollama/files/15427125/llama_cpp_win.log) [server_windows.log](https://github.com/ollama/ollama/files/15427126/server_windows.log)
GiteaMirror commented 2026-04-12 13:13:44 -05:00
Author
Owner
Copy Link

@gabilanbrc commented on GitHub (May 28, 2024):

Hi!
I'm facing a similar issue with a RTX 3050 in Windows, similar responses of ollama ps and nvidia-smi.exe. When I use
curl -X POST -H "Content-type: application/json" --data "{\"model\": \"llama2\", \"prompt\": \"Who are you?\"}" http://localhost:11434/api/generate
I have a response that takes almost 4 minutes. Is that normal or I should expect to have a response much sooner?

<!-- gh-comment-id:2135524177 --> @gabilanbrc commented on GitHub (May 28, 2024): Hi! I'm facing a similar issue with a RTX 3050 in Windows, similar responses of ollama ps and nvidia-smi.exe. When I use `curl -X POST -H "Content-type: application/json" --data "{\"model\": \"llama2\", \"prompt\": \"Who are you?\"}" http://localhost:11434/api/generate` I have a response that takes almost 4 minutes. Is that normal or I should expect to have a response much sooner?
GiteaMirror commented 2026-04-12 13:13:44 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 28, 2024):

@Menghuan1918 you can try forcing more layers via num_gpu and rely on Windows VRAM paging support to see if that yields better or worse performance than the CPU offload of those 3 layers.

@gabilanbrc what does ollama ps show after you run the prompt?

<!-- gh-comment-id:2136222700 --> @dhiltgen commented on GitHub (May 28, 2024): @Menghuan1918 you can try forcing more layers via `num_gpu` and rely on Windows VRAM paging support to see if that yields better or worse performance than the CPU offload of those 3 layers. @gabilanbrc what does `ollama ps` show after you run the prompt?
GiteaMirror commented 2026-04-12 13:13:45 -05:00
Author
Owner
Copy Link

@Menghuan1918 commented on GitHub (May 29, 2024):

@Menghuan1918 you can try forcing more layers via num_gpu and rely on Windows VRAM paging support to see if that yields better or worse performance than the CPU offload of those 3 layers.

@gabilanbrc what does ollama ps show after you run the prompt?

If offload of those 3 layers , it does slow it down a bit more...

But I've found that both at 3%/97% CPU/GPU, I'll get almost twice the speed of the native Windows version running with WSL2, which is now an acceptable speed.

Thank you for your reply！

20240529_095642

20240529_095949

<!-- gh-comment-id:2136393672 --> @Menghuan1918 commented on GitHub (May 29, 2024): > @Menghuan1918 you can try forcing more layers via `num_gpu` and rely on Windows VRAM paging support to see if that yields better or worse performance than the CPU offload of those 3 layers. > > @gabilanbrc what does `ollama ps` show after you run the prompt? If offload of those 3 layers , it does slow it down a bit more... But I've found that both at 3%/97% CPU/GPU, I'll get almost twice the speed of the native Windows version running with WSL2, which is now an acceptable speed. Thank you for your reply！ ![20240529_095642](https://github.com/ollama/ollama/assets/122662527/88e81d2c-dc68-458b-9f16-440055b68b5a) ![20240529_095949](https://github.com/ollama/ollama/assets/122662527/3227e5e7-27a2-4ac3-adc2-ba3c87f0b72d)
GiteaMirror commented 2026-04-12 13:13:45 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 29, 2024):

Glad to hear you found an acceptable configuration. I'll close this issue now.

<!-- gh-comment-id:2138059183 --> @dhiltgen commented on GitHub (May 29, 2024): Glad to hear you found an acceptable configuration. I'll close this issue now.
GiteaMirror commented 2026-04-12 13:13:46 -05:00
Author
Owner
Copy Link

@gabilanbrc commented on GitHub (May 31, 2024):

@Menghuan1918 you can try forcing more layers via num_gpu and rely on Windows VRAM paging support to see if that yields better or worse performance than the CPU offload of those 3 layers.

@gabilanbrc what does ollama ps show after you run the prompt?

Hi there, after the last ollama update it started to work much better, anyway here is the ps response

NAME            ID              SIZE    PROCESSOR       UNTIL
llama2:latest   78e26419b446    5.6 GB  4%/96% CPU/GPU  4 minutes from now
<!-- gh-comment-id:2141933342 --> @gabilanbrc commented on GitHub (May 31, 2024): > @Menghuan1918 you can try forcing more layers via `num_gpu` and rely on Windows VRAM paging support to see if that yields better or worse performance than the CPU offload of those 3 layers. > > @gabilanbrc what does `ollama ps` show after you run the prompt? Hi there, after the last ollama update it started to work much better, anyway here is the ps response ``` NAME ID SIZE PROCESSOR UNTIL llama2:latest 78e26419b446 5.6 GB 4%/96% CPU/GPU 4 minutes from now ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug nvidia windows
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#2876
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?