github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #2936] Does not using all threads on NUMA configuration (server motherboards 2, 4, 6 multisocket CPU) on Windows #63838

New Issue
Open
opened 2026-05-03 15:08:53 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-05-03 15:08:53 -05:00
Owner
Copy Link

Originally created by @GermanAizek on GitHub (Mar 5, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2936

Originally assigned to: @dhiltgen on GitHub.

Very old problem since 00s early Microsoft concerns setThreadAffinity which by default does not cover all logical processors in the system, this was fixed 20 years later only in Windows 11 and more Windows Server, but count threads is most likely incorrectly calculated. I do not know Golang, but if it were in С/C++ I would be able to help. I have already fixed this issue here before:

https://github.com/ggerganov/llama.cpp/issues/5524

https://github.com/x64dbg/x64dbg/pull/3272

https://github.com/giampaolo/psutil/issues/771

d1fa25f376

image

Originally created by @GermanAizek on GitHub (Mar 5, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2936 Originally assigned to: @dhiltgen on GitHub. Very old problem since 00s early Microsoft concerns `setThreadAffinity` which by default does not cover all logical processors in the system, this was fixed 20 years later only in Windows 11 and more Windows Server, but count threads is most likely incorrectly calculated. I do not know Golang, but if it were in С/C++ I would be able to help. I have already fixed this issue here before: https://github.com/ggerganov/llama.cpp/issues/5524 https://github.com/x64dbg/x64dbg/pull/3272 https://github.com/giampaolo/psutil/issues/771 https://github.com/GermanAizek/llvm-project/commit/d1fa25f37631b8b33a71fbe9eb4ea89e3a47b723 ![image](https://github.com/ollama/ollama/assets/21138600/a517d7a8-b9ba-48de-af90-488b0039474c)
GiteaMirror added the bugwindows labels 2026-05-03 15:08:53 -05:00
GiteaMirror commented 2026-05-03 15:08:57 -05:00
Author
Owner
Copy Link

@remy415 commented on GitHub (Mar 5, 2024):

https://github.com/ggerganov/llama.cpp/issues/5524

Ollama uses llama.cpp as backend. If llama.cpp pushes a fix for the issue, Ollama should automatically use it in its next release.

<!-- gh-comment-id:1979664096 --> @remy415 commented on GitHub (Mar 5, 2024): > https://github.com/ggerganov/llama.cpp/issues/5524 Ollama uses llama.cpp as backend. If llama.cpp pushes a fix for the issue, Ollama should automatically use it in its next release.
GiteaMirror commented 2026-05-03 15:08:59 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jun 1, 2024):

@GermanAizek skimming through the llama.cpp commit log, I didn't notice a commit from you fixing this, and it looks like the issue was closed due to inactivity. As @remy415 points out, this logic in ollama is based on the llama.cpp C++ code, so if you contribute a fix to llama.cpp, we'll pick it up automatically next time we bump our dependency.

<!-- gh-comment-id:2143588006 --> @dhiltgen commented on GitHub (Jun 1, 2024): @GermanAizek skimming through the llama.cpp commit log, I didn't notice a commit from you fixing this, and it looks like the issue was closed due to inactivity. As @remy415 points out, this logic in ollama is based on the llama.cpp C++ code, so if you contribute a fix to llama.cpp, we'll pick it up automatically next time we bump our dependency.
GiteaMirror commented 2026-05-03 15:09:00 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Aug 9, 2024):

This should be working better now on Linux with #6186, but windows support is still a gap.

<!-- gh-comment-id:2278859555 --> @dhiltgen commented on GitHub (Aug 9, 2024): This should be working better now on Linux with #6186, but windows support is still a gap.
GiteaMirror commented 2026-05-03 15:09:02 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Oct 15, 2024):

We've merged new logic to discover the available CPU Sockets, Cores (efficiency and performance) and hyperthreads (logical cpus) for MacOS, Windows, and Linux. In testing on a NUMA system on both Linux and Windows, there's still more work to do, so we'll only allocate the number of physical cores in one socket for now to avoid thrashing. Inference speed should be better, but will still be using only a single socket.

<!-- gh-comment-id:2415305393 --> @dhiltgen commented on GitHub (Oct 15, 2024): We've [merged new logic](https://github.com/ollama/ollama/pull/6264) to discover the available CPU Sockets, Cores (efficiency and performance) and hyperthreads (logical cpus) for MacOS, Windows, and Linux. In testing on a NUMA system on both Linux and Windows, there's still more work to do, so we'll only allocate the number of physical cores in one socket for now to avoid thrashing. Inference speed should be better, but will still be using only a single socket.
GiteaMirror commented 2026-05-03 15:09:05 -05:00
Author
Owner
Copy Link

@james-irwin commented on GitHub (Feb 4, 2025):

Notwithstanding being made in the context of manual override (reducing) of the number of threads and via an environment variable (for CPU-based targets) #8792 appears to make more observable and consistent effect than /set parameter num_thread invoked from the client. It could be used to dial them up, too with the same command-line invocation placing those thread-counts on specific cores (via taskset or numactl). The motive for mentioning this here is while I applaud the efforts for automatic and out-of-the-box great default experience, the manual overrides are appropriate for my use case, and it appears for others too.

<!-- gh-comment-id:2634241229 --> @james-irwin commented on GitHub (Feb 4, 2025): Notwithstanding being made in the context of manual override (reducing) of the number of threads and via an environment variable (for CPU-based targets) #8792 appears to make more observable and consistent effect than `/set parameter num_thread` invoked from the client. It could be used to dial them up, too with the same command-line invocation placing those thread-counts on specific cores (via taskset or numactl). The motive for mentioning this here is while I applaud the efforts for automatic and out-of-the-box great default experience, the manual overrides are appropriate for my use case, and it appears for others too.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug windows
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#63838
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?