github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #2283] Low utilization on a large machine. #78993

New Issue
Closed
opened 2026-05-09 03:43:45 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-05-09 03:43:45 -05:00
Owner
Copy Link

Originally created by @nevakrien on GitHub (Jan 31, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2283

so I am running mixtral I completely removed the context length limit by setting it to 2 million and I dumped a full Wikipedia page in.
u would expect my memory use to grow linearly in time and if no limits are put a crash (which is nice to not need to deal with)

instead I am seeing only around 10gb out of a 128gb. it allocates them when the program starts and dosent relaly alocate more which is very suspicious and I can see its clearly not compute bound because my cpu utilization is not that high and my gpu utilization is like 6%.

my system has 134gb of ram an RTX 4090 and an i9-13900K

what this looks like is some sort of disk mapping or something but I didnt set anything like that up and its clearly not fit for my system.
it also seems to be context switching a lot because it changes cpus willy nilly and I think thats not the best for it either so if there is a nice way to make it just occupy like 30 cores stick to them and take around 100gb of memory and just go to town that would be very nice.

either way VERY happy it is able to run at all and even seems to have access to the full context to some extent. great work this is an excellent repo

Originally created by @nevakrien on GitHub (Jan 31, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2283 so I am running mixtral I completely removed the context length limit by setting it to 2 million and I dumped a full Wikipedia page in. u would expect my memory use to grow linearly in time and if no limits are put a crash (which is nice to not need to deal with) instead I am seeing only around 10gb out of a 128gb. it allocates them when the program starts and dosent relaly alocate more which is very suspicious and I can see its clearly not compute bound because my cpu utilization is not that high and my gpu utilization is like 6%. my system has 134gb of ram an RTX 4090 and an i9-13900K what this looks like is some sort of disk mapping or something but I didnt set anything like that up and its clearly not fit for my system. it also seems to be context switching a lot because it changes cpus willy nilly and I think thats not the best for it either so if there is a nice way to make it just occupy like 30 cores stick to them and take around 100gb of memory and just go to town that would be very nice. either way VERY happy it is able to run at all and even seems to have access to the full context to some extent. great work this is an excellent repo
GiteaMirror commented 2026-05-09 03:43:49 -05:00
Author
Owner
Copy Link

@easp commented on GitHub (Feb 1, 2024):

Regarding CPU utilization, text generation is memory bandwidth bound. Ollama defaults to using as many threads as physical cores, so it will never exceed 50% unless you configure more threads (which is unlikely to help). As for it hopping between cores, that's the OS's schedulers choice.

Regarding GPU utilization, the default 4-bit quantization + full context size won't fit in VRAM, so part of the model is in RAM and running on CPU. GPU has to wait on CPU, and vice versa, as each process their portion of the model for each token. So, GPU utilization will be relatively low. Check the Ollama log to confirm whether or not the GPU is being used.

Regarding memory footprint. First, weights are memory mapped, so they don't show up in process memory, they are instead accounted for in the file cache size. Second, it doesn't matter if you set the context limit to 2 million, Mixtral's context is 32K tokens. Third, 32k tokens would be far above average for a wikipedia page that isn't a list or an extensive timeline/history.

There are some models with larger context sizes (yi, mistral-yarn), but I don't think any of them are chat/instruct models.

<!-- gh-comment-id:1921906145 --> @easp commented on GitHub (Feb 1, 2024): Regarding CPU utilization, text generation is memory bandwidth bound. Ollama defaults to using as many threads as physical cores, so it will never exceed 50% unless you configure more threads (which is unlikely to help). As for it hopping between cores, that's the OS's schedulers choice. Regarding GPU utilization, the default 4-bit quantization + full context size won't fit in VRAM, so part of the model is in RAM and running on CPU. GPU has to wait on CPU, and vice versa, as each process their portion of the model for each token. So, GPU utilization will be relatively low. Check the Ollama log to confirm whether or not the GPU is being used. Regarding memory footprint. First, weights are memory mapped, so they don't show up in process memory, they are instead accounted for in the file cache size. Second, it doesn't matter if you set the context limit to 2 million, Mixtral's context is 32K tokens. Third, 32k tokens would be far above average for a wikipedia page that isn't a list or an extensive timeline/history. There are some models with larger context sizes (yi, mistral-yarn), but I don't think any of them are chat/instruct models.
GiteaMirror commented 2026-05-09 03:43:50 -05:00
Author
Owner
Copy Link

@bmizerany commented on GitHub (Mar 11, 2024):

@nevakrien I hope your found the thorough response from @easp helpful. I'm going to close this, but if you have more issues please let us know.

<!-- gh-comment-id:1989565068 --> @bmizerany commented on GitHub (Mar 11, 2024): @nevakrien I hope your found the thorough response from @easp helpful. I'm going to close this, but if you have more issues please let us know.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#78993
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?