github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #5724] Avoid blocking requests to already loaded models while loading another model #3565

New Issue
Open
opened 2026-04-12 14:17:29 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-12 14:17:29 -05:00
Owner
Copy Link

Originally created by @ghost on GitHub (Jul 16, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5724

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

I have noticed that when GPU VRAM gets near-full, but ollama has decided to load 2 models into VRAM, incoming requests to one model simply stall until the other model pops out of memory. This is most noticeable with an embedding model plus a larger model that takes up most of my 16 GB of VRAM. When the embedding model goes out of VRAM, the request to the other model processes immediately. So it's pretty clear that ollama is queueing up the request, but it seems like some tweaks need to be made to the algorithm for dispatching the requests in concurrent scenarios, or the logic for deciding to load multiple models into VRAM.

OS

Linux

GPU

AMD

CPU

AMD

Ollama version

0.2.5

Originally created by @ghost on GitHub (Jul 16, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5724 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? I have noticed that when GPU VRAM gets near-full, but ollama has decided to load 2 models into VRAM, incoming requests to one model simply stall until the other model pops out of memory. This is most noticeable with an embedding model plus a larger model that takes up most of my 16 GB of VRAM. When the embedding model goes out of VRAM, the request to the other model processes immediately. So it's pretty clear that ollama is queueing up the request, but it seems like some tweaks need to be made to the algorithm for dispatching the requests in concurrent scenarios, or the logic for deciding to load multiple models into VRAM. ### OS Linux ### GPU AMD ### CPU AMD ### Ollama version 0.2.5
GiteaMirror added the bug label 2026-04-12 14:17:29 -05:00
GiteaMirror commented 2026-04-12 14:17:30 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jul 16, 2024):

The current algorithm queues up all requests when we're adjusting the set of loaded models to ensure we get the VRAM information correct and don't hit OOM crashes. A side effect of this is requests for a model that's already loaded, and wont be unloaded are blocked ~unnecessarily.

I think it makes sense to refine this so that once we have identified which existing model is going to be unloaded for the new model request, we allow other requests to model(s) that aren't getting unloaded to proceed unblocked.

<!-- gh-comment-id:2231790317 --> @dhiltgen commented on GitHub (Jul 16, 2024): The current algorithm queues up all requests when we're adjusting the set of loaded models to ensure we get the VRAM information correct and don't hit OOM crashes. A side effect of this is requests for a model that's already loaded, and wont be unloaded are blocked ~unnecessarily. I think it makes sense to refine this so that once we have identified which existing model is going to be unloaded for the new model request, we allow other requests to model(s) that aren't getting unloaded to proceed unblocked.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#3565
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?