github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #4350] Configurable model loading timeout #2713

New Issue
Closed
opened 2026-04-12 13:01:45 -05:00 by GiteaMirror · 10 comments
GiteaMirror commented 2026-04-12 13:01:45 -05:00
Owner
Copy Link

Originally created by @ProjectMoon on GitHub (May 11, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4350

The model loading timeout, the time to wait for the llama runner, is hard coded. It would be nice to be able to configure this to increase or decrease it (for me, mostly increase). This would allow experimenting with big models that take forever to load, but might run fine once loaded.

Originally created by @ProjectMoon on GitHub (May 11, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4350 The model loading timeout, the time to wait for the llama runner, is hard coded. It would be nice to be able to configure this to increase or decrease it (for me, mostly increase). This would allow experimenting with big models that take forever to load, but might run fine once loaded.
GiteaMirror added the feature request label 2026-04-12 13:01:45 -05:00
GiteaMirror commented 2026-04-12 13:01:46 -05:00
Author
Owner
Copy Link

@lengrongfu commented on GitHub (May 13, 2024):

I can try to add this feature.

<!-- gh-comment-id:2106552500 --> @lengrongfu commented on GitHub (May 13, 2024): I can try to add this feature.
GiteaMirror commented 2026-04-12 13:01:46 -05:00
Author
Owner
Copy Link

@lengrongfu commented on GitHub (May 13, 2024):

/assign

<!-- gh-comment-id:2106552567 --> @lengrongfu commented on GitHub (May 13, 2024): /assign
GiteaMirror commented 2026-04-12 13:01:47 -05:00
Author
Owner
Copy Link

@lengrongfu commented on GitHub (May 13, 2024):

@ProjectMoon Your problem is that when ollama runs a model that does not exist locally, does pulling it from the registry time out?

<!-- gh-comment-id:2106556703 --> @lengrongfu commented on GitHub (May 13, 2024): @ProjectMoon Your problem is that when ollama runs a model that does not exist locally, does pulling it from the registry time out?
GiteaMirror commented 2026-04-12 13:01:47 -05:00
Author
Owner
Copy Link

@ProjectMoon commented on GitHub (May 13, 2024):

No, it's the loading of the model from disk. There's a hard coded timeout of 10 minutes. I forget the exact file where this is, but the comment above the line says something like "be generous, as long models can take a long time to load."

<!-- gh-comment-id:2106778204 --> @ProjectMoon commented on GitHub (May 13, 2024): No, it's the loading of the model from disk. There's a hard coded timeout of 10 minutes. I forget the exact file where this is, but the comment above the line says something like "be generous, as long models can take a long time to load."
GiteaMirror commented 2026-04-12 13:01:47 -05:00
Author
Owner
Copy Link

@ProjectMoon commented on GitHub (May 13, 2024):

An example of why this would be useful (to me): I can load Mixtral 8x7b using the Q2_K quant (smallest available file). It loads in 565 seconds, just under the 10 minute timeout limit. But once it's loaded, it generates at 16 tokens/second. It would be lovely if I could try the higher quants and see what happens.

<!-- gh-comment-id:2108395407 --> @ProjectMoon commented on GitHub (May 13, 2024): An example of why this would be useful (to me): I can load Mixtral 8x7b using the Q2_K quant (smallest available file). It loads in 565 seconds, just under the 10 minute timeout limit. But once it's loaded, it generates at 16 tokens/second. It would be lovely if I could try the higher quants and see what happens.
GiteaMirror commented 2026-04-12 13:01:48 -05:00
Author
Owner
Copy Link

@ProjectMoon commented on GitHub (Jun 14, 2024):

I am noticing some flakiness on slow model loads. I have one model that gets to 80% and then gives up. Should I reopen this issue or make a new issue to change the stall time?

<!-- gh-comment-id:2168154875 --> @ProjectMoon commented on GitHub (Jun 14, 2024): I am noticing some flakiness on slow model loads. I have one model that gets to 80% and then gives up. Should I reopen this issue or make a new issue to change the stall time?
GiteaMirror commented 2026-04-12 13:01:48 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jun 14, 2024):

@ProjectMoon can you try disabling mmap to see if that significantly changes the load times for your setup?

curl http://localhost:11434/api/generate -d '{
  "model": "llama3:70b",
  "prompt": "Why is the sky blue?",
  "stream": false, "options": {"use_mmap": false}
}'
<!-- gh-comment-id:2168318383 --> @dhiltgen commented on GitHub (Jun 14, 2024): @ProjectMoon can you try disabling mmap to see if that significantly changes the load times for your setup? ``` curl http://localhost:11434/api/generate -d '{ "model": "llama3:70b", "prompt": "Why is the sky blue?", "stream": false, "options": {"use_mmap": false} }' ```
GiteaMirror commented 2026-04-12 13:01:48 -05:00
Author
Owner
Copy Link

@ProjectMoon commented on GitHub (Jun 14, 2024):

@dhiltgen Disabling mmap makes a world of difference. All the models actually load now, properly split across CPU and GPU. They also load a billion times faster. It's like magic. But mmap doesn't seem to be globally toggleable, and OpenWebUI seems to have only "on" or "default" as options for mmap, instead of also having an "off" value (this isn't part of the Ollama project, but it is odd).

I will now be making some new modelfiles with ollama disabled.

<!-- gh-comment-id:2168492936 --> @ProjectMoon commented on GitHub (Jun 14, 2024): @dhiltgen Disabling mmap makes a world of difference. All the models actually load now, properly split across CPU and GPU. They also load a billion times faster. It's like magic. But mmap doesn't seem to be globally toggleable, and OpenWebUI seems to have only "on" or "default" as options for mmap, instead of also having an "off" value (this isn't part of the Ollama project, but it is odd). I will now be making some new modelfiles with ollama disabled.
GiteaMirror commented 2026-04-12 13:01:49 -05:00
Author
Owner
Copy Link

@ProjectMoon commented on GitHub (Jun 14, 2024):

Have noticed, though, that some models have issues loading with mmap disabled if I increase the context size. They crash for various reasons, and not always out of memory issues.

<!-- gh-comment-id:2168645543 --> @ProjectMoon commented on GitHub (Jun 14, 2024): Have noticed, though, that some models have issues loading with mmap disabled if I increase the context size. They crash for various reasons, and not always out of memory issues.
GiteaMirror commented 2026-04-12 13:01:49 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jun 14, 2024):

We'll track this via #3940

<!-- gh-comment-id:2168738082 --> @dhiltgen commented on GitHub (Jun 14, 2024): We'll track this via #3940
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#2713
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?