Hi! Ollama recently updated and added the following:
keep_alive parameter: control how long models stay loaded
When making API requests, the new keep_alive parameter can be used to control how long a model stays loaded in memory:
curl http://localhost:11434/api/generate -d '{
"model": "mistral",
"prompt": "Why is the sky blue?",
"keep_alive": "30s"
}'
If set to a positive duration (e.g. 20m, 1hr or 30), the model will stay loaded for the provided duration
If set to a negative duration (e.g. -1), the model will stay loaded indefinitely
If set to 0, the model will be unloaded immediately once finished
If not set, the model will stay loaded for 5 minutes by default
I believe this would be great to have in the 'Advanced' tab in ollama-webui's settings, for someone who regularly uses the same model I hate having to wait for it to load after it unloads after sitting idle. I can also make a pull request to implement this when I have free time too, I'm just very busy for this next week.
Thanks!
Originally created by @nick-tonjum on GitHub (Feb 7, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/670
Hi! Ollama recently updated and added the following:
```
keep_alive parameter: control how long models stay loaded
When making API requests, the new keep_alive parameter can be used to control how long a model stays loaded in memory:
curl http://localhost:11434/api/generate -d '{
"model": "mistral",
"prompt": "Why is the sky blue?",
"keep_alive": "30s"
}'
If set to a positive duration (e.g. 20m, 1hr or 30), the model will stay loaded for the provided duration
If set to a negative duration (e.g. -1), the model will stay loaded indefinitely
If set to 0, the model will be unloaded immediately once finished
If not set, the model will stay loaded for 5 minutes by default
```
I believe this would be great to have in the 'Advanced' tab in ollama-webui's settings, for someone who regularly uses the same model I hate having to wait for it to load after it unloads after sitting idle. I can also make a pull request to implement this when I have free time too, I'm just very busy for this next week.
Thanks!
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @nick-tonjum on GitHub (Feb 7, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/670
Hi! Ollama recently updated and added the following:
I believe this would be great to have in the 'Advanced' tab in ollama-webui's settings, for someone who regularly uses the same model I hate having to wait for it to load after it unloads after sitting idle. I can also make a pull request to implement this when I have free time too, I'm just very busy for this next week.
Thanks!
@justinh-rahb commented on GitHub (Feb 7, 2024):
Duplicate: #596