github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #4681] Cant we load two models in memory forever #2946

New Issue
Closed
opened 2026-04-12 13:19:17 -05:00 by GiteaMirror · 13 comments
GiteaMirror commented 2026-04-12 13:19:17 -05:00
Owner
Copy Link

Originally created by @Greatz08 on GitHub (May 28, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/4681

What i mean by this is that right now if i wanna load model in memory forever i have to use this command :
-> curl http://localhost:11434/api/generate -d '{"model": "dolphin-llama3", "keep_alive": -1}'
but if i load second model it replace that with the first mode which means that first model gets offload from memory and second one takes place. I have 8gb vram (RTX 4060) i know its not sufficient but i have seen many comments in past saying that it will use full vram and then use some system ram to load and process altho it will be slow but it will do it automatically so my question is that is it not applicable in this case or like ollama doesnt support this or any other thing.

Originally created by @Greatz08 on GitHub (May 28, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/4681 What i mean by this is that right now if i wanna load model in memory forever i have to use this command : -> curl http://localhost:11434/api/generate -d '{"model": "dolphin-llama3", "keep_alive": -1}' but if i load second model it replace that with the first mode which means that first model gets offload from memory and second one takes place. I have 8gb vram (RTX 4060) i know its not sufficient but i have seen many comments in past saying that it will use full vram and then use some system ram to load and process altho it will be slow but it will do it automatically so my question is that is it not applicable in this case or like ollama doesnt support this or any other thing.
GiteaMirror commented 2026-04-12 13:19:19 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 28, 2024):

Hey @HakaishinShwet you can use the OLLAMA_KEEP_ALIVE env variable to set the duration. More details in the FAQ.

<!-- gh-comment-id:2136039504 --> @pdevine commented on GitHub (May 28, 2024): Hey @HakaishinShwet you can use the `OLLAMA_KEEP_ALIVE` env variable to set the duration. More details in the [FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-keep-a-model-loaded-in-memory-or-make-it-unload-immediately).
GiteaMirror commented 2026-04-12 13:19:20 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 28, 2024):

Also, if there isn't space for the second model, it will unfortunately have to unload the first model.

<!-- gh-comment-id:2136042025 --> @pdevine commented on GitHub (May 28, 2024): Also, if there isn't space for the second model, it will unfortunately have to unload the first model.
GiteaMirror commented 2026-04-12 13:19:20 -05:00
Author
Owner
Copy Link

@Greatz08 commented on GitHub (May 29, 2024):

@pdevine will it not load second model in ram automatically or use remaining vram and load other part in ram and help in using two models?

<!-- gh-comment-id:2136853005 --> @Greatz08 commented on GitHub (May 29, 2024): @pdevine will it not load second model in ram automatically or use remaining vram and load other part in ram and help in using two models?
GiteaMirror commented 2026-04-12 13:19:21 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 29, 2024):

It depends on if OLLAMA_MAX_LOADED_MODELS is set. By default (for now) it will default to a single model, but if you set it to a higher number it will try to keep up to that number of models in memory before unloading one to make room for another.

<!-- gh-comment-id:2137881436 --> @pdevine commented on GitHub (May 29, 2024): It depends on if `OLLAMA_MAX_LOADED_MODELS` is set. By default (for now) it will default to a single model, but if you set it to a higher number it will try to keep up to that number of models in memory before unloading one to make room for another.
GiteaMirror commented 2026-04-12 13:19:22 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 29, 2024):

Also, starting in 0.1.39 you can do ollama serve -h which will give you a list of the different environment variables you can set for the server.

<!-- gh-comment-id:2137950014 --> @pdevine commented on GitHub (May 29, 2024): Also, starting in `0.1.39` you can do `ollama serve -h` which will give you a list of the different environment variables you can set for the server.
GiteaMirror commented 2026-04-12 13:19:22 -05:00
Author
Owner
Copy Link

@Greatz08 commented on GitHub (May 31, 2024):

@pdevine thankyou very much for clear explanation. do i have to set this environmental variable in system service or in /etc/environment or in bashrc or zshrc file? or somewhere else and what will be exact format and procedure please guide for it too

<!-- gh-comment-id:2141435172 --> @Greatz08 commented on GitHub (May 31, 2024): @pdevine thankyou very much for clear explanation. do i have to set this environmental variable in system service or in /etc/environment or in bashrc or zshrc file? or somewhere else and what will be exact format and procedure please guide for it too
GiteaMirror commented 2026-04-12 13:19:23 -05:00
Author
Owner
Copy Link

@Cownjackson commented on GitHub (May 31, 2024):

I'm having a similar issue. Running ollama on an Nvidia L40s (48GB VRAM) and plenty of CPU RAM (384GB RAM) windows server. I have OLLAMA_MAX_LOADED_MODELS=4 and OLLAMA_NUM_PARALLEL=4 and it is still swapping the llama3 model with the smaller gemma model I've been using to test parallelization. I recall seeing both models loaded and performing inference together closer to when parallelization was released but perhaps I was confused.

Is there a way to guarentee a model persists even if it is inactive and another model has been requested?

<!-- gh-comment-id:2142728050 --> @Cownjackson commented on GitHub (May 31, 2024): I'm having a similar issue. Running ollama on an Nvidia L40s (48GB VRAM) and plenty of CPU RAM (384GB RAM) windows server. I have OLLAMA_MAX_LOADED_MODELS=4 and OLLAMA_NUM_PARALLEL=4 and it is still swapping the llama3 model with the smaller gemma model I've been using to test parallelization. I recall seeing both models loaded and performing inference together closer to when parallelization was released but perhaps I was confused. Is there a way to guarentee a model persists even if it is inactive and another model has been requested?
GiteaMirror commented 2026-04-12 13:19:25 -05:00
Author
Owner
Copy Link

@Cownjackson commented on GitHub (May 31, 2024):

I'm having a similar issue. Running ollama on an Nvidia L40s (48GB VRAM) and plenty of CPU RAM (384GB RAM) windows server. I have OLLAMA_MAX_LOADED_MODELS=4 and OLLAMA_NUM_PARALLEL=4 and it is still swapping the llama3 model with the smaller gemma model I've been using to test parallelization. I recall seeing both models loaded and performing inference together closer to when parallelization was released but perhaps I was confused.

Is there a way to guarentee a model persists even if it is inactive and another model has been requested?

Looks like my VRAM is the thing thats booting the model I must have been testing it with smaller models. So if the VRAM is maxed out, secondary models can't be loaded into the CPU memory while the primary model remains on the GPU?

<!-- gh-comment-id:2142734534 --> @Cownjackson commented on GitHub (May 31, 2024): > I'm having a similar issue. Running ollama on an Nvidia L40s (48GB VRAM) and plenty of CPU RAM (384GB RAM) windows server. I have OLLAMA_MAX_LOADED_MODELS=4 and OLLAMA_NUM_PARALLEL=4 and it is still swapping the llama3 model with the smaller gemma model I've been using to test parallelization. I recall seeing both models loaded and performing inference together closer to when parallelization was released but perhaps I was confused. > > Is there a way to guarentee a model persists even if it is inactive and another model has been requested? Looks like my VRAM is the thing thats booting the model I must have been testing it with smaller models. So if the VRAM is maxed out, secondary models can't be loaded into the CPU memory while the primary model remains on the GPU?
GiteaMirror commented 2026-04-12 13:19:26 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (May 31, 2024):

@Cownjackson That's correct. If your VRAM is maxed out it will swap in whatever model is trying to be used.

<!-- gh-comment-id:2142808359 --> @pdevine commented on GitHub (May 31, 2024): @Cownjackson That's correct. If your VRAM is maxed out it will swap in whatever model is trying to be used.
GiteaMirror commented 2026-04-12 13:19:26 -05:00
Author
Owner
Copy Link

@Greatz08 commented on GitHub (Jun 1, 2024):

@pdevine cant there be changes to how this works? i have seen lot more comments regarding this same thing. Like if i have 10 gb vram and 32 gb ddr 5 5600 Mhz great ram available and if one model take for example 5.5-6.5 gb vram and that model is coding based model but i wanna run another model for better chat response and other stuff and it will again consume 6-7 gb vram but i dont have that much available completely still 3.5-4 gb vram for example if i still have and wanna use with some amount of faster ram and cpu then according to current ollama system i will never be able to use it and that much vram left over will be wasted plus i cant take advantage of ram and cpu so this current ollama system is not looking too good for many users with respect to this type of situation . Right now it cares for only vram if you have that much it will run else unload and load game of models start but people do have great cpu's and ram and can still share some remaining part of vram and ram and cpu and run two models simultaneously is what i think.What i request from you is to maybe have conversation with other devs regarding this again and maybe find/create better solution to deal with this situation. Rest devs wish thats all from my side, Many cant afford too much vram and many run on laptops only in which we get 8-10 gb vram ones only for best price so running one small but still capable model is easy but running 2 is not possible in current situation

<!-- gh-comment-id:2143491012 --> @Greatz08 commented on GitHub (Jun 1, 2024): @pdevine cant there be changes to how this works? i have seen lot more comments regarding this same thing. Like if i have 10 gb vram and 32 gb ddr 5 5600 Mhz great ram available and if one model take for example 5.5-6.5 gb vram and that model is coding based model but i wanna run another model for better chat response and other stuff and it will again consume 6-7 gb vram but i dont have that much available completely still 3.5-4 gb vram for example if i still have and wanna use with some amount of faster ram and cpu then according to current ollama system i will never be able to use it and that much vram left over will be wasted plus i cant take advantage of ram and cpu so this current ollama system is not looking too good for many users with respect to this type of situation . Right now it cares for only vram if you have that much it will run else unload and load game of models start but people do have great cpu's and ram and can still share some remaining part of vram and ram and cpu and run two models simultaneously is what i think.What i request from you is to maybe have conversation with other devs regarding this again and maybe find/create better solution to deal with this situation. Rest devs wish thats all from my side, Many cant afford too much vram and many run on laptops only in which we get 8-10 gb vram ones only for best price so running one small but still capable model is easy but running 2 is not possible in current situation
GiteaMirror commented 2026-04-12 13:19:27 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Jun 7, 2024):

So the thinking here is that the model loaded entirely onto VRAM will always be considerably faster than a hybrid loaded model, so it's almost always faster to just load the model back into VRAM.

cc @dhiltgen

<!-- gh-comment-id:2155667109 --> @pdevine commented on GitHub (Jun 7, 2024): So the thinking here is that the model loaded entirely onto VRAM will always be considerably faster than a hybrid loaded model, so it's almost always faster to just load the model back into VRAM. cc @dhiltgen
GiteaMirror commented 2026-04-12 13:19:28 -05:00
Author
Owner
Copy Link

@Greatz08 commented on GitHub (Jun 9, 2024):

@pdevine can we increase the model loading speed by any mean? i am using linux btw.

<!-- gh-comment-id:2156263735 --> @Greatz08 commented on GitHub (Jun 9, 2024): @pdevine can we increase the model loading speed by any mean? i am using linux btw.
GiteaMirror commented 2026-04-12 13:19:29 -05:00
Author
Owner
Copy Link

@TIME-GATE commented on GitHub (Jun 20, 2024):

Why is this issue closed? can we understand that, according to the loading speed, VRAM > RAM > SSD > HDD, since we can't load all the models in VRAM, why don't we cache the model in RAM? in that case, model loading is the best, which is much faster than SSD or HDD?

<!-- gh-comment-id:2180657940 --> @TIME-GATE commented on GitHub (Jun 20, 2024): Why is this issue closed? can we understand that, according to the loading speed, VRAM > RAM > SSD > HDD, since we can't load all the models in VRAM, why don't we cache the model in RAM? in that case, model loading is the best, which is much faster than SSD or HDD?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#2946
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?