github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #7206] Help, is it possible to copy and deploy a small size model onto multiple GPU cards? #66631

New Issue
Closed
opened 2026-05-04 07:39:29 -05:00 by GiteaMirror · 2 comments
GiteaMirror commented 2026-05-04 07:39:29 -05:00
Owner
Copy Link

Originally created by @tigflanker on GitHub (Oct 15, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7206

The current situation is as follows:

I have 4 T4 GPU cards, each with 16G of memory, and my model size is the quantized version, which is only 5G.

When considering the multi-concurrency scenario in the production environment, I want to load this model onto all 4 GPUs simultaneously (for subsequent concurrent invocation).

Is there a way to do this? Thanks.

Originally created by @tigflanker on GitHub (Oct 15, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7206 The current situation is as follows: I have 4 T4 GPU cards, each with 16G of memory, and my model size is the quantized version, which is only 5G. When considering the multi-concurrency scenario in the production environment, I want to load this model onto all 4 GPUs simultaneously (for subsequent concurrent invocation). Is there a way to do this? Thanks.
GiteaMirror added the feature request label 2026-05-04 07:39:29 -05:00
GiteaMirror commented 2026-05-04 07:39:31 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Oct 15, 2024):

ollama doesn't currently support loading the same model more than once. https://github.com/ollama/ollama/issues/3902 is tracking the work for loading a model multiple times but there's no progress so far.

If you want to run multiple copies of the same model, the easiest way at the moment is to start an ollama server on a different port for each GPU, using CUDA_VISIBLE_DEVICES to bind each GPU to it's server, and then use a reverse proxy (eg nginx or ollama_proxy_server) to distribute the requests to the ollama servers.

Another approach would be to set OLLAMA_SCHED_SPREAD=1 to force the model to be spread across all the GPUs and set OLLAMA_NUM_PARALLEL=4 to allow concurrent processing. I don't know what the performance of this type of configuration would be.

<!-- gh-comment-id:2413928300 --> @rick-github commented on GitHub (Oct 15, 2024): ollama doesn't currently support loading the same model more than once. https://github.com/ollama/ollama/issues/3902 is tracking the work for loading a model multiple times but there's no progress so far. If you want to run multiple copies of the same model, the easiest way at the moment is to start an ollama server on a different port for each GPU, using `CUDA_VISIBLE_DEVICES` to bind each GPU to it's server, and then use a reverse proxy (eg [nginx](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/) or [ollama_proxy_server](https://github.com/ParisNeo/ollama_proxy_server)) to distribute the requests to the ollama servers. Another approach would be to set `OLLAMA_SCHED_SPREAD=1` to force the model to be spread across all the GPUs and set `OLLAMA_NUM_PARALLEL=4` to allow concurrent processing. I don't know what the performance of this type of configuration would be.
GiteaMirror commented 2026-05-04 07:39:31 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Oct 15, 2024):

Tracking via #3902

<!-- gh-comment-id:2415221414 --> @dhiltgen commented on GitHub (Oct 15, 2024): Tracking via #3902
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#66631
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?