github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9394] Specify GPU to run model #31888

New Issue
Closed
opened 2026-04-22 12:40:52 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-22 12:40:52 -05:00
Owner
Copy Link

Originally created by @wangwuli-AAA on GitHub (Feb 27, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9394

I have multiple graphics cards and I want to specify the model to run on a specific graphics card. But I found that configuring environment variables cannot achieve this, and it was not mentioned in the documentation.

Originally created by @wangwuli-AAA on GitHub (Feb 27, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9394 I have multiple graphics cards and I want to specify the model to run on a specific graphics card. But I found that configuring environment variables cannot achieve this, and it was not mentioned in the documentation.
GiteaMirror added the feature request label 2026-04-22 12:40:52 -05:00
GiteaMirror commented 2026-04-22 12:40:54 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 27, 2025):

Start an ollama server on a different port for each GPU, using CUDA_VISIBLE_DEVICES to bind each GPU to it's server, and then use a reverse proxy (eg nginx or ollama_proxy_server) to distribute the requests to the ollama servers.

https://github.com/ollama/ollama/issues/3902

<!-- gh-comment-id:2688049181 --> @rick-github commented on GitHub (Feb 27, 2025): Start an ollama server on a different port for each GPU, using `CUDA_VISIBLE_DEVICES` to bind each GPU to it's server, and then use a reverse proxy (eg [nginx](https://github.com/ollama/ollama/issues/7570#issuecomment-2464469733) or [ollama_proxy_server](https://github.com/ParisNeo/ollama_proxy_server)) to distribute the requests to the ollama servers. https://github.com/ollama/ollama/issues/3902
GiteaMirror commented 2026-04-22 12:40:55 -05:00
Author
Owner
Copy Link

@wangwuli-AAA commented on GitHub (Feb 27, 2025):

Start an ollama server on a different port for each GPU, using to bind each GPU to it's server, and then use a reverse proxy (eg nginx or ollama_proxy_server) to distribute the requests to the ollama servers.CUDA_VISIBLE_DEVICES

#3902

I think you misunderstood.
For example, I have 4 GPU and I only want to load the model on the first two.
I want to use these two for something else instead of using ollama.
I tried CUDA_VISIBLEDEVICES but couldn't do it.

<!-- gh-comment-id:2688120920 --> @wangwuli-AAA commented on GitHub (Feb 27, 2025): > Start an ollama server on a different port for each GPU, using to bind each GPU to it's server, and then use a reverse proxy (eg [nginx](https://github.com/ollama/ollama/issues/7570#issuecomment-2464469733) or [ollama_proxy_server](https://github.com/ParisNeo/ollama_proxy_server)) to distribute the requests to the ollama servers.`CUDA_VISIBLE_DEVICES` > > [#3902](https://github.com/ollama/ollama/issues/3902) I think you misunderstood. For example, I have 4 GPU and I only want to load the model on the first two. I want to use these two for something else instead of using ollama. I tried CUDA_VISIBLEDEVICES but couldn't do it.
GiteaMirror commented 2026-04-22 12:40:56 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 27, 2025):

If you can supply server logs it may be possible to figure out why it's not working.

<!-- gh-comment-id:2688129164 --> @rick-github commented on GitHub (Feb 27, 2025): If you can supply [server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) it may be possible to figure out why it's not working.
GiteaMirror commented 2026-04-22 12:40:56 -05:00
Author
Owner
Copy Link

@wangwuli-AAA commented on GitHub (Feb 27, 2025):

2/ 27 19:55:27  ollama[2573567]: time=2025-02-27T19:55:27.532+08:00 level=INFO source=server.go:596 msg="llama runner started in 10.04 seconds"
2/ 27 19:55:43  ollama[2573567]: [GIN] 2025/02/27 - 19:55:43 | 200 | 28.505243458s |       127.0.0.1 | POST     "/api/chat"
2/ 27 19:56:10  ollama[2573567]: [GIN] 2025/02/27 - 19:56:10 | 200 | 25.336561327s |       127.0.0.1 | POST     "/api/chat"
2/ 27 19:56:31  ollama[2573567]: [GIN] 2025/02/27 - 19:56:31 | 200 | 21.592891546s |       127.0.0.1 | POST     "/api/chat"
2/ 27 19:57:05  ollama[2573567]: [GIN] 2025/02/27 - 19:57:05 | 200 |      63.252µs |       127.0.0.1 | HEAD     "/"
2/ 27 19:57:05  ollama[2573567]: [GIN] 2025/02/27 - 19:57:05 | 200 |     293.266µs |       127.0.0.1 | GET      "/api/ps"
2/ 27 19:57:48  ollama[2573567]: [GIN] 2025/02/27 - 19:57:48 | 200 |         1m16s |       127.0.0.1 | POST     "/api/chat"
2/ 27 20:00:35  ollama[2573567]: [GIN] 2025/02/27 - 20:00:35 | 200 |       57.57µs |       127.0.0.1 | HEAD     "/"
2/ 27 20:00:35  ollama[2573567]: [GIN] 2025/02/27 - 20:00:35 | 200 |      76.289µs |       127.0.0.1 | GET      "/api/ps"
2/ 27 20:00:46  ollama[2573567]: [GIN] 2025/02/27 - 20:00:46 | 200 |      23.136µs |       127.0.0.1 | HEAD     "/"
2/ 27 20:00:46  ollama[2573567]: [GIN] 2025/02/27 - 20:00:46 | 200 |     1.14952ms |       127.0.0.1 | POST     "/api/generate"
2/ 27 20:00:48  ollama[2573567]: [GIN] 2025/02/27 - 20:00:48 | 200 |      61.004µs |       127.0.0.1 | HEAD     "/"
2/ 27 20:00:48  ollama[2573567]: [GIN] 2025/02/27 - 20:00:48 | 200 |        17.4µs |       127.0.0.1 | GET      "/api/ps"
2/ 27 20:01:24  ollama[2573567]: [GIN] 2025/02/27 - 20:01:24 | 200 |      45.953µs |       127.0.0.1 | HEAD     "/"
2/ 27 20:01:24  ollama[2573567]: [GIN] 2025/02/27 - 20:01:24 | 200 |   28.406345ms |       127.0.0.1 | POST     "/api/show"
2/ 27 20:01:25  ollama[2573567]: time=2025-02-27T20:01:25.252+08:00 level=INFO source=sched.go:730 msg="new model will fit in available VRAM, loading" model=/mnt/mydisk/mod>
2/ 27 20:01:26  ollama[2573567]: time=2025-02-27T20:01:26.110+08:00 level=INFO source=server.go:100 msg="system memory" total="629.5 GiB" free="490.3 GiB" free_swap="20.1 M>
2/ 27 20:01:26  ollama[2573567]: time=2025-02-27T20:01:26.111+08:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=81 layers.offload>
2/ 27 20:01:26  ollama[2573567]: time=2025-02-27T20:01:26.111+08:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /m>
2/ 27 20:01:26  ollama[2573567]: time=2025-02-27T20:01:26.111+08:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
2/ 27 20:01:26  ollama[2573567]: time=2025-02-27T20:01:26.112+08:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding"
2/ 27 20:01:26  ollama[2573567]: time=2025-02-27T20:01:26.112+08:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error"
2/ 27 20:01:26  ollama[2573567]: time=2025-02-27T20:01:26.133+08:00 level=INFO source=runner.go:936 msg="starting go runner"
2/ 27 20:01:26  ollama[2573567]: time=2025-02-27T20:01:26.133+08:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" t>
2/ 27 20:01:26  ollama[2573567]: time=2025-02-27T20:01:26.134+08:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:42979"
2/ 27 20:01:26  ollama[2573567]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
2/ 27 20:01:26  ollama[2573567]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
2/ 27 20:01:26  ollama[2573567]: ggml_cuda_init: found 8 CUDA devices:
2/ 27 20:01:26  ollama[2573567]:   Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
2/ 27 20:01:26  ollama[2573567]:   Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
2/ 27 20:01:26  ollama[2573567]:   Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
2/ 27 20:01:26  ollama[2573567]:   Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
2/ 27 20:01:26  ollama[2573567]:   Device 4: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
2/ 27 20:01:26  ollama[2573567]:   Device 5: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
2/ 27 20:01:26  ollama[2573567]:   Device 6: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
2/ 27 20:01:26  ollama[2573567]:   Device 7: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes

This is my log. I specified 4 GPUs to be visible, but it still uses all of them

<!-- gh-comment-id:2688321149 --> @wangwuli-AAA commented on GitHub (Feb 27, 2025): ``` 2/ 27 19:55:27 ollama[2573567]: time=2025-02-27T19:55:27.532+08:00 level=INFO source=server.go:596 msg="llama runner started in 10.04 seconds" 2/ 27 19:55:43 ollama[2573567]: [GIN] 2025/02/27 - 19:55:43 | 200 | 28.505243458s | 127.0.0.1 | POST "/api/chat" 2/ 27 19:56:10 ollama[2573567]: [GIN] 2025/02/27 - 19:56:10 | 200 | 25.336561327s | 127.0.0.1 | POST "/api/chat" 2/ 27 19:56:31 ollama[2573567]: [GIN] 2025/02/27 - 19:56:31 | 200 | 21.592891546s | 127.0.0.1 | POST "/api/chat" 2/ 27 19:57:05 ollama[2573567]: [GIN] 2025/02/27 - 19:57:05 | 200 | 63.252µs | 127.0.0.1 | HEAD "/" 2/ 27 19:57:05 ollama[2573567]: [GIN] 2025/02/27 - 19:57:05 | 200 | 293.266µs | 127.0.0.1 | GET "/api/ps" 2/ 27 19:57:48 ollama[2573567]: [GIN] 2025/02/27 - 19:57:48 | 200 | 1m16s | 127.0.0.1 | POST "/api/chat" 2/ 27 20:00:35 ollama[2573567]: [GIN] 2025/02/27 - 20:00:35 | 200 | 57.57µs | 127.0.0.1 | HEAD "/" 2/ 27 20:00:35 ollama[2573567]: [GIN] 2025/02/27 - 20:00:35 | 200 | 76.289µs | 127.0.0.1 | GET "/api/ps" 2/ 27 20:00:46 ollama[2573567]: [GIN] 2025/02/27 - 20:00:46 | 200 | 23.136µs | 127.0.0.1 | HEAD "/" 2/ 27 20:00:46 ollama[2573567]: [GIN] 2025/02/27 - 20:00:46 | 200 | 1.14952ms | 127.0.0.1 | POST "/api/generate" 2/ 27 20:00:48 ollama[2573567]: [GIN] 2025/02/27 - 20:00:48 | 200 | 61.004µs | 127.0.0.1 | HEAD "/" 2/ 27 20:00:48 ollama[2573567]: [GIN] 2025/02/27 - 20:00:48 | 200 | 17.4µs | 127.0.0.1 | GET "/api/ps" 2/ 27 20:01:24 ollama[2573567]: [GIN] 2025/02/27 - 20:01:24 | 200 | 45.953µs | 127.0.0.1 | HEAD "/" 2/ 27 20:01:24 ollama[2573567]: [GIN] 2025/02/27 - 20:01:24 | 200 | 28.406345ms | 127.0.0.1 | POST "/api/show" 2/ 27 20:01:25 ollama[2573567]: time=2025-02-27T20:01:25.252+08:00 level=INFO source=sched.go:730 msg="new model will fit in available VRAM, loading" model=/mnt/mydisk/mod> 2/ 27 20:01:26 ollama[2573567]: time=2025-02-27T20:01:26.110+08:00 level=INFO source=server.go:100 msg="system memory" total="629.5 GiB" free="490.3 GiB" free_swap="20.1 M> 2/ 27 20:01:26 ollama[2573567]: time=2025-02-27T20:01:26.111+08:00 level=INFO source=memory.go:356 msg="offload to cuda" layers.requested=-1 layers.model=81 layers.offload> 2/ 27 20:01:26 ollama[2573567]: time=2025-02-27T20:01:26.111+08:00 level=INFO source=server.go:380 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /m> 2/ 27 20:01:26 ollama[2573567]: time=2025-02-27T20:01:26.111+08:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 2/ 27 20:01:26 ollama[2573567]: time=2025-02-27T20:01:26.112+08:00 level=INFO source=server.go:557 msg="waiting for llama runner to start responding" 2/ 27 20:01:26 ollama[2573567]: time=2025-02-27T20:01:26.112+08:00 level=INFO source=server.go:591 msg="waiting for server to become available" status="llm server error" 2/ 27 20:01:26 ollama[2573567]: time=2025-02-27T20:01:26.133+08:00 level=INFO source=runner.go:936 msg="starting go runner" 2/ 27 20:01:26 ollama[2573567]: time=2025-02-27T20:01:26.133+08:00 level=INFO source=runner.go:937 msg=system info="CPU : LLAMAFILE = 1 | CPU : LLAMAFILE = 1 | cgo(gcc)" t> 2/ 27 20:01:26 ollama[2573567]: time=2025-02-27T20:01:26.134+08:00 level=INFO source=runner.go:995 msg="Server listening on 127.0.0.1:42979" 2/ 27 20:01:26 ollama[2573567]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no 2/ 27 20:01:26 ollama[2573567]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no 2/ 27 20:01:26 ollama[2573567]: ggml_cuda_init: found 8 CUDA devices: 2/ 27 20:01:26 ollama[2573567]: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes 2/ 27 20:01:26 ollama[2573567]: Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes 2/ 27 20:01:26 ollama[2573567]: Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes 2/ 27 20:01:26 ollama[2573567]: Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes 2/ 27 20:01:26 ollama[2573567]: Device 4: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes 2/ 27 20:01:26 ollama[2573567]: Device 5: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes 2/ 27 20:01:26 ollama[2573567]: Device 6: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes 2/ 27 20:01:26 ollama[2573567]: Device 7: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes ``` This is my log. I specified 4 GPUs to be visible, but it still uses all of them
GiteaMirror commented 2026-04-22 12:40:57 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 27, 2025):

A full server log, from when the server started.

<!-- gh-comment-id:2688326267 --> @rick-github commented on GitHub (Feb 27, 2025): A full server log, from when the server started.
GiteaMirror commented 2026-04-22 12:40:58 -05:00
Author
Owner
Copy Link

@fxmbsw7 commented on GitHub (Mar 2, 2025):

u set , two as visible
and start another ollama serve with the other two visible

<!-- gh-comment-id:2692752690 --> @fxmbsw7 commented on GitHub (Mar 2, 2025): u set , two as visible and start another `ollama serve` with the other two visible
GiteaMirror commented 2026-04-22 12:40:58 -05:00
Author
Owner
Copy Link

@wangwuli-AAA commented on GitHub (Mar 6, 2025):

I suddenly realized that using Docker can do this.........

<!-- gh-comment-id:2703687631 --> @wangwuli-AAA commented on GitHub (Mar 6, 2025): I suddenly realized that using Docker can do this.........
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#31888
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?