github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6840] no nvidia devices detected #4320

New Issue
Closed
opened 2026-04-12 15:14:57 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-04-12 15:14:57 -05:00
Owner
Copy Link

Originally created by @deardeer7 on GitHub (Sep 17, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6840

What is the issue?

I deployed ollama with docker on a debian12 machine, but it can't use the GPU, it can only use the cpu for inference.
The ollama docker log:

LAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"

time=2024-09-17T15:04:01.101Z level=INFO source=images.go:753 msg="total blobs: 11"

time=2024-09-17T15:04:01.101Z level=INFO source=images.go:760 msg="total unused blobs removed: 0"

time=2024-09-17T15:04:01.101Z level=INFO source=routes.go:1172 msg="Listening on [::]:11434 (version 0.3.10)"

time=2024-09-17T15:04:01.102Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama303164785/runners

time=2024-09-17T15:04:08.212Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102]"

time=2024-09-17T15:04:08.212Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs"

time=2024-09-17T15:04:08.215Z level=INFO source=gpu.go:560 msg="no nvidia devices detected" library=/usr/lib/x86_64-linux-gnu/libcuda.so.550.107.02

time=2024-09-17T15:04:08.220Z level=INFO source=gpu.go:347 msg="no compatible GPUs were discovered"

time=2024-09-17T15:04:08.220Z level=INFO source=types.go:107 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="31.3 GiB" available="29.4 GiB"

My GPU is a GTX 1060, driver is installed, nvidia-smi:

Tue Sep 17 23:09:22 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1060 6GB    Off |   00000000:2B:00.0 Off |                  N/A |
| 57%   43C    P0             28W /  120W |       0MiB /   6144MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

docker compose:

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    environment:
      - OLLAMA_ORIGINS=*
    volumes:
      - /vol1/1000/os/docker/ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    restart: always

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

latest

Originally created by @deardeer7 on GitHub (Sep 17, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6840 ### What is the issue? I deployed ollama with docker on a debian12 machine, but it can't use the GPU, it can only use the cpu for inference. The ollama docker log: ```log LAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[* http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]" time=2024-09-17T15:04:01.101Z level=INFO source=images.go:753 msg="total blobs: 11" time=2024-09-17T15:04:01.101Z level=INFO source=images.go:760 msg="total unused blobs removed: 0" time=2024-09-17T15:04:01.101Z level=INFO source=routes.go:1172 msg="Listening on [::]:11434 (version 0.3.10)" time=2024-09-17T15:04:01.102Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama303164785/runners time=2024-09-17T15:04:08.212Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102]" time=2024-09-17T15:04:08.212Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs" time=2024-09-17T15:04:08.215Z level=INFO source=gpu.go:560 msg="no nvidia devices detected" library=/usr/lib/x86_64-linux-gnu/libcuda.so.550.107.02 time=2024-09-17T15:04:08.220Z level=INFO source=gpu.go:347 msg="no compatible GPUs were discovered" time=2024-09-17T15:04:08.220Z level=INFO source=types.go:107 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="31.3 GiB" available="29.4 GiB" ``` My GPU is a GTX 1060, driver is installed, nvidia-smi: ``` Tue Sep 17 23:09:22 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.107.02 Driver Version: 550.107.02 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce GTX 1060 6GB Off | 00000000:2B:00.0 Off | N/A | | 57% 43C P0 28W / 120W | 0MiB / 6144MiB | 2% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ``` docker compose: ``` version: '3.8' services: ollama: image: ollama/ollama:latest container_name: ollama ports: - "11434:11434" environment: - OLLAMA_ORIGINS=* volumes: - /vol1/1000/os/docker/ollama:/root/.ollama deploy: resources: reservations: devices: - capabilities: [gpu] restart: always ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version latest
GiteaMirror added the bug label 2026-04-12 15:14:57 -05:00
GiteaMirror commented 2026-04-12 15:14:59 -05:00
Author
Owner
Copy Link

@hainakus commented on GitHub (Sep 17, 2024):

For Nvidia test with sudo docker run --rm --runtime=nvidia
-e NVIDIA_VISIBLE_DEVICES=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

<!-- gh-comment-id:2356677531 --> @hainakus commented on GitHub (Sep 17, 2024): For Nvidia test with sudo docker run --rm --runtime=nvidia \ -e NVIDIA_VISIBLE_DEVICES=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
GiteaMirror commented 2026-04-12 15:15:00 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 18, 2024):

What's the output of docker exec ollama nvidia-smi?

<!-- gh-comment-id:2357226047 --> @rick-github commented on GitHub (Sep 18, 2024): What's the output of `docker exec ollama nvidia-smi`?
GiteaMirror commented 2026-04-12 15:15:00 -05:00
Author
Owner
Copy Link

@deardeer7 commented on GitHub (Sep 18, 2024):

What's the output of docker exec ollama nvidia-smi?

Failed to initialize NVML: Unknown Error

<!-- gh-comment-id:2357309547 --> @deardeer7 commented on GitHub (Sep 18, 2024): > What's the output of `docker exec ollama nvidia-smi`? `Failed to initialize NVML: Unknown Error`
GiteaMirror commented 2026-04-12 15:15:01 -05:00
Author
Owner
Copy Link

@deardeer7 commented on GitHub (Sep 18, 2024):

sudo docker run --rm --runtime=nvidia
-e NVIDIA_VISIBLE_DEVICES=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

sudo docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all -v ollama:/root/.ollama -p 11434:11434 --name ollama2 ollama/ollama

output:

Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.
Your new public key is:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN20Gy8QTobWu9uJrobect66kdma2CMbDiOSbdnlOQ9G

2024/09/18 01:35:39 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-09-18T01:35:39.240Z level=INFO source=images.go:753 msg="total blobs: 0"
time=2024-09-18T01:35:39.240Z level=INFO source=images.go:760 msg="total unused blobs removed: 0"
time=2024-09-18T01:35:39.240Z level=INFO source=routes.go:1172 msg="Listening on [::]:11434 (version 0.3.10)"
time=2024-09-18T01:35:39.240Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama994917163/runners
time=2024-09-18T01:35:46.472Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102 cpu]"
time=2024-09-18T01:35:46.472Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs"
time=2024-09-18T01:35:46.475Z level=INFO source=gpu.go:560 msg="no nvidia devices detected" library=/usr/lib/x86_64-linux-gnu/libcuda.so.550.107.02
time=2024-09-18T01:35:46.480Z level=INFO source=gpu.go:347 msg="no compatible GPUs were discovered"
time=2024-09-18T01:35:46.480Z level=INFO source=types.go:107 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="31.3 GiB" available="29.2 GiB"
<!-- gh-comment-id:2357314532 --> @deardeer7 commented on GitHub (Sep 18, 2024): > sudo docker run --rm --runtime=nvidia > -e NVIDIA_VISIBLE_DEVICES=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama ```shell sudo docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all -v ollama:/root/.ollama -p 11434:11434 --name ollama2 ollama/ollama ``` output: ``` Couldn't find '/root/.ollama/id_ed25519'. Generating new private key. Your new public key is: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN20Gy8QTobWu9uJrobect66kdma2CMbDiOSbdnlOQ9G 2024/09/18 01:35:39 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]" time=2024-09-18T01:35:39.240Z level=INFO source=images.go:753 msg="total blobs: 0" time=2024-09-18T01:35:39.240Z level=INFO source=images.go:760 msg="total unused blobs removed: 0" time=2024-09-18T01:35:39.240Z level=INFO source=routes.go:1172 msg="Listening on [::]:11434 (version 0.3.10)" time=2024-09-18T01:35:39.240Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama994917163/runners time=2024-09-18T01:35:46.472Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm_v60102 cpu]" time=2024-09-18T01:35:46.472Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs" time=2024-09-18T01:35:46.475Z level=INFO source=gpu.go:560 msg="no nvidia devices detected" library=/usr/lib/x86_64-linux-gnu/libcuda.so.550.107.02 time=2024-09-18T01:35:46.480Z level=INFO source=gpu.go:347 msg="no compatible GPUs were discovered" time=2024-09-18T01:35:46.480Z level=INFO source=types.go:107 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="31.3 GiB" available="29.2 GiB" ```
GiteaMirror commented 2026-04-12 15:15:01 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 18, 2024):

What does docker run --rm --gpus all --privileged --entrypoint nvidia-smi ollama/ollama output?

<!-- gh-comment-id:2357335269 --> @rick-github commented on GitHub (Sep 18, 2024): What does `docker run --rm --gpus all --privileged --entrypoint nvidia-smi ollama/ollama` output?
GiteaMirror commented 2026-04-12 15:15:02 -05:00
Author
Owner
Copy Link

@deardeer7 commented on GitHub (Sep 18, 2024):

docker run --rm --gpus all --privileged --entrypoint nvidia-smi ollama/ollama

looks like ok?

Wed Sep 18 01:59:18 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1060 6GB    Off |   00000000:2B:00.0 Off |                  N/A |
| 36%   50C    P8             11W /  120W |     202MiB /   6144MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+
<!-- gh-comment-id:2357339512 --> @deardeer7 commented on GitHub (Sep 18, 2024): > docker run --rm --gpus all --privileged --entrypoint nvidia-smi ollama/ollama looks like ok? ``` Wed Sep 18 01:59:18 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.107.02 Driver Version: 550.107.02 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce GTX 1060 6GB Off | 00000000:2B:00.0 Off | N/A | | 36% 50C P8 11W / 120W | 202MiB / 6144MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| +-----------------------------------------------------------------------------------------+ ```
GiteaMirror commented 2026-04-12 15:15:02 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 18, 2024):

Uncomment #no-cgroups = false in /etc/nvidia-container-runtime/config.toml, then

sudo systemctl restart docker
docker run --rm --gpus all --entrypoint nvidia-smi ollama/ollama
<!-- gh-comment-id:2357344001 --> @rick-github commented on GitHub (Sep 18, 2024): Uncomment `#no-cgroups = false` in `/etc/nvidia-container-runtime/config.toml`, then ``` sudo systemctl restart docker docker run --rm --gpus all --entrypoint nvidia-smi ollama/ollama ```
GiteaMirror commented 2026-04-12 15:15:03 -05:00
Author
Owner
Copy Link

@deardeer7 commented on GitHub (Sep 18, 2024):

Uncomment #no-cgroups = false in /etc/nvidia-container-runtime/config.toml, then

sudo systemctl restart docker
docker run --rm --gpus all --entrypoint nvidia-smi ollama/ollama

Solved my problem! Thank you very much!

  1. changed to no-cgroups = false in /etc/nvidia-container-runtime/config.toml
  2. restart docker sudo systemctl restart docker
<!-- gh-comment-id:2357351552 --> @deardeer7 commented on GitHub (Sep 18, 2024): > Uncomment `#no-cgroups = false` in `/etc/nvidia-container-runtime/config.toml`, then > > ``` > sudo systemctl restart docker > docker run --rm --gpus all --entrypoint nvidia-smi ollama/ollama > ``` Solved my problem! Thank you very much! 1. changed to `no-cgroups = false` in `/etc/nvidia-container-runtime/config.toml` 2. restart docker `sudo systemctl restart docker`
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#4320
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?