github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8674] No compatible GPUs were discovered #83305

New Issue
Closed
opened 2026-05-09 17:46:22 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-05-09 17:46:22 -05:00
Owner
Copy Link

Originally created by @mikedolx on GitHub (Jan 29, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8674

What is the issue?

Hi,

i'm currently trying to setup ollama within docker. I am using the following docker-compose.yml:


services:
  ollama:
    container_name: ollama
    restart: unless-stopped
    image: ollama/ollama:latest
    ports:
      - 11434:11434
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    networks:
      - ollama-docker
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: "1"
              capabilities: [gpu]

  ollama-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: ollama-webui
    volumes:
      - webui:/app/backend/data
    depends_on:
      - ollama
    ports:
      - 11080:8080
    environment: # https://docs.openwebui.com/getting-started/env-configuration#default_models
      - OLLAMA_BASE_URLS=http://host.docker.internal:7869 #comma separated ollama hosts
      - ENV=dev
      - WEBUI_AUTH=False
      - WEBUI_NAME=valiantlynx AI
      - WEBUI_URL=http://localhost:8080
      - WEBUI_SECRET_KEY=t0p-s3cr3t
    extra_hosts:
      - host.docker.internal:host-gateway
    restart: unless-stopped
    networks:
      - ollama-docker

volumes:
  webui:
  ollama:

networks:
  ollama-docker:
    external: false

When i start the containers and check the logs of the ollama container i can see the following logs.

[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

 - using env:	export GIN_MODE=release

 - using code:	gin.SetMode(gin.ReleaseMode)

2025/01/29 21:41:11 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:24h0m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"

[GIN-debug] POST   /api/pull                 --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)

time=2025-01-29T21:41:11.597Z level=INFO source=images.go:432 msg="total blobs: 0"

[GIN-debug] POST   /api/generate             --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)

time=2025-01-29T21:41:11.597Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"

[GIN-debug] POST   /api/chat                 --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)

time=2025-01-29T21:41:11.597Z level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7-0-ga420a45-dirty)"

[GIN-debug] POST   /api/embed                --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers)

time=2025-01-29T21:41:11.598Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 cuda_v11_avx cuda_v12_avx]"

[GIN-debug] POST   /api/embeddings           --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)

time=2025-01-29T21:41:11.598Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs"

[GIN-debug] POST   /api/create               --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers)

time=2025-01-29T21:41:11.601Z level=WARN source=gpu.go:623 msg="unknown error initializing cuda driver library /usr/lib/x86_64-linux-gnu/libcuda.so.535.216.03: cuda driver library init failure: 999. see https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md for more information"

Apperently, ollama is unable to recognize my GPU.

I can run nvidia-smi on the host and get the following result (which tells me that at least on the host everything is correctly installed):

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.216.03             Driver Version: 535.216.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3060        On  | 00000000:00:10.0 Off |                  N/A |
|  0%   47C    P5              18W / 170W |      1MiB / 12288MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

I can run the same command within the ollama container and get this result:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.216.03             Driver Version: 535.216.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3060        On  | 00000000:00:10.0 Off |                  N/A |
|  0%   50C    P5              18W / 170W |      1MiB / 12288MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

On the host machine i have installed the nvidia-driver using this method: https://ubuntu.com/server/docs/nvidia-drivers-installation.
I have also installed the cuda toolkit following these instructions: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
I have also installed the nvidia-container-toolkit and set it up in the /etc/docker/daemon.json accordingly.

I have read all the troubleshooting tipps here: https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md and tried the hints. (Yes, i've reloaded the nvidia_uvm module several times).

I have passed through an ASUS RTX 3060 12GB via IOMMU from my proxmox host (v7.x) to the docker-vm, which runs on ubuntu. Apperently, the GPU is correctly working, but fails to be recognized by the ollama container.

This is also how my /etc/docker/daemon.json looks like:

{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    },
    "exec-opts": ["native.cgroupdriver=cgroupfs"]
}

Any ideas what i could try?

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

ollama version is 0.5.7-0-ga420a45-dirty

Originally created by @mikedolx on GitHub (Jan 29, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8674 ### What is the issue? Hi, i'm currently trying to setup ollama within docker. I am using the following `docker-compose.yml`: ```yaml services: ollama: container_name: ollama restart: unless-stopped image: ollama/ollama:latest ports: - 11434:11434 environment: - OLLAMA_KEEP_ALIVE=24h networks: - ollama-docker volumes: - ollama:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: "1" capabilities: [gpu] ollama-webui: image: ghcr.io/open-webui/open-webui:main container_name: ollama-webui volumes: - webui:/app/backend/data depends_on: - ollama ports: - 11080:8080 environment: # https://docs.openwebui.com/getting-started/env-configuration#default_models - OLLAMA_BASE_URLS=http://host.docker.internal:7869 #comma separated ollama hosts - ENV=dev - WEBUI_AUTH=False - WEBUI_NAME=valiantlynx AI - WEBUI_URL=http://localhost:8080 - WEBUI_SECRET_KEY=t0p-s3cr3t extra_hosts: - host.docker.internal:host-gateway restart: unless-stopped networks: - ollama-docker volumes: webui: ollama: networks: ollama-docker: external: false ``` When i start the containers and check the logs of the ollama container i can see the following logs. ``` [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached. [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production. - using env: export GIN_MODE=release - using code: gin.SetMode(gin.ReleaseMode) 2025/01/29 21:41:11 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:24h0m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" [GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers) time=2025-01-29T21:41:11.597Z level=INFO source=images.go:432 msg="total blobs: 0" [GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers) time=2025-01-29T21:41:11.597Z level=INFO source=images.go:439 msg="total unused blobs removed: 0" [GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers) time=2025-01-29T21:41:11.597Z level=INFO source=routes.go:1238 msg="Listening on [::]:11434 (version 0.5.7-0-ga420a45-dirty)" [GIN-debug] POST /api/embed --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers) time=2025-01-29T21:41:11.598Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cpu cpu_avx cpu_avx2 cuda_v11_avx cuda_v12_avx]" [GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers) time=2025-01-29T21:41:11.598Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs" [GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers) time=2025-01-29T21:41:11.601Z level=WARN source=gpu.go:623 msg="unknown error initializing cuda driver library /usr/lib/x86_64-linux-gnu/libcuda.so.535.216.03: cuda driver library init failure: 999. see https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md for more information" ```` Apperently, ollama is unable to recognize my GPU. I can run `nvidia-smi` on the host and get the following result (which tells me that at least on the host everything is correctly installed): ```bash +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.216.03 Driver Version: 535.216.03 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce RTX 3060 On | 00000000:00:10.0 Off | N/A | | 0% 47C P5 18W / 170W | 1MiB / 12288MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | No running processes found | +---------------------------------------------------------------------------------------+ ``` I can run the same command within the ollama container and get this result: ```bash +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.216.03 Driver Version: 535.216.03 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce RTX 3060 On | 00000000:00:10.0 Off | N/A | | 0% 50C P5 18W / 170W | 1MiB / 12288MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | No running processes found | +---------------------------------------------------------------------------------------+ ``` On the host machine i have installed the nvidia-driver using this method: https://ubuntu.com/server/docs/nvidia-drivers-installation. I have also installed the cuda toolkit following these instructions: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html I have also installed the nvidia-container-toolkit and set it up in the `/etc/docker/daemon.json` accordingly. I have read all the troubleshooting tipps here: https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md and tried the hints. (Yes, i've reloaded the nvidia_uvm module several times). I have passed through an ASUS RTX 3060 12GB via IOMMU from my proxmox host (v7.x) to the docker-vm, which runs on ubuntu. Apperently, the GPU is correctly working, but fails to be recognized by the ollama container. This is also how my `/etc/docker/daemon.json` looks like: ```json { "runtimes": { "nvidia": { "args": [], "path": "nvidia-container-runtime" } }, "exec-opts": ["native.cgroupdriver=cgroupfs"] } ``` Any ideas what i could try? ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version ollama version is 0.5.7-0-ga420a45-dirty
GiteaMirror added the bug label 2026-05-09 17:46:22 -05:00
GiteaMirror commented 2026-05-09 17:46:24 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 29, 2025):

https://github.com/ollama/ollama/issues/8638#issuecomment-2619166616

<!-- gh-comment-id:2622966876 --> @rick-github commented on GitHub (Jan 29, 2025): https://github.com/ollama/ollama/issues/8638#issuecomment-2619166616
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#83305
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?