github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #14287] Ollama docker doesn't recognize GPU on new RTX Pro 6000 Blackwell GPU with Ubuntu 24.04 #9300

New Issue
Closed
opened 2026-04-12 22:10:01 -05:00 by GiteaMirror · 30 comments
GiteaMirror commented 2026-04-12 22:10:01 -05:00
Owner
Copy Link

Originally created by @akhilec on GitHub (Feb 16, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/14287

What is the issue?

details-Log.txt
I have 10 GPU devices which are NVIDIA RTX PRO 6000 Blackwell on ubunutu 24.04. GPUs are not getting discovered. I tried with all latest nvidia drivers 580., 590. but no luck. I have also tried with CUDA 13.1

 NVIDIA-SMI 570.211.01             Driver Version: 570.211.01     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:54:00.0 Off |                    0 |
| N/A   29C    P8             33W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:57:00.0 Off |                    0 |
| N/A   29C    P8             34W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:5A:00.0 Off |                    0 |
| N/A   30C    P8             36W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:5D:00.0 Off |                    0 |
| N/A   29C    P8             33W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:5E:00.0 Off |                    0 |
| N/A   30C    P8             36W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:D3:00.0 Off |                    0 |
| N/A   30C    P8             35W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:D6:00.0 Off |                    0 |
| N/A   29C    P8             34W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:D9:00.0 Off |                    0 |
| N/A   28C    P8             34W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   8  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:DC:00.0 Off |                    0 |
| N/A   29C    P8             36W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   9  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:DD:00.0 Off |                    0 |
| N/A   29C    P8             35W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Relevant log output

OS

No response

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @akhilec on GitHub (Feb 16, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/14287 ### What is the issue? [details-Log.txt](https://github.com/user-attachments/files/25351233/details-Log.txt) I have 10 GPU devices which are NVIDIA RTX PRO 6000 Blackwell on ubunutu 24.04. GPUs are not getting discovered. I tried with all latest nvidia drivers 580.*, 590.* but no luck. I have also tried with CUDA 13.1 ``` NVIDIA-SMI 570.211.01 Driver Version: 570.211.01 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX PRO 6000 Blac... Off | 00000000:54:00.0 Off | 0 | | N/A 29C P8 33W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX PRO 6000 Blac... Off | 00000000:57:00.0 Off | 0 | | N/A 29C P8 34W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA RTX PRO 6000 Blac... Off | 00000000:5A:00.0 Off | 0 | | N/A 30C P8 36W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA RTX PRO 6000 Blac... Off | 00000000:5D:00.0 Off | 0 | | N/A 29C P8 33W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 4 NVIDIA RTX PRO 6000 Blac... Off | 00000000:5E:00.0 Off | 0 | | N/A 30C P8 36W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 5 NVIDIA RTX PRO 6000 Blac... Off | 00000000:D3:00.0 Off | 0 | | N/A 30C P8 35W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 6 NVIDIA RTX PRO 6000 Blac... Off | 00000000:D6:00.0 Off | 0 | | N/A 29C P8 34W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 7 NVIDIA RTX PRO 6000 Blac... Off | 00000000:D9:00.0 Off | 0 | | N/A 28C P8 34W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 8 NVIDIA RTX PRO 6000 Blac... Off | 00000000:DC:00.0 Off | 0 | | N/A 29C P8 36W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 9 NVIDIA RTX PRO 6000 Blac... Off | 00000000:DD:00.0 Off | 0 | | N/A 29C P8 35W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ``` ### Relevant log output ```shell ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_
GiteaMirror added the bug label 2026-04-12 22:10:01 -05:00
GiteaMirror commented 2026-04-12 22:10:05 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 16, 2026):

Is that nvidia-smi output from inside or outside of the container?

<!-- gh-comment-id:3910805458 --> @rick-github commented on GitHub (Feb 16, 2026): Is that `nvidia-smi` output from inside or outside of the container?
GiteaMirror commented 2026-04-12 22:10:05 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 17, 2026):

Is that nvidia-smi output from inside or outside of the container?

It is from inside. I could see all devices from inside container

<!-- gh-comment-id:3914536713 --> @akhilec commented on GitHub (Feb 17, 2026): > Is that `nvidia-smi` output from inside or outside of the container? It is from inside. I could see all devices from inside container
GiteaMirror commented 2026-04-12 22:10:06 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 17, 2026):

Set OLLAMA_DEBUG=2 in the server environment and post the log from start to the line that says inference compute.

<!-- gh-comment-id:3914810718 --> @rick-github commented on GitHub (Feb 17, 2026): Set `OLLAMA_DEBUG=2` in the server environment and post the log from start to the line that says `inference compute`.
GiteaMirror commented 2026-04-12 22:10:06 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 17, 2026):

Set OLLAMA_DEBUG=2 in the server environment and post the log from start to the line that says inference compute.

Sure, attached is the detail log you asked for. Also below commands I used to pull docker image and run a model

root@hq-it-ai:~# sudo docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama -e OLLAMA_DEBUG=2 ollama/ollama
Unable to find image 'ollama/ollama:latest' locally
latest: Pulling from ollama/ollama
1c7d52019ebf: Pull complete 
086e43f94810: Pull complete 
26f128d62ae6: Pull complete 
b1d3ba5a8ea3: Download complete 
Digest: sha256:0764cf55b4a33bcecca10f718394d097ef7d464b75669a14f0cd4ac1a8b9a0c5
Status: Downloaded newer image for ollama/ollama:latest
2864df567639067f8c2cd1429ea959b0dfd683564fb50c21e95ad5adb0928703
root@hq-it-ai:~# sudo docker exec -it ollama ollama run gemma3:27B
>>> hi
Hi there! 👋 

How can I help you today? Just let me know what you're thinking, or if you just wanted to say hi, that's 
great too! 😊 

I can:

* **Answer questions:** About pretty much anything!
* **Generate creative content:** Like stories, poems, code, scripts, musical pieces, email, letters, 
etc.
* **Translate languages:** 
* **Summarize text:**
* **Help with brainstorming:**
* **Just chat!**





>>> /bye
root@hq-it-ai:~# sudo docker exec -it ollama ollama ps
NAME          ID              SIZE     PROCESSOR    CONTEXT    UNTIL              
gemma3:27b    a418f5838eaf    19 GB    100% CPU     4096       4 minutes from now    
root@hq-it-ai:~#

2864df567639067f8c2cd1429ea959b0dfd683564fb50c21e95ad5adb0928703-json.log

<!-- gh-comment-id:3915485465 --> @akhilec commented on GitHub (Feb 17, 2026): > Set `OLLAMA_DEBUG=2` in the server environment and post the log from start to the line that says `inference compute`. Sure, attached is the detail log you asked for. Also below commands I used to pull docker image and run a model ``` root@hq-it-ai:~# sudo docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama -e OLLAMA_DEBUG=2 ollama/ollama Unable to find image 'ollama/ollama:latest' locally latest: Pulling from ollama/ollama 1c7d52019ebf: Pull complete 086e43f94810: Pull complete 26f128d62ae6: Pull complete b1d3ba5a8ea3: Download complete Digest: sha256:0764cf55b4a33bcecca10f718394d097ef7d464b75669a14f0cd4ac1a8b9a0c5 Status: Downloaded newer image for ollama/ollama:latest 2864df567639067f8c2cd1429ea959b0dfd683564fb50c21e95ad5adb0928703 root@hq-it-ai:~# sudo docker exec -it ollama ollama run gemma3:27B >>> hi Hi there! 👋 How can I help you today? Just let me know what you're thinking, or if you just wanted to say hi, that's great too! 😊 I can: * **Answer questions:** About pretty much anything! * **Generate creative content:** Like stories, poems, code, scripts, musical pieces, email, letters, etc. * **Translate languages:** * **Summarize text:** * **Help with brainstorming:** * **Just chat!** >>> /bye root@hq-it-ai:~# sudo docker exec -it ollama ollama ps NAME ID SIZE PROCESSOR CONTEXT UNTIL gemma3:27b a418f5838eaf 19 GB 100% CPU 4096 4 minutes from now root@hq-it-ai:~# ``` [2864df567639067f8c2cd1429ea959b0dfd683564fb50c21e95ad5adb0928703-json.log](https://github.com/user-attachments/files/25368916/2864df567639067f8c2cd1429ea959b0dfd683564fb50c21e95ad5adb0928703-json.log)
GiteaMirror commented 2026-04-12 22:10:07 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 17, 2026):

Please post plain text logs.

ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version

What's the output of nvidia-smi outside of the container?

<!-- gh-comment-id:3915590358 --> @rick-github commented on GitHub (Feb 17, 2026): Please post plain text logs. ``` ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version ``` What's the output of `nvidia-smi` outside of the container?
GiteaMirror commented 2026-04-12 22:10:07 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 17, 2026):

Please post plain text logs.

ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version

What's the output of nvidia-smi outside of the container?

Same as it was shown from within container

| NVIDIA-SMI 570.211.01             Driver Version: 570.211.01     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:54:00.0 Off |                    0 |
| N/A   29C    P8             34W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:57:00.0 Off |                    0 |
| N/A   29C    P8             33W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:5A:00.0 Off |                    0 |
| N/A   30C    P8             35W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:5D:00.0 Off |                    0 |
| N/A   29C    P8             33W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:5E:00.0 Off |                    0 |
| N/A   29C    P8             36W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:D3:00.0 Off |                    0 |
| N/A   29C    P8             35W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:D6:00.0 Off |                    0 |
| N/A   29C    P8             34W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:D9:00.0 Off |                    0 |
| N/A   28C    P8             34W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   8  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:DC:00.0 Off |                    0 |
| N/A   29C    P8             36W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   9  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:DD:00.0 Off |                    0 |
| N/A   29C    P8             35W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found
<!-- gh-comment-id:3915656456 --> @akhilec commented on GitHub (Feb 17, 2026): > Please post plain text logs. > > ``` > ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version > ``` > > What's the output of `nvidia-smi` outside of the container? Same as it was shown from within container ``` | NVIDIA-SMI 570.211.01 Driver Version: 570.211.01 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX PRO 6000 Blac... Off | 00000000:54:00.0 Off | 0 | | N/A 29C P8 34W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX PRO 6000 Blac... Off | 00000000:57:00.0 Off | 0 | | N/A 29C P8 33W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA RTX PRO 6000 Blac... Off | 00000000:5A:00.0 Off | 0 | | N/A 30C P8 35W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA RTX PRO 6000 Blac... Off | 00000000:5D:00.0 Off | 0 | | N/A 29C P8 33W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 4 NVIDIA RTX PRO 6000 Blac... Off | 00000000:5E:00.0 Off | 0 | | N/A 29C P8 36W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 5 NVIDIA RTX PRO 6000 Blac... Off | 00000000:D3:00.0 Off | 0 | | N/A 29C P8 35W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 6 NVIDIA RTX PRO 6000 Blac... Off | 00000000:D6:00.0 Off | 0 | | N/A 29C P8 34W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 7 NVIDIA RTX PRO 6000 Blac... Off | 00000000:D9:00.0 Off | 0 | | N/A 28C P8 34W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 8 NVIDIA RTX PRO 6000 Blac... Off | 00000000:DC:00.0 Off | 0 | | N/A 29C P8 36W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 9 NVIDIA RTX PRO 6000 Blac... Off | 00000000:DD:00.0 Off | 0 | | N/A 29C P8 35W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found ```
GiteaMirror commented 2026-04-12 22:10:07 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 17, 2026):

Please post plain text logs.

ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version

What's the output of nvidia-smi outside of the container?

Same as it was shown from within container

See outside container result and from within container result

Image Image
<!-- gh-comment-id:3915710858 --> @akhilec commented on GitHub (Feb 17, 2026): > > Please post plain text logs. > > ``` > > ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version > > ``` > > > > > > > > > > > > > > > > > > > > > > > > What's the output of `nvidia-smi` outside of the container? > > Same as it was shown from within container See outside container result and from within container result <img width="654" height="977" alt="Image" src="https://github.com/user-attachments/assets/a442cf61-5067-42d4-990d-947dc9eba1af" /> <img width="886" height="981" alt="Image" src="https://github.com/user-attachments/assets/de439d90-f05e-49d3-8be0-6f53b33d063b" />
GiteaMirror commented 2026-04-12 22:10:07 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 17, 2026):

Are the GPUs discovered if you run ollama natively?

<!-- gh-comment-id:3915820012 --> @rick-github commented on GitHub (Feb 17, 2026): Are the GPUs discovered if you run ollama natively?
GiteaMirror commented 2026-04-12 22:10:08 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 17, 2026):

Are the GPUs discovered if you run ollama natively?

no same error. Attached detail logs when ollama is run locally

ollama_native-detail-logs.txt

<!-- gh-comment-id:3916466235 --> @akhilec commented on GitHub (Feb 17, 2026): > Are the GPUs discovered if you run ollama natively? no same error. Attached detail logs when ollama is run locally [ollama_native-detail-logs.txt](https://github.com/user-attachments/files/25372085/ollama_native-detail-logs.txt)
GiteaMirror commented 2026-04-12 22:10:08 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 18, 2026):

Both the v12 and v13 libraries get initialization errors but it's not clear why. What happens if you restrict the GPUs to one device by setting CUDA_VISIBLE_DEVICES=0? What's the output of nvidia-smi -q?

<!-- gh-comment-id:3918174066 --> @rick-github commented on GitHub (Feb 18, 2026): Both the v12 and v13 libraries get initialization errors but it's not clear why. What happens if you restrict the GPUs to one device by setting `CUDA_VISIBLE_DEVICES=0`? What's the output of `nvidia-smi -q`?
GiteaMirror commented 2026-04-12 22:10:09 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

CUDA_VISIBLE_DEVICES

I set CUDA_VISIBLE_DEVICES="0" as -e variable to my docker, but in nvidia-smi from within container, i see all 10 devices.

So for root@hq-it-ai:~# sudo docker exec -it ollama nvidia-smi -q, i got data for all 10 devices. but I copied data for one device and pasted below (see towards the end)

Detail log attached

root@hq-it-ai:~# sudo docker run -d --gpus=all -v ollama:/root/.ollama -p 11435:11434 --name ollama -e OLLAMA_DEBUG=2 -e CUDA_VISIBLE_DEVICES="0" ollama/ollama
Unable to find image 'ollama/ollama:latest' locally
latest: Pulling from ollama/ollama
086e43f94810: Pull complete 
a3629ac5b9f4: Pull complete 
26f128d62ae6: Pull complete 
1c7d52019ebf: Pull complete 
b1d3ba5a8ea3: Download complete 
Digest: sha256:0764cf55b4a33bcecca10f718394d097ef7d464b75669a14f0cd4ac1a8b9a0c5
Status: Downloaded newer image for ollama/ollama:latest
353e4b6a2b7660e29454306ee2c7d0d802783d56ab7cf15b6c7a6a1933c43c56
root@hq-it-ai:~# sudo docker exec -it ollama nvidia-smi
Wed Feb 18 05:25:52 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09             Driver Version: 580.126.09     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:54:00.0 Off |                    0 |
| N/A   29C    P8             34W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:57:00.0 Off |                    0 |
| N/A   29C    P8             33W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:5A:00.0 Off |                    0 |
| N/A   30C    P8             36W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:5D:00.0 Off |                    0 |
| N/A   29C    P8             33W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:5E:00.0 Off |                    0 |
| N/A   30C    P8             36W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:D3:00.0 Off |                    0 |
| N/A   30C    P8             36W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:D6:00.0 Off |                    0 |
| N/A   29C    P8             34W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:D9:00.0 Off |                    0 |
| N/A   28C    P8             34W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   8  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:DC:00.0 Off |                    0 |
| N/A   30C    P8             36W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   9  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:DD:00.0 Off |                    0 |
| N/A   29C    P8             35W /  600W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+










root@hq-it-ai:~# sudo docker exec -it ollama nvidia-smi -q

GPU 00000000:DD:00.0
    Product Name                                       : NVIDIA RTX PRO 6000 Blackwell Server Edition
    Product Brand                                      : NVIDIA
    Product Architecture                               : Blackwell
    Display Mode                                       : Requested functionality has been deprecated
    Display Attached                                   : Yes
    Display Active                                     : Disabled
    Persistence Mode                                   : Disabled
    Addressing Mode                                    : HMM
    MIG Mode
        Current                                        : Disabled
        Pending                                        : Disabled
    Accounting Mode                                    : Disabled
    Accounting Mode Buffer Size                        : 4000
    Driver Model
        Current                                        : N/A
        Pending                                        : N/A
    Serial Number                                      : 1794125013164
    GPU UUID                                           : GPU-5afaa004-f27b-fd5d-e4e4-a974bce8803c
    GPU PDI                                            : 0xe9f2e5343f319f33
    Minor Number                                       : 9
    VBIOS Version                                      : 98.02.67.00.0A
    MultiGPU Board                                     : No
    Board ID                                           : 0xdd00
    Board Part Number                                  : 900-2G153-0000-000
    GPU Part Number                                    : 2BB5-895-A1
    FRU Part Number                                    : N/A
    Platform Info
        Chassis Serial Number                          : 
        Slot Number                                    : 0
        Tray Index                                     : 0
        Host ID                                        : 1
        Peer Type                                      : Direct Connected
        Module Id                                      : 1
        GPU Fabric GUID                                : 0x0000000000000000
    Inforom Version
        Image Version                                  : G153.0210.00.02
        OEM Object                                     : 2.1
        ECC Object                                     : 7.16
        Power Management Object                        : N/A
    Inforom BBX Object Flush
        Latest Timestamp                               : N/A
        Latest Duration                                : N/A
    GPU Operation Mode
        Current                                        : N/A
        Pending                                        : N/A
    GPU C2C Mode                                       : Disabled
    GPU Virtualization Mode
        Virtualization Mode                            : None
        Host VGPU Mode                                 : N/A
        vGPU Heterogeneous Mode                        : N/A
    GPU Recovery Action                                : None
    GSP Firmware Version                               : 580.126.09
    IBMNPU
        Relaxed Ordering Mode                          : N/A
    PCI
        Bus                                            : 0xDD
        Device                                         : 0x00
        Domain                                         : 0x0000
        Base Classcode                                 : 0x3
        Sub Classcode                                  : 0x2
        Device Id                                      : 0x2BB510DE
        Bus Id                                         : 00000000:DD:00.0
        Sub System Id                                  : 0x204E10DE
        GPU Link Info
            PCIe Generation
                Max                                    : 5
                Current                                : 1
                Device Current                         : 1
                Device Max                             : 5
                Host Max                               : 5
            Link Width
                Max                                    : 16x
                Current                                : 16x
        Bridge Chip
            Type                                       : N/A
            Firmware                                   : N/A
        Replays Since Reset                            : 0
        Replay Number Rollovers                        : 0
        Tx Throughput                                  : 498 KB/s
        Rx Throughput                                  : 506 KB/s
        Atomic Caps Outbound                           : N/A
        Atomic Caps Inbound                            : FETCHADD_32 FETCHADD_64 SWAP_32 SWAP_64 CAS_32 CAS_64 
    Fan Speed                                          : N/A
    Performance State                                  : P8
    Clocks Event Reasons
        Idle                                           : Not Active
        Applications Clocks Setting                    : Not Active
        SW Power Cap                                   : Not Active
        HW Slowdown                                    : Not Active
            HW Thermal Slowdown                        : Not Active
            HW Power Brake Slowdown                    : Not Active
        Sync Boost                                     : Not Active
        SW Thermal Slowdown                            : Not Active
        Display Clock Setting                          : Not Active
    Clocks Event Reasons Counters
        SW Power Capping                               : 1218674 us
        Sync Boost                                     : 0 us
        SW Thermal Slowdown                            : 0 us
        HW Thermal Slowdown                            : 0 us
        HW Power Braking                               : 0 us
    Sparse Operation Mode                              : N/A
    FB Memory Usage
        Total                                          : 97887 MiB
        Reserved                                       : 638 MiB
        Used                                           : 0 MiB
        Free                                           : 97250 MiB
    BAR1 Memory Usage
        Total                                          : 131072 MiB
        Used                                           : 1 MiB
        Free                                           : 131071 MiB
    Conf Compute Protected Memory Usage
        Total                                          : 0 MiB
        Used                                           : 0 MiB
        Free                                           : 0 MiB
    Compute Mode                                       : Default
    Utilization
        GPU                                            : 0 %
        Memory                                         : 0 %
        Encoder                                        : 0 %
        Decoder                                        : 0 %
        JPEG                                           : 0 %
        OFA                                            : 0 %
    Encoder Stats
        Active Sessions                                : 0
        Average FPS                                    : 0
        Average Latency                                : 0
    FBC Stats
        Active Sessions                                : 0
        Average FPS                                    : 0
        Average Latency                                : 0
    DRAM Encryption Mode
        Current                                        : Disabled
        Pending                                        : Disabled
    ECC Mode
        Current                                        : Enabled
        Pending                                        : Enabled
    ECC Errors
        Volatile
            SRAM Correctable                           : 0
            SRAM Uncorrectable Parity                  : 0
            SRAM Uncorrectable SEC-DED                 : 0
            DRAM Correctable                           : 0
            DRAM Uncorrectable                         : 0
        Aggregate
            SRAM Correctable                           : 0
            SRAM Uncorrectable Parity                  : 0
            SRAM Uncorrectable SEC-DED                 : 0
            DRAM Correctable                           : 0
            DRAM Uncorrectable                         : 0
            SRAM Threshold Exceeded                    : No
        Aggregate Uncorrectable SRAM Sources
            SRAM L2                                    : 0
            SRAM SM                                    : 0
            SRAM Microcontroller                       : 0
            SRAM PCIE                                  : 0
            SRAM Other                                 : 0
        Channel Repair Pending                         : No
        TPC Repair Pending                             : No
    Retired Pages
        Single Bit ECC                                 : N/A
        Double Bit ECC                                 : N/A
        Pending Page Blacklist                         : N/A
    Remapped Rows
        Correctable Error                              : 0
        Uncorrectable Error                            : 0
        Pending                                        : No
        Remapping Failure Occurred                     : No
        Bank Remap Availability Histogram
            Max                                        : 512 bank(s)
            High                                       : 0 bank(s)
            Partial                                    : 0 bank(s)
            Low                                        : 0 bank(s)
            None                                       : 0 bank(s)
    Temperature
        GPU Current Temp                               : 29 C
        GPU T.Limit Temp                               : 56 C
        GPU Shutdown T.Limit Temp                      : -5 C
        GPU Slowdown T.Limit Temp                      : -2 C
        GPU Max Operating T.Limit Temp                 : 0 C
        GPU Target Temperature                         : N/A
        Memory Current Temp                            : N/A
        Memory Max Operating T.Limit Temp              : N/A
    GPU Power Readings
        Average Power Draw                             : 35.15 W
        Instantaneous Power Draw                       : 35.15 W
        Current Power Limit                            : 600.00 W
        Requested Power Limit                          : 600.00 W
        Default Power Limit                            : 600.00 W
        Min Power Limit                                : 300.00 W
        Max Power Limit                                : 600.00 W
    GPU Memory Power Readings 
        Average Power Draw                             : N/A
        Instantaneous Power Draw                       : N/A
    Module Power Readings
        Average Power Draw                             : N/A
        Instantaneous Power Draw                       : N/A
        Current Power Limit                            : N/A
        Requested Power Limit                          : N/A
        Default Power Limit                            : N/A
        Min Power Limit                                : N/A
        Max Power Limit                                : N/A
    Power Smoothing                                    : N/A
    Workload Power Profiles
        Requested Profiles                             : N/A
        Enforced Profiles                              : N/A
    Clocks
        Graphics                                       : 180 MHz
        SM                                             : 180 MHz
        Memory                                         : 405 MHz
        Video                                          : 600 MHz
    Applications Clocks
        Graphics                                       : 2430 MHz
        Memory                                         : 12481 MHz
    Default Applications Clocks
        Graphics                                       : 2430 MHz
        Memory                                         : 12481 MHz
    Deferred Clocks
        Memory                                         : N/A
    Max Clocks
        Graphics                                       : 2430 MHz
        SM                                             : 2430 MHz
        Memory                                         : 12481 MHz
        Video                                          : 2107 MHz
    Max Customer Boost Clocks
        Graphics                                       : 2430 MHz
    Clock Policy
        Auto Boost                                     : N/A
        Auto Boost Default                             : N/A
    Fabric
        State                                          : N/A
        Status                                         : N/A
        CliqueId                                       : N/A
        ClusterUUID                                    : N/A
        Health
            Summary                                    : N/A
            Bandwidth                                  : N/A
            Route Recovery in progress                 : N/A
            Route Unhealthy                            : N/A
            Access Timeout Recovery                    : N/A
            Incorrect Configuration                    : N/A
            Partition Assigned                         : N/A
    Processes                                          : None
    Capabilities
        EGM                                            : disabled

detail_log_with CUDA_VISIBLE_.txt

<!-- gh-comment-id:3918824674 --> @akhilec commented on GitHub (Feb 18, 2026): > CUDA_VISIBLE_DEVICES I set CUDA_VISIBLE_DEVICES="0" as -e variable to my docker, but in nvidia-smi from within container, i see all 10 devices. So for root@hq-it-ai:~# sudo docker exec -it ollama nvidia-smi -q, i got data for all 10 devices. but I copied data for one device and pasted below (see towards the end) Detail log attached ``` root@hq-it-ai:~# sudo docker run -d --gpus=all -v ollama:/root/.ollama -p 11435:11434 --name ollama -e OLLAMA_DEBUG=2 -e CUDA_VISIBLE_DEVICES="0" ollama/ollama Unable to find image 'ollama/ollama:latest' locally latest: Pulling from ollama/ollama 086e43f94810: Pull complete a3629ac5b9f4: Pull complete 26f128d62ae6: Pull complete 1c7d52019ebf: Pull complete b1d3ba5a8ea3: Download complete Digest: sha256:0764cf55b4a33bcecca10f718394d097ef7d464b75669a14f0cd4ac1a8b9a0c5 Status: Downloaded newer image for ollama/ollama:latest 353e4b6a2b7660e29454306ee2c7d0d802783d56ab7cf15b6c7a6a1933c43c56 root@hq-it-ai:~# sudo docker exec -it ollama nvidia-smi Wed Feb 18 05:25:52 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.126.09 Driver Version: 580.126.09 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX PRO 6000 Blac... Off | 00000000:54:00.0 Off | 0 | | N/A 29C P8 34W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX PRO 6000 Blac... Off | 00000000:57:00.0 Off | 0 | | N/A 29C P8 33W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA RTX PRO 6000 Blac... Off | 00000000:5A:00.0 Off | 0 | | N/A 30C P8 36W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA RTX PRO 6000 Blac... Off | 00000000:5D:00.0 Off | 0 | | N/A 29C P8 33W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 4 NVIDIA RTX PRO 6000 Blac... Off | 00000000:5E:00.0 Off | 0 | | N/A 30C P8 36W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 5 NVIDIA RTX PRO 6000 Blac... Off | 00000000:D3:00.0 Off | 0 | | N/A 30C P8 36W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 6 NVIDIA RTX PRO 6000 Blac... Off | 00000000:D6:00.0 Off | 0 | | N/A 29C P8 34W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 7 NVIDIA RTX PRO 6000 Blac... Off | 00000000:D9:00.0 Off | 0 | | N/A 28C P8 34W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 8 NVIDIA RTX PRO 6000 Blac... Off | 00000000:DC:00.0 Off | 0 | | N/A 30C P8 36W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 9 NVIDIA RTX PRO 6000 Blac... Off | 00000000:DD:00.0 Off | 0 | | N/A 29C P8 35W / 600W | 0MiB / 97887MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ root@hq-it-ai:~# sudo docker exec -it ollama nvidia-smi -q GPU 00000000:DD:00.0 Product Name : NVIDIA RTX PRO 6000 Blackwell Server Edition Product Brand : NVIDIA Product Architecture : Blackwell Display Mode : Requested functionality has been deprecated Display Attached : Yes Display Active : Disabled Persistence Mode : Disabled Addressing Mode : HMM MIG Mode Current : Disabled Pending : Disabled Accounting Mode : Disabled Accounting Mode Buffer Size : 4000 Driver Model Current : N/A Pending : N/A Serial Number : 1794125013164 GPU UUID : GPU-5afaa004-f27b-fd5d-e4e4-a974bce8803c GPU PDI : 0xe9f2e5343f319f33 Minor Number : 9 VBIOS Version : 98.02.67.00.0A MultiGPU Board : No Board ID : 0xdd00 Board Part Number : 900-2G153-0000-000 GPU Part Number : 2BB5-895-A1 FRU Part Number : N/A Platform Info Chassis Serial Number : Slot Number : 0 Tray Index : 0 Host ID : 1 Peer Type : Direct Connected Module Id : 1 GPU Fabric GUID : 0x0000000000000000 Inforom Version Image Version : G153.0210.00.02 OEM Object : 2.1 ECC Object : 7.16 Power Management Object : N/A Inforom BBX Object Flush Latest Timestamp : N/A Latest Duration : N/A GPU Operation Mode Current : N/A Pending : N/A GPU C2C Mode : Disabled GPU Virtualization Mode Virtualization Mode : None Host VGPU Mode : N/A vGPU Heterogeneous Mode : N/A GPU Recovery Action : None GSP Firmware Version : 580.126.09 IBMNPU Relaxed Ordering Mode : N/A PCI Bus : 0xDD Device : 0x00 Domain : 0x0000 Base Classcode : 0x3 Sub Classcode : 0x2 Device Id : 0x2BB510DE Bus Id : 00000000:DD:00.0 Sub System Id : 0x204E10DE GPU Link Info PCIe Generation Max : 5 Current : 1 Device Current : 1 Device Max : 5 Host Max : 5 Link Width Max : 16x Current : 16x Bridge Chip Type : N/A Firmware : N/A Replays Since Reset : 0 Replay Number Rollovers : 0 Tx Throughput : 498 KB/s Rx Throughput : 506 KB/s Atomic Caps Outbound : N/A Atomic Caps Inbound : FETCHADD_32 FETCHADD_64 SWAP_32 SWAP_64 CAS_32 CAS_64 Fan Speed : N/A Performance State : P8 Clocks Event Reasons Idle : Not Active Applications Clocks Setting : Not Active SW Power Cap : Not Active HW Slowdown : Not Active HW Thermal Slowdown : Not Active HW Power Brake Slowdown : Not Active Sync Boost : Not Active SW Thermal Slowdown : Not Active Display Clock Setting : Not Active Clocks Event Reasons Counters SW Power Capping : 1218674 us Sync Boost : 0 us SW Thermal Slowdown : 0 us HW Thermal Slowdown : 0 us HW Power Braking : 0 us Sparse Operation Mode : N/A FB Memory Usage Total : 97887 MiB Reserved : 638 MiB Used : 0 MiB Free : 97250 MiB BAR1 Memory Usage Total : 131072 MiB Used : 1 MiB Free : 131071 MiB Conf Compute Protected Memory Usage Total : 0 MiB Used : 0 MiB Free : 0 MiB Compute Mode : Default Utilization GPU : 0 % Memory : 0 % Encoder : 0 % Decoder : 0 % JPEG : 0 % OFA : 0 % Encoder Stats Active Sessions : 0 Average FPS : 0 Average Latency : 0 FBC Stats Active Sessions : 0 Average FPS : 0 Average Latency : 0 DRAM Encryption Mode Current : Disabled Pending : Disabled ECC Mode Current : Enabled Pending : Enabled ECC Errors Volatile SRAM Correctable : 0 SRAM Uncorrectable Parity : 0 SRAM Uncorrectable SEC-DED : 0 DRAM Correctable : 0 DRAM Uncorrectable : 0 Aggregate SRAM Correctable : 0 SRAM Uncorrectable Parity : 0 SRAM Uncorrectable SEC-DED : 0 DRAM Correctable : 0 DRAM Uncorrectable : 0 SRAM Threshold Exceeded : No Aggregate Uncorrectable SRAM Sources SRAM L2 : 0 SRAM SM : 0 SRAM Microcontroller : 0 SRAM PCIE : 0 SRAM Other : 0 Channel Repair Pending : No TPC Repair Pending : No Retired Pages Single Bit ECC : N/A Double Bit ECC : N/A Pending Page Blacklist : N/A Remapped Rows Correctable Error : 0 Uncorrectable Error : 0 Pending : No Remapping Failure Occurred : No Bank Remap Availability Histogram Max : 512 bank(s) High : 0 bank(s) Partial : 0 bank(s) Low : 0 bank(s) None : 0 bank(s) Temperature GPU Current Temp : 29 C GPU T.Limit Temp : 56 C GPU Shutdown T.Limit Temp : -5 C GPU Slowdown T.Limit Temp : -2 C GPU Max Operating T.Limit Temp : 0 C GPU Target Temperature : N/A Memory Current Temp : N/A Memory Max Operating T.Limit Temp : N/A GPU Power Readings Average Power Draw : 35.15 W Instantaneous Power Draw : 35.15 W Current Power Limit : 600.00 W Requested Power Limit : 600.00 W Default Power Limit : 600.00 W Min Power Limit : 300.00 W Max Power Limit : 600.00 W GPU Memory Power Readings Average Power Draw : N/A Instantaneous Power Draw : N/A Module Power Readings Average Power Draw : N/A Instantaneous Power Draw : N/A Current Power Limit : N/A Requested Power Limit : N/A Default Power Limit : N/A Min Power Limit : N/A Max Power Limit : N/A Power Smoothing : N/A Workload Power Profiles Requested Profiles : N/A Enforced Profiles : N/A Clocks Graphics : 180 MHz SM : 180 MHz Memory : 405 MHz Video : 600 MHz Applications Clocks Graphics : 2430 MHz Memory : 12481 MHz Default Applications Clocks Graphics : 2430 MHz Memory : 12481 MHz Deferred Clocks Memory : N/A Max Clocks Graphics : 2430 MHz SM : 2430 MHz Memory : 12481 MHz Video : 2107 MHz Max Customer Boost Clocks Graphics : 2430 MHz Clock Policy Auto Boost : N/A Auto Boost Default : N/A Fabric State : N/A Status : N/A CliqueId : N/A ClusterUUID : N/A Health Summary : N/A Bandwidth : N/A Route Recovery in progress : N/A Route Unhealthy : N/A Access Timeout Recovery : N/A Incorrect Configuration : N/A Partition Assigned : N/A Processes : None Capabilities EGM : disabled ``` [detail_log_with CUDA_VISIBLE_.txt](https://github.com/user-attachments/files/25382163/detail_log_with.CUDA_VISIBLE_.txt)
GiteaMirror commented 2026-04-12 22:10:09 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 18, 2026):

Try this:

docker run  --gpus=all -e OLLAMA_DEBUG=2 ollama/ollama:0.12.3

It uses an older version that has extra debugging during device discovery. Run it, wait for the "inference compute" line, then ^C and post the output.

<!-- gh-comment-id:3920207381 --> @rick-github commented on GitHub (Feb 18, 2026): Try this: ``` docker run --gpus=all -e OLLAMA_DEBUG=2 ollama/ollama:0.12.3 ``` It uses an older version that has extra debugging during device discovery. Run it, wait for the "inference compute" line, then ^C and post the output.
GiteaMirror commented 2026-04-12 22:10:09 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

Try this:

docker run  --gpus=all -e OLLAMA_DEBUG=2 ollama/ollama:0.12.3

It uses an older version that has extra debugging during device discovery. Run it, wait for the "inference compute" line, then ^C and post the output.

root@hq-it-ai:~# docker run --gpus=all -e OLLAMA_DEBUG=2 ollama/ollama:0.12.3
Unable to find image 'ollama/ollama:0.12.3' locally
0.12.3: Pulling from ollama/ollama
36591e7dd4a3: Pull complete
804f1b698a9f: Pull complete
66ef1ccd9b48: Pull complete
953cdd413371: Pull complete
b4f95af85236: Download complete
Digest: sha256:c622a7adec67cf5bd7fe1802b7e26aa583a955a54e91d132889301f50c3e0bd0
Status: Downloaded newer image for ollama/ollama:0.12.3
Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.
Your new public key is:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBXRDpA/veXS8w4StHVGy9q8QSo0zfsZxr8yV5FSuO1x

time=2026-02-18T15:23:39.656Z level=INFO source=routes.go:1475 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG-4 OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2026-02-18T15:23:39.656Z level=INFO source=images.go:518 msg="total blobs: 0"
time=2026-02-18T15:23:39.656Z level=INFO source=images.go:525 msg="total unused blobs removed: 0"
time=2026-02-18T15:23:39.657Z level=INFO source=routes.go:1528 msg="Listening on [::]:11434 (version 0.12.3)"
time=2026-02-18T15:23:39.657Z level=DEBUG source=sched.go:121 msg="starting llm scheduler"
time=2026-02-18T15:23:39.657Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2026-02-18T15:23:39.673Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2026-02-18T15:23:39.673Z level=DEBUG source=gpu.go:520 msg="Searching for GPU library" name=libcuda.so
time=2026-02-18T15:23:39.673Z level=DEBUG source=gpu.go:544 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets//lib/libcuda.so /usr/lib/-linux-gnu/nvidia/current/libcuda.so /usr/lib/-linux-gnu/libcuda.so /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers//libcuda.so /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2026-02-18T15:23:39.675Z level=DEBUG source=gpu.go:577 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09]
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09
dlsym: cuInit - 0x7624fa527850
dlsym: cuDriverGetVersion - 0x7624fa527910
dlsym: cuDeviceGetCount - 0x7624fa527a90
dlsym: cuDeviceGet - 0x7624fa5279d0
dlsym: cuDeviceGetAttribute - 0x7624fa527f10
dlsym: cuDeviceGetUuid - 0x7624fa57eb10
dlsym: cuDeviceGetName - 0x7624fa527b50
dlsym: cuCtxCreate_v3 - 0x7624fa57c2b0
dlsym: cuMemGetInfo_v2 - 0x7624fa52b780
dlsym: cuCtxDestroy - 0x7624fa57e1b0
calling cuInit
cuInit err: 3
time=2026-02-18T15:23:39.875Z level=INFO source=gpu.go:631 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09: cuda driver library init failure: 3"
time=2026-02-18T15:23:39.875Z level=DEBUG source=gpu.go:520 msg="Searching for GPU library" name=libcudart.so*
time=2026-02-18T15:23:39.875Z level=DEBUG source=gpu.go:544 msg="gpu library search" globs="[/usr/lib/ollama/libcudart.so* /usr/local/nvidia/lib/libcudart.so* /usr/local/nvidia/lib64/libcudart.so* /usr/lib/ollama/cuda_v*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers//libcudart.so /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2026-02-18T15:23:39.876Z level=DEBUG source=gpu.go:577 msg="discovered GPU libraries" paths="[/usr/lib/ollama/cuda_v12/libcudart.so.12.8.90 /usr/lib/ollama/cuda_v13/libcudart.so.13.0.88]"
cudaSetDevice err: 3
time=2026-02-18T15:23:40.051Z level=DEBUG source=gpu.go:593 msg="Unable to load cudart library /usr/lib/ollama/cuda_v12/libcudart.so.12.8.90: cudart init failure: 3"
cudaSetDevice err: 3
time=2026-02-18T15:23:40.224Z level=DEBUG source=gpu.go:593 msg="Unable to load cudart library /usr/lib/ollama/cuda_v13/libcudart.so.13.0.88: cudart init failure: 3"
time=2026-02-18T15:23:40.224Z level=DEBUG source=amd_linux.go:423 msg="amdgpu driver not detected /sys/module/amdgpu"
time=2026-02-18T15:23:40.224Z level=INFO source=gpu.go:396 msg="no compatible GPUs were discovered"
time=2026-02-18T15:23:40.224Z level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="2267.2 GiB" available="2237.1 GiB"

<!-- gh-comment-id:3921452194 --> @akhilec commented on GitHub (Feb 18, 2026): > Try this: > > ``` > docker run --gpus=all -e OLLAMA_DEBUG=2 ollama/ollama:0.12.3 > ``` > > It uses an older version that has extra debugging during device discovery. Run it, wait for the "inference compute" line, then ^C and post the output. root@hq-it-ai:~# docker run --gpus=all -e OLLAMA_DEBUG=2 ollama/ollama:0.12.3 Unable to find image 'ollama/ollama:0.12.3' locally 0.12.3: Pulling from ollama/ollama 36591e7dd4a3: Pull complete 804f1b698a9f: Pull complete 66ef1ccd9b48: Pull complete 953cdd413371: Pull complete b4f95af85236: Download complete Digest: sha256:c622a7adec67cf5bd7fe1802b7e26aa583a955a54e91d132889301f50c3e0bd0 Status: Downloaded newer image for ollama/ollama:0.12.3 Couldn't find '/root/.ollama/id_ed25519'. Generating new private key. Your new public key is: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBXRDpA/veXS8w4StHVGy9q8QSo0zfsZxr8yV5FSuO1x time=2026-02-18T15:23:39.656Z level=INFO source=routes.go:1475 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG-4 OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2026-02-18T15:23:39.656Z level=INFO source=images.go:518 msg="total blobs: 0" time=2026-02-18T15:23:39.656Z level=INFO source=images.go:525 msg="total unused blobs removed: 0" time=2026-02-18T15:23:39.657Z level=INFO source=routes.go:1528 msg="Listening on [::]:11434 (version 0.12.3)" time=2026-02-18T15:23:39.657Z level=DEBUG source=sched.go:121 msg="starting llm scheduler" time=2026-02-18T15:23:39.657Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2026-02-18T15:23:39.673Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2026-02-18T15:23:39.673Z level=DEBUG source=gpu.go:520 msg="Searching for GPU library" name=libcuda.so* time=2026-02-18T15:23:39.673Z level=DEBUG source=gpu.go:544 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2026-02-18T15:23:39.675Z level=DEBUG source=gpu.go:577 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09] initializing /usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09 dlsym: cuInit - 0x7624fa527850 dlsym: cuDriverGetVersion - 0x7624fa527910 dlsym: cuDeviceGetCount - 0x7624fa527a90 dlsym: cuDeviceGet - 0x7624fa5279d0 dlsym: cuDeviceGetAttribute - 0x7624fa527f10 dlsym: cuDeviceGetUuid - 0x7624fa57eb10 dlsym: cuDeviceGetName - 0x7624fa527b50 dlsym: cuCtxCreate_v3 - 0x7624fa57c2b0 dlsym: cuMemGetInfo_v2 - 0x7624fa52b780 dlsym: cuCtxDestroy - 0x7624fa57e1b0 calling cuInit cuInit err: 3 time=2026-02-18T15:23:39.875Z level=INFO source=gpu.go:631 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09: cuda driver library init failure: 3" time=2026-02-18T15:23:39.875Z level=DEBUG source=gpu.go:520 msg="Searching for GPU library" name=libcudart.so* time=2026-02-18T15:23:39.875Z level=DEBUG source=gpu.go:544 msg="gpu library search" globs="[/usr/lib/ollama/libcudart.so* /usr/local/nvidia/lib/libcudart.so* /usr/local/nvidia/lib64/libcudart.so* /usr/lib/ollama/cuda_v*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]" time=2026-02-18T15:23:39.876Z level=DEBUG source=gpu.go:577 msg="discovered GPU libraries" paths="[/usr/lib/ollama/cuda_v12/libcudart.so.12.8.90 /usr/lib/ollama/cuda_v13/libcudart.so.13.0.88]" cudaSetDevice err: 3 time=2026-02-18T15:23:40.051Z level=DEBUG source=gpu.go:593 msg="Unable to load cudart library /usr/lib/ollama/cuda_v12/libcudart.so.12.8.90: cudart init failure: 3" cudaSetDevice err: 3 time=2026-02-18T15:23:40.224Z level=DEBUG source=gpu.go:593 msg="Unable to load cudart library /usr/lib/ollama/cuda_v13/libcudart.so.13.0.88: cudart init failure: 3" time=2026-02-18T15:23:40.224Z level=DEBUG source=amd_linux.go:423 msg="amdgpu driver not detected /sys/module/amdgpu" time=2026-02-18T15:23:40.224Z level=INFO source=gpu.go:396 msg="no compatible GPUs were discovered" time=2026-02-18T15:23:40.224Z level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="2267.2 GiB" available="2237.1 GiB"
GiteaMirror commented 2026-04-12 22:10:09 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 18, 2026):

cuInit err: 3

From the CUDA Toolkit documentation:
cudaErrorInitializationError = 3

  • The API call failed because the CUDA driver and runtime could not be initialized.

Which really doesn't shed a lot more light on the situation. What's the output of

grep -i nvidia /var/log/dmesg
lsmod | grep nv
ls -l /dev/nvidia*
<!-- gh-comment-id:3921521738 --> @rick-github commented on GitHub (Feb 18, 2026): ``` cuInit err: 3 ``` From the [CUDA Toolkit](https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES) documentation: [cudaErrorInitializationError](https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1gg3f51e3575c2178246db0a94a430e0038e942e4cbbd2bef6e92e293253f055613:~:text=cudaErrorInitializationError) = 3 - The API call failed because the CUDA driver and runtime could not be initialized. Which really doesn't shed a lot more light on the situation. What's the output of ``` grep -i nvidia /var/log/dmesg lsmod | grep nv ls -l /dev/nvidia* ```
GiteaMirror commented 2026-04-12 22:10:10 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

grep -i nvidia /var/log/dmesg

root@hq-it-ai:/usr/lib# grep -i nvidia /var/log/dmesg
[ 24.873726] kernel: nvidia: loading out-of-tree module taints kernel.
[ 24.873734] kernel: nvidia: module verification failed: signature and/or required key missing - tainting kernel
[ 24.946863] kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 510
[ 24.985371] kernel: nvidia 0000:54:00.0: enabling device (0000 -> 0002)
[ 25.022394] kernel: nvidia 0000:57:00.0: enabling device (0000 -> 0002)
[ 25.063685] kernel: nvidia 0000:5a:00.0: enabling device (0000 -> 0002)
[ 25.130840] kernel: nvidia 0000:5d:00.0: enabling device (0000 -> 0002)
[ 25.154457] kernel: nvidia 0000:5e:00.0: enabling device (0000 -> 0002)
[ 25.225437] kernel: nvidia 0000:d3:00.0: enabling device (0000 -> 0002)
[ 25.262423] kernel: nvidia 0000:d6:00.0: enabling device (0000 -> 0002)
[ 25.297391] kernel: nvidia 0000:d9:00.0: enabling device (0000 -> 0002)
[ 25.332523] kernel: nvidia 0000:dc:00.0: enabling device (0000 -> 0002)
[ 25.355507] kernel: nvidia 0000:dd:00.0: enabling device (0000 -> 0002)
[ 25.366279] kernel: NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 580.126.09 Release Build (dvs-builder@U22-I3-AM02-24-3) Wed Jan 7 22:51:36 UTC 2026
[ 25.383176] kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 580.126.09 Release Build (dvs-builder@U22-I3-AM02-24-3) Wed Jan 7 22:33:56 UTC 2026
[ 25.387260] kernel: [drm] [nvidia-drm] [GPU ID 0x00005400] Loading driver
[ 26.974766] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:54:00.0 on minor 1
[ 26.974797] kernel: nvidia 0000:54:00.0: [drm] No compatible format found
[ 26.974800] kernel: nvidia 0000:54:00.0: [drm] Cannot find any crtc or sizes
[ 26.974835] kernel: [drm] [nvidia-drm] [GPU ID 0x00005700] Loading driver
[ 31.684692] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:57:00.0 on minor 2
[ 31.684736] kernel: nvidia 0000:57:00.0: [drm] No compatible format found
[ 31.684739] kernel: nvidia 0000:57:00.0: [drm] Cannot find any crtc or sizes
[ 31.684779] kernel: [drm] [nvidia-drm] [GPU ID 0x00005a00] Loading driver
[ 31.709490] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:5a:00.0 on minor 3
[ 31.709515] kernel: nvidia 0000:5a:00.0: [drm] No compatible format found
[ 31.709517] kernel: nvidia 0000:5a:00.0: [drm] Cannot find any crtc or sizes
[ 31.709554] kernel: [drm] [nvidia-drm] [GPU ID 0x00005d00] Loading driver
[ 31.714236] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:5d:00.0 on minor 4
[ 31.714261] kernel: nvidia 0000:5d:00.0: [drm] No compatible format found
[ 31.714263] kernel: nvidia 0000:5d:00.0: [drm] Cannot find any crtc or sizes
[ 31.714290] kernel: [drm] [nvidia-drm] [GPU ID 0x00005e00] Loading driver
[ 31.718846] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:5e:00.0 on minor 5
[ 31.718867] kernel: nvidia 0000:5e:00.0: [drm] No compatible format found
[ 31.718868] kernel: nvidia 0000:5e:00.0: [drm] Cannot find any crtc or sizes
[ 31.718921] kernel: [drm] [nvidia-drm] [GPU ID 0x0000d300] Loading driver
[ 31.723516] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:d3:00.0 on minor 6
[ 31.723529] kernel: nvidia 0000:d3:00.0: [drm] No compatible format found
[ 31.723531] kernel: nvidia 0000:d3:00.0: [drm] Cannot find any crtc or sizes
[ 31.723554] kernel: [drm] [nvidia-drm] [GPU ID 0x0000d600] Loading driver
[ 31.728083] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:d6:00.0 on minor 7
[ 31.728093] kernel: nvidia 0000:d6:00.0: [drm] No compatible format found
[ 31.728095] kernel: nvidia 0000:d6:00.0: [drm] Cannot find any crtc or sizes
[ 31.728116] kernel: [drm] [nvidia-drm] [GPU ID 0x0000d900] Loading driver
[ 31.732799] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:d9:00.0 on minor 8
[ 31.732809] kernel: nvidia 0000:d9:00.0: [drm] No compatible format found
[ 31.732811] kernel: nvidia 0000:d9:00.0: [drm] Cannot find any crtc or sizes
[ 31.732835] kernel: [drm] [nvidia-drm] [GPU ID 0x0000dc00] Loading driver
[ 31.737556] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:dc:00.0 on minor 9
[ 31.737568] kernel: nvidia 0000:dc:00.0: [drm] No compatible format found
[ 31.737570] kernel: nvidia 0000:dc:00.0: [drm] Cannot find any crtc or sizes
[ 31.737593] kernel: [drm] [nvidia-drm] [GPU ID 0x0000dd00] Loading driver
[ 31.742228] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:dd:00.0 on minor 10
[ 31.742238] kernel: nvidia 0000:dd:00.0: [drm] No compatible format found
[ 31.742240] kernel: nvidia 0000:dd:00.0: [drm] Cannot find any crtc or sizes

<!-- gh-comment-id:3921588493 --> @akhilec commented on GitHub (Feb 18, 2026): > grep -i nvidia /var/log/dmesg root@hq-it-ai:/usr/lib# grep -i nvidia /var/log/dmesg [ 24.873726] kernel: nvidia: loading out-of-tree module taints kernel. [ 24.873734] kernel: nvidia: module verification failed: signature and/or required key missing - tainting kernel [ 24.946863] kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 510 [ 24.985371] kernel: nvidia 0000:54:00.0: enabling device (0000 -> 0002) [ 25.022394] kernel: nvidia 0000:57:00.0: enabling device (0000 -> 0002) [ 25.063685] kernel: nvidia 0000:5a:00.0: enabling device (0000 -> 0002) [ 25.130840] kernel: nvidia 0000:5d:00.0: enabling device (0000 -> 0002) [ 25.154457] kernel: nvidia 0000:5e:00.0: enabling device (0000 -> 0002) [ 25.225437] kernel: nvidia 0000:d3:00.0: enabling device (0000 -> 0002) [ 25.262423] kernel: nvidia 0000:d6:00.0: enabling device (0000 -> 0002) [ 25.297391] kernel: nvidia 0000:d9:00.0: enabling device (0000 -> 0002) [ 25.332523] kernel: nvidia 0000:dc:00.0: enabling device (0000 -> 0002) [ 25.355507] kernel: nvidia 0000:dd:00.0: enabling device (0000 -> 0002) [ 25.366279] kernel: NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 580.126.09 Release Build (dvs-builder@U22-I3-AM02-24-3) Wed Jan 7 22:51:36 UTC 2026 [ 25.383176] kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 580.126.09 Release Build (dvs-builder@U22-I3-AM02-24-3) Wed Jan 7 22:33:56 UTC 2026 [ 25.387260] kernel: [drm] [nvidia-drm] [GPU ID 0x00005400] Loading driver [ 26.974766] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:54:00.0 on minor 1 [ 26.974797] kernel: nvidia 0000:54:00.0: [drm] No compatible format found [ 26.974800] kernel: nvidia 0000:54:00.0: [drm] Cannot find any crtc or sizes [ 26.974835] kernel: [drm] [nvidia-drm] [GPU ID 0x00005700] Loading driver [ 31.684692] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:57:00.0 on minor 2 [ 31.684736] kernel: nvidia 0000:57:00.0: [drm] No compatible format found [ 31.684739] kernel: nvidia 0000:57:00.0: [drm] Cannot find any crtc or sizes [ 31.684779] kernel: [drm] [nvidia-drm] [GPU ID 0x00005a00] Loading driver [ 31.709490] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:5a:00.0 on minor 3 [ 31.709515] kernel: nvidia 0000:5a:00.0: [drm] No compatible format found [ 31.709517] kernel: nvidia 0000:5a:00.0: [drm] Cannot find any crtc or sizes [ 31.709554] kernel: [drm] [nvidia-drm] [GPU ID 0x00005d00] Loading driver [ 31.714236] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:5d:00.0 on minor 4 [ 31.714261] kernel: nvidia 0000:5d:00.0: [drm] No compatible format found [ 31.714263] kernel: nvidia 0000:5d:00.0: [drm] Cannot find any crtc or sizes [ 31.714290] kernel: [drm] [nvidia-drm] [GPU ID 0x00005e00] Loading driver [ 31.718846] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:5e:00.0 on minor 5 [ 31.718867] kernel: nvidia 0000:5e:00.0: [drm] No compatible format found [ 31.718868] kernel: nvidia 0000:5e:00.0: [drm] Cannot find any crtc or sizes [ 31.718921] kernel: [drm] [nvidia-drm] [GPU ID 0x0000d300] Loading driver [ 31.723516] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:d3:00.0 on minor 6 [ 31.723529] kernel: nvidia 0000:d3:00.0: [drm] No compatible format found [ 31.723531] kernel: nvidia 0000:d3:00.0: [drm] Cannot find any crtc or sizes [ 31.723554] kernel: [drm] [nvidia-drm] [GPU ID 0x0000d600] Loading driver [ 31.728083] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:d6:00.0 on minor 7 [ 31.728093] kernel: nvidia 0000:d6:00.0: [drm] No compatible format found [ 31.728095] kernel: nvidia 0000:d6:00.0: [drm] Cannot find any crtc or sizes [ 31.728116] kernel: [drm] [nvidia-drm] [GPU ID 0x0000d900] Loading driver [ 31.732799] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:d9:00.0 on minor 8 [ 31.732809] kernel: nvidia 0000:d9:00.0: [drm] No compatible format found [ 31.732811] kernel: nvidia 0000:d9:00.0: [drm] Cannot find any crtc or sizes [ 31.732835] kernel: [drm] [nvidia-drm] [GPU ID 0x0000dc00] Loading driver [ 31.737556] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:dc:00.0 on minor 9 [ 31.737568] kernel: nvidia 0000:dc:00.0: [drm] No compatible format found [ 31.737570] kernel: nvidia 0000:dc:00.0: [drm] Cannot find any crtc or sizes [ 31.737593] kernel: [drm] [nvidia-drm] [GPU ID 0x0000dd00] Loading driver [ 31.742228] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:dd:00.0 on minor 10 [ 31.742238] kernel: nvidia 0000:dd:00.0: [drm] No compatible format found [ 31.742240] kernel: nvidia 0000:dd:00.0: [drm] Cannot find any crtc or sizes
GiteaMirror commented 2026-04-12 22:10:10 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

root@hq-it-ai:/usr/lib# lsmod | grep nv
nvidia_uvm 2158592 4
nvidia_drm 139264 0
nvidia_modeset 1814528 1 nvidia_drm
nvidia 14409728 47 nvidia_uvm,nvidia_modeset
video 77824 1 nvidia_modeset
ecc 45056 1 nvidia
nvme 61440 7
nvme_core 212992 8 nvme
nvme_auth 28672 1 nvme_core

<!-- gh-comment-id:3921591647 --> @akhilec commented on GitHub (Feb 18, 2026): root@hq-it-ai:/usr/lib# lsmod | grep nv nvidia_uvm 2158592 4 nvidia_drm 139264 0 nvidia_modeset 1814528 1 nvidia_drm nvidia 14409728 47 nvidia_uvm,nvidia_modeset video 77824 1 nvidia_modeset ecc 45056 1 nvidia nvme 61440 7 nvme_core 212992 8 nvme nvme_auth 28672 1 nvme_core
GiteaMirror commented 2026-04-12 22:10:10 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

root@hq-it-ai:/usr/lib# ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195, 0 Feb 18 05:05 /dev/nvidia0
crw-rw-rw- 1 root root 195, 1 Feb 18 05:05 /dev/nvidia1
crw-rw-rw- 1 root root 195, 2 Feb 18 05:05 /dev/nvidia2
crw-rw-rw- 1 root root 195, 3 Feb 18 05:05 /dev/nvidia3
crw-rw-rw- 1 root root 195, 4 Feb 18 05:05 /dev/nvidia4
crw-rw-rw- 1 root root 195, 5 Feb 18 05:05 /dev/nvidia5
crw-rw-rw- 1 root root 195, 6 Feb 18 05:05 /dev/nvidia6
crw-rw-rw- 1 root root 195, 7 Feb 18 05:05 /dev/nvidia7
crw-rw-rw- 1 root root 195, 8 Feb 18 05:05 /dev/nvidia8
crw-rw-rw- 1 root root 195, 9 Feb 18 05:05 /dev/nvidia9
crw-rw-rw- 1 root root 195, 255 Feb 18 05:05 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 Feb 18 05:05 /dev/nvidia-modeset
crw-rw-rw- 1 root root 508, 0 Feb 18 05:05 /dev/nvidia-uvm
crw-rw-rw- 1 root root 508, 1 Feb 18 05:05 /dev/nvidia-uvm-tools

/dev/nvidia-caps:
total 0
cr-------- 1 root root 511, 1 Feb 18 05:05 nvidia-cap1
cr--r--r-- 1 root root 511, 2 Feb 18 05:05 nvidia-cap2

<!-- gh-comment-id:3921593807 --> @akhilec commented on GitHub (Feb 18, 2026): root@hq-it-ai:/usr/lib# ls -l /dev/nvidia* crw-rw-rw- 1 root root 195, 0 Feb 18 05:05 /dev/nvidia0 crw-rw-rw- 1 root root 195, 1 Feb 18 05:05 /dev/nvidia1 crw-rw-rw- 1 root root 195, 2 Feb 18 05:05 /dev/nvidia2 crw-rw-rw- 1 root root 195, 3 Feb 18 05:05 /dev/nvidia3 crw-rw-rw- 1 root root 195, 4 Feb 18 05:05 /dev/nvidia4 crw-rw-rw- 1 root root 195, 5 Feb 18 05:05 /dev/nvidia5 crw-rw-rw- 1 root root 195, 6 Feb 18 05:05 /dev/nvidia6 crw-rw-rw- 1 root root 195, 7 Feb 18 05:05 /dev/nvidia7 crw-rw-rw- 1 root root 195, 8 Feb 18 05:05 /dev/nvidia8 crw-rw-rw- 1 root root 195, 9 Feb 18 05:05 /dev/nvidia9 crw-rw-rw- 1 root root 195, 255 Feb 18 05:05 /dev/nvidiactl crw-rw-rw- 1 root root 195, 254 Feb 18 05:05 /dev/nvidia-modeset crw-rw-rw- 1 root root 508, 0 Feb 18 05:05 /dev/nvidia-uvm crw-rw-rw- 1 root root 508, 1 Feb 18 05:05 /dev/nvidia-uvm-tools /dev/nvidia-caps: total 0 cr-------- 1 root root 511, 1 Feb 18 05:05 nvidia-cap1 cr--r--r-- 1 root root 511, 2 Feb 18 05:05 nvidia-cap2
GiteaMirror commented 2026-04-12 22:10:10 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 18, 2026):

Try this:

echo 'options nvidia_uvm uvm_disable_hmm=1' > /etc/modprobe.d/nvidia-uvm.conf
modprobe -r nvidia_uvm
modprobe nvidia_uvm
<!-- gh-comment-id:3921639413 --> @rick-github commented on GitHub (Feb 18, 2026): Try this: ``` echo 'options nvidia_uvm uvm_disable_hmm=1' > /etc/modprobe.d/nvidia-uvm.conf modprobe -r nvidia_uvm modprobe nvidia_uvm ```
GiteaMirror commented 2026-04-12 22:10:11 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

Try this:

echo 'options nvidia_uvm uvm_disable_hmm=1' > /etc/modprobe.d/nvidia-uvm.conf
modprobe -r nvidia_uvm
modprobe nvidia_uvm

at the host level?

<!-- gh-comment-id:3921642443 --> @akhilec commented on GitHub (Feb 18, 2026): > Try this: > > ``` > echo 'options nvidia_uvm uvm_disable_hmm=1' > /etc/modprobe.d/nvidia-uvm.conf > modprobe -r nvidia_uvm > modprobe nvidia_uvm > ``` at the host level?
GiteaMirror commented 2026-04-12 22:10:11 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 18, 2026):

Yes.

<!-- gh-comment-id:3921643809 --> @rick-github commented on GitHub (Feb 18, 2026): Yes.
GiteaMirror commented 2026-04-12 22:10:11 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

Try this:

echo 'options nvidia_uvm uvm_disable_hmm=1' > /etc/modprobe.d/nvidia-uvm.conf
modprobe -r nvidia_uvm
modprobe nvidia_uvm

root@hq-it-ai:/usr/lib# echo 'options nvidia_uvm uvm_disable_hmm=1' > /etc/modprobe.d/nvidia-uvm.conf
root@hq-it-ai:/usr/lib# modprobe -r nvidia_uvm
root@hq-it-ai:/usr/lib# modprobe nvidia_uvm
root@hq-it-ai:/usr/lib#

I tired all these commands, dind't see any out put. Was i expected to see anything

<!-- gh-comment-id:3921649474 --> @akhilec commented on GitHub (Feb 18, 2026): > Try this: > > ``` > echo 'options nvidia_uvm uvm_disable_hmm=1' > /etc/modprobe.d/nvidia-uvm.conf > modprobe -r nvidia_uvm > modprobe nvidia_uvm > ``` root@hq-it-ai:/usr/lib# echo 'options nvidia_uvm uvm_disable_hmm=1' > /etc/modprobe.d/nvidia-uvm.conf root@hq-it-ai:/usr/lib# modprobe -r nvidia_uvm root@hq-it-ai:/usr/lib# modprobe nvidia_uvm root@hq-it-ai:/usr/lib# I tired all these commands, dind't see any out put. Was i expected to see anything
GiteaMirror commented 2026-04-12 22:10:12 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 18, 2026):

If there were no errors, now try:

docker run  --gpus=all -e OLLAMA_DEBUG=2 ollama/ollama:0.12.3
<!-- gh-comment-id:3921656543 --> @rick-github commented on GitHub (Feb 18, 2026): If there were no errors, now try: ``` docker run --gpus=all -e OLLAMA_DEBUG=2 ollama/ollama:0.12.3 ```
GiteaMirror commented 2026-04-12 22:10:12 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

docker run --gpus=all -e OLLAMA_DEBUG=2 ollama/ollama:0.12.3

root@hq-it-ai:~# docker logs -f 22c3299c409d
Couldn't find '/root/.ollama/id_ed25519'. Generating new private key.
Your new public key is:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICC7kq7tAKGc+OqHhoUYl51S9zqrgwoNyGNQX0Oa3zi3

time=2026-02-18T16:09:54.367Z level=INFO source=routes.go:1475 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG-4 OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2026-02-18T16:09:54.368Z level=INFO source=images.go:518 msg="total blobs: 0"
time=2026-02-18T16:09:54.368Z level=INFO source=images.go:525 msg="total unused blobs removed: 0"
time=2026-02-18T16:09:54.368Z level=INFO source=routes.go:1528 msg="Listening on [::]:11434 (version 0.12.3)"
time=2026-02-18T16:09:54.368Z level=DEBUG source=sched.go:121 msg="starting llm scheduler"
time=2026-02-18T16:09:54.368Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2026-02-18T16:09:54.376Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2026-02-18T16:09:54.376Z level=DEBUG source=gpu.go:520 msg="Searching for GPU library" name=libcuda.so
time=2026-02-18T16:09:54.376Z level=DEBUG source=gpu.go:544 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets//lib/libcuda.so /usr/lib/-linux-gnu/nvidia/current/libcuda.so /usr/lib/-linux-gnu/libcuda.so /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers//libcuda.so /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2026-02-18T16:09:54.377Z level=DEBUG source=gpu.go:577 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09]
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09
dlsym: cuInit - 0x700b0a527850
dlsym: cuDriverGetVersion - 0x700b0a527910
dlsym: cuDeviceGetCount - 0x700b0a527a90
dlsym: cuDeviceGet - 0x700b0a5279d0
dlsym: cuDeviceGetAttribute - 0x700b0a527f10
dlsym: cuDeviceGetUuid - 0x700b0a57eb10
dlsym: cuDeviceGetName - 0x700b0a527b50
dlsym: cuCtxCreate_v3 - 0x700b0a57c2b0
dlsym: cuMemGetInfo_v2 - 0x700b0a52b780
dlsym: cuCtxDestroy - 0x700b0a57e1b0
calling cuInit
calling cuDriverGetVersion
raw version 0x32c8
CUDA driver version: 13.0
calling cuDeviceGetCount
device count 10
time=2026-02-18T16:09:55.615Z level=DEBUG source=gpu.go:125 msg="detected GPUs" count=10 library=/usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09
[GPU-dec32acf-20bb-2380-c2a0-291da4e33fba] CUDA totalMem 97249mb
[GPU-dec32acf-20bb-2380-c2a0-291da4e33fba] CUDA freeMem 96687mb
[GPU-dec32acf-20bb-2380-c2a0-291da4e33fba] Compute Capability 12.0
[GPU-39d80ed7-aa0f-4a25-f778-e1f2f2ffd9e9] CUDA totalMem 97249mb
[GPU-39d80ed7-aa0f-4a25-f778-e1f2f2ffd9e9] CUDA freeMem 96687mb
[GPU-39d80ed7-aa0f-4a25-f778-e1f2f2ffd9e9] Compute Capability 12.0
[GPU-2d43c411-a53f-a2fb-0374-5636d1b750b4] CUDA totalMem 97249mb
[GPU-2d43c411-a53f-a2fb-0374-5636d1b750b4] CUDA freeMem 96687mb
[GPU-2d43c411-a53f-a2fb-0374-5636d1b750b4] Compute Capability 12.0
[GPU-eb51bc4d-4f67-c734-23f7-5fab1ef2e885] CUDA totalMem 97249mb
[GPU-eb51bc4d-4f67-c734-23f7-5fab1ef2e885] CUDA freeMem 96687mb
[GPU-eb51bc4d-4f67-c734-23f7-5fab1ef2e885] Compute Capability 12.0
[GPU-f474b2ec-e25b-2723-be82-cac7b99609fa] CUDA totalMem 97249mb
[GPU-f474b2ec-e25b-2723-be82-cac7b99609fa] CUDA freeMem 96687mb
[GPU-f474b2ec-e25b-2723-be82-cac7b99609fa] Compute Capability 12.0
[GPU-fc02a6f3-15da-232f-cd45-503f90a1c4b7] CUDA totalMem 97249mb
[GPU-fc02a6f3-15da-232f-cd45-503f90a1c4b7] CUDA freeMem 96687mb
[GPU-fc02a6f3-15da-232f-cd45-503f90a1c4b7] Compute Capability 12.0
[GPU-720c4c0f-9858-3779-be3e-9718bfd01653] CUDA totalMem 97249mb
[GPU-720c4c0f-9858-3779-be3e-9718bfd01653] CUDA freeMem 96687mb
[GPU-720c4c0f-9858-3779-be3e-9718bfd01653] Compute Capability 12.0
[GPU-ff1d1807-c731-089a-5711-759443a6a60d] CUDA totalMem 97249mb
[GPU-ff1d1807-c731-089a-5711-759443a6a60d] CUDA freeMem 96687mb
[GPU-ff1d1807-c731-089a-5711-759443a6a60d] Compute Capability 12.0
[GPU-e17849d5-ecca-44e0-88f1-482b72195ea4] CUDA totalMem 97249mb
[GPU-e17849d5-ecca-44e0-88f1-482b72195ea4] CUDA freeMem 96687mb
[GPU-e17849d5-ecca-44e0-88f1-482b72195ea4] Compute Capability 12.0
[GPU-5afaa004-f27b-fd5d-e4e4-a974bce8803c] CUDA totalMem 97249mb
[GPU-5afaa004-f27b-fd5d-e4e4-a974bce8803c] CUDA freeMem 96687mb
[GPU-5afaa004-f27b-fd5d-e4e4-a974bce8803c] Compute Capability 12.0
time=2026-02-18T16:09:57.159Z level=DEBUG source=amd_linux.go:423 msg="amdgpu driver not detected /sys/module/amdgpu"
releasing cuda driver library
time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-dec32acf-20bb-2380-c2a0-291da4e33fba library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"
time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-39d80ed7-aa0f-4a25-f778-e1f2f2ffd9e9 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"
time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-2d43c411-a53f-a2fb-0374-5636d1b750b4 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"
time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-eb51bc4d-4f67-c734-23f7-5fab1ef2e885 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"
time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-f474b2ec-e25b-2723-be82-cac7b99609fa library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"
time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-fc02a6f3-15da-232f-cd45-503f90a1c4b7 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"
time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-720c4c0f-9858-3779-be3e-9718bfd01653 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"
time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-ff1d1807-c731-089a-5711-759443a6a60d library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"
time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-e17849d5-ecca-44e0-88f1-482b72195ea4 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"
time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-5afaa004-f27b-fd5d-e4e4-a974bce8803c library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"

<!-- gh-comment-id:3921717697 --> @akhilec commented on GitHub (Feb 18, 2026): > docker run --gpus=all -e OLLAMA_DEBUG=2 ollama/ollama:0.12.3 root@hq-it-ai:~# docker logs -f 22c3299c409d Couldn't find '/root/.ollama/id_ed25519'. Generating new private key. Your new public key is: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICC7kq7tAKGc+OqHhoUYl51S9zqrgwoNyGNQX0Oa3zi3 time=2026-02-18T16:09:54.367Z level=INFO source=routes.go:1475 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG-4 OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2026-02-18T16:09:54.368Z level=INFO source=images.go:518 msg="total blobs: 0" time=2026-02-18T16:09:54.368Z level=INFO source=images.go:525 msg="total unused blobs removed: 0" time=2026-02-18T16:09:54.368Z level=INFO source=routes.go:1528 msg="Listening on [::]:11434 (version 0.12.3)" time=2026-02-18T16:09:54.368Z level=DEBUG source=sched.go:121 msg="starting llm scheduler" time=2026-02-18T16:09:54.368Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2026-02-18T16:09:54.376Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2026-02-18T16:09:54.376Z level=DEBUG source=gpu.go:520 msg="Searching for GPU library" name=libcuda.so* time=2026-02-18T16:09:54.376Z level=DEBUG source=gpu.go:544 msg="gpu library search" globs="[/usr/lib/ollama/libcuda.so* /usr/local/nvidia/lib/libcuda.so* /usr/local/nvidia/lib64/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2026-02-18T16:09:54.377Z level=DEBUG source=gpu.go:577 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09] initializing /usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09 dlsym: cuInit - 0x700b0a527850 dlsym: cuDriverGetVersion - 0x700b0a527910 dlsym: cuDeviceGetCount - 0x700b0a527a90 dlsym: cuDeviceGet - 0x700b0a5279d0 dlsym: cuDeviceGetAttribute - 0x700b0a527f10 dlsym: cuDeviceGetUuid - 0x700b0a57eb10 dlsym: cuDeviceGetName - 0x700b0a527b50 dlsym: cuCtxCreate_v3 - 0x700b0a57c2b0 dlsym: cuMemGetInfo_v2 - 0x700b0a52b780 dlsym: cuCtxDestroy - 0x700b0a57e1b0 calling cuInit calling cuDriverGetVersion raw version 0x32c8 CUDA driver version: 13.0 calling cuDeviceGetCount device count 10 time=2026-02-18T16:09:55.615Z level=DEBUG source=gpu.go:125 msg="detected GPUs" count=10 library=/usr/lib/x86_64-linux-gnu/libcuda.so.580.126.09 [GPU-dec32acf-20bb-2380-c2a0-291da4e33fba] CUDA totalMem 97249mb [GPU-dec32acf-20bb-2380-c2a0-291da4e33fba] CUDA freeMem 96687mb [GPU-dec32acf-20bb-2380-c2a0-291da4e33fba] Compute Capability 12.0 [GPU-39d80ed7-aa0f-4a25-f778-e1f2f2ffd9e9] CUDA totalMem 97249mb [GPU-39d80ed7-aa0f-4a25-f778-e1f2f2ffd9e9] CUDA freeMem 96687mb [GPU-39d80ed7-aa0f-4a25-f778-e1f2f2ffd9e9] Compute Capability 12.0 [GPU-2d43c411-a53f-a2fb-0374-5636d1b750b4] CUDA totalMem 97249mb [GPU-2d43c411-a53f-a2fb-0374-5636d1b750b4] CUDA freeMem 96687mb [GPU-2d43c411-a53f-a2fb-0374-5636d1b750b4] Compute Capability 12.0 [GPU-eb51bc4d-4f67-c734-23f7-5fab1ef2e885] CUDA totalMem 97249mb [GPU-eb51bc4d-4f67-c734-23f7-5fab1ef2e885] CUDA freeMem 96687mb [GPU-eb51bc4d-4f67-c734-23f7-5fab1ef2e885] Compute Capability 12.0 [GPU-f474b2ec-e25b-2723-be82-cac7b99609fa] CUDA totalMem 97249mb [GPU-f474b2ec-e25b-2723-be82-cac7b99609fa] CUDA freeMem 96687mb [GPU-f474b2ec-e25b-2723-be82-cac7b99609fa] Compute Capability 12.0 [GPU-fc02a6f3-15da-232f-cd45-503f90a1c4b7] CUDA totalMem 97249mb [GPU-fc02a6f3-15da-232f-cd45-503f90a1c4b7] CUDA freeMem 96687mb [GPU-fc02a6f3-15da-232f-cd45-503f90a1c4b7] Compute Capability 12.0 [GPU-720c4c0f-9858-3779-be3e-9718bfd01653] CUDA totalMem 97249mb [GPU-720c4c0f-9858-3779-be3e-9718bfd01653] CUDA freeMem 96687mb [GPU-720c4c0f-9858-3779-be3e-9718bfd01653] Compute Capability 12.0 [GPU-ff1d1807-c731-089a-5711-759443a6a60d] CUDA totalMem 97249mb [GPU-ff1d1807-c731-089a-5711-759443a6a60d] CUDA freeMem 96687mb [GPU-ff1d1807-c731-089a-5711-759443a6a60d] Compute Capability 12.0 [GPU-e17849d5-ecca-44e0-88f1-482b72195ea4] CUDA totalMem 97249mb [GPU-e17849d5-ecca-44e0-88f1-482b72195ea4] CUDA freeMem 96687mb [GPU-e17849d5-ecca-44e0-88f1-482b72195ea4] Compute Capability 12.0 [GPU-5afaa004-f27b-fd5d-e4e4-a974bce8803c] CUDA totalMem 97249mb [GPU-5afaa004-f27b-fd5d-e4e4-a974bce8803c] CUDA freeMem 96687mb [GPU-5afaa004-f27b-fd5d-e4e4-a974bce8803c] Compute Capability 12.0 time=2026-02-18T16:09:57.159Z level=DEBUG source=amd_linux.go:423 msg="amdgpu driver not detected /sys/module/amdgpu" releasing cuda driver library time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-dec32acf-20bb-2380-c2a0-291da4e33fba library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB" time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-39d80ed7-aa0f-4a25-f778-e1f2f2ffd9e9 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB" time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-2d43c411-a53f-a2fb-0374-5636d1b750b4 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB" time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-eb51bc4d-4f67-c734-23f7-5fab1ef2e885 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB" time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-f474b2ec-e25b-2723-be82-cac7b99609fa library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB" time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-fc02a6f3-15da-232f-cd45-503f90a1c4b7 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB" time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-720c4c0f-9858-3779-be3e-9718bfd01653 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB" time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-ff1d1807-c731-089a-5711-759443a6a60d library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB" time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-e17849d5-ecca-44e0-88f1-482b72195ea4 library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB" time=2026-02-18T16:09:57.159Z level=INFO source=types.go:131 msg="inference compute" id=GPU-5afaa004-f27b-fd5d-e4e4-a974bce8803c library=cuda variant=v13 compute=12.0 driver=13.0 name="NVIDIA RTX PRO 6000 Blackwell Server Edition" total="95.0 GiB" available="94.4 GiB"
GiteaMirror commented 2026-04-12 22:10:12 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

Looks like it was able to detect now

<!-- gh-comment-id:3921718860 --> @akhilec commented on GitHub (Feb 18, 2026): Looks like it was able to detect now
GiteaMirror commented 2026-04-12 22:10:13 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

root@hq-it-ai:~# docker exec -it 22c3299c409d ollama run gemma3:27b
pulling manifest
pulling e796792eba26: 100% ▕█████████████████████████████████████████████▏ 17 GB
pulling e0a42594d802: 100% ▕█████████████████████████████████████████████▏ 358 B
pulling dd084c7d92a3: 100% ▕█████████████████████████████████████████████▏ 8.4 KB
pulling 3116c5225075: 100% ▕█████████████████████████████████████████████▏ 77 B
pulling f838f048d368: 100% ▕█████████████████████████████████████████████▏ 490 B
verifying sha256 digest
writing manifest
success

hi
Hi there! 👋

How can I help you today? Just let me know what you're thinking, or if you just wanted to say hi,
that's perfectly fine too! 😊

I can:

  • Answer questions: About pretty much anything!
  • Generate creative content: Stories, poems, scripts, musical pieces, email, letters, etc.
  • Translate languages
  • Summarize text
  • Help with brainstorming
  • Just chat!

/bye
root@hq-it-ai:# docker exec -it 22c3299c409d ollama ps
NAME ID SIZE PROCESSOR CONTEXT UNTIL
gemma3:27b a418f5838eaf 20 GB 100% GPU 4096 4 minutes from now
root@hq-it-ai:#

NAME ID SIZE PROCESSOR CONTEXT UNTIL
gemma3:27b a418f5838eaf 20 GB 100% GPU 4096 4 minutes from now

<!-- gh-comment-id:3921765851 --> @akhilec commented on GitHub (Feb 18, 2026): root@hq-it-ai:~# docker exec -it 22c3299c409d ollama run gemma3:27b pulling manifest pulling e796792eba26: 100% ▕█████████████████████████████████████████████▏ 17 GB pulling e0a42594d802: 100% ▕█████████████████████████████████████████████▏ 358 B pulling dd084c7d92a3: 100% ▕█████████████████████████████████████████████▏ 8.4 KB pulling 3116c5225075: 100% ▕█████████████████████████████████████████████▏ 77 B pulling f838f048d368: 100% ▕█████████████████████████████████████████████▏ 490 B verifying sha256 digest writing manifest success >>> hi Hi there! 👋 How can I help you today? Just let me know what you're thinking, or if you just wanted to say hi, that's perfectly fine too! 😊 I can: * **Answer questions:** About pretty much anything! * **Generate creative content:** Stories, poems, scripts, musical pieces, email, letters, etc. * **Translate languages** * **Summarize text** * **Help with brainstorming** * **Just chat!** >>> /bye root@hq-it-ai:~# docker exec -it 22c3299c409d ollama ps NAME ID SIZE PROCESSOR CONTEXT UNTIL gemma3:27b a418f5838eaf 20 GB 100% GPU 4096 4 minutes from now root@hq-it-ai:~# NAME ID SIZE PROCESSOR CONTEXT UNTIL gemma3:27b a418f5838eaf 20 GB 100% GPU 4096 4 minutes from now
GiteaMirror commented 2026-04-12 22:10:13 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

Image
<!-- gh-comment-id:3921779092 --> @akhilec commented on GitHub (Feb 18, 2026): <img width="659" height="963" alt="Image" src="https://github.com/user-attachments/assets/29ab6874-b6ed-4bd2-a63c-5f5e44e185e5" />
GiteaMirror commented 2026-04-12 22:10:14 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

Thanks Rick, you are the best and the troubleshooting approach you did to hash out the problem was just amazing.

<!-- gh-comment-id:3921783832 --> @akhilec commented on GitHub (Feb 18, 2026): Thanks Rick, you are the best and the troubleshooting approach you did to hash out the problem was just amazing.
GiteaMirror commented 2026-04-12 22:10:14 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 18, 2026):

No worries, glad we resolved it. Have fun with your 10x6000s, I'm a little bit envious.

<!-- gh-comment-id:3921791734 --> @rick-github commented on GitHub (Feb 18, 2026): No worries, glad we resolved it. Have fun with your 10x6000s, I'm a little bit envious.
GiteaMirror commented 2026-04-12 22:10:14 -05:00
Author
Owner
Copy Link

@akhilec commented on GitHub (Feb 18, 2026):

Rick, one question, Should I continue to be on 0.12.3 or can try latest?

<!-- gh-comment-id:3921971101 --> @akhilec commented on GitHub (Feb 18, 2026): Rick, one question, Should I continue to be on 0.12.3 or can try latest?
GiteaMirror commented 2026-04-12 22:10:15 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 18, 2026):

Latest should now work.

<!-- gh-comment-id:3921978705 --> @rick-github commented on GitHub (Feb 18, 2026): Latest should now work.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#9300
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?