github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #9298] Ollama can't find Nvidia GPU via CUDA while other CUDA programs can #6065

New Issue
Closed
opened 2026-04-12 17:23:38 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-04-12 17:23:38 -05:00
Owner
Copy Link

Originally created by @OpenSauce04 on GitHub (Feb 23, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9298

What is the issue?

When attempting to use ollama with CUDA, I recieve the following messages in the logs:

time=2025-02-23T15:29:54.892Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cpu cuda_v12]"                                                                                                
time=2025-02-23T15:29:54.892Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs"                                                                                                                       
time=2025-02-23T15:29:54.908Z level=INFO source=gpu.go:620 msg="no nvidia devices detected by library /usr/lib64/libcuda.so.550.144.03"                                                                            
time=2025-02-23T15:29:54.917Z level=INFO source=gpu.go:392 msg="no compatible GPUs were discovered"                                                                                                                
time=2025-02-23T15:29:54.917Z level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="31.2 GiB" available="25.9 GiB"

Ollama then proceeds to exclusively use the CPU for processing with no GPU or VRAM activity to be seen.

I then went and found this gist which tests if CUDA is working correctly, and I recieve the following output, indicating that CUDA itself is likely not the reason the GPU can't be detected:

$ ./cuda_check
Found 1 device(s).
Device: 0
  Name: NVIDIA GeForce RTX 3060 Ti
  Compute Capability: 8.6
  Multiprocessors: 38
  CUDA Cores: 4864
  Concurrent threads: 58368
  GPU clock: 1695 MHz
  Memory clock: 7001 MHz
  Total Memory: 7963 MiB
  Free Memory: 6980 MiB

I am using ollama on Gentoo Linux as it is built by the app-misc/ollama ebuild in the GURU repository with the cuda USE flag enabled.

Here are my CPU flags from emerge --info if that's useful somehow:

CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3"

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.5.7

Originally created by @OpenSauce04 on GitHub (Feb 23, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9298 ### What is the issue? When attempting to use ollama with CUDA, I recieve the following messages in the logs: ``` time=2025-02-23T15:29:54.892Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners="[cpu cuda_v12]" time=2025-02-23T15:29:54.892Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs" time=2025-02-23T15:29:54.908Z level=INFO source=gpu.go:620 msg="no nvidia devices detected by library /usr/lib64/libcuda.so.550.144.03" time=2025-02-23T15:29:54.917Z level=INFO source=gpu.go:392 msg="no compatible GPUs were discovered" time=2025-02-23T15:29:54.917Z level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="31.2 GiB" available="25.9 GiB" ``` Ollama then proceeds to exclusively use the CPU for processing with no GPU or VRAM activity to be seen. I then went and found [this gist](https://gist.github.com/f0k/0d6431e3faa60bffc788f8b4daa029b1) which tests if CUDA is working correctly, and I recieve the following output, indicating that CUDA itself is likely not the reason the GPU can't be detected: ``` $ ./cuda_check Found 1 device(s). Device: 0 Name: NVIDIA GeForce RTX 3060 Ti Compute Capability: 8.6 Multiprocessors: 38 CUDA Cores: 4864 Concurrent threads: 58368 GPU clock: 1695 MHz Memory clock: 7001 MHz Total Memory: 7963 MiB Free Memory: 6980 MiB ``` I am using ollama on Gentoo Linux as it is built by the `app-misc/ollama` ebuild in the GURU repository with the `cuda` USE flag enabled. Here are my CPU flags from `emerge --info` if that's useful somehow: ``` CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3" ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.7
GiteaMirror added the needs more infobug labels 2026-04-12 17:23:38 -05:00
GiteaMirror commented 2026-04-12 17:23:39 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 23, 2025):

What does the following return:

ldd ./cuda_check
ls $(dirname $(dirname $(command -v ollama))) $(dirname $(dirname $(command -v ollama)))/lib/ollama
<!-- gh-comment-id:2676961703 --> @rick-github commented on GitHub (Feb 23, 2025): What does the following return: ``` ldd ./cuda_check ls $(dirname $(dirname $(command -v ollama))) $(dirname $(dirname $(command -v ollama)))/lib/ollama ```
GiteaMirror commented 2026-04-12 17:23:40 -05:00
Author
Owner
Copy Link

@OpenSauce04 commented on GitHub (Feb 23, 2025):

What does the following return:

ldd ./cuda_check
ls $(dirname $(dirname $(command -v ollama))) $(dirname $(dirname $(command -v ollama)))/lib/ollama

 $ ldd ./cuda_check
	linux-vdso.so.1 (0x00007f131ea90000)
	libcuda.so.1 => /usr/lib64/libcuda.so.1 (0x00007f131cc00000)
	libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/14/libstdc++.so.6 (0x00007f131c800000)
	libm.so.6 => /usr/lib64/libm.so.6 (0x00007f131e8f8000)
	libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/14/libgcc_s.so.1 (0x00007f131e8cb000)
	libc.so.6 => /usr/lib64/libc.so.6 (0x00007f131c62c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f131ea92000)
	libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f131e8c4000)
	libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f131e8bf000)
	librt.so.1 => /usr/lib64/librt.so.1 (0x00007f131e8ba000)

$ ls $(dirname $(dirname $(command -v ollama))) $(dirname $(dirname $(command -v ollama)))/lib/ollama
/usr:
bin  include  lib  lib64  libexec  local  sbin	share  src  x86_64-pc-linux-gnu

/usr/lib/ollama:
libcublas.so.12  libcublas.so.12.6.1.4	libcublasLt.so.12  libcublasLt.so.12.6.1.4  libcudart.so.12  libcudart.so.12.6.68  runners
<!-- gh-comment-id:2677037239 --> @OpenSauce04 commented on GitHub (Feb 23, 2025): > What does the following return: > > ``` > ldd ./cuda_check > ls $(dirname $(dirname $(command -v ollama))) $(dirname $(dirname $(command -v ollama)))/lib/ollama > ``` ``` $ ldd ./cuda_check linux-vdso.so.1 (0x00007f131ea90000) libcuda.so.1 => /usr/lib64/libcuda.so.1 (0x00007f131cc00000) libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/14/libstdc++.so.6 (0x00007f131c800000) libm.so.6 => /usr/lib64/libm.so.6 (0x00007f131e8f8000) libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/14/libgcc_s.so.1 (0x00007f131e8cb000) libc.so.6 => /usr/lib64/libc.so.6 (0x00007f131c62c000) /lib64/ld-linux-x86-64.so.2 (0x00007f131ea92000) libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f131e8c4000) libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f131e8bf000) librt.so.1 => /usr/lib64/librt.so.1 (0x00007f131e8ba000) $ ls $(dirname $(dirname $(command -v ollama))) $(dirname $(dirname $(command -v ollama)))/lib/ollama /usr: bin include lib lib64 libexec local sbin share src x86_64-pc-linux-gnu /usr/lib/ollama: libcublas.so.12 libcublas.so.12.6.1.4 libcublasLt.so.12 libcublasLt.so.12.6.1.4 libcudart.so.12 libcudart.so.12.6.68 runners ```
GiteaMirror commented 2026-04-12 17:23:40 -05:00
Author
Owner
Copy Link

@OpenSauce04 commented on GitHub (Feb 25, 2025):

Similar issue here -- although I am running Ollama in Docker inside a container, between Saturday and today Ollama no longer picks up the GPU:

time=2025-02-25T16:32:26.162Z level=INFO source=routes.go:1256 msg="Listening on [::]:11434 (version 0.5.12)"
time=2025-02-25T16:32:26.162Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-02-25T16:32:26.436Z level=WARN source=gpu.go:605 msg="unknown error initializing cuda driver library /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03: cuda driver library init failure: 999. see https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md for more information"

Testing with nvidia-smi inside the container reports the GPU as expected.

The error message in your message seems to indicate that it is unrelated to the issue I was experiencing. For me, CUDA itself was working fine but couldn't find the GPU, whereas in your case it seems like CUDA itself isn't working properly.

<!-- gh-comment-id:2682910528 --> @OpenSauce04 commented on GitHub (Feb 25, 2025): > Similar issue here -- although I am running Ollama in Docker inside a container, between Saturday and today Ollama no longer picks up the GPU: > > time=2025-02-25T16:32:26.162Z level=INFO source=routes.go:1256 msg="Listening on [::]:11434 (version 0.5.12)" > time=2025-02-25T16:32:26.162Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" > time=2025-02-25T16:32:26.436Z level=WARN source=gpu.go:605 msg="unknown error initializing cuda driver library /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03: cuda driver library init failure: 999. see https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md for more information" > > Testing with nvidia-smi inside the container reports the GPU as expected. The error message in your message seems to indicate that it is unrelated to the issue I was experiencing. For me, CUDA itself was working fine but couldn't find the GPU, whereas in your case it seems like CUDA itself isn't working properly.
GiteaMirror commented 2026-04-12 17:23:41 -05:00
Author
Owner
Copy Link

@jasonhouston commented on GitHub (Feb 25, 2025):

Similar issue here -- although I am running Ollama in Docker inside a container, between Saturday and today Ollama no longer picks up the GPU:
time=2025-02-25T16:32:26.162Z level=INFO source=routes.go:1256 msg="Listening on [::]:11434 (version 0.5.12)"
time=2025-02-25T16:32:26.162Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-02-25T16:32:26.436Z level=WARN source=gpu.go:605 msg="unknown error initializing cuda driver library /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03: cuda driver library init failure: 999. see https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md for more information"
Testing with nvidia-smi inside the container reports the GPU as expected.

The error message in your message seems to indicate that it is unrelated to the issue I was experiencing. For me, CUDA itself was working fine but couldn't find the GPU, whereas in your case it seems like CUDA itself isn't working properly.

You are correct -- I found the misconfiguration here, my apologies

<!-- gh-comment-id:2683360507 --> @jasonhouston commented on GitHub (Feb 25, 2025): > > Similar issue here -- although I am running Ollama in Docker inside a container, between Saturday and today Ollama no longer picks up the GPU: > > time=2025-02-25T16:32:26.162Z level=INFO source=routes.go:1256 msg="Listening on [::]:11434 (version 0.5.12)" > > time=2025-02-25T16:32:26.162Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" > > time=2025-02-25T16:32:26.436Z level=WARN source=gpu.go:605 msg="unknown error initializing cuda driver library /usr/lib/x86_64-linux-gnu/libcuda.so.550.144.03: cuda driver library init failure: 999. see https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md for more information" > > Testing with nvidia-smi inside the container reports the GPU as expected. > > The error message in your message seems to indicate that it is unrelated to the issue I was experiencing. For me, CUDA itself was working fine but couldn't find the GPU, whereas in your case it seems like CUDA itself isn't working properly. You are correct -- I found the misconfiguration here, my apologies
GiteaMirror commented 2026-04-12 17:23:41 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Feb 27, 2025):

Set OLLAMA_DEBUG=1 in the server environment and post the logs.

<!-- gh-comment-id:2688087061 --> @rick-github commented on GitHub (Feb 27, 2025): Set `OLLAMA_DEBUG=1` in the server environment and post the logs.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug needs more info
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6065
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?