github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-09 01:59:50 -05:00
Code Issues 14.7k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #11593] init failure on RTX 6000 PRO Blackwell #54164

New Issue
Closed
opened 2026-04-29 05:18:05 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-29 05:18:05 -05:00
Owner
Copy Link

Originally created by @tjwebb on GitHub (Jul 31, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11593

What is the issue?

Ubuntu 25.04
NVIDIA-SMI 570.158.01
CUDA 12.8.90

Ollama will not use GPU, fails with "cudart init failure: 3"

I've checked, the file it is looking for does exist.

Relevant log output

OLLAMA_DEBUG=1 ollama serve 2>&1 | tee server.log
time=2025-07-31T02:39:58.545Z level=INFO source=routes.go:1238 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY:http://34.56.223.105:8888 NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/mnt/big2/usr/share/ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-07-31T02:39:58.545Z level=INFO source=images.go:476 msg="total blobs: 0"
time=2025-07-31T02:39:58.545Z level=INFO source=images.go:483 msg="total unused blobs removed: 0"
time=2025-07-31T02:39:58.545Z level=INFO source=routes.go:1291 msg="Listening on 127.0.0.1:11434 (version 0.10.0)"
time=2025-07-31T02:39:58.545Z level=DEBUG source=sched.go:106 msg="starting llm scheduler"
time=2025-07-31T02:39:58.545Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-07-31T02:39:58.552Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-07-31T02:39:58.552Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so*
time=2025-07-31T02:39:58.552Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/local/lib/ollama/libcuda.so* /usr/lib/x86_64-linux-gnu/libcuda.so* /mnt/big2/home/tjwebb/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2025-07-31T02:39:58.555Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.570.158.01]
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.158.01
dlsym: cuInit - 0x7e53b7d0ca70
dlsym: cuDriverGetVersion - 0x7e53b7d0ca90
dlsym: cuDeviceGetCount - 0x7e53b7d0cad0
dlsym: cuDeviceGet - 0x7e53b7d0cab0
dlsym: cuDeviceGetAttribute - 0x7e53b7d0cbb0
dlsym: cuDeviceGetUuid - 0x7e53b7d0cb10
dlsym: cuDeviceGetName - 0x7e53b7d0caf0
dlsym: cuCtxCreate_v3 - 0x7e53b7d0cd90
dlsym: cuMemGetInfo_v2 - 0x7e53b7d2d130
dlsym: cuCtxDestroy - 0x7e53b7d6ba50
calling cuInit
cuInit err: 3
time=2025-07-31T02:39:58.615Z level=INFO source=gpu.go:612 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcuda.so.570.158.01: cuda driver library init failure: 3"
time=2025-07-31T02:39:58.615Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcudart.so*
time=2025-07-31T02:39:58.615Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/local/lib/ollama/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /mnt/big2/home/tjwebb/libcudart.so* /usr/local/lib/ollama/cuda_v*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2025-07-31T02:39:58.617Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[/usr/local/lib/ollama/libcudart.so.12.8.90 /usr/lib/x86_64-linux-gnu/libcudart.so.12.8.90 /usr/lib/x86_64-linux-gnu/libcudart.so.12.2.140]"
cudaSetDevice err: 3
time=2025-07-31T02:39:58.677Z level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /usr/local/lib/ollama/libcudart.so.12.8.90: cudart init failure: 3"
cudaSetDevice err: 3
time=2025-07-31T02:39:58.729Z level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcudart.so.12.8.90: cudart init failure: 3"
cudaSetDevice err: 3
time=2025-07-31T02:39:58.770Z level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcudart.so.12.2.140: cudart init failure: 3"
time=2025-07-31T02:39:58.770Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu"
time=2025-07-31T02:39:58.770Z level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"
time=2025-07-31T02:39:58.770Z level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="1003.2 GiB" available="992.6 GiB"

OS

Ubuntu 25.04

GPU

1x RTX 6000 Pro Max Q Blackwell

CPU

Xeon 6

Ollama version

0.10

Originally created by @tjwebb on GitHub (Jul 31, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11593 ### What is the issue? Ubuntu 25.04 NVIDIA-SMI 570.158.01 CUDA 12.8.90 Ollama will not use GPU, fails with "cudart init failure: 3" I've checked, the file it is looking for does exist. ### Relevant log output ```shell OLLAMA_DEBUG=1 ollama serve 2>&1 | tee server.log time=2025-07-31T02:39:58.545Z level=INFO source=routes.go:1238 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY:http://34.56.223.105:8888 NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/mnt/big2/usr/share/ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]" time=2025-07-31T02:39:58.545Z level=INFO source=images.go:476 msg="total blobs: 0" time=2025-07-31T02:39:58.545Z level=INFO source=images.go:483 msg="total unused blobs removed: 0" time=2025-07-31T02:39:58.545Z level=INFO source=routes.go:1291 msg="Listening on 127.0.0.1:11434 (version 0.10.0)" time=2025-07-31T02:39:58.545Z level=DEBUG source=sched.go:106 msg="starting llm scheduler" time=2025-07-31T02:39:58.545Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-07-31T02:39:58.552Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2025-07-31T02:39:58.552Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so* time=2025-07-31T02:39:58.552Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/local/lib/ollama/libcuda.so* /usr/lib/x86_64-linux-gnu/libcuda.so* /mnt/big2/home/tjwebb/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2025-07-31T02:39:58.555Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.570.158.01] initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.158.01 dlsym: cuInit - 0x7e53b7d0ca70 dlsym: cuDriverGetVersion - 0x7e53b7d0ca90 dlsym: cuDeviceGetCount - 0x7e53b7d0cad0 dlsym: cuDeviceGet - 0x7e53b7d0cab0 dlsym: cuDeviceGetAttribute - 0x7e53b7d0cbb0 dlsym: cuDeviceGetUuid - 0x7e53b7d0cb10 dlsym: cuDeviceGetName - 0x7e53b7d0caf0 dlsym: cuCtxCreate_v3 - 0x7e53b7d0cd90 dlsym: cuMemGetInfo_v2 - 0x7e53b7d2d130 dlsym: cuCtxDestroy - 0x7e53b7d6ba50 calling cuInit cuInit err: 3 time=2025-07-31T02:39:58.615Z level=INFO source=gpu.go:612 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcuda.so.570.158.01: cuda driver library init failure: 3" time=2025-07-31T02:39:58.615Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcudart.so* time=2025-07-31T02:39:58.615Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/local/lib/ollama/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /mnt/big2/home/tjwebb/libcudart.so* /usr/local/lib/ollama/cuda_v*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]" time=2025-07-31T02:39:58.617Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[/usr/local/lib/ollama/libcudart.so.12.8.90 /usr/lib/x86_64-linux-gnu/libcudart.so.12.8.90 /usr/lib/x86_64-linux-gnu/libcudart.so.12.2.140]" cudaSetDevice err: 3 time=2025-07-31T02:39:58.677Z level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /usr/local/lib/ollama/libcudart.so.12.8.90: cudart init failure: 3" cudaSetDevice err: 3 time=2025-07-31T02:39:58.729Z level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcudart.so.12.8.90: cudart init failure: 3" cudaSetDevice err: 3 time=2025-07-31T02:39:58.770Z level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcudart.so.12.2.140: cudart init failure: 3" time=2025-07-31T02:39:58.770Z level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu" time=2025-07-31T02:39:58.770Z level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered" time=2025-07-31T02:39:58.770Z level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="1003.2 GiB" available="992.6 GiB" ``` ### OS Ubuntu 25.04 ### GPU 1x RTX 6000 Pro Max Q Blackwell ### CPU Xeon 6 ### Ollama version 0.10
GiteaMirror added the bugnvidialinux labels 2026-04-29 05:18:06 -05:00
GiteaMirror commented 2026-04-29 05:18:08 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jul 31, 2025):

According to the NVIDIA API docs, this error code is related to the driver not being initialized

    /**
     * This indicates that the CUDA driver has not been initialized with
     * ::cuInit() or that initialization has failed.
     */
    CUDA_ERROR_NOT_INITIALIZED                = 3,

Does nvidia-smi show your GPU?
Do you see any nvidia errors in dmesg related to your GPU?

<!-- gh-comment-id:3141381710 --> @dhiltgen commented on GitHub (Jul 31, 2025): According to the NVIDIA API docs, this error code is related to the driver not being initialized ``` /** * This indicates that the CUDA driver has not been initialized with * ::cuInit() or that initialization has failed. */ CUDA_ERROR_NOT_INITIALIZED = 3, ``` Does nvidia-smi show your GPU? Do you see any nvidia errors in `dmesg` related to your GPU?
GiteaMirror commented 2026-04-29 05:18:08 -05:00
Author
Owner
Copy Link

@tjwebb commented on GitHub (Jul 31, 2025):

Does nvidia-smi show your GPU?

Yes

Do you see any nvidia errors in dmesg related to your GPU?

No I've checked per the docs, there are no nvidia-related errors that I see

<!-- gh-comment-id:3141481943 --> @tjwebb commented on GitHub (Jul 31, 2025): > Does nvidia-smi show your GPU? Yes > Do you see any nvidia errors in dmesg related to your GPU? No I've checked per the docs, there are no nvidia-related errors that I see
GiteaMirror commented 2026-04-29 05:18:09 -05:00
Author
Owner
Copy Link

@tjwebb commented on GitHub (Aug 2, 2025):

Same thing happens with the 575 driver and CUDA 12.9 btw:

time=2025-08-02T22:42:20.643Z level=INFO source=gpu.go:612 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08: cuda driver library init failure: 3"

Is this GPU not supported by ollama?

<!-- gh-comment-id:3146802729 --> @tjwebb commented on GitHub (Aug 2, 2025): Same thing happens with the 575 driver and CUDA 12.9 btw: ``` time=2025-08-02T22:42:20.643Z level=INFO source=gpu.go:612 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08: cuda driver library init failure: 3" ``` Is this GPU not supported by ollama?
GiteaMirror commented 2026-04-29 05:18:09 -05:00
Author
Owner
Copy Link

@tjwebb commented on GitHub (Aug 22, 2025):

Ok I fixed this after a lot of debugging, issue was an Nvidia driver bug.

I needed to create /etc/modprobe.d/uvm.conf and set this:

options nvidia_uvm uvm_disable_hmm=1

https://forums.developer.nvidia.com/t/solved-cuda-driver-initialization-failed-2x-rtx-5090/334578/4

<!-- gh-comment-id:3215176257 --> @tjwebb commented on GitHub (Aug 22, 2025): Ok I fixed this after a lot of debugging, issue was an Nvidia driver bug. I needed to create /etc/modprobe.d/uvm.conf and set this: ``` options nvidia_uvm uvm_disable_hmm=1 ``` https://forums.developer.nvidia.com/t/solved-cuda-driver-initialization-failed-2x-rtx-5090/334578/4
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug linux nvidia
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#54164
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?