github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 16:40:08 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #1879] Jetson Orin NX 16gb not seeing much CUDA usage with Ubuntu 22 and Jetpack 6 even after applying documented LD path work around #63114

New Issue
Closed
opened 2026-05-03 12:12:11 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-05-03 12:12:11 -05:00
Owner
Copy Link

Originally created by @carolynhudson on GitHub (Jan 9, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/1879

Originally assigned to: @dhiltgen on GitHub.

I recently rebuilt my Orin NX and chose the newest release OS and Jetpack edition as I wanted a clean slate to try ollama in. I saw no difference in the performance before or after following the given workaround. When I close the service instance and intentionally opened a new terminal window to run ollama serve in the service loads, says it sees CUDA but when it does the GPU check it looks in the modified LD path for a libnvidia-ml.so, fails, and then reports no GPUs available. I conformed using jtop that all CPU cores were at or near 100% when running mistral and the CUDA cores were mostly idle with only occasional usage blips. I also tried other paths such as the cuda12.2 folder rather than the base CUDA and where I did see a libnvida-ml.so which just causes another error over libnvidia.so.1 and still no “GPU” detection and no CUDA usage. I went so far as to run through the Nvidia portion of the setup script and made sure everything was installed as directed by it.

I think I will try rebuilding it again with Jetpack 5.1 just to see if it works there. But I wanted to report it anyway just in case it is a Jetpack 6.0 vs. 5.1 issue. I will update if that fixes it.

Originally created by @carolynhudson on GitHub (Jan 9, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/1879 Originally assigned to: @dhiltgen on GitHub. I recently rebuilt my Orin NX and chose the newest release OS and Jetpack edition as I wanted a clean slate to try ollama in. I saw no difference in the performance before or after following the given workaround. When I close the service instance and intentionally opened a new terminal window to run ollama serve in the service loads, says it sees CUDA but when it does the GPU check it looks in the modified LD path for a libnvidia-ml.so, fails, and then reports no GPUs available. I conformed using jtop that all CPU cores were at or near 100% when running mistral and the CUDA cores were mostly idle with only occasional usage blips. I also tried other paths such as the cuda12.2 folder rather than the base CUDA and where I did see a libnvida-ml.so which just causes another error over libnvidia.so.1 and still no “GPU” detection and no CUDA usage. I went so far as to run through the Nvidia portion of the setup script and made sure everything was installed as directed by it. I think I will try rebuilding it again with Jetpack 5.1 just to see if it works there. But I wanted to report it anyway just in case it is a Jetpack 6.0 vs. 5.1 issue. I will update if that fixes it.
GiteaMirror commented 2026-05-03 12:12:14 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jan 10, 2024):

Can you share the server log showing the failed attempt to lookup GPU details via libnvidia-ml.so, along with the path where the library is found on your system?

<!-- gh-comment-id:1884012512 --> @dhiltgen commented on GitHub (Jan 10, 2024): Can you share the server log showing the failed attempt to lookup GPU details via libnvidia-ml.so, along with the path where the library is found on your system?
GiteaMirror commented 2026-05-03 12:12:15 -05:00
Author
Owner
Copy Link

@carolynhudson commented on GitHub (Jan 10, 2024):

Here is the log, in my attempts one of the things i did seemed to force it to that WSL directory no idea why as it does't exist. Hence why rather than trying try figure out what did that since i know pointing to the location where the libnvidia-ml.so is doesn't work i figured i would just reload and go back to jetpack 5.1.2 and ubuntu 20
Dec 31 16:00:30 bunnybot systemd[1]: Started Ollama Service.
Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 images.go:834: total blobs: 6
Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 images.go:841: total unused blobs removed: 0
Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 routes.go:929: Listening on 127.0.0.1:11434 (version 0.1.18)
Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 shim_ext_server.go:142: Dynamic LLM variants [cuda]
Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 gpu.go:34: Detecting GPU type
Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 gpu.go:39: CUDA not detected: Unable to load libnvidia-ml.so library to query for Nvidia GPUs: /usr/lib/wsl/lib/li>
Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 gpu.go:45: ROCm not detected: Unable to load librocm_smi64.so library to query for Radeon GPUs: /opt/rocm/lib/libr>
Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 routes.go:952: no GPU detected

This is the locations in /usr where i could find that library.
/usr/lib/aarch64-linux-gnu/nvidia/libnvidia-ml.so.1
/usr/local/cuda-12.2/targets/aarch64-linux/lib/stubs/libnvidia-ml.so

<!-- gh-comment-id:1884087292 --> @carolynhudson commented on GitHub (Jan 10, 2024): Here is the log, in my attempts one of the things i did seemed to force it to that WSL directory no idea why as it does't exist. Hence why rather than trying try figure out what did that since i know pointing to the location where the libnvidia-ml.so is doesn't work i figured i would just reload and go back to jetpack 5.1.2 and ubuntu 20 Dec 31 16:00:30 bunnybot systemd[1]: Started Ollama Service. Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 images.go:834: total blobs: 6 Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 images.go:841: total unused blobs removed: 0 Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 routes.go:929: Listening on 127.0.0.1:11434 (version 0.1.18) Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 shim_ext_server.go:142: Dynamic LLM variants [cuda] Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 gpu.go:34: Detecting GPU type Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 gpu.go:39: CUDA not detected: Unable to load libnvidia-ml.so library to query for Nvidia GPUs: /usr/lib/wsl/lib/li> Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 gpu.go:45: ROCm not detected: Unable to load librocm_smi64.so library to query for Radeon GPUs: /opt/rocm/lib/libr> Dec 31 16:00:31 bunnybot ollama[1094]: 1969/12/31 16:00:31 routes.go:952: no GPU detected This is the locations in /usr where i could find that library. /usr/lib/aarch64-linux-gnu/nvidia/libnvidia-ml.so.1 /usr/local/cuda-12.2/targets/aarch64-linux/lib/stubs/libnvidia-ml.so
GiteaMirror commented 2026-05-03 12:12:16 -05:00
Author
Owner
Copy Link

@carolynhudson commented on GitHub (Jan 10, 2024):

Small update, built a new Ubuntu box and setup the jetson orin nx with Ubuntu 20 and Jetpack 5.1.2, I even built it from a new VM made from a fresh Ubuntu 20 Desktop iso. After all setup was complete, apt update/upgrade run, rebooted, and ran the instructions from: https://github.com/jmorganca/ollama/blob/main/docs/tutorials/nvidia-jetson.md

It does not seem to be respecting the LD_LIBRARY_PATH as specified.
This is the log from the ollama service added to the system:
Jan 10 01:19:17 bunnybot systemd[1]: Started Ollama Service.
Jan 10 01:19:17 bunnybot ollama[1114]: 2024/01/10 01:19:17 images.go:808: total blobs: 0
Jan 10 01:19:17 bunnybot ollama[1114]: 2024/01/10 01:19:17 images.go:815: total unused blobs removed: 0
Jan 10 01:19:17 bunnybot ollama[1114]: 2024/01/10 01:19:17 routes.go:930: Listening on 127.0.0.1:11434 (version 0.1.19)
Jan 10 01:19:18 bunnybot ollama[1114]: 2024/01/10 01:19:18 shim_ext_server.go:142: Dynamic LLM variants [cuda]
Jan 10 01:19:18 bunnybot ollama[1114]: 2024/01/10 01:19:18 gpu.go:35: Detecting GPU type
Jan 10 01:19:18 bunnybot ollama[1114]: 2024/01/10 01:19:18 gpu.go:40: CUDA not detected: Unable to load libnvidia-ml.so library to query for Nvidia GPUs: /usr/lib/wsl/lib/libnvidia-ml.so.1: cannot open shared object file: No such file or directory
Jan 10 01:19:18 bunnybot ollama[1114]: 2024/01/10 01:19:18 gpu.go:46: ROCm not detected: Unable to load librocm_smi64.so library to query for Radeon GPUs: /opt/rocm/lib/librocm_smi64.so: cannot open shared object file: No such file or directory
Jan 10 01:19:18 bunnybot ollama[1114]: 2024/01/10 01:19:18 routes.go:953: no GPU detected
Jan 10 01:20:52 bunnybot systemd[1]: Stopping Ollama Service...
Jan 10 01:20:52 bunnybot systemd[1]: ollama.service: Succeeded.
Jan 10 01:20:52 bunnybot systemd[1]: Stopped Ollama Service.
image

I would post the text from the tmuxed ollama_jetson window but honestly other than the service stop message it is identical. In the past I have used other tools to run Jetson CUDA optimized LLMs and they were much faster, but required more work and time converting LLMs to get working so I was excited to try ollama as we have been toying with integrating various other off the shelf tools and having the ability to test many models is very tempting. So no matter what thank you!

<!-- gh-comment-id:1884544164 --> @carolynhudson commented on GitHub (Jan 10, 2024): Small update, built a new Ubuntu box and setup the jetson orin nx with Ubuntu 20 and Jetpack 5.1.2, I even built it from a new VM made from a fresh Ubuntu 20 Desktop iso. After all setup was complete, apt update/upgrade run, rebooted, and ran the instructions from: https://github.com/jmorganca/ollama/blob/main/docs/tutorials/nvidia-jetson.md It does not seem to be respecting the LD_LIBRARY_PATH as specified. This is the log from the ollama service added to the system: Jan 10 01:19:17 bunnybot systemd[1]: Started Ollama Service. Jan 10 01:19:17 bunnybot ollama[1114]: 2024/01/10 01:19:17 images.go:808: total blobs: 0 Jan 10 01:19:17 bunnybot ollama[1114]: 2024/01/10 01:19:17 images.go:815: total unused blobs removed: 0 Jan 10 01:19:17 bunnybot ollama[1114]: 2024/01/10 01:19:17 routes.go:930: Listening on 127.0.0.1:11434 (version 0.1.19) Jan 10 01:19:18 bunnybot ollama[1114]: 2024/01/10 01:19:18 shim_ext_server.go:142: Dynamic LLM variants [cuda] Jan 10 01:19:18 bunnybot ollama[1114]: 2024/01/10 01:19:18 gpu.go:35: Detecting GPU type Jan 10 01:19:18 bunnybot ollama[1114]: 2024/01/10 01:19:18 gpu.go:40: CUDA not detected: Unable to load libnvidia-ml.so library to query for Nvidia GPUs: /usr/lib/wsl/lib/libnvidia-ml.so.1: cannot open shared object file: No such file or directory Jan 10 01:19:18 bunnybot ollama[1114]: 2024/01/10 01:19:18 gpu.go:46: ROCm not detected: Unable to load librocm_smi64.so library to query for Radeon GPUs: /opt/rocm/lib/librocm_smi64.so: cannot open shared object file: No such file or directory Jan 10 01:19:18 bunnybot ollama[1114]: 2024/01/10 01:19:18 routes.go:953: no GPU detected Jan 10 01:20:52 bunnybot systemd[1]: Stopping Ollama Service... Jan 10 01:20:52 bunnybot systemd[1]: ollama.service: Succeeded. Jan 10 01:20:52 bunnybot systemd[1]: Stopped Ollama Service. ![image](https://github.com/jmorganca/ollama/assets/59717105/1ab5e30a-4452-4bcf-a750-94996bc221ab) I would post the text from the tmuxed ollama_jetson window but honestly other than the service stop message it is identical. In the past I have used other tools to run Jetson CUDA optimized LLMs and they were much faster, but required more work and time converting LLMs to get working so I was excited to try ollama as we have been toying with integrating various other off the shelf tools and having the ability to test many models is very tempting. So no matter what thank you!
GiteaMirror commented 2026-05-03 12:12:17 -05:00
Author
Owner
Copy Link

@carolynhudson commented on GitHub (Jan 11, 2024):

Thanks!

<!-- gh-comment-id:1886077707 --> @carolynhudson commented on GitHub (Jan 11, 2024): Thanks!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#63114
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?