github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 08:02:14 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #1071] Enhancement: Enable GPU Support for NVIDIA JetPack and the NVIDIA Jetson Lineup #62565

New Issue
Closed
opened 2026-05-03 09:35:14 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-05-03 09:35:14 -05:00
Owner
Copy Link

Originally created by @bnodnarb on GitHub (Nov 10, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1071

Hi all,

I recently purchased an NVIDIA Jetson Orin Developer Kit and am hoping to get Ollama running on it.

These little powerhouses are specifically built for AI applications and they have a ton of capability crammed into a tiny form factor.

The Jetson devices run a flavor of Linux called JetPack, which is also packed with AI features. Description of JetPack below:

NVIDIA JetPack SDK is the most comprehensive solution for building end-to-end accelerated AI applications. JetPack provides a full development environment for hardware-accelerated AI-at-the-edge development on Nvidia Jetson modules. JetPack includes Jetson Linux with bootloader, Linux kernel, Ubuntu desktop environment, and a complete set of libraries for acceleration of GPU computing, multimedia, graphics, and computer vision.

Ollama works in a "CPU only" fashion when installed via the standard curl https://ollama.ai/install.sh | sh command when running on JetPack, but it seems to ignore all of the GPU power that is available on the Jetson board.

It would be awesome if we could get Ollama running at full speed on these devices.

Does anyone know where we could begin? I'm happy to help, but don't really know where to start.

Originally created by @bnodnarb on GitHub (Nov 10, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1071 Hi all, I recently purchased an [NVIDIA Jetson Orin Developer Kit](https://store.nvidia.com/en-us/jetson/store/?page=1&limit=9&locale=en-us) and am hoping to get Ollama running on it. These little powerhouses are specifically built for AI applications and they have a ton of capability crammed into a tiny form factor. The Jetson devices run a flavor of Linux called [JetPack](https://developer.nvidia.com/embedded/jetpack), which is also packed with AI features. Description of JetPack below: > NVIDIA JetPack SDK is the most comprehensive solution for building end-to-end accelerated AI applications. JetPack provides a full development environment for hardware-accelerated AI-at-the-edge development on Nvidia Jetson modules. JetPack includes [Jetson Linux](https://developer.nvidia.com/embedded/jetson-linux) with bootloader, Linux kernel, Ubuntu desktop environment, and a complete set of libraries for acceleration of GPU computing, multimedia, graphics, and computer vision. Ollama works in a "CPU only" fashion when installed via the standard `curl https://ollama.ai/install.sh | sh` command when running on JetPack, but it seems to ignore all of the GPU power that is available on the Jetson board. It would be awesome if we could get Ollama running at full speed on these devices. Does anyone know where we could begin? I'm happy to help, but don't really know where to start.
GiteaMirror commented 2026-05-03 09:35:15 -05:00
Author
Owner
Copy Link

@65a commented on GitHub (Nov 10, 2023):

Does it work in ggerganov/llama.cpp with acceleration?

<!-- gh-comment-id:1805156015 --> @65a commented on GitHub (Nov 10, 2023): Does it work in ggerganov/llama.cpp with acceleration?
GiteaMirror commented 2026-05-03 09:35:16 -05:00
Author
Owner
Copy Link

@bnodnarb commented on GitHub (Nov 10, 2023):

I know that @dusty-nv from NVIDIA got llama.cpp working and makes it available in a Jetson Docker Container.

I'm not sure what tweaks needed to be made, but it does mean that llama.cpp can be made to run with CUDA enabled.

<!-- gh-comment-id:1805170651 --> @bnodnarb commented on GitHub (Nov 10, 2023): I know that @dusty-nv from NVIDIA got llama.cpp working and makes it available in a [Jetson Docker Container](https://github.com/dusty-nv/jetson-containers/tree/master/packages/llm/llama_cpp). I'm not sure what tweaks needed to be made, but it does mean that llama.cpp can be made to run with CUDA enabled.
GiteaMirror commented 2026-05-03 09:35:16 -05:00
Author
Owner
Copy Link

@bnodnarb commented on GitHub (Nov 10, 2023):

This is the output that I get after running the Linux install script:

>>> Downloading ollama...
######################################################################## 100.0%##O=#  #                                                                      
>>> Installing ollama to /usr/local/bin...
>>> Creating ollama user...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
>>> Enabling and starting ollama service...
Created symlink /etc/systemd/system/default.target.wants/ollama.service → /etc/systemd/system/ollama.service.
>>> Installing NVIDIA repository...
curl: (22) The requested URL returned error: 404

CUDA libraries are already installed at /usr/local/cuda/lib64, so I'm trying to integrate it with ollama serve as shown below

$ LD_LIBRARY_PATH=/usr/local/cuda/lib64 ollama serve
2023/11/09 22:53:28 images.go:824: total blobs: 0
2023/11/09 22:53:28 images.go:831: total unused blobs removed: 0
2023/11/09 22:53:28 routes.go:680: Listening on 127.0.0.1:11434 (version 0.1.8)
2023/11/09 22:53:28 routes.go:700: Warning: GPU support may not be enabled, check you have installed GPU drivers: nvidia-smi command failed

I don't exactly know how to test to see if it's working without ising ollama run

<!-- gh-comment-id:1805263035 --> @bnodnarb commented on GitHub (Nov 10, 2023): This is the output that I get after running the Linux install script: ``` >>> Downloading ollama... ######################################################################## 100.0%##O=# # >>> Installing ollama to /usr/local/bin... >>> Creating ollama user... >>> Adding current user to ollama group... >>> Creating ollama systemd service... >>> Enabling and starting ollama service... Created symlink /etc/systemd/system/default.target.wants/ollama.service → /etc/systemd/system/ollama.service. >>> Installing NVIDIA repository... curl: (22) The requested URL returned error: 404 ``` CUDA libraries are already installed at `/usr/local/cuda/lib64`, so I'm trying to integrate it with ollama serve as shown below ``` $ LD_LIBRARY_PATH=/usr/local/cuda/lib64 ollama serve 2023/11/09 22:53:28 images.go:824: total blobs: 0 2023/11/09 22:53:28 images.go:831: total unused blobs removed: 0 2023/11/09 22:53:28 routes.go:680: Listening on 127.0.0.1:11434 (version 0.1.8) 2023/11/09 22:53:28 routes.go:700: Warning: GPU support may not be enabled, check you have installed GPU drivers: nvidia-smi command failed ``` I don't exactly know how to test to see if it's working without ising `ollama run`
GiteaMirror commented 2026-05-03 09:35:18 -05:00
Author
Owner
Copy Link

@bnodnarb commented on GitHub (Nov 10, 2023):

Eureka! I was able to get this to work by combining a handful of tickets. I'll create a pull request with documented instructions on how to get it up and running :)

<!-- gh-comment-id:1805465000 --> @bnodnarb commented on GitHub (Nov 10, 2023): Eureka! I was able to get this to work by combining a handful of tickets. I'll create a pull request with documented instructions on how to get it up and running :)
GiteaMirror commented 2026-05-03 09:35:20 -05:00
Author
Owner
Copy Link

@jp2000 commented on GitHub (Nov 10, 2023):

Cant wait, just ran into this with mine.
llama.cpp did work great for me with no changes, but yeah ollama is stuck in cpu-only mode.

<!-- gh-comment-id:1806477866 --> @jp2000 commented on GitHub (Nov 10, 2023): Cant wait, just ran into this with mine. llama.cpp did work great for me with no changes, but yeah ollama is stuck in cpu-only mode.
GiteaMirror commented 2026-05-03 09:35:21 -05:00
Author
Owner
Copy Link

@bnodnarb commented on GitHub (Nov 11, 2023):

@jp2000 - working on the PR now. Will send a note and link once it's submitted. Thanks!

<!-- gh-comment-id:1806755186 --> @bnodnarb commented on GitHub (Nov 11, 2023): @jp2000 - working on the PR now. Will send a note and link once it's submitted. Thanks!
GiteaMirror commented 2026-05-03 09:35:22 -05:00
Author
Owner
Copy Link

@bnodnarb commented on GitHub (Nov 12, 2023):

@jp2000 - working on the PR now. Will send a note and link once it's submitted. Thanks!

Hi all - please see submitted pull request here: https://github.com/jmorganca/ollama/pull/1098

Closing this issue :)

Thanks!

<!-- gh-comment-id:1807092874 --> @bnodnarb commented on GitHub (Nov 12, 2023): > @jp2000 - working on the PR now. Will send a note and link once it's submitted. Thanks! Hi all - please see submitted pull request here: https://github.com/jmorganca/ollama/pull/1098 Closing this issue :) Thanks!
GiteaMirror commented 2026-05-03 09:35:23 -05:00
Author
Owner
Copy Link

@mraiser commented on GitHub (Nov 12, 2023):

@jp2000 - working on the PR now. Will send a note and link once it's submitted. Thanks!

Hi all - please see submitted pull request here: #1098

Closing this issue :)

Thanks!

Can confirm... This works great on 8GB Jetson Orin Nano with Jetpack 5.1.2 on Ubuntu 20.04. Thank you!!!

<!-- gh-comment-id:1807134437 --> @mraiser commented on GitHub (Nov 12, 2023): > > @jp2000 - working on the PR now. Will send a note and link once it's submitted. Thanks! > > Hi all - please see submitted pull request here: #1098 > > Closing this issue :) > > Thanks! Can confirm... This works great on 8GB Jetson Orin Nano with Jetpack 5.1.2 on Ubuntu 20.04. Thank you!!!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#62565
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?