github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #2670] Build Cuda ready Docker image #79265

New Issue
Closed
opened 2026-05-09 04:36:17 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-05-09 04:36:17 -05:00
Owner
Copy Link

Originally created by @thiner on GitHub (Feb 22, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2670

Originally assigned to: @dhiltgen on GitHub.

Currently, the official ollama container image doesn't contain necessary cuda libraries. This is really inconvenient when run it on server. I see you have provided [rocm] images for AMD gpus, can you also provide cuda ready images? If that's not feasible, how about provide the specific Dockerfile?

Originally created by @thiner on GitHub (Feb 22, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2670 Originally assigned to: @dhiltgen on GitHub. Currently, the official ollama container image doesn't contain necessary cuda libraries. This is really inconvenient when run it on server. I see you have provided [rocm] images for AMD gpus, can you also provide cuda ready images? If that's not feasible, how about provide the specific Dockerfile?
GiteaMirror added the question label 2026-05-09 04:36:17 -05:00
GiteaMirror commented 2026-05-09 04:36:27 -05:00
Author
Owner
Copy Link

@aaronnewsome commented on GitHub (Feb 22, 2024):

I'm using Ollama container "ollama/ollama:0.1.26" and cuda libraries are in there. Make sure you've installed Nvidia container runtime before starting Ollama.

<!-- gh-comment-id:1959510717 --> @aaronnewsome commented on GitHub (Feb 22, 2024): I'm using Ollama container "ollama/ollama:0.1.26" and cuda libraries are in there. Make sure you've installed Nvidia container runtime before starting Ollama.
GiteaMirror commented 2026-05-09 04:36:29 -05:00
Author
Owner
Copy Link

@thiner commented on GitHub (Feb 23, 2024):

@aaronnewsome yes, there is nvidia-smi command in the docker image. But it lacks of other libraries. You can simply compare the size of the images between current standard and ROCM images. A container contains those runtime libraries is quite obvious, its size usually over 2GB.
The reason you can run the standard image locally, very likely you installed those dependencies in host machine, but that's not a good practice for server environment.

<!-- gh-comment-id:1960664594 --> @thiner commented on GitHub (Feb 23, 2024): @aaronnewsome yes, there is `nvidia-smi` command in the docker image. But it lacks of other libraries. You can simply compare the size of the images between current standard and ROCM images. A container contains those runtime libraries is quite obvious, its size usually over 2GB. The reason you can run the standard image locally, very likely you installed those dependencies in host machine, but that's not a good practice for server environment.
GiteaMirror commented 2026-05-09 04:36:31 -05:00
Author
Owner
Copy Link

@mxyng commented on GitHub (Feb 23, 2024):

rocm libraries are ridiculously large. cuda is much more reasonable. using cuda in docker requires nvidia-container-toolkit and the container must be started with --gpus flag. these two prerequisites with the ollama/ollama image should give you acceleration out of the box.

image size is precisely why there's a separate rocm docker image. we originally wanted a single image which can handle both cpu, cuda, and rocm but the final image was way too large. the original docker image was 200-400MB. the additional rocm requirements bumped that up to 2GB which we felt was a significant and unacceptable bump in image size. especially since most users will want one of cuda and rocm, never both.

<!-- gh-comment-id:1961747366 --> @mxyng commented on GitHub (Feb 23, 2024): rocm libraries are ridiculously large. cuda is much more reasonable. using cuda in docker requires nvidia-container-toolkit and the container must be started with `--gpus` flag. these two prerequisites with the `ollama/ollama` image should give you acceleration out of the box. image size is precisely why there's a separate rocm docker image. we originally wanted a single image which can handle both cpu, cuda, and rocm but the final image was way too large. the original docker image was 200-400MB. the additional rocm requirements bumped that up to 2GB which we felt was a significant and unacceptable bump in image size. especially since most users will want one of cuda and rocm, never both.
GiteaMirror commented 2026-05-09 04:36:36 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 11, 2024):

@thiner our ollama/ollama image should work on container systems that have the nvidia container runtime installed and configured. The CUDA v11 libraries are currently embedded within the ollama linux binary and are extracted at runtime. The discovery of the GPU is through the nvidia management library which is supposed to be mounted from the host system when things are set up properly. If you're seeing the container fail to discover the GPUs, that most likely means the nvidia container runtime isn't properly configured.

All that said, we did fix a bug a few weeks back where we weren't setting all the env vars necessary for things to work on all container runtimes. My suspicion is you may have hit that defect, and if you try the latest image it should work.

If you're still seeing problems, please share more information about what container runtime you're using (Docker, podman, etc.) and if you've followed the nvidia container runtime setup guide, and a server log so we can see what's going wrong. (setting -e OLLAMA_DEBUG=1 will yield more verbose logs and help troubleshoot)

<!-- gh-comment-id:1989567739 --> @dhiltgen commented on GitHub (Mar 11, 2024): @thiner our `ollama/ollama` image should work on container systems that have the nvidia container runtime installed and configured. The CUDA v11 libraries are currently embedded within the ollama linux binary and are extracted at runtime. The discovery of the GPU is through the nvidia management library which is supposed to be mounted from the host system when things are set up properly. If you're seeing the container fail to discover the GPUs, that most likely means the nvidia container runtime isn't properly configured. All that said, we did fix a bug a few weeks back where we weren't setting all the env vars necessary for things to work on all container runtimes. My suspicion is you may have hit that defect, and if you try the latest image it should work. If you're still seeing problems, please share more information about what container runtime you're using (Docker, podman, etc.) and if you've followed the nvidia container runtime setup guide, and a server log so we can see what's going wrong. (setting `-e OLLAMA_DEBUG=1` will yield more verbose logs and help troubleshoot)
GiteaMirror commented 2026-05-09 04:36:39 -05:00
Author
Owner
Copy Link

@thiner commented on GitHub (Mar 12, 2024):

@dhiltgen Daniel, thanks for your clarification. I just tried latest docker image(v0.1.28), and it's working now.

<!-- gh-comment-id:1990167390 --> @thiner commented on GitHub (Mar 12, 2024): @dhiltgen Daniel, thanks for your clarification. I just tried latest docker image(v0.1.28), and it's working now.
GiteaMirror commented 2026-05-09 04:36:43 -05:00
Author
Owner
Copy Link

@Pathsis commented on GitHub (May 4, 2024):

@thiner our ollama/ollama image should work on container systems that have the nvidia container runtime installed and configured. The CUDA v11 libraries are currently embedded within the ollama linux binary and are extracted at runtime. The discovery of the GPU is through the nvidia management library which is supposed to be mounted from the host system when things are set up properly. If you're seeing the container fail to discover the GPUs, that most likely means the nvidia container runtime isn't properly configured.

All that said, we did fix a bug a few weeks back where we weren't setting all the env vars necessary for things to work on all container runtimes. My suspicion is you may have hit that defect, and if you try the latest image it should work.

If you're still seeing problems, please share more information about what container runtime you're using (Docker, podman, etc.) and if you've followed the nvidia container runtime setup guide, and a server log so we can see what's going wrong. (setting -e OLLAMA_DEBUG=1 will yield more verbose logs and help troubleshoot)

Hi @dhiltgen Does this mean that ollama cannot be used with other versions of CUDA? The CUDA version installed in my Ubuntu machine is 12.2, however, I saw in nividia-smi that ollama uses cuda_v11. I also see that there are variables about cuda-12.2 in ollama.service:

Environment="PATH=/home/midtail/.local/share/pnpm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/usr/local/cuda-12.2/bin"

Thank you so much!

<!-- gh-comment-id:2094206164 --> @Pathsis commented on GitHub (May 4, 2024): > @thiner our `ollama/ollama` image should work on container systems that have the nvidia container runtime installed and configured. The CUDA v11 libraries are currently embedded within the ollama linux binary and are extracted at runtime. The discovery of the GPU is through the nvidia management library which is supposed to be mounted from the host system when things are set up properly. If you're seeing the container fail to discover the GPUs, that most likely means the nvidia container runtime isn't properly configured. > > All that said, we did fix a bug a few weeks back where we weren't setting all the env vars necessary for things to work on all container runtimes. My suspicion is you may have hit that defect, and if you try the latest image it should work. > > If you're still seeing problems, please share more information about what container runtime you're using (Docker, podman, etc.) and if you've followed the nvidia container runtime setup guide, and a server log so we can see what's going wrong. (setting `-e OLLAMA_DEBUG=1` will yield more verbose logs and help troubleshoot) Hi @dhiltgen Does this mean that ollama cannot be used with other versions of CUDA? The CUDA version installed in my Ubuntu machine is 12.2, however, I saw in `nividia-smi` that ollama uses cuda_v11. I also see that there are variables about cuda-12.2 in ollama.service: ```shell Environment="PATH=/home/midtail/.local/share/pnpm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/usr/local/cuda-12.2/bin" ``` Thank you so much!
GiteaMirror commented 2026-05-09 04:36:49 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 4, 2024):

Newer drivers are backwards compatible with older cuda libraries. We currently build our official binaries with v11 to maximize compatibility across GPUs and driver versions. If our container image does not detect your GPU, please open a new issue and include server logs so we can investigate why.

<!-- gh-comment-id:2094276861 --> @dhiltgen commented on GitHub (May 4, 2024): Newer drivers are backwards compatible with older cuda libraries. We currently build our official binaries with v11 to maximize compatibility across GPUs and driver versions. If our container image does not detect your GPU, please open a new issue and include server logs so we can investigate why.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#79265
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?