github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-22 05:31:43 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #1091] ollama install messed the CUDA setup, ollama unable to use CUDA #541

New Issue
Closed
opened 2026-04-12 10:13:47 -05:00 by GiteaMirror · 13 comments
GiteaMirror commented 2026-04-12 10:13:47 -05:00
Owner
Copy Link

Originally created by @ArsBinarii on GitHub (Nov 11, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1091

Originally assigned to: @dhiltgen on GitHub.

Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

downloaded and installed:

https://developer.nvidia.com/cuda-12-0-0-download-archive

GPU: 1080

after cuda install, nvidia-smi reports:
NVIDIA-SMI 525.60.13 Driver Version: 525.60.13 CUDA Version: 12.0

installed ollama: curl https://ollama.ai/install.sh | sh
strange message appears: >>> NVIDIA GPU installed.
run dophin2.2: ollama run dolphin2.2-mistral
very very low performance
check nvidia-smi: No devices were found
reinstall CUDA from: https://developer.nvidia.com/cuda-12-0-0-download-archive
check nvidia-smi: No devices were found

Originally created by @ArsBinarii on GitHub (Nov 11, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1091 Originally assigned to: @dhiltgen on GitHub. Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy downloaded and installed: https://developer.nvidia.com/cuda-12-0-0-download-archive GPU: 1080 after cuda install, nvidia-smi reports: NVIDIA-SMI 525.60.13 Driver Version: 525.60.13 CUDA Version: 12.0 installed ollama: curl https://ollama.ai/install.sh | sh strange message appears: >>> NVIDIA GPU installed. run dophin2.2: ollama run dolphin2.2-mistral very very low performance check nvidia-smi: No devices were found reinstall CUDA from: https://developer.nvidia.com/cuda-12-0-0-download-archive check nvidia-smi: No devices were found
GiteaMirror added the bugnvidia labels 2026-04-12 10:13:47 -05:00
GiteaMirror commented 2026-04-12 10:13:48 -05:00
Author
Owner
Copy Link

@ArsBinarii commented on GitHub (Nov 11, 2023):

confirmed works with:
NVIDIA-SMI 545.23.06 Driver Version: 545.23.06 CUDA Version: 12.3

it seems there is no CUDA version detect, which is fine BUT:

  1. an error message should be presented to the user telling that ollama requirements are CUDA version X and that the system has installed version Y
  2. NEVER EVER EVER BREAK THE CUDA ENV/SETUP ON THE USER'S MACHINE.
    and I mean break, I did a purge, remove via the nvidia run file, then reinstalled, nvidia-smi and all cuda integration was DONE. I had to redo the VM machine and this time I snapshot it
<!-- gh-comment-id:1806893716 --> @ArsBinarii commented on GitHub (Nov 11, 2023): confirmed works with: NVIDIA-SMI 545.23.06 Driver Version: 545.23.06 CUDA Version: 12.3 it seems there is no CUDA version detect, which is fine BUT: 1) an error message should be presented to the user telling that ollama requirements are CUDA version X and that the system has installed version Y 2) NEVER EVER EVER BREAK THE CUDA ENV/SETUP ON THE USER'S MACHINE. and I mean break, I did a purge, remove via the nvidia run file, then reinstalled, nvidia-smi and all cuda integration was DONE. I had to redo the VM machine and this time I snapshot it
GiteaMirror commented 2026-04-12 10:13:48 -05:00
Author
Owner
Copy Link

@chymian commented on GitHub (Nov 12, 2023):

2. NEVER EVER EVER BREAK THE CUDA ENV/SETUP ON THE USERS MACHINE.
and I mean break, I did a purge, remove via the nvidia run file, then reinstalled, nvidia-smi and all cuda integration was DONE. I had to redo the VM machine and this time I snapshot it

I totally agrree. you are an app, not devops. don't E.V.E.R. tinker with the systemsetup outside of your venv!
you are supposed to deliver a service, not brake a system.

<!-- gh-comment-id:1807040842 --> @chymian commented on GitHub (Nov 12, 2023): > 2\. NEVER EVER EVER BREAK THE CUDA ENV/SETUP ON THE USERS MACHINE. > and I mean break, I did a purge, remove via the nvidia run file, then reinstalled, nvidia-smi and all cuda integration was DONE. I had to redo the VM machine and this time I snapshot it I totally agrree. you are an app, not devops. don't E.V.E.R. tinker with the systemsetup outside of your venv! you are supposed to deliver a service, not brake a system.
GiteaMirror commented 2026-04-12 10:13:48 -05:00
Author
Owner
Copy Link

@orlyandico commented on GitHub (Nov 17, 2023):

Had a similar issue but on WSL on Windows 11 with Nvidia 535 drivers and CUDA 12.2 - after running for a little while everything would lock up (even "nvidia-smi" would lock hard and become unkillable).

It did not destroy the cuda installation though (can't see how that can happen) and wsl --shutdown killed off the stuck processes.

Upgrading to the Nvidia driver 546 (on the host OS) and Cuda 12.3 (in the Ubuntu guest) fixed the problem.

<!-- gh-comment-id:1815590624 --> @orlyandico commented on GitHub (Nov 17, 2023): Had a similar issue but on WSL on Windows 11 with Nvidia 535 drivers and CUDA 12.2 - after running for a little while everything would lock up (even "nvidia-smi" would lock hard and become unkillable). It did not destroy the cuda installation though (can't see how that can happen) and wsl --shutdown killed off the stuck processes. Upgrading to the Nvidia driver 546 (on the host OS) and Cuda 12.3 (in the Ubuntu guest) fixed the problem.
GiteaMirror commented 2026-04-12 10:13:49 -05:00
Author
Owner
Copy Link

@ArsBinarii commented on GitHub (Nov 17, 2023):

I had that before when messing with different versions of CUDA and Nvidia drivers, nvidia-smi would get stuck permanently, from my experience it is a incompatibility between CUDA and driver, or driver and GPU.

<!-- gh-comment-id:1815664652 --> @ArsBinarii commented on GitHub (Nov 17, 2023): I had that before when messing with different versions of CUDA and Nvidia drivers, nvidia-smi would get stuck permanently, from my experience it is a incompatibility between CUDA and driver, or driver and GPU.
GiteaMirror commented 2026-04-12 10:13:50 -05:00
Author
Owner
Copy Link

@wijjj commented on GitHub (Nov 24, 2023):

same here with ubuntu 22.04
ollama suddenly stopped using GPU between a system reboot. I really can't tell what changed in the meantime. Except there's a .ollama directory in the user home, but also ollama in /usr/share.

<!-- gh-comment-id:1825049707 --> @wijjj commented on GitHub (Nov 24, 2023): same here with ubuntu 22.04 ollama suddenly stopped using GPU between a system reboot. I really can't tell what changed in the meantime. Except there's a `.ollama` directory in the user home, but also `ollama` in `/usr/share`.
GiteaMirror commented 2026-04-12 10:13:50 -05:00
Author
Owner
Copy Link

@ArsBinarii commented on GitHub (Nov 24, 2023):

do a watch (half a sec) on nvidia-smi and see if the python process shows. did something update, check nvidia-smi versions ? can you redo the process/setup ?

<!-- gh-comment-id:1825268107 --> @ArsBinarii commented on GitHub (Nov 24, 2023): do a watch (half a sec) on nvidia-smi and see if the python process shows. did something update, check nvidia-smi versions ? can you redo the process/setup ?
GiteaMirror commented 2026-04-12 10:13:51 -05:00
Author
Owner
Copy Link

@anashr18 commented on GitHub (Nov 24, 2023):

Post Ollama setup and experiment with different models in Ollama, multiple versions of cuda installation are found in the VM .
12K lib
28K share
7.8G cuda-11.7
712K dcgm
9.6G cuda-12.2
9.9G cuda-12.0
9.9G cuda-11.8
0 cuda
9.6G cuda-12.1
395M bin

Do model require different version of them?

<!-- gh-comment-id:1825307052 --> @anashr18 commented on GitHub (Nov 24, 2023): Post Ollama setup and experiment with different models in Ollama, multiple versions of cuda installation are found in the VM . 12K lib 28K share 7.8G cuda-11.7 712K dcgm 9.6G cuda-12.2 9.9G cuda-12.0 9.9G cuda-11.8 0 cuda 9.6G cuda-12.1 395M bin Do model require different version of them?
GiteaMirror commented 2026-04-12 10:13:51 -05:00
Author
Owner
Copy Link

@ArsBinarii commented on GitHub (Nov 25, 2023):

no, cuda version can be found via running nvidia-smi, 1 cuda version will be used for all models as the llama does the integration with cuda; models have no idea what to run, more or less the model describes the neural net arch and the weights, llama uses cuda to load the model in the GPU and run it

<!-- gh-comment-id:1826410309 --> @ArsBinarii commented on GitHub (Nov 25, 2023): no, cuda version can be found via running nvidia-smi, 1 cuda version will be used for all models as the llama does the integration with cuda; models have no idea what to run, more or less the model describes the neural net arch and the weights, llama uses cuda to load the model in the GPU and run it
GiteaMirror commented 2026-04-12 10:13:51 -05:00
Author
Owner
Copy Link

@siikdUde commented on GitHub (Dec 22, 2023):

I got ollama to start using my rtx 4090 by:

  1. Uninstalling Ubuntu
  2. Uninstalling WSL
  3. Reboot
  4. Installing WSL
  5. Installing Ubuntu
  6. (Crucial Part): Basically this is optional for you but it makes the process streamlined:
  • Installed oobabooga via the one click installer for WSL in my root folder.
  • Input all the values for my system and such (such as specifying I have an nvidia GPU) and it went ahead and downloaded all CUDA drivers, toolkit, pytorch and all other dependencies.
  • Again, this part is optional as it is for installing oobabooga, but as a welcomed side effect, it installed everything I needed to get Ollama working with my GPU. As a result, my GPU usage now is between 40% - 100% and CPU around 60% while the model is working. Before it was at 0% with my CPU being at around 70%.

Also, it installs the 12.1 version of the toolkit, which I believe is the one that works (at least for me). When I updated to 12.3, my GPU stopped working with Ollama, so be mindful of that.

Hope this helps anyone that comes across this thread.

<!-- gh-comment-id:1868109979 --> @siikdUde commented on GitHub (Dec 22, 2023): I got ollama to start using my rtx 4090 by: 1. Uninstalling Ubuntu 2. Uninstalling WSL 3. Reboot 4. Installing WSL 5. Installing Ubuntu 6. (Crucial Part): Basically this is optional for you but it makes the process streamlined: - Installed oobabooga via the one click installer for WSL in my root folder. - Input all the values for my system and such (such as specifying I have an nvidia GPU) and it went ahead and downloaded all CUDA drivers, toolkit, pytorch and all other dependencies. - Again, this part is optional as it is for installing oobabooga, but as a welcomed side effect, it installed everything I needed to get Ollama working with my GPU. As a result, my GPU usage now is between 40% - 100% and CPU around 60% while the model is working. Before it was at 0% with my CPU being at around 70%. Also, it installs the 12.1 version of the toolkit, which I believe is the one that works (at least for me). When I updated to 12.3, my GPU stopped working with Ollama, so be mindful of that. Hope this helps anyone that comes across this thread.
GiteaMirror commented 2026-04-12 10:13:52 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Jan 14, 2024):

Hi folks, sorry about this. The installer needs to be better at respecting and not touching existing CUDA setups (vs attempting to interfere or upgrade)

<!-- gh-comment-id:1891054362 --> @jmorganca commented on GitHub (Jan 14, 2024): Hi folks, sorry about this. The installer needs to be better at respecting and not touching existing CUDA setups (vs attempting to interfere or upgrade)
GiteaMirror commented 2026-04-12 10:13:53 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 12, 2024):

Is this still a problem for folks? The install script does attempt to discover an existing functional nvidia-smi and only install if not detected. Could someone who has seen this error run the installer with sh -x install.sh so we can see more details on why these checks aren't working as intended? One possible thing I can think of is maybe in these cases nvidia-smi isn't in the users path when they run the script? If so, we could enhance the script to check the path, and if that doesn't work, check ~standard location(s) where cuda gets installed too.

<!-- gh-comment-id:1992008055 --> @dhiltgen commented on GitHub (Mar 12, 2024): Is this still a problem for folks? The install script does attempt to discover an existing functional nvidia-smi and only install if not detected. Could someone who has seen this error run the installer with `sh -x install.sh` so we can see more details on why these checks aren't working as intended? One possible thing I can think of is maybe in these cases `nvidia-smi` isn't in the users path when they run the script? If so, we could enhance the script to check the path, and if that doesn't work, check ~standard location(s) where cuda gets installed too.
GiteaMirror commented 2026-04-12 10:13:54 -05:00
Author
Owner
Copy Link

@wijjj commented on GitHub (Mar 12, 2024):

Is this still a problem for folks? The install script does attempt to discover an existing functional nvidia-smi and only install if not detected. Could someone who has seen this error run the installer with sh -x install.sh so we can see more details on why these checks aren't working as intended? One possible thing I can think of is maybe in these cases nvidia-smi isn't in the users path when they run the script? If so, we could enhance the script to check the path, and if that doesn't work, check ~standard location(s) where cuda gets installed too.

I guess it's fine now with the newer default installation script. Installed it a few more times on various systems. Albeit reluctant because in the past it fucked up existing Ubuntu 22 CUDA (systemwide) installations. But everything seems to be fine now. Also GPUs are used as expected.

<!-- gh-comment-id:1992627419 --> @wijjj commented on GitHub (Mar 12, 2024): > Is this still a problem for folks? The install script does attempt to discover an existing functional nvidia-smi and only install if not detected. Could someone who has seen this error run the installer with `sh -x install.sh` so we can see more details on why these checks aren't working as intended? One possible thing I can think of is maybe in these cases `nvidia-smi` isn't in the users path when they run the script? If so, we could enhance the script to check the path, and if that doesn't work, check ~standard location(s) where cuda gets installed too. I guess it's fine now with the newer default installation script. Installed it a few more times on various systems. Albeit reluctant because in the past it fucked up existing Ubuntu 22 CUDA (systemwide) installations. But everything seems to be fine now. Also GPUs are used as expected.
GiteaMirror commented 2026-04-12 10:13:55 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 12, 2024):

Sounds like we can close this as resolved.

<!-- gh-comment-id:2052611362 --> @dhiltgen commented on GitHub (Apr 12, 2024): Sounds like we can close this as resolved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug nvidia
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#541
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?