github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #3321] request to add flag to ollama serve to #27801

New Issue
Closed
opened 2026-04-22 05:23:51 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-22 05:23:51 -05:00
Owner
Copy Link

Originally created by @straydog-1212 on GitHub (Mar 24, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3321

Originally assigned to: @dhiltgen on GitHub.

What are you trying to do?

my GPU is unsupported and I used to run it in CPU-only mode. After reinstalling, however, it thinks my 6700 is a Nvidia card and won't let me run Ollama serve.

`ollama serve  ✔  23s 
time=2024-03-23T22:00:19.302-07:00 level=INFO source=images.go:806 msg="total blobs: 0"
time=2024-03-23T22:00:19.302-07:00 level=INFO source=images.go:813 msg="total unused blobs removed: 0"
time=2024-03-23T22:00:19.302-07:00 level=INFO source=routes.go:1110 msg="Listening on 127.0.0.1:11434 (version 0.1.29)"
time=2024-03-23T22:00:19.302-07:00 level=INFO source=payload_common.go:112 msg="Extracting dynamic libraries to /tmp/ollama156389064/runners ..."
time=2024-03-23T22:00:21.954-07:00 level=INFO source=payload_common.go:139 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60000]"
time=2024-03-23T22:00:21.954-07:00 level=INFO source=gpu.go:77 msg="Detecting GPU type"
time=2024-03-23T22:00:21.955-07:00 level=INFO source=gpu.go:191 msg="Searching for GPU management library libnvidia-ml.so"
time=2024-03-23T22:00:21.980-07:00 level=INFO source=gpu.go:237 msg="Discovered GPU libraries: [/usr/lib/libnvidia-ml.so.550.54.14 /usr/lib64/libnvidia-ml.so.550.54.14 /opt/cuda/targets/x86_64-linux/lib/stubs/libnvidia-ml.so]"
time=2024-03-23T22:00:21.996-07:00 level=INFO source=gpu.go:249 msg="Unable to load CUDA management library /usr/lib/libnvidia-ml.so.550.54.14: nvml vram init failure: 9"
time=2024-03-23T22:00:22.011-07:00 level=INFO source=gpu.go:249 msg="Unable to load CUDA management library /usr/lib64/libnvidia-ml.so.550.54.14: nvml vram init failure: 9"

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
WARNING:

You should always run with libnvidia-ml.so that is installed with your
NVIDIA Display Driver. By default it's installed in /usr/lib and /usr/lib64.
libnvidia-ml.so in GDK package is a stub library that is attached only for
build purposes (e.g. machine that you build your application doesn't have
to have Display Driver installed).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Linked to libnvidia-ml library at wrong path : /opt/cuda/targets/x86_64-linux/lib/stubs/libnvidia-ml.so

time=2024-03-23T22:00:22.054-07:00 level=INFO source=gpu.go:249 msg="Unable to load CUDA management library /opt/cuda/targets/x86_64-linux/lib/stubs/libnvidia-ml.so: nvml vram init failure: 9"
time=2024-03-23T22:00:22.054-07:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-23T22:00:22.054-07:00 level=WARN source=amd_linux.go:53 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers: amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-03-23T22:00:22.055-07:00 level=INFO source=amd_linux.go:88 msg="detected amdgpu versions [gfx1031]"
time=2024-03-23T22:00:22.055-07:00 level=WARN source=amd_linux.go:350 msg="amdgpu detected, but no compatible rocm library found. Either install rocm v6, or follow manual install instructions at https://github.com/ollama/ollama/blob/main/docs/linux.md#manual-install"
time=2024-03-23T22:00:22.055-07:00 level=WARN source=amd_linux.go:99 msg="unable to verify rocm library, will use cpu: no suitable rocm found, falling back to CPU"
time=2024-03-23T22:00:22.055-07:00 level=INFO source=routes.go:1133 msg="no GPU detected"
`

How should we solve this?

please add a flag to ollama serve to start in CPU only mode until more gpus are supported

What is the impact of not solving this?

ollama won't run at all and from another issue ticket the solution I found was '/set parameter num_gpu 0'. unfortunately, I cant run that because ollama won't even run.Ollama

Anything else?

No response

Originally created by @straydog-1212 on GitHub (Mar 24, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3321 Originally assigned to: @dhiltgen on GitHub. ### What are you trying to do? my GPU is unsupported and I used to run it in CPU-only mode. After reinstalling, however, it thinks my 6700 is a Nvidia card and won't let me run Ollama serve. `ollama serve  ✔  23s  time=2024-03-23T22:00:19.302-07:00 level=INFO source=images.go:806 msg="total blobs: 0" time=2024-03-23T22:00:19.302-07:00 level=INFO source=images.go:813 msg="total unused blobs removed: 0" time=2024-03-23T22:00:19.302-07:00 level=INFO source=routes.go:1110 msg="Listening on 127.0.0.1:11434 (version 0.1.29)" time=2024-03-23T22:00:19.302-07:00 level=INFO source=payload_common.go:112 msg="Extracting dynamic libraries to /tmp/ollama156389064/runners ..." time=2024-03-23T22:00:21.954-07:00 level=INFO source=payload_common.go:139 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2 cuda_v11 rocm_v60000]" time=2024-03-23T22:00:21.954-07:00 level=INFO source=gpu.go:77 msg="Detecting GPU type" time=2024-03-23T22:00:21.955-07:00 level=INFO source=gpu.go:191 msg="Searching for GPU management library libnvidia-ml.so" time=2024-03-23T22:00:21.980-07:00 level=INFO source=gpu.go:237 msg="Discovered GPU libraries: [/usr/lib/libnvidia-ml.so.550.54.14 /usr/lib64/libnvidia-ml.so.550.54.14 /opt/cuda/targets/x86_64-linux/lib/stubs/libnvidia-ml.so]" time=2024-03-23T22:00:21.996-07:00 level=INFO source=gpu.go:249 msg="Unable to load CUDA management library /usr/lib/libnvidia-ml.so.550.54.14: nvml vram init failure: 9" time=2024-03-23T22:00:22.011-07:00 level=INFO source=gpu.go:249 msg="Unable to load CUDA management library /usr/lib64/libnvidia-ml.so.550.54.14: nvml vram init failure: 9" !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! WARNING: You should always run with libnvidia-ml.so that is installed with your NVIDIA Display Driver. By default it's installed in /usr/lib and /usr/lib64. libnvidia-ml.so in GDK package is a stub library that is attached only for build purposes (e.g. machine that you build your application doesn't have to have Display Driver installed). !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Linked to libnvidia-ml library at wrong path : /opt/cuda/targets/x86_64-linux/lib/stubs/libnvidia-ml.so time=2024-03-23T22:00:22.054-07:00 level=INFO source=gpu.go:249 msg="Unable to load CUDA management library /opt/cuda/targets/x86_64-linux/lib/stubs/libnvidia-ml.so: nvml vram init failure: 9" time=2024-03-23T22:00:22.054-07:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-03-23T22:00:22.054-07:00 level=WARN source=amd_linux.go:53 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers: amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory" time=2024-03-23T22:00:22.055-07:00 level=INFO source=amd_linux.go:88 msg="detected amdgpu versions [gfx1031]" time=2024-03-23T22:00:22.055-07:00 level=WARN source=amd_linux.go:350 msg="amdgpu detected, but no compatible rocm library found. Either install rocm v6, or follow manual install instructions at https://github.com/ollama/ollama/blob/main/docs/linux.md#manual-install" time=2024-03-23T22:00:22.055-07:00 level=WARN source=amd_linux.go:99 msg="unable to verify rocm library, will use cpu: no suitable rocm found, falling back to CPU" time=2024-03-23T22:00:22.055-07:00 level=INFO source=routes.go:1133 msg="no GPU detected" ` ### How should we solve this? please add a flag to ollama serve to start in CPU only mode until more gpus are supported ### What is the impact of not solving this? ollama won't run at all and from another issue ticket the solution I found was '/set parameter num_gpu 0'. unfortunately, I cant run that because ollama won't even run.Ollama ### Anything else? _No response_
GiteaMirror added the gpubug labels 2026-04-22 05:23:52 -05:00
GiteaMirror commented 2026-04-22 05:23:53 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 25, 2024):

Those messages look normal for an unsupported GPU, and it should fallback to CPU mode automatically. Can you explain what you mean by "won't let me run Ollama serve"

<!-- gh-comment-id:2018837029 --> @dhiltgen commented on GitHub (Mar 25, 2024): Those messages look normal for an unsupported GPU, and it should fallback to CPU mode automatically. Can you explain what you mean by "won't let me run Ollama serve"
GiteaMirror commented 2026-04-22 05:23:54 -05:00
Author
Owner
Copy Link

@straydog-1212 commented on GitHub (Mar 26, 2024):

The process hangs indefinitely at the end of the no GPU detected line whenever I run Ollama serve. it never switches to CPU mode. just sits there incomplete until I kill it 20 mins later

<!-- gh-comment-id:2019317680 --> @straydog-1212 commented on GitHub (Mar 26, 2024): The process hangs indefinitely at the end of the no GPU detected line whenever I run Ollama serve. it never switches to CPU mode. just sits there incomplete until I kill it 20 mins later
GiteaMirror commented 2026-04-22 05:23:54 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 26, 2024):

What behavior do you see in the other terminal window where you run the client ollama run llama2 or whatever model you're trying to load into the CPU? Does it show the spinner indefinitely?

<!-- gh-comment-id:2019385453 --> @dhiltgen commented on GitHub (Mar 26, 2024): What behavior do you see in the other terminal window where you run the client `ollama run llama2` or whatever model you're trying to load into the CPU? Does it show the spinner indefinitely?
GiteaMirror commented 2026-04-22 05:23:54 -05:00
Author
Owner
Copy Link

@oatmealm commented on GitHub (Mar 26, 2024):

I think it'd be helpful to be able to specify some override. I'm passing HSA_OVERRIDE_GFX_VERSION=9.0.0 to have my gfx9012 be used. So if the installer can ask or use this environment variable when installing on a machine with AMD GPU/iGPU, that would be very helpful.

<!-- gh-comment-id:2021058263 --> @oatmealm commented on GitHub (Mar 26, 2024): I think it'd be helpful to be able to specify some override. I'm passing ```HSA_OVERRIDE_GFX_VERSION=9.0.0``` to have my gfx9012 be used. So if the installer can ask or use this environment variable when installing on a machine with AMD GPU/iGPU, that would be very helpful.
GiteaMirror commented 2026-04-22 05:23:55 -05:00
Author
Owner
Copy Link

@straydog-1212 commented on GitHub (Mar 27, 2024):

Error: could not connect to ollama app, is it running?

<!-- gh-comment-id:2021713635 --> @straydog-1212 commented on GitHub (Mar 27, 2024): Error: could not connect to ollama app, is it running?
GiteaMirror commented 2026-04-22 05:23:55 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 27, 2024):

I haven't been able to reproduce this failure mode. I'm guessing that you didn't use our install script, which is why we didn't install the ROCm library, and none is found. The gfx1031 card isn't supported automatically, but you might have luck with HSA_OVERRIDE_GFX_VERSION=10.3.0 once you get ROCm v6 installed on the system through some means.

We might be able to tell what's going wrong with OLLAMA_DEBUG=1 set on the server to generate more verbose logging.

<!-- gh-comment-id:2023434572 --> @dhiltgen commented on GitHub (Mar 27, 2024): I haven't been able to reproduce this failure mode. I'm guessing that you didn't use our install script, which is why we didn't install the ROCm library, and none is found. The gfx1031 card isn't supported automatically, but you might have luck with `HSA_OVERRIDE_GFX_VERSION=10.3.0` once you get ROCm v6 installed on the system through some means. We might be able to tell what's going wrong with `OLLAMA_DEBUG=1` set on the server to generate more verbose logging.
GiteaMirror commented 2026-04-22 05:23:56 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Apr 15, 2024):

Please give the above suggestions a try, and if you're still having problems, share your server log with OLLAMA_DEBUG=1 enabled so we can take a look.

<!-- gh-comment-id:2057952059 --> @dhiltgen commented on GitHub (Apr 15, 2024): Please give the above suggestions a try, and if you're still having problems, share your server log with `OLLAMA_DEBUG=1` enabled so we can take a look.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug gpu
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#27801
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?