github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #15079] GPU detection silently falls back to CPU when working directory is inaccessible to service user #35433

New Issue
Open
opened 2026-04-22 19:56:04 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-04-22 19:56:04 -05:00
Owner
Copy Link

Originally created by @Frank-Schruefer on GitHub (Mar 26, 2026).
Original GitHub issue: https://github.com/ollama/ollama/issues/15079

Description

When ollama serve is started as a service user (e.g. www-data, wwwrun, nobody) and the current working directory is not accessible to that user, GPU detection silently fails and ollama falls back to library=cpu with no error message or warning.

Steps to Reproduce

# Fails silently → library=cpu
sudo -u www-data bash -c 'cd /root; ollama serve'

# Works → library=cuda
sudo -u www-data bash -c 'cd /tmp; ollama serve'

The only difference is the working directory. The environment (PATH, OLLAMA_HOST, CUDA env vars, DISPLAY, XAUTHORITY) is identical.

Environment

  • OS: openSUSE Tumbleweed (Linux 6.x)
  • GPU: NVIDIA RTX A3000 12GB (Optimus/PRIME, Dell Precision 7770)
  • ollama built from source (current main)
  • Service user: wwwrun (equivalent of www-data)

Root Cause

The GPU discovery subprocess (the internal ollama runner --ollama-engine process spawned by bootstrapDevices()) inherits the working directory of the parent process.

During CUDA backend initialization, libcuda.so (or a related NVIDIA library) accesses the working directory — most likely to write its JIT kernel cache. If the cwd is a directory the service user cannot access (e.g. /root with mode 700), the CUDA backend fails to initialize. Since the discovery subprocess discards stdout/stderr unless OLLAMA_DEBUG=2 (trace level), this failure is completely invisible to the operator. Ollama then falls back to CPU without any log entry at the default or info log level.

Impact

This is a silent failure — operators get no indication that GPU detection failed. The symptom (running on CPU instead of GPU) has many possible causes, and nothing in the logs points to the working directory as the culprit. Debugging this is extremely time-consuming.

Suggested Fix

Before spawning the discovery subprocess in bootstrapDevices(), change to a safe working directory:

os.Chdir(os.TempDir())

Alternatively (or additionally): log a warning at Info level when a GPU backend fails to load, so the failure is visible without requiring trace-level debugging.

Workaround

Ensure ollama is started from a directory accessible to the service user, e.g. by adding cd /tmp (or any world-accessible directory) before exec ollama serve in the start script.

Originally created by @Frank-Schruefer on GitHub (Mar 26, 2026). Original GitHub issue: https://github.com/ollama/ollama/issues/15079 ## Description When `ollama serve` is started as a service user (e.g. `www-data`, `wwwrun`, `nobody`) and the current working directory is not accessible to that user, GPU detection silently fails and ollama falls back to `library=cpu` with no error message or warning. ## Steps to Reproduce ```bash # Fails silently → library=cpu sudo -u www-data bash -c 'cd /root; ollama serve' # Works → library=cuda sudo -u www-data bash -c 'cd /tmp; ollama serve' ``` The only difference is the working directory. The environment (PATH, OLLAMA_HOST, CUDA env vars, DISPLAY, XAUTHORITY) is identical. ## Environment - OS: openSUSE Tumbleweed (Linux 6.x) - GPU: NVIDIA RTX A3000 12GB (Optimus/PRIME, Dell Precision 7770) - ollama built from source (current main) - Service user: `wwwrun` (equivalent of `www-data`) ## Root Cause The GPU discovery subprocess (the internal `ollama runner --ollama-engine` process spawned by `bootstrapDevices()`) inherits the working directory of the parent process. During CUDA backend initialization, `libcuda.so` (or a related NVIDIA library) accesses the working directory — most likely to write its JIT kernel cache. If the cwd is a directory the service user cannot access (e.g. `/root` with mode 700), the CUDA backend fails to initialize. Since the discovery subprocess discards stdout/stderr unless `OLLAMA_DEBUG=2` (trace level), this failure is completely invisible to the operator. Ollama then falls back to CPU without any log entry at the default or info log level. ## Impact This is a **silent failure** — operators get no indication that GPU detection failed. The symptom (running on CPU instead of GPU) has many possible causes, and nothing in the logs points to the working directory as the culprit. Debugging this is extremely time-consuming. ## Suggested Fix Before spawning the discovery subprocess in `bootstrapDevices()`, change to a safe working directory: ```go os.Chdir(os.TempDir()) ``` Alternatively (or additionally): log a warning at `Info` level when a GPU backend fails to load, so the failure is visible without requiring trace-level debugging. ## Workaround Ensure ollama is started from a directory accessible to the service user, e.g. by adding `cd /tmp` (or any world-accessible directory) before `exec ollama serve` in the start script.
GiteaMirror added the bug label 2026-04-22 19:56:04 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#35433
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?