github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 20:54:54 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8199] Ollama can recorganize RTX 4090 on linux via nvidia driver 12.7, but it insist in using CPU & MEM not GPU VRAM (24GB), via running codellama #67289

New Issue
Closed
opened 2026-05-04 09:49:21 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-05-04 09:49:21 -05:00
Owner
Copy Link

Originally created by @kylelee on GitHub (Dec 21, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/8199

What is the issue?

Ollama can recorganize RTX 4090 on linux via nvidia driver 12.7, but it insist in using CPU & MEM not GPU VRAM (24GB), via running codellama

The old ollama version (0.1.14) works well with the same other things, but after I upgraded the ollama from 0.1.14 to 0.5.4, the everything goes very slow.

It's seemed that ollama do not choose GPU processor and vram, as show the belowing

When I run ollama

ollama run codellama

the nvidia and ollama query:

nvidia-smi && ollama ps

showed:

Sat Dec 21 21:41:09 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0  On |                  Off |
|  0%   29C    P5             12W /  450W |    1780MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      3711      G   /usr/bin/gnome-shell                          710MiB |
|    0   N/A  N/A      4847    C+G   ...e/.local/zed.app/libexec/zed-editor        143MiB |
|    0   N/A  N/A      5134      G   /usr/bin/Xwayland                             540MiB |
|    0   N/A  N/A     17341      G   xxxxxxxx                                        6MiB |
+-----------------------------------------------------------------------------------------+
NAME                ID              SIZE      PROCESSOR    UNTIL              
codellama:latest    8fdf8f752f6e    5.6 GB    100% GPU     4 minutes from now

The following is ollama(debug on) log output:
ollama.log

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.5.4

Originally created by @kylelee on GitHub (Dec 21, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/8199 ### What is the issue? Ollama can recorganize RTX 4090 on linux via nvidia driver 12.7, but it insist in using CPU & MEM not GPU VRAM (24GB), via running codellama The old ollama version (0.1.14) works well with the same other things, but after I upgraded the ollama from 0.1.14 to 0.5.4, the everything goes very slow. It's seemed that ollama do not choose GPU processor and vram, as show the belowing When I run ollama ```bash ollama run codellama ``` the nvidia and ollama query: ``` nvidia-smi && ollama ps ``` showed: ```bash Sat Dec 21 21:41:09 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 565.57.01 Driver Version: 565.57.01 CUDA Version: 12.7 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 4090 Off | 00000000:01:00.0 On | Off | | 0% 29C P5 12W / 450W | 1780MiB / 24564MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 3711 G /usr/bin/gnome-shell 710MiB | | 0 N/A N/A 4847 C+G ...e/.local/zed.app/libexec/zed-editor 143MiB | | 0 N/A N/A 5134 G /usr/bin/Xwayland 540MiB | | 0 N/A N/A 17341 G xxxxxxxx 6MiB | +-----------------------------------------------------------------------------------------+ NAME ID SIZE PROCESSOR UNTIL codellama:latest 8fdf8f752f6e 5.6 GB 100% GPU 4 minutes from now ``` The following is ollama(debug on) log output: [ollama.log](https://github.com/user-attachments/files/18218700/ollama.log) ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.4
GiteaMirror added the bug label 2026-05-04 09:49:21 -05:00
GiteaMirror commented 2026-05-04 09:49:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 22, 2024):

time=2024-12-21T21:40:16.880+08:00 level=INFO source=routes.go:1339 msg="Dynamic LLM libraries" runners=[cpu]

You don't have any runners that use the GPU. How did you install ollama?

<!-- gh-comment-id:2558284176 --> @rick-github commented on GitHub (Dec 22, 2024): ``` time=2024-12-21T21:40:16.880+08:00 level=INFO source=routes.go:1339 msg="Dynamic LLM libraries" runners=[cpu] ``` You don't have any runners that use the GPU. How did you install ollama?
GiteaMirror commented 2026-05-04 09:49:25 -05:00
Author
Owner
Copy Link

@kylelee commented on GitHub (Dec 22, 2024):

time=2024-12-21T21:40:16.880+08:00 level=INFO source=routes.go:1339 msg="Dynamic LLM libraries" runners=[cpu]

You don't have any runners that use the GPU. How did you install ollama?

I have installed an old version (0.1.14) long time ago, and i use root acount and online install shell script install this version. But I found the online install script do not update the /usr/bin/ollama, the cmd line showed client version is not updated.

So I install it again manually as ollama docs said, download and extract all files to /usr folder, and restart the systemd and even OS. And I check the log, ollama found my GPU, but it do not use as runner, it's very strange.

<!-- gh-comment-id:2558352460 --> @kylelee commented on GitHub (Dec 22, 2024): > ``` > time=2024-12-21T21:40:16.880+08:00 level=INFO source=routes.go:1339 msg="Dynamic LLM libraries" runners=[cpu] > ``` > > You don't have any runners that use the GPU. How did you install ollama? I have installed an old version (0.1.14) long time ago, and i use root acount and online install shell script install this version. But I found the online install script do not update the /usr/bin/ollama, the cmd line showed client version is not updated. So I install it again manually as ollama docs said, download and extract all files to /usr folder, and restart the systemd and even OS. And I check the log, ollama found my GPU, but it do not use as runner, it's very strange.
GiteaMirror commented 2026-05-04 09:49:25 -05:00
Author
Owner
Copy Link

@kylelee commented on GitHub (Dec 22, 2024):

I've tried set envs in service file:

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="PATH=$PATH"
Environment="GGML_CUDA_ENABLE_UNIFIED_MEMORY=0"
Environment="OLLAMA_DEBUG=1"
Environment="OLLAMA_MODELS=/data/ollama/.ollama/models"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_SCHED_SPREAD=1"
Environment="OLLAMA_GPU_OVERHEAD=0"
# Environment="OLLAMA_LLM_LIBRARY=cuda_v12"

[Install]
WantedBy=default.target
<!-- gh-comment-id:2558358172 --> @kylelee commented on GitHub (Dec 22, 2024): I've tried set envs in service file: ```service [Unit] Description=Ollama Service After=network-online.target [Service] ExecStart=/usr/local/bin/ollama serve User=ollama Group=ollama Restart=always RestartSec=3 Environment="PATH=$PATH" Environment="GGML_CUDA_ENABLE_UNIFIED_MEMORY=0" Environment="OLLAMA_DEBUG=1" Environment="OLLAMA_MODELS=/data/ollama/.ollama/models" Environment="OLLAMA_MAX_LOADED_MODELS=1" Environment="OLLAMA_NUM_PARALLEL=1" Environment="OLLAMA_SCHED_SPREAD=1" Environment="OLLAMA_GPU_OVERHEAD=0" # Environment="OLLAMA_LLM_LIBRARY=cuda_v12" [Install] WantedBy=default.target ```
GiteaMirror commented 2026-05-04 09:49:27 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Dec 22, 2024):

I suggest deleting /usr/bin/ollama and doing a normal install with curl|sh, and then use sudo systemctl edit ollama to set the environment variables. Then run the model and it still fails to use the GPU, post the new logs to this thread.

<!-- gh-comment-id:2558369039 --> @rick-github commented on GitHub (Dec 22, 2024): I suggest deleting /usr/bin/ollama and doing a normal install with `curl|sh`, and then use `sudo systemctl edit ollama` to set the environment variables. Then `run` the model and it still fails to use the GPU, post the new logs to this thread.
GiteaMirror commented 2026-05-04 09:49:30 -05:00
Author
Owner
Copy Link

@kylelee commented on GitHub (Dec 22, 2024):

I suggest deleting /usr/bin/ollama and doing a normal install with curl|sh, and then use sudo systemctl edit ollama to set the environment variables. Then run the model and it still fails to use the GPU, post the new logs to this thread.

Works now, as your help, i rm -rf /usr/bin/ollama and install online, thanks for your help

<!-- gh-comment-id:2558459336 --> @kylelee commented on GitHub (Dec 22, 2024): > I suggest deleting /usr/bin/ollama and doing a normal install with `curl|sh`, and then use `sudo systemctl edit ollama` to set the environment variables. Then `run` the model and it still fails to use the GPU, post the new logs to this thread. Works now, as your help, i ```rm -rf /usr/bin/ollama``` and install online, thanks for your help
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#67289
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?