github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-15 12:48:05 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #11162] gemma3:27b-it-fp16 llayer offloaded to CPU despite VRAM available #7361

New Issue
Closed
opened 2026-04-12 19:25:10 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-12 19:25:10 -05:00
Owner
Copy Link

Originally created by @GevatterGaul on GitHub (Jun 22, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11162

What is the issue?

Hi, when running gemma3:27b-it-fp16 on a dual 5090 setup, one layer always gets offloaded to CPU despite VRAM still available.
The model has unusually an uneven number of layers: 63, according to logs.

How to reproduce:

Run gemma3:27b-it-fp16 on a dual 5090 setup, ollama 0.9.2, check ollama ps.

Relevant log output

Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.306+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-c8e6d566-a20a-dda9-6663-133a93ee472c name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.4 GiB" before.free="30.9 GiB" now.total="31.4 GiB" now.free="30.9 GiB" now.used="519.1 MiB"
Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.442+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-fb5a01d7-2738-8543-7e7c-a3e48ed7f644 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.4 GiB" before.free="30.8 GiB" now.total="31.4 GiB" now.free="30.8 GiB" now.used="544.7 MiB"
...
 Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.450+02:00 level=INFO source=server.go:168 msg=offload library=cuda layers.requested=256 layers.model=63 layers.offload=62 layers.split=31,31 memory.available="[30.9 GiB 30.8 GiB]" memory.gpu_overhead="0 B" memory.required.full="58.6 GiB" memory.required.partial="54.2 GiB" memory.required.kv="944.0 MiB" memory.required.allocations="[27.0 GiB 27.2 GiB]" memory.weights.total="50.3 GiB" memory.weights.repeating="47.7 GiB" memory.weights.nonrepeating="2.6 GiB" memory.graph.full="1.6 GiB" memory.graph.partial="1.6 GiB" projector.weights="795.9 MiB" projector.graph="1.0 GiB"
...
Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.491+02:00 level=INFO source=server.go:431 msg="starting llama server" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-07ca3450446e07c4e3dfd55d34e3f426963a15f1db00c3093d9214c202d12e25 --ctx-size 4096 --batch-size 512 --n-gpu-layers 62 --threads 24 --parallel 1 --tensor-split 31,31 --port 35149"
...
Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.743+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.768+02:00 level=INFO source=ggml.go:351 msg="model weights" buffer=CUDA0 size="23.8 GiB"
Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.768+02:00 level=INFO source=ggml.go:351 msg="model weights" buffer=CUDA1 size="23.8 GiB"
Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.768+02:00 level=INFO source=ggml.go:351 msg="model weights" buffer=CPU size="6.0 GiB"


nvidia-smi:

Sun Jun 22 23:54:08 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        Off |   00000000:41:00.0 Off |                  N/A |
|  0%   51C    P8             31W /  575W |      18MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 5090        Off |   00000000:81:00.0 Off |                  N/A |
|  0%   49C    P8             22W /  575W |      44MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            3208      G   /usr/lib/xorg/Xorg                        4MiB |
|    1   N/A  N/A            3208      G   /usr/lib/xorg/Xorg                       10MiB |
|    1   N/A  N/A            3325      G   /usr/bin/gnome-shell                     10MiB |
+-----------------------------------------------------------------------------------------+

OS

Linux

GPU

Nvidia

CPU

AMD

Ollama version

0.9.2

Originally created by @GevatterGaul on GitHub (Jun 22, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11162 ### What is the issue? Hi, when running gemma3:27b-it-fp16 on a dual 5090 setup, one layer always gets offloaded to CPU despite VRAM still available. The model has unusually an uneven number of layers: 63, according to logs. How to reproduce: Run gemma3:27b-it-fp16 on a dual 5090 setup, ollama 0.9.2, check `ollama ps`. ### Relevant log output ```shell Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.306+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-c8e6d566-a20a-dda9-6663-133a93ee472c name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.4 GiB" before.free="30.9 GiB" now.total="31.4 GiB" now.free="30.9 GiB" now.used="519.1 MiB" Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.442+02:00 level=DEBUG source=gpu.go:441 msg="updating cuda memory data" gpu=GPU-fb5a01d7-2738-8543-7e7c-a3e48ed7f644 name="NVIDIA GeForce RTX 5090" overhead="0 B" before.total="31.4 GiB" before.free="30.8 GiB" now.total="31.4 GiB" now.free="30.8 GiB" now.used="544.7 MiB" ... Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.450+02:00 level=INFO source=server.go:168 msg=offload library=cuda layers.requested=256 layers.model=63 layers.offload=62 layers.split=31,31 memory.available="[30.9 GiB 30.8 GiB]" memory.gpu_overhead="0 B" memory.required.full="58.6 GiB" memory.required.partial="54.2 GiB" memory.required.kv="944.0 MiB" memory.required.allocations="[27.0 GiB 27.2 GiB]" memory.weights.total="50.3 GiB" memory.weights.repeating="47.7 GiB" memory.weights.nonrepeating="2.6 GiB" memory.graph.full="1.6 GiB" memory.graph.partial="1.6 GiB" projector.weights="795.9 MiB" projector.graph="1.0 GiB" ... Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.491+02:00 level=INFO source=server.go:431 msg="starting llama server" cmd="/usr/local/bin/ollama runner --ollama-engine --model /usr/share/ollama/.ollama/models/blobs/sha256-07ca3450446e07c4e3dfd55d34e3f426963a15f1db00c3093d9214c202d12e25 --ctx-size 4096 --batch-size 512 --n-gpu-layers 62 --threads 24 --parallel 1 --tensor-split 31,31 --port 35149" ... Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.743+02:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.768+02:00 level=INFO source=ggml.go:351 msg="model weights" buffer=CUDA0 size="23.8 GiB" Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.768+02:00 level=INFO source=ggml.go:351 msg="model weights" buffer=CUDA1 size="23.8 GiB" Jun 22 20:07:23 bigboi ollama[1215321]: time=2025-06-22T20:07:23.768+02:00 level=INFO source=ggml.go:351 msg="model weights" buffer=CPU size="6.0 GiB" nvidia-smi: Sun Jun 22 23:54:08 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.124.06 Driver Version: 570.124.06 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 5090 Off | 00000000:41:00.0 Off | N/A | | 0% 51C P8 31W / 575W | 18MiB / 32607MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 5090 Off | 00000000:81:00.0 Off | N/A | | 0% 49C P8 22W / 575W | 44MiB / 32607MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 3208 G /usr/lib/xorg/Xorg 4MiB | | 1 N/A N/A 3208 G /usr/lib/xorg/Xorg 10MiB | | 1 N/A N/A 3325 G /usr/bin/gnome-shell 10MiB | +-----------------------------------------------------------------------------------------+ ``` ### OS Linux ### GPU Nvidia ### CPU AMD ### Ollama version 0.9.2
GiteaMirror added the bug label 2026-04-12 19:25:10 -05:00
GiteaMirror commented 2026-04-12 19:25:11 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 22, 2025):

This likely due to the input layer assignment. The input layer is always loaded on the CPU because the CUDA backend doesn't support some of the required operations on quantized tensors. The performance impact is minimal. You can set OLLAMA_DEBUG=2 to verify the layer assignment.

<!-- gh-comment-id:2994479568 --> @rick-github commented on GitHub (Jun 22, 2025): This likely due to the input layer assignment. The input layer is always loaded on the CPU because the CUDA backend doesn't support some of the required operations on quantized tensors. The performance impact is minimal. You can set `OLLAMA_DEBUG=2` to verify the layer assignment.
GiteaMirror commented 2026-04-12 19:25:12 -05:00
Author
Owner
Copy Link

@GevatterGaul commented on GitHub (Jun 22, 2025):

Interesting, thx. How do the smaller gemma models manage to run on GPU only then? Don´t they have an input layer?

innovation-hacking@bigboi:~$ ollama ps
NAME                  ID              SIZE     PROCESSOR    UNTIL              
gemma3:12b-it-fp16    6b1ba564b78d    30 GB    100% GPU     4 minutes from now

In contrast to a llama, the layer assignment is not recorded even with OLLAMA_DEBUG=2

<!-- gh-comment-id:2994544856 --> @GevatterGaul commented on GitHub (Jun 22, 2025): Interesting, thx. How do the smaller gemma models manage to run on GPU only then? Don´t they have an input layer? ``` innovation-hacking@bigboi:~$ ollama ps NAME ID SIZE PROCESSOR UNTIL gemma3:12b-it-fp16 6b1ba564b78d 30 GB 100% GPU 4 minutes from now ``` In contrast to a llama, the layer assignment is not recorded even with `OLLAMA_DEBUG=2`
GiteaMirror commented 2026-04-12 19:25:13 -05:00
Author
Owner
Copy Link

@GevatterGaul commented on GitHub (Jun 25, 2025):

This behaviour can also be observed on an M4 MAX with 64GB RAM

<!-- gh-comment-id:3006168624 --> @GevatterGaul commented on GitHub (Jun 25, 2025): This behaviour can also be observed on an M4 MAX with 64GB RAM
GiteaMirror commented 2026-04-12 19:25:14 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 25, 2025):

Server logs with OLLAMA_DEBUG=2 may provide some insight.

<!-- gh-comment-id:3006181596 --> @rick-github commented on GitHub (Jun 25, 2025): Server logs with `OLLAMA_DEBUG=2` may provide some insight.
GiteaMirror commented 2026-04-12 19:25:14 -05:00
Author
Owner
Copy Link

@mercuriy94 commented on GitHub (Aug 11, 2025):

Hi! i have 3 RTX 3090 and I see the same problem when I set num_ctx more than 7k.

<!-- gh-comment-id:3177011788 --> @mercuriy94 commented on GitHub (Aug 11, 2025): Hi! i have 3 RTX 3090 and I see the same problem when I set num_ctx more than 7k.
GiteaMirror commented 2026-04-12 19:25:15 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 11, 2025):

Server logs will aid in debugging.

<!-- gh-comment-id:3177015096 --> @rick-github commented on GitHub (Aug 11, 2025): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will aid in debugging.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#7361
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?