github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-11-10 15:02:13 -06:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[PR #11525] Increase performance for Gemma3n models on NVGPUs by enabling CUDA Graph execution #13568

New Issue
Closed
opened 2026-04-13 00:30:18 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-04-13 00:30:18 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/ollama/ollama/pull/11525

State: closed
Merged: Yes

This PR enables the execution of Gemma3n as CUDA Graphs on NVGPUs by porting https://github.com/ggml-org/llama.cpp/pull/14741 to ollama. Since the model graph is defined differently in ollama compared to llama.cpp, the heuristic used to identify and exclude the per_layer_projection from batch-size determination needed to be modified a bit. As a consequence, the patch will need to be maintained even after llama.cpp is updated to a commit that contains https://github.com/ggml-org/llama.cpp/pull/14741.

On a RTX PRO 6000 Max-Q under Windows, this PR improves perf by ~2.5x, see

Model Configuration Tokens/sec
gemma3n:e2b CG ON 103
gemma3n:e2b CG OFF 43
gemma3n:e4b CG ON 79
gemma3n:e4b CG OFF 35

Thanks @mxyng for providing changes to gemma3n model graph definition in c4de3eaa3e that make the checking more robust.

**Original Pull Request:** https://github.com/ollama/ollama/pull/11525 **State:** closed **Merged:** Yes --- This PR enables the execution of Gemma3n as CUDA Graphs on NVGPUs by porting https://github.com/ggml-org/llama.cpp/pull/14741 to ollama. Since the model graph is defined differently in ollama compared to llama.cpp, the heuristic used to identify and exclude the `per_layer_projection` from batch-size determination needed to be modified a bit. As a consequence, the patch will need to be maintained even after llama.cpp is updated to a commit that contains https://github.com/ggml-org/llama.cpp/pull/14741. On a RTX PRO 6000 Max-Q under Windows, this PR improves perf by ~2.5x, see Model | Configuration | Tokens/sec -- | -- | -- gemma3n:e2b | CG ON | 103 gemma3n:e2b | CG OFF | 43 gemma3n:e4b | CG ON | 79 gemma3n:e4b | CG OFF | 35 Thanks @mxyng for providing changes to gemma3n model graph definition in c4de3eaa3e11eee48ed061ebbae5737f7a7cd2b2 that make the checking more robust.
GiteaMirror added the pull-request label 2026-04-13 00:30:18 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No labels pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#13568