github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-12 03:08:28 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #10735] Cannot run granite3.3:2b with context size 32768 on an H100 #53563

New Issue
Closed
opened 2026-04-29 03:48:07 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-04-29 03:48:07 -05:00
Owner
Copy Link

Originally created by @JCPOCSIN on GitHub (May 16, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/10735

What is the issue?

Hi all,

I've got a weird issue. When I try to run granite3.3:2b with num_ctx 32768 on an H100 Linux VM, it crashes as it tries to allocate too much VRAM: about 51GB.

The same LLM works well with num_ctx 40960 on my laptop (which has no usable GPU and 8GB RAM).

Looking at the logs, Ollama tries to allocate a kv_size = 655360 on the H100 VM, and 65536 on my laptop.

What did I miss?

ollama version is 0.7.0
OS: Linux RHEL 9
Driver Version: 550.144.03 CUDA Version: 12.4

Relevant log output

May 16 15:11:15 myH100VM ollama[1405]: llama_context: n_ctx_per_seq (32768) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
May 16 15:11:15 myH100VM ollama[1405]: llama_context:  CUDA_Host  output buffer size =     3.91 MiB
May 16 15:11:15 myH100VM ollama[1405]: llama_kv_cache_unified: kv_size = 655360, type_k = 'f16', type_v = 'f16', n_layer = 40, can_shift = 1, padding = 32
May 16 15:11:15 myH100VM ollama[1405]: llama_kv_cache_unified:      CUDA0 KV buffer size = 51200.00 MiB
May 16 15:11:15 myH100VM ollama[1405]: llama_kv_cache_unified: KV self size  = 51200.00 MiB, K (f16): 25600.00 MiB, V (f16): 25600.00 MiB
May 16 15:11:15 myH100VM ollama[1405]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 42256.00 MiB on device 0: cudaMalloc failed: out of memory
May 16 15:11:15 myH100VM ollama[1405]: ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 44308631552
May 16 15:11:15 myH100VM ollama[1405]: llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
May 16 15:11:15 myH100VM ollama[1405]: panic: unable to create llama context
May 16 15:11:15 myH100VM ollama[1405]: goroutine 36 [running]:
May 16 15:11:15 myH100VM ollama[1405]: github.com/ollama/ollama/runner/llamarunner.(*Server).loadModel(0xc00052c000, {0x29, 0x0, 0x1, {0x0, 0x0, 0x0}, 0xc0005822e0, 0x0}, {0x7ffd8c6e9c3f, ...}, ...)
May 16 15:11:15 myH100VM ollama[1405]:         github.com/ollama/ollama/runner/llamarunner/runner.go:757 +0x389
May 16 15:11:15 myH100VM ollama[1405]: created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1
May 16 15:11:15 myH100VM ollama[1405]:         github.com/ollama/ollama/runner/llamarunner/runner.go:848 +0xb57
May 16 15:11:16 myH100VM ollama[1405]: time=2025-05-16T15:11:16.129+02:00 level=ERROR source=server.go:457 msg="llama runner terminated" error="exit status 2"
May 16 15:11:16 myH100VM ollama[1405]: time=2025-05-16T15:11:16.167+02:00 level=ERROR source=sched.go:478 msg="error loading llama server" error="llama runner process has terminated: cudaMalloc failed: out of memory\nggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 44308631552"
May 16 15:11:16 myH100VM ollama[1405]: [GIN] 2025/05/16 - 15:11:16 | 500 |  2.204450678s |       127.0.0.1 | POST     "/api/chat"

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.7.0

Originally created by @JCPOCSIN on GitHub (May 16, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/10735 ### What is the issue? Hi all, I've got a weird issue. When I try to run granite3.3:2b with num_ctx 32768 on an H100 Linux VM, it crashes as it tries to allocate too much VRAM: about 51GB. The same LLM works well with num_ctx 40960 on my laptop (which has no usable GPU and 8GB RAM). Looking at the logs, Ollama tries to allocate a kv_size = 655360 on the H100 VM, and 65536 on my laptop. What did I miss? ollama version is 0.7.0 OS: Linux RHEL 9 Driver Version: 550.144.03 CUDA Version: 12.4 ### Relevant log output ```shell May 16 15:11:15 myH100VM ollama[1405]: llama_context: n_ctx_per_seq (32768) < n_ctx_train (131072) -- the full capacity of the model will not be utilized May 16 15:11:15 myH100VM ollama[1405]: llama_context: CUDA_Host output buffer size = 3.91 MiB May 16 15:11:15 myH100VM ollama[1405]: llama_kv_cache_unified: kv_size = 655360, type_k = 'f16', type_v = 'f16', n_layer = 40, can_shift = 1, padding = 32 May 16 15:11:15 myH100VM ollama[1405]: llama_kv_cache_unified: CUDA0 KV buffer size = 51200.00 MiB May 16 15:11:15 myH100VM ollama[1405]: llama_kv_cache_unified: KV self size = 51200.00 MiB, K (f16): 25600.00 MiB, V (f16): 25600.00 MiB May 16 15:11:15 myH100VM ollama[1405]: ggml_backend_cuda_buffer_type_alloc_buffer: allocating 42256.00 MiB on device 0: cudaMalloc failed: out of memory May 16 15:11:15 myH100VM ollama[1405]: ggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 44308631552 May 16 15:11:15 myH100VM ollama[1405]: llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers May 16 15:11:15 myH100VM ollama[1405]: panic: unable to create llama context May 16 15:11:15 myH100VM ollama[1405]: goroutine 36 [running]: May 16 15:11:15 myH100VM ollama[1405]: github.com/ollama/ollama/runner/llamarunner.(*Server).loadModel(0xc00052c000, {0x29, 0x0, 0x1, {0x0, 0x0, 0x0}, 0xc0005822e0, 0x0}, {0x7ffd8c6e9c3f, ...}, ...) May 16 15:11:15 myH100VM ollama[1405]: github.com/ollama/ollama/runner/llamarunner/runner.go:757 +0x389 May 16 15:11:15 myH100VM ollama[1405]: created by github.com/ollama/ollama/runner/llamarunner.Execute in goroutine 1 May 16 15:11:15 myH100VM ollama[1405]: github.com/ollama/ollama/runner/llamarunner/runner.go:848 +0xb57 May 16 15:11:16 myH100VM ollama[1405]: time=2025-05-16T15:11:16.129+02:00 level=ERROR source=server.go:457 msg="llama runner terminated" error="exit status 2" May 16 15:11:16 myH100VM ollama[1405]: time=2025-05-16T15:11:16.167+02:00 level=ERROR source=sched.go:478 msg="error loading llama server" error="llama runner process has terminated: cudaMalloc failed: out of memory\nggml_gallocr_reserve_n: failed to allocate CUDA0 buffer of size 44308631552" May 16 15:11:16 myH100VM ollama[1405]: [GIN] 2025/05/16 - 15:11:16 | 500 | 2.204450678s | 127.0.0.1 | POST "/api/chat" ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.7.0
GiteaMirror added the bug label 2026-04-29 03:48:07 -05:00
GiteaMirror commented 2026-04-29 03:48:10 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (May 16, 2025):

Full log will make it easier to debug.

<!-- gh-comment-id:2886818570 --> @rick-github commented on GitHub (May 16, 2025): Full log will make it easier to debug.
GiteaMirror commented 2026-04-29 03:48:12 -05:00
Author
Owner
Copy Link

@JCPOCSIN commented on GitHub (May 16, 2025):

Here are the complete logs of the session

ollama-log-granite-crash.txt

<!-- gh-comment-id:2886843235 --> @JCPOCSIN commented on GitHub (May 16, 2025): Here are the complete logs of the session [ollama-log-granite-crash.txt](https://github.com/user-attachments/files/20247012/ollama-log-granite-crash.txt)
GiteaMirror commented 2026-04-29 03:48:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (May 16, 2025):

May 16 16:05:29 MyH100VM ollama[1405]: time=2025-05-16T16:05:29.037+02:00 level=INFO source=server.go:168
 msg=offload  library=cuda layers.requested=-1 layers.model=41 layers.offload=41 layers.split=""
 memory.available="[90.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="86.5 GiB"
 memory.required.partial="86.5 GiB" memory.required.kv="50.0 GiB" memory.required.allocations="[86.5 GiB]"
 memory.weights.total="1.4 GiB" memory.weights.repeating="1.4 GiB" memory.weights.nonrepeating="78.8 MiB"
 memory.graph.full="33.3 GiB" memory.graph.partial="33.3 GiB"

You have OLLAMA_NUM_PARALLEL=20, this increases KV requirement. 90.9G available and ollama has estimated 85.5G is needed to load the model. The runner OOMs when allocating buffers. See here for ways to mitigate OOM conditions.

<!-- gh-comment-id:2886893502 --> @rick-github commented on GitHub (May 16, 2025): ``` May 16 16:05:29 MyH100VM ollama[1405]: time=2025-05-16T16:05:29.037+02:00 level=INFO source=server.go:168 msg=offload library=cuda layers.requested=-1 layers.model=41 layers.offload=41 layers.split="" memory.available="[90.9 GiB]" memory.gpu_overhead="0 B" memory.required.full="86.5 GiB" memory.required.partial="86.5 GiB" memory.required.kv="50.0 GiB" memory.required.allocations="[86.5 GiB]" memory.weights.total="1.4 GiB" memory.weights.repeating="1.4 GiB" memory.weights.nonrepeating="78.8 MiB" memory.graph.full="33.3 GiB" memory.graph.partial="33.3 GiB" ``` You have `OLLAMA_NUM_PARALLEL=20`, this increases KV requirement. 90.9G available and ollama has estimated 85.5G is needed to load the model. The runner OOMs when allocating buffers. See [here](https://github.com/ollama/ollama/issues/8597#issuecomment-2614533288) for ways to mitigate OOM conditions.
GiteaMirror commented 2026-04-29 03:48:15 -05:00
Author
Owner
Copy Link

@JCPOCSIN commented on GitHub (May 16, 2025):

OK...
Now, I'm wondering how to allow inferences in parallel. But that would be another question ;)

Thank you for your help @rick-github !

<!-- gh-comment-id:2886906137 --> @JCPOCSIN commented on GitHub (May 16, 2025): OK... Now, I'm wondering how to allow inferences in parallel. But that would be another question ;) Thank you for your help @rick-github !
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#53563
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?