github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #3554] Potential problems with the llm/ext_server/server.cpp not accepting --ubatch-size option #2195

New Issue
Open
opened 2026-04-12 12:26:36 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-12 12:26:36 -05:00
Owner
Copy Link

Originally created by @jukofyork on GitHub (Apr 9, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3554

Originally assigned to: @jessegross on GitHub.

What is the issue?

Not sure what to list this issue under (it's a potential bug I think).

Recently llama.cpp has added an option called --ubatch-size and appears to have changed the default value (and possibly meaning of) the old --batch-size option:

https://github.com/ggerganov/llama.cpp/pull/6017

The PR says:

By default n_batch is 4096, n_ubatch is 512

but I think it is actually 2048 and 512 now.

This better explains the difference between the options:

https://github.com/ggerganov/llama.cpp/discussions/6328

There are several potential problems and implications of this change for Ollama:

  • It looks like llm/ext_server/server.cpp is using some old code that handles the n_batch in the old way, but this in turn seems to be importing headers, etc from a newer version of lamma.cpp (b2581), who's examples/server.cpp code does appear to have the new n_batch / n_ubatch logic in it.
  • It looks like api/types.go is setting the n_batch value back down to 512, when the new default is supposed to be 2048.
  • Since Ollama is using the --embedding option there are some potential problems mentioned here: https://github.com/ggerganov/llama.cpp/issues/6263
  • Not being able to set the n_ubatch value is causing regressions for people using multiple GPUs (https://github.com/ggerganov/llama.cpp/issues/6075 & https://github.com/ggerganov/llama.cpp/issues/6476).

https://github.com/ollama/ollama/blob/main/llm/ext_server/server.cpp:

    printf("  --pooling {none,mean,cls}\n");
    printf("                        pooling type for embeddings, use model default if unspecified\n");
    printf("  -b N, --batch-size N      batch size for prompt processing (default: %d)\n", params.n_batch);
    printf("  --memory-f32              use f32 instead of f16 for memory key+value (default: disabled)\n");

vs:

37e7854c10/examples/server/server.cpp:

    printf("  --pooling {none,mean,cls} pooling type for embeddings, use model default if unspecified\n");
    printf("  -dt N, --defrag-thold N\n");
    printf("                            KV cache defragmentation threshold (default: %.1f, < 0 - disabled)\n", params.defrag_thold);
    printf("  -b N, --batch-size N      logical maximum batch size (default: %d)\n", params.n_batch);
    printf("  -ub N, --ubatch-size N    physical maximum batch size (default: %d)\n", params.n_ubatch);
    printf("  --memory-f32              use f32 instead of f16 for memory key+value (default: disabled)\n");

- `-b N`, `--batch-size N`: Set the batch size for prompt processing. Default: `2048`
- `-ub N`, `--ubatch-size N`: Physical maximum batch size. Default: `512`
Originally created by @jukofyork on GitHub (Apr 9, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3554 Originally assigned to: @jessegross on GitHub. ### What is the issue? Not sure what to list this issue under (it's a potential bug I think). Recently `llama.cpp` has added an option called `--ubatch-size ` and appears to have changed the default value (and possibly meaning of) the old `--batch-size ` option: https://github.com/ggerganov/llama.cpp/pull/6017 The PR says: > By default `n_batch` is 4096, `n_ubatch` is 512 but I think it is actually 2048 and 512 now. This better explains the difference between the options: https://github.com/ggerganov/llama.cpp/discussions/6328 There are several potential problems and implications of this change for Ollama: - It looks like `llm/ext_server/server.cpp` is using some old code that handles the `n_batch` in the old way, but this in turn seems to be importing headers, etc from a newer version of `lamma.cpp` (b2581), who's `examples/server.cpp` code does appear to have the new `n_batch` / `n_ubatch` logic in it. - It looks like `api/types.go` is setting the `n_batch` value back down to 512, when the new default is supposed to be 2048. - Since Ollama is using the `--embedding` option there are some potential problems mentioned here: https://github.com/ggerganov/llama.cpp/issues/6263 - Not being able to set the `n_ubatch` value is causing regressions for people using multiple GPUs (https://github.com/ggerganov/llama.cpp/issues/6075 & https://github.com/ggerganov/llama.cpp/issues/6476). --- https://github.com/ollama/ollama/blob/main/llm/ext_server/server.cpp: ``` printf(" --pooling {none,mean,cls}\n"); printf(" pooling type for embeddings, use model default if unspecified\n"); printf(" -b N, --batch-size N batch size for prompt processing (default: %d)\n", params.n_batch); printf(" --memory-f32 use f32 instead of f16 for memory key+value (default: disabled)\n"); ``` vs: https://github.com/ggerganov/llama.cpp/blob/37e7854c104301c5b5323ccc40e07699f3a62c3e/examples/server/server.cpp: ``` printf(" --pooling {none,mean,cls} pooling type for embeddings, use model default if unspecified\n"); printf(" -dt N, --defrag-thold N\n"); printf(" KV cache defragmentation threshold (default: %.1f, < 0 - disabled)\n", params.defrag_thold); printf(" -b N, --batch-size N logical maximum batch size (default: %d)\n", params.n_batch); printf(" -ub N, --ubatch-size N physical maximum batch size (default: %d)\n", params.n_ubatch); printf(" --memory-f32 use f32 instead of f16 for memory key+value (default: disabled)\n"); ``` --- ``` - `-b N`, `--batch-size N`: Set the batch size for prompt processing. Default: `2048` - `-ub N`, `--ubatch-size N`: Physical maximum batch size. Default: `512` ```
GiteaMirror added the bug label 2026-04-12 12:26:36 -05:00
GiteaMirror commented 2026-04-12 12:26:37 -05:00
Author
Owner
Copy Link

@s-kostyaev commented on GitHub (Nov 23, 2024):

Looks like I have related issue. When I try to calculate batch embeddings with ollama on m1 mac I have 500 status code and this in logs:

llama.cpp:17743: GGML_ASSERT((cparams.causal_attn || cparams.n_ubatch >= n_tokens_all) && "non-causal attention requires n_ubatch >= n_tokens") failed

ollama_log.txt

ollama version is 0.4.3

ollama version is 0.4.4
<!-- gh-comment-id:2495528068 --> @s-kostyaev commented on GitHub (Nov 23, 2024): Looks like I have related issue. When I try to calculate batch embeddings with ollama on m1 mac I have 500 status code and this in logs: >llama.cpp:17743: GGML_ASSERT((cparams.causal_attn || cparams.n_ubatch >= n_tokens_all) && "non-causal attention requires n_ubatch >= n_tokens") failed [ollama_log.txt](https://github.com/user-attachments/files/17880443/ollama_log.txt) ``` ollama version is 0.4.3 ``` ``` ollama version is 0.4.4 ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#2195
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?