github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #1396] Continuous batching support #78416

New Issue
Closed
opened 2026-05-08 22:59:15 -05:00 by GiteaMirror · 15 comments
GiteaMirror commented 2026-05-08 22:59:15 -05:00
Owner
Copy Link

Originally created by @Huvinesh-Rajendran-12 on GitHub (Dec 6, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1396

Does Ollama support continuous batching for concurrent requests? I couldn't find anything in the documentation.

Originally created by @Huvinesh-Rajendran-12 on GitHub (Dec 6, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1396 Does Ollama support continuous batching for concurrent requests? I couldn't find anything in the documentation.
GiteaMirror commented 2026-05-08 22:59:20 -05:00
Author
Owner
Copy Link

@easp commented on GitHub (Dec 6, 2023):

It doesn't.

<!-- gh-comment-id:1843753777 --> @easp commented on GitHub (Dec 6, 2023): It doesn't.
GiteaMirror commented 2026-05-08 22:59:23 -05:00
Author
Owner
Copy Link

@trenta3 commented on GitHub (Dec 7, 2023):

llama.cpp (which is the engine at the base of Ollama) does indeed support it, I'd also like for a configuration parameter in Ollama to be set to enable continuous batching.

Ref: https://github.com/ggerganov/llama.cpp/discussions/3471

<!-- gh-comment-id:1845051509 --> @trenta3 commented on GitHub (Dec 7, 2023): llama.cpp (which is the engine at the base of Ollama) does indeed support it, I'd also like for a configuration parameter in Ollama to be set to enable continuous batching. Ref: https://github.com/ggerganov/llama.cpp/discussions/3471
GiteaMirror commented 2026-05-08 22:59:24 -05:00
Author
Owner
Copy Link

@sodre commented on GitHub (Dec 8, 2023):

@trenta3, how do we turn it in in the llama.cpp case?

<!-- gh-comment-id:1847886771 --> @sodre commented on GitHub (Dec 8, 2023): @trenta3, how do we turn it in in the llama.cpp case?
GiteaMirror commented 2026-05-08 22:59:24 -05:00
Author
Owner
Copy Link

@Huvinesh-Rajendran-12 commented on GitHub (Dec 10, 2023):

pass in the -cb flag when running the server

<!-- gh-comment-id:1848870193 --> @Huvinesh-Rajendran-12 commented on GitHub (Dec 10, 2023): pass in the -cb flag when running the server
GiteaMirror commented 2026-05-08 22:59:25 -05:00
Author
Owner
Copy Link

@trenta3 commented on GitHub (Dec 14, 2023):

Yes indeed, does someone know if there is a way in ollama to pass options directly to the underlying llama.cpp?

<!-- gh-comment-id:1856782153 --> @trenta3 commented on GitHub (Dec 14, 2023): Yes indeed, does someone know if there is a way in ollama to pass options directly to the underlying llama.cpp?
GiteaMirror commented 2026-05-08 22:59:27 -05:00
Author
Owner
Copy Link

@sodre commented on GitHub (Dec 19, 2023):

The issue is less about passing the parameters down and more about ensuring that the different connection at the Ollama side use different slots of llama.cpp.

<!-- gh-comment-id:1862005618 --> @sodre commented on GitHub (Dec 19, 2023): The issue is less about passing the parameters down and more about ensuring that the different connection at the Ollama side use different slots of llama.cpp.
GiteaMirror commented 2026-05-08 22:59:27 -05:00
Author
Owner
Copy Link

@jkuehn commented on GitHub (Feb 27, 2024):

Hey, just to start the conversation: how about adding a new endpoint to Ollama that can handle batching? After we see it's working well, we could make it part of the main generate endpoint.

Like, EricLLM uses a queue and an inference loop for batching. I think it's a good and easy way to do it. People could start using it and if something comes up, we still could switch to a more sophisticated solution. I believe this would be a major feature for Ollama!
EricLLM: https://github.com/epolewski/EricLLM/blob/main/ericLLM.py

What do you think about the approach?

<!-- gh-comment-id:1966260717 --> @jkuehn commented on GitHub (Feb 27, 2024): Hey, just to start the conversation: how about adding a new endpoint to Ollama that can handle batching? After we see it's working well, we could make it part of the main generate endpoint. Like, EricLLM uses a queue and an inference loop for batching. I think it's a good and easy way to do it. People could start using it and if something comes up, we still could switch to a more sophisticated solution. I believe this would be a major feature for Ollama! EricLLM: https://github.com/epolewski/EricLLM/blob/main/ericLLM.py What do you think about the approach?
GiteaMirror commented 2026-05-08 22:59:28 -05:00
Author
Owner
Copy Link

@ianpurton commented on GitHub (Feb 27, 2024):

For me it would be great to switch on the continuous batching via the command line or env car.

Then I could use the existing open air end points.

Can anyone explain how this works with llama.cpp?

<!-- gh-comment-id:1967290265 --> @ianpurton commented on GitHub (Feb 27, 2024): For me it would be great to switch on the continuous batching via the command line or env car. Then I could use the existing open air end points. Can anyone explain how this works with llama.cpp?
GiteaMirror commented 2026-05-08 22:59:29 -05:00
Author
Owner
Copy Link

@Huvinesh-Rajendran-12 commented on GitHub (Feb 28, 2024):

pass in the -cb flag when running the server

@9876691 follow this.

<!-- gh-comment-id:1968187190 --> @Huvinesh-Rajendran-12 commented on GitHub (Feb 28, 2024): > pass in the -cb flag when running the server > > @9876691 follow this.
GiteaMirror commented 2026-05-08 22:59:30 -05:00
Author
Owner
Copy Link

@dantheman0207 commented on GitHub (Mar 20, 2024):

I would also be interested in this functionality.

<!-- gh-comment-id:2009320579 --> @dantheman0207 commented on GitHub (Mar 20, 2024): I would also be interested in this functionality.
GiteaMirror commented 2026-05-08 22:59:31 -05:00
Author
Owner
Copy Link

@MarcellM01 commented on GitHub (Mar 24, 2024):

This would be a great feature to have and would increase the utility of Ollama by an order of magnitude.

<!-- gh-comment-id:2016732866 --> @MarcellM01 commented on GitHub (Mar 24, 2024): This would be a great feature to have and would increase the utility of Ollama by an order of magnitude.
GiteaMirror commented 2026-05-08 22:59:31 -05:00
Author
Owner
Copy Link

@pawelgnatowski commented on GitHub (Jul 2, 2024):

Any news on this one?
Has anyone tried the saturation of the ollama server?

<!-- gh-comment-id:2203534924 --> @pawelgnatowski commented on GitHub (Jul 2, 2024): Any news on this one? Has anyone tried the saturation of the ollama server?
GiteaMirror commented 2026-05-08 22:59:32 -05:00
Author
Owner
Copy Link

@fzyzcjy commented on GitHub (Aug 23, 2024):

Hi, is there any updates? Thanks!

<!-- gh-comment-id:2306351638 --> @fzyzcjy commented on GitHub (Aug 23, 2024): Hi, is there any updates? Thanks!
GiteaMirror commented 2026-05-08 22:59:33 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Sep 4, 2024):

Hi there, this is enabled by default now with concurrency: https://github.com/ollama/ollama/releases/tag/v0.2.0

<!-- gh-comment-id:2327847509 --> @jmorganca commented on GitHub (Sep 4, 2024): Hi there, this is enabled by default now with concurrency: https://github.com/ollama/ollama/releases/tag/v0.2.0
GiteaMirror commented 2026-05-08 22:59:34 -05:00
Author
Owner
Copy Link

@tudormunteanu commented on GitHub (Jul 10, 2025):

Interested in this too. An equivalent to /v1/batches for OpenAI compatibility would be amazing.

What is the current blocker?

<!-- gh-comment-id:3057155735 --> @tudormunteanu commented on GitHub (Jul 10, 2025): Interested in this too. An equivalent to `/v1/batches` for OpenAI compatibility would be amazing. What is the current blocker?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#78416
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?