[GH-ISSUE #358] Parallel requests #62196

New Issue

GiteaMirror · 2026-05-03T07:49:13-05:00

GiteaMirror commented

2026-05-03 07:49:13 -05:00

Originally created by @youssef02 on GitHub (Aug 16, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/358

Originally assigned to: @dhiltgen on GitHub.

the app is amazing but the problem is If I want to create a multiagent from one api, I need to create a Queue system, as it can reply only to one request at a time, is there a way to improve this or do I have to implement a Queue system?

I just started here so sorry for any mistake, ;)

Originally created by @youssef02 on GitHub (Aug 16, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/358 Originally assigned to: @dhiltgen on GitHub. the app is amazing but the problem is If I want to create a multiagent from one api, I need to create a Queue system, as it can reply only to one request at a time, is there a way to improve this or do I have to implement a Queue system? I just started here so sorry for any mistake, ;)

GiteaMirror added the feature request label 2026-05-03 07:49:13 -05:00

GiteaMirror closed this issue

2026-05-03 07:49:14 -05:00

GiteaMirror commented

2026-05-03 07:49:16 -05:00

@jmorganca commented on GitHub (Aug 23, 2023):

Not a mistake – Ollama will serve one generation at a time currently, but supporting 2+ concurrent requests is definitely on the roadmap

@jmorganca commented on GitHub (Aug 23, 2023): Not a mistake – Ollama will serve one generation at a time currently, but supporting 2+ concurrent requests is definitely on the roadmap

GiteaMirror commented

2026-05-03 07:49:17 -05:00

@LeHaroun commented on GitHub (Oct 17, 2023):

Is there a way to run multiple instances on the same machine

the app is amazing but the problem is If I want to create a multiagent from one api, I need to create a Queue system, as it can reply only to one request at a time, is there a way to improve this or do I have to implement a Queue system?

I just started here so sorry for any mistake, ;)

I am working on a similar implementation using MAS, If you have beefy hardware, I recommend you run several instances in closed envs. Save completions or generation and process them later on.

Also check LiteLLM APIs on top of Ollama. Quite helpful.

@LeHaroun commented on GitHub (Oct 17, 2023): Is there a way to run multiple instances on the same machine > the app is amazing but the problem is If I want to create a multiagent from one api, I need to create a Queue system, as it can reply only to one request at a time, is there a way to improve this or do I have to implement a Queue system? > > I just started here so sorry for any mistake, ;) I am working on a similar implementation using MAS, If you have beefy hardware, I recommend you run several instances in closed envs. Save completions or generation and process them later on. Also check [LiteLLM](https://docs.litellm.ai/docs/providers/ollama) APIs on top of Ollama. Quite helpful.

GiteaMirror commented

2026-05-03 07:49:18 -05:00

@skye0402 commented on GitHub (Jan 9, 2024):

Would be definitely a great addition to Ollama:

Concurrency of requests
Using GPU mem for several models

I'm running it on cloud using a T4 with 16GB GPU memory and having a phi-2 and codellama both in the V-RAM would be no issue at all. Ideally other models are kept in regular RAM instead of loading it from disk.
Adding to it: Users might switch models, so the queuing approach would apply to model switches, too 😃

@skye0402 commented on GitHub (Jan 9, 2024): Would be definitely a great addition to Ollama: - Concurrency of requests - Using GPU mem for several models I'm running it on cloud using a T4 with 16GB GPU memory and having a phi-2 and codellama both in the V-RAM would be no issue at all. Ideally other models are kept in regular RAM instead of loading it from disk. Adding to it: Users might switch models, so the queuing approach would apply to model switches, too 😃

GiteaMirror commented

2026-05-03 07:49:19 -05:00

@ivanfioravanti commented on GitHub (Jan 18, 2024):

Any news on this one? Parallel requests can be a real game changer for Ollama

@ivanfioravanti commented on GitHub (Jan 18, 2024): Any news on this one? Parallel requests can be a real game changer for Ollama

GiteaMirror commented

2026-05-03 07:49:21 -05:00

@ParisNeo commented on GitHub (Jan 30, 2024):

If you have enough VRAM to run multiple models, you can create multiple instances of ollama with different port numbers then use my proxy to manage access and route to the requests to each:
ParisNeo's ollama_proxy_server

@ParisNeo commented on GitHub (Jan 30, 2024): If you have enough VRAM to run multiple models, you can create multiple instances of ollama with different port numbers then use my proxy to manage access and route to the requests to each: [ParisNeo's ollama_proxy_server](https://github.com/ParisNeo/ollama_proxy_server)

GiteaMirror commented

2026-05-03 07:49:21 -05:00

@ehartford commented on GitHub (Jan 30, 2024):

I would really like to have an m2 ultra 192gb on my company's intranet that can service the whole R&D department (a dozen people)

as long as I have enough ram, I wish to be able to run multiple inference request at the same time. Thank you for considering my wish!

@ehartford commented on GitHub (Jan 30, 2024): I would really like to have an m2 ultra 192gb on my company's intranet that can service the whole R&D department (a dozen people) as long as I have enough ram, I wish to be able to run multiple inference request at the same time. Thank you for considering my wish!

GiteaMirror commented

2026-05-03 07:49:23 -05:00

@ivanfioravanti commented on GitHub (Jan 30, 2024):

same here, multiple 7B models served by an M2 Ultra. My dream! 🙏

@ivanfioravanti commented on GitHub (Jan 30, 2024): same here, multiple 7B models served by an M2 Ultra. My dream! 🙏

GiteaMirror commented

2026-05-03 07:49:24 -05:00

@Adphi commented on GitHub (Jan 31, 2024):

At first glance, when I started examining the source code, I thought that the problem of concurrent requests was due to the fact that the current implementation referred to the use of global variables in its llama.cpp binding. After paying a little more attention to the original llapma.cpp source code, I realized that the original implementation wasn't exactly geared towards multithreaded or server-side use, but rather towards to a local development/experimentation use case.
So apart from setting up an API using queued workers (managed using gRPC for example) based on a fork/exec model to get around the lack of batch processing on the llama.cpp side, which doesn't seem to procure too much attention from the developpers (reference missing, but found in the github project's conversations), which Local AI does in a way, I don't see exactly how it could be implemented here.

@Adphi commented on GitHub (Jan 31, 2024): At first glance, when I started examining the source code, I thought that the problem of concurrent requests was due to the fact that the current implementation referred to the use of global variables in its llama.cpp binding. After paying a little more attention to the original llapma.cpp source code, I realized that the original implementation wasn't exactly geared towards multithreaded or server-side use, but rather towards to a local development/experimentation use case. So apart from setting up an API using queued workers (managed using gRPC for example) based on a fork/exec model to get around the lack of batch processing on the llama.cpp side, which doesn't seem to procure too much attention from the developpers (reference missing, but found in the github project's conversations), which Local AI does in a way, I don't see exactly how it could be implemented here.

GiteaMirror commented

2026-05-03 07:49:25 -05:00

@ParisNeo commented on GitHub (Jan 31, 2024):

That's why I had to build a proxy. You can install multiple servers on a single or multiple machines then use my proxy to service multiple users with a multi queue. For example if you have 2 servers then you can servicec at most 2 clients simultaniously and when the two are buzy, the current message is queued on the server that has the least full queue. In practice, since the generation is very fast, you don't get too much collisions. We rarely get more than 1 person in the queue.

I also added security features as well as logging features. It works fine and serves all of us perfectly.

Also, some dude built a docker docker version and I accepted his PR:
https://github.com/ParisNeo/ollama_proxy_server

If you are interested, you can try it, it is open source, you can also read the code, get inspired, help enhance it etc. It is Apache 2.0 so you can do whatever you want with it 100% free.

We get around 300 tokens/s for each user so it is not causing any significant delay.

@ParisNeo commented on GitHub (Jan 31, 2024): That's why I had to build a proxy. You can install multiple servers on a single or multiple machines then use my proxy to service multiple users with a multi queue. For example if you have 2 servers then you can servicec at most 2 clients simultaniously and when the two are buzy, the current message is queued on the server that has the least full queue. In practice, since the generation is very fast, you don't get too much collisions. We rarely get more than 1 person in the queue. I also added security features as well as logging features. It works fine and serves all of us perfectly. Also, some dude built a docker docker version and I accepted his PR: https://github.com/ParisNeo/ollama_proxy_server If you are interested, you can try it, it is open source, you can also read the code, get inspired, help enhance it etc. It is Apache 2.0 so you can do whatever you want with it 100% free. We get around 300 tokens/s for each user so it is not causing any significant delay.

GiteaMirror commented

2026-05-03 07:49:26 -05:00

@ehartford commented on GitHub (Jan 31, 2024):

llama.cpp can be run as multiple process, multiple thread, or as server.
but it's totally fair for the feature request to be deprioritized, of course.

@ehartford commented on GitHub (Jan 31, 2024): llama.cpp can be run as multiple process, multiple thread, or as server. but it's totally fair for the feature request to be deprioritized, of course.

GiteaMirror commented

2026-05-03 07:49:27 -05:00

@farhanhubble commented on GitHub (Feb 1, 2024):

The easiest way to multiplex Ollama, at least on linux system should be with a reverse proxy load balancer like HAProxy. Launch multiple instances of ollama serve on different ports and map them to a single port using HAproxy.

Note that the approach can sometimes deteriorate performance due to CPU contention. I have a decent system with 64 cores and 24GB of GPU RAM. When I run 3 instances of Ollama with HAproxy to generate embeddings it does speed up the process however if I try to generate text, the processing time is much worse than with a single instance.

@farhanhubble commented on GitHub (Feb 1, 2024): The easiest way to multiplex Ollama, at least on linux system should be with a reverse proxy load balancer like HAProxy. Launch multiple instances of `ollama serve` on different ports and map them to a single port using HAproxy. Note that the approach can sometimes deteriorate performance due to CPU contention. I have a decent system with 64 cores and 24GB of GPU RAM. When I run 3 instances of Ollama with HAproxy to generate embeddings it does speed up the process however if I try to generate text, the processing time is much worse than with a single instance.

GiteaMirror commented

2026-05-03 07:49:28 -05:00

@trymeouteh commented on GitHub (Mar 1, 2024):

Would like to see the ability for using the same LLM in two or more apps at the same time and would like to see the ability to use multiple LLMs in two or more apps at the same time

@trymeouteh commented on GitHub (Mar 1, 2024): Would like to see the ability for using the same LLM in two or more apps at the same time and would like to see the ability to use multiple LLMs in two or more apps at the same time

GiteaMirror commented

2026-05-03 07:49:30 -05:00

@ParisNeo commented on GitHub (Mar 1, 2024):

well technically, you can run multiple instances of the same model by runnin multiple instances of ollama with different port numbers, configure them in the proxy config file then they can be accessed from multiple clients at once.

@ParisNeo commented on GitHub (Mar 1, 2024): well technically, you can run multiple instances of the same model by runnin multiple instances of ollama with different port numbers, configure them in the proxy config file then they can be accessed from multiple clients at once.

GiteaMirror commented

2026-05-03 07:49:32 -05:00

@ehartford commented on GitHub (Mar 1, 2024):

That's just wrong 😂

@ehartford commented on GitHub (Mar 1, 2024): That's just wrong 😂

GiteaMirror commented

2026-05-03 07:49:35 -05:00

@ParisNeo commented on GitHub (Mar 1, 2024):

ollama does not support batching and unless they build something like vllm, I don't see how you can do it.

@ParisNeo commented on GitHub (Mar 1, 2024): ollama does not support batching and unless they build something like vllm, I don't see how you can do it.

GiteaMirror commented

2026-05-03 07:49:38 -05:00

@ehartford commented on GitHub (Mar 1, 2024):

manage a process/thread pool. Load balance. It's not rocket science.

@ehartford commented on GitHub (Mar 1, 2024): manage a process/thread pool. Load balance. It's not rocket science.

GiteaMirror commented

2026-05-03 07:49:41 -05:00

@youssef02 commented on GitHub (Mar 1, 2024):

This is somehow a problem with scaling, I think. While ollama imodel can only handle one request at a time. It can't be concurrent to multiple request as it will need to run new models, which take more resources. I really want for someone to share their finding when trying to handle multiple request like in a uni environment with many student.

How to approach this problem

@youssef02 commented on GitHub (Mar 1, 2024): This is somehow a problem with scaling, I think. While ollama imodel can only handle one request at a time. It can't be concurrent to multiple request as it will need to run new models, which take more resources. I really want for someone to share their finding when trying to handle multiple request like in a uni environment with many student. How to approach this problem

GiteaMirror commented

2026-05-03 07:49:42 -05:00

@ehartford commented on GitHub (Mar 2, 2024):

The idea is the system (a MacBook or Mac studio with 128gb+ of unified RAM) can handle several dozen 4-bit 7b models at once.

So taking more resources is ok, no problem.

@ehartford commented on GitHub (Mar 2, 2024): The idea is the system (a MacBook or Mac studio with 128gb+ of unified RAM) can handle several dozen 4-bit 7b models at once. So taking more resources is ok, no problem.

GiteaMirror commented

2026-05-03 07:49:43 -05:00

@ParisNeo commented on GitHub (Mar 2, 2024):

The idea is the system (a MacBook or Mac studio with 128gb+ of unified RAM) can handle several dozen 4-bit 7b models at once.

So taking more resources is ok, no problem.

If you want that way, then you can use my ollama_proxy project, it does exactly what you asked for.
If multiplying the number of resources doesn't bother you then this is a simple solution. Each ollama process runs on its own and the proxy manages the routing and queues and everything:
https://github.com/ParisNeo/ollama_proxy_server

I was really wanting a better more optimized way. Neural nets can do inferance on a batch of inputs. You don't need to duplicate the model weights. That's what we talk about when we say real multigeneration system. You can perform fast inference with that, just as in vllm.

@ParisNeo commented on GitHub (Mar 2, 2024): > The idea is the system (a MacBook or Mac studio with 128gb+ of unified RAM) can handle several dozen 4-bit 7b models at once. > > So taking more resources is ok, no problem. If you want that way, then you can use my ollama_proxy project, it does exactly what you asked for. If multiplying the number of resources doesn't bother you then this is a simple solution. Each ollama process runs on its own and the proxy manages the routing and queues and everything: [https://github.com/ParisNeo/ollama_proxy_server](https://github.com/ParisNeo/ollama_proxy_server) I was really wanting a better more optimized way. Neural nets can do inferance on a batch of inputs. You don't need to duplicate the model weights. That's what we talk about when we say real multigeneration system. You can perform fast inference with that, just as in vllm.

GiteaMirror commented

2026-05-03 07:49:44 -05:00

@ehartford commented on GitHub (Mar 2, 2024):

I appreciate that you have provided a workaround.

Basically that's what ollama needs to implement internally.

I want ollama to be smart, and internally spawn and destroy llama.cpp threads/processes as needed to handle parallel requests without an external proxy.

That's what I'm asking for. it should "just work" ollama-style.

@ehartford commented on GitHub (Mar 2, 2024): I appreciate that you have provided a workaround. Basically that's what ollama needs to implement internally. I want ollama to be smart, and internally spawn and destroy llama.cpp threads/processes as needed to handle parallel requests without an external proxy. That's what I'm asking for. it should "just work" ollama-style.

GiteaMirror commented

2026-05-03 07:49:46 -05:00

@ffaubert commented on GitHub (Mar 18, 2024):

+1 for this. It would be a game changer. @bmizerany is this something in progress?

@ffaubert commented on GitHub (Mar 18, 2024): +1 for this. It would be a game changer. @bmizerany is this something in progress?

GiteaMirror commented

2026-05-03 07:49:47 -05:00

@triet0612 commented on GitHub (Mar 19, 2024):

I notice that new requests will wait for old requests to complete despite that we interrupt and stop old request when they're running or waiting to be executed. Have anyone have this issue? Thank you and sorry for taking your time.

@triet0612 commented on GitHub (Mar 19, 2024): I notice that new requests will wait for old requests to complete despite that we interrupt and stop old request when they're running or waiting to be executed. Have anyone have this issue? Thank you and sorry for taking your time.

GiteaMirror commented

2026-05-03 07:49:48 -05:00

@ehartford commented on GitHub (Mar 19, 2024):

I notice similar sometimes I have to kill and restart ollama because of it but I think that deserves it's own issue.

@ehartford commented on GitHub (Mar 19, 2024): I notice similar sometimes I have to kill and restart ollama because of it but I think that deserves it's own issue.

GiteaMirror commented

2026-05-03 07:49:49 -05:00

@jammsen commented on GitHub (Mar 20, 2024):

Not a mistake – Ollama will serve one generation at a time currently, but supporting 2+ concurrent requests is definitely on the roadmap

@jmorganca you wrote its on the roadmap, could you explain for when? Do you have a date?

Something beeing on a roadmap doesnt in reality mean it gets solved/done/programmed.

Could you help us/the community out please? The request is from 6months++ ago.

@jammsen commented on GitHub (Mar 20, 2024): > Not a mistake – Ollama will serve one generation at a time currently, but supporting 2+ concurrent requests is definitely on the roadmap @jmorganca you wrote its on the roadmap, could you explain for when? Do you have a date? Something beeing on a roadmap doesnt in reality mean it gets solved/done/programmed. Could you help us/the community out please? The request is from 6months++ ago.

GiteaMirror commented

2026-05-03 07:49:50 -05:00

@ehartford commented on GitHub (Mar 20, 2024):

this feature would allow ollama to be a production backend. Low traffic for sure but still, this would get a lot of startup running / hosting MVP with minimal effort.

@ehartford commented on GitHub (Mar 20, 2024): this feature would allow ollama to be a production backend. Low traffic for sure but still, this would get a lot of startup running / hosting MVP with minimal effort.

GiteaMirror commented

2026-05-03 07:49:51 -05:00

@ParisNeo commented on GitHub (Mar 24, 2024):

In the meantime you can try my ollama proxy :)
It can be configured to fire up multiple ollama services, then you can also define users with keys to access the service and it balances the loads between the instances of ollama with multiple queues.

@ParisNeo commented on GitHub (Mar 24, 2024): In the meantime you can try my ollama proxy :) It can be configured to fire up multiple ollama services, then you can also define users with keys to access the service and it balances the loads between the instances of ollama with multiple queues.

GiteaMirror commented

2026-05-03 07:49:52 -05:00

@jammsen commented on GitHub (Mar 24, 2024):

In the meantime you can try my ollama proxy :) It can be configured to fire up multiple ollama services, then you can also define users with keys to access the service and it balances the loads between the instances of ollama with multiple queues.

@ParisNeo
Do you have a link for me please? Also how does that works from an architectur point of view? I guessing sticky-session, so people usng this wont get mixed up and have a english cooking recipe with c++ snippets and german dictionaries in there right?

@jammsen commented on GitHub (Mar 24, 2024): > In the meantime you can try my ollama proxy :) It can be configured to fire up multiple ollama services, then you can also define users with keys to access the service and it balances the loads between the instances of ollama with multiple queues. @ParisNeo Do you have a link for me please? Also how does that works from an architectur point of view? I guessing sticky-session, so people usng this wont get mixed up and have a english cooking recipe with c++ snippets and german dictionaries in there right?

GiteaMirror commented

2026-05-03 07:49:53 -05:00

@m4r1k commented on GitHub (Mar 26, 2024):

While the ollama_proxy_server is a valuable starting point, Ollama's lack of native parallel request handling hinders its scalability. This is particularly evident in cloud environments where Ollama pods/instances could dynamically scale based on load.

The current approach in ollama_proxy_server requires hardcoded backend configuration, complicating consistent and dynamic scaling.

@m4r1k commented on GitHub (Mar 26, 2024): While the [`ollama_proxy_server`](https://github.com/ParisNeo/ollama_proxy_server) is a valuable starting point, Ollama's lack of native parallel request handling hinders its scalability. This is particularly evident in cloud environments where Ollama pods/instances could dynamically scale based on load. The current approach in `ollama_proxy_server` requires hardcoded backend configuration, complicating consistent and dynamic scaling.

GiteaMirror commented

2026-05-03 07:49:54 -05:00

@0x77dev commented on GitHub (Mar 27, 2024):

This issue seems to be stale for a while. Ollama is a promising project with the potential to expand beyond just "running on laptops." I believe there are many projects that wish to use it in production despite facing challenges (pre-pulling a model on start, custom model registry, etc), with parallel requests being a significant obstacle in my opinion.

Merged PR #3348 has newer version of llama.cpp, where parallel decoding is supported for sure.

I will try to delve into the Ollama codebase to address this issue. It would be helpful if a maintainer or someone familiar with the /llm section of the codebase could identify any blockers, maybe potential blocking of parallel processing of requests on the Golang side.

Discord thread
https://github.com/ollama/ollama/issues/380#issuecomment-1762925095
The llama.cpp server example demonstrates support for parallel decoding.

@0x77dev commented on GitHub (Mar 27, 2024): [This issue seems to be stale for a while.](https://github.com/ollama/ollama/issues/358#issuecomment-2010280146) Ollama is a promising project with the potential to expand beyond just "running on laptops." I believe there are many projects that wish [to use it in production despite facing challenges](https://github.com/ollama/ollama/issues/358#issuecomment-2010706535) ([pre-pulling a model on start](https://github.com/ollama/ollama/issues/3369#issue-2210418734), custom model registry, etc), with parallel requests being a significant obstacle in my opinion. Merged PR #3348 has newer version of llama.cpp, where parallel decoding is supported for sure. I will try to delve into the Ollama codebase to address this issue. It would be helpful if a maintainer or someone familiar with the `/llm` section of the codebase could identify any blockers, maybe potential blocking of parallel processing of requests on the Golang side. Related: - [Discord thread](https://discordapp.com/channels/1128867683291627614/1128867684130508875/1222495299113713758) - https://github.com/ollama/ollama/issues/380#issuecomment-1762925095 - The [llama.cpp server example](https://github.com/ggerganov/llama.cpp/tree/master/examples/server#llamacpp-http-server) demonstrates support for parallel decoding.

GiteaMirror commented

2026-05-03 07:49:56 -05:00

@0x77dev commented on GitHub (Mar 28, 2024):

There has been some progress, but there are still issues to address. For instance, support for calling different models simultaneously (maybe calling different models simultaneously can be skipped, but it may cause DX confusion), and loaded.mu.{Lock,Unlock}() is not implemented correctly in my changes below.

This will require more testing and work on a better design for model loading

https://github.com/ollama/ollama/assets/46429701/00d84851-ca18-4e0a-a939-be367d49226d

git diff

diff --git a/llm/dyn_ext_server.go b/llm/dyn_ext_server.go
index 6e43333..5b0e435 100644
--- a/llm/dyn_ext_server.go
+++ b/llm/dyn_ext_server.go
@@ -95,7 +95,7 @@ func newDynExtServer(library, model string, adapters, projectors []string, opts
 	sparams.n_batch = C.uint(opts.NumBatch)
 	sparams.n_gpu_layers = C.int(opts.NumGPU)
 	sparams.main_gpu = C.int(opts.MainGPU)
-	sparams.n_parallel = 1 // TODO - wire up concurrency
+	sparams.n_parallel = 4 // TODO - wire up concurrency
 
 	// Always use the value encoded in the model
 	sparams.rope_freq_base = 0.0
diff --git a/server/routes.go b/server/routes.go
index 5582f3a..1ce55e6 100644
--- a/server/routes.go
+++ b/server/routes.go
@@ -67,8 +67,10 @@ var loaded struct {
 
 var defaultSessionDuration = 5 * time.Minute
 
-// load a model into memory if it is not already loaded, it is up to the caller to lock loaded.mu before calling this function
 func load(c *gin.Context, model *Model, opts api.Options, sessionDuration time.Duration) error {
+	loaded.mu.Lock()
+	defer loaded.mu.Unlock()
+
 	needLoad := loaded.runner == nil || // is there a model loaded?
 		loaded.ModelPath != model.ModelPath || // has the base model changed?
 		!reflect.DeepEqual(loaded.AdapterPaths, model.AdapterPaths) || // have the adapters changed?
@@ -145,9 +147,6 @@ func isSupportedImageType(image []byte) bool {
 }
 
 func GenerateHandler(c *gin.Context) {
-	loaded.mu.Lock()
-	defer loaded.mu.Unlock()
-
 	checkpointStart := time.Now()
 	var req api.GenerateRequest
 	err := c.ShouldBindJSON(&req)

@0x77dev commented on GitHub (Mar 28, 2024): There has been some progress, but there are still issues to address. For instance, support for calling different models simultaneously (maybe calling different models simultaneously can be skipped, but it may cause _DX confusion_), and `loaded.mu.{Lock,Unlock}()` is not implemented correctly in my changes below. This will require more testing and work on a better design for model loading https://github.com/ollama/ollama/assets/46429701/00d84851-ca18-4e0a-a939-be367d49226d <details> <summary> git diff </summary> ```diff diff --git a/llm/dyn_ext_server.go b/llm/dyn_ext_server.go index 6e43333..5b0e435 100644 --- a/llm/dyn_ext_server.go +++ b/llm/dyn_ext_server.go @@ -95,7 +95,7 @@ func newDynExtServer(library, model string, adapters, projectors []string, opts sparams.n_batch = C.uint(opts.NumBatch) sparams.n_gpu_layers = C.int(opts.NumGPU) sparams.main_gpu = C.int(opts.MainGPU) - sparams.n_parallel = 1 // TODO - wire up concurrency + sparams.n_parallel = 4 // TODO - wire up concurrency // Always use the value encoded in the model sparams.rope_freq_base = 0.0 diff --git a/server/routes.go b/server/routes.go index 5582f3a..1ce55e6 100644 --- a/server/routes.go +++ b/server/routes.go @@ -67,8 +67,10 @@ var loaded struct { var defaultSessionDuration = 5 * time.Minute -// load a model into memory if it is not already loaded, it is up to the caller to lock loaded.mu before calling this function func load(c *gin.Context, model *Model, opts api.Options, sessionDuration time.Duration) error { + loaded.mu.Lock() + defer loaded.mu.Unlock() + needLoad := loaded.runner == nil || // is there a model loaded? loaded.ModelPath != model.ModelPath || // has the base model changed? !reflect.DeepEqual(loaded.AdapterPaths, model.AdapterPaths) || // have the adapters changed? @@ -145,9 +147,6 @@ func isSupportedImageType(image []byte) bool { } func GenerateHandler(c *gin.Context) { - loaded.mu.Lock() - defer loaded.mu.Unlock() - checkpointStart := time.Now() var req api.GenerateRequest err := c.ShouldBindJSON(&req) ``` </details>

GiteaMirror commented

2026-05-03 07:49:56 -05:00

@easp commented on GitHub (Mar 30, 2024):

Looks relevant: https://github.com/ollama/ollama/pull/3418

@easp commented on GitHub (Mar 30, 2024): Looks relevant: https://github.com/ollama/ollama/pull/3418

GiteaMirror commented

2026-05-03 07:49:57 -05:00

@darwinvelez58 commented on GitHub (Apr 9, 2024):

any update?

@darwinvelez58 commented on GitHub (Apr 9, 2024): any update?

GiteaMirror commented

2026-05-03 07:49:58 -05:00

@0x77dev commented on GitHub (Apr 9, 2024):

@darwinvelez58 there is some work being done in https://github.com/ollama/ollama/pull/3418 ~~and a9195b3efa~~

was testing both options, at the moment seems very unstable

@0x77dev commented on GitHub (Apr 9, 2024): @darwinvelez58 there is some work being done in https://github.com/ollama/ollama/pull/3418 ~~and https://github.com/Adphi/ollama/commit/a9195b3efa3a63ea8dee2907ecc7c06dccc59e6f~~ was testing both options, at the moment seems very unstable

GiteaMirror commented

2026-05-03 07:49:59 -05:00

@ParisNeo commented on GitHub (Apr 9, 2024):

In the meantime you can try my ollama proxy :) It can be configured to fire up multiple ollama services, then you can also define users with keys to access the service and it balances the loads between the instances of ollama with multiple queues.

@ParisNeo Do you have a link for me please? Also how does that works from an architectur point of view? I guessing sticky-session, so people usng this wont get mixed up and have a english cooking recipe with c++ snippets and german dictionaries in there right?

Hi, sorry I was very very buzy lately and had no time to look at the github messages.
The proxy server can be found here:
https://github.com/ParisNeo/ollama_proxy_server

It has also an optional security to allow you to have a users base with a personal key for each user.
The architecture is simple:
You specify multiple ollama instances viewable only on the server side each one has a different port number, then the proxy manages both the authentication of the users, the logging of their access, and distribution of the users over multiple queues.
Each queue is behind one instance of ollama and the proxy will always put you in the least filled queue.
So if you run 4 ollama instances, you can simultaniously serve 4 users and the others will be queued.

I agree that you can do more if ollama manages this all inside to mutualize the weights and stuff. I have built my proxy because I needed the security and the management and back then ollama had no plans to do this so got to move, but I'll be happy if they integrate multiusers directly inside.

@ParisNeo commented on GitHub (Apr 9, 2024): > > In the meantime you can try my ollama proxy :) It can be configured to fire up multiple ollama services, then you can also define users with keys to access the service and it balances the loads between the instances of ollama with multiple queues. > > @ParisNeo Do you have a link for me please? Also how does that works from an architectur point of view? I guessing sticky-session, so people usng this wont get mixed up and have a english cooking recipe with c++ snippets and german dictionaries in there right? Hi, sorry I was very very buzy lately and had no time to look at the github messages. The proxy server can be found here: [https://github.com/ParisNeo/ollama_proxy_server](https://github.com/ParisNeo/ollama_proxy_server) It has also an optional security to allow you to have a users base with a personal key for each user. The architecture is simple: You specify multiple ollama instances viewable only on the server side each one has a different port number, then the proxy manages both the authentication of the users, the logging of their access, and distribution of the users over multiple queues. Each queue is behind one instance of ollama and the proxy will always put you in the least filled queue. So if you run 4 ollama instances, you can simultaniously serve 4 users and the others will be queued. I agree that you can do more if ollama manages this all inside to mutualize the weights and stuff. I have built my proxy because I needed the security and the management and back then ollama had no plans to do this so got to move, but I'll be happy if they integrate multiusers directly inside.

GiteaMirror commented

2026-05-03 07:50:01 -05:00

@TheMasterFX commented on GitHub (Apr 10, 2024):

vLLM uses a PagedAttention. Is this a part which must be integrated in ollama or in the llama.cpp part?

@TheMasterFX commented on GitHub (Apr 10, 2024): vLLM uses a [PagedAttention](https://blog.vllm.ai/2023/06/20/vllm.html). Is this a part which must be integrated in ollama or in the llama.cpp part?

GiteaMirror commented

2026-05-03 07:50:03 -05:00

@0x77dev commented on GitHub (Apr 10, 2024):

@TheMasterFX this is llama.cpp side of things

@0x77dev commented on GitHub (Apr 10, 2024): @TheMasterFX this is llama.cpp side of things

GiteaMirror commented

2026-05-03 07:50:04 -05:00

@TheMasterFX commented on GitHub (Apr 12, 2024):

Seems like there are improvements coming:
https://twitter.com/AlexReibman/status/1778695203957977148

@TheMasterFX commented on GitHub (Apr 12, 2024): Seems like there are improvements coming: https://twitter.com/AlexReibman/status/1778695203957977148

GiteaMirror commented

2026-05-03 07:50:05 -05:00

@guitmonk-1290 commented on GitHub (Apr 21, 2024):

I was reading ray serve's docs and it seems they support Dynamic Request Batching, which creates a queue system.
https://docs.ray.io/en/latest/serve/advanced-guides/dyn-req-batch.html

I am new to this, so can we use this with Ollama for batch inference?

@guitmonk-1290 commented on GitHub (Apr 21, 2024): I was reading ray serve's docs and it seems they support Dynamic Request Batching, which creates a queue system. https://docs.ray.io/en/latest/serve/advanced-guides/dyn-req-batch.html I am new to this, so can we use this with Ollama for batch inference?

GiteaMirror commented

2026-05-03 07:50:06 -05:00

@jpmcb commented on GitHub (Apr 24, 2024):

I've had some success load balancing multiple Ollama instances on Kubernetes with a pool of GPU nodes:

❯ kubectl get nodes -A
NAME                     STATUS   ROLES    AGE   VERSION
defaultpool-88943984-0   Ready    <none>   5d    v1.29.2
defaultpool-88943984-1   Ready    <none>   5d    v1.29.2
gpupool-42074538-0       Ready    <none>   41h   v1.29.2
gpupool-42074538-1       Ready    <none>   41h   v1.29.2
gpupool-42074538-2       Ready    <none>   41h   v1.29.2
gpupool-42074538-3       Ready    <none>   41h   v1.29.2
gpupool-42074538-4       Ready    <none>   41h   v1.29.2

Notice the 5 "gpupool" nodes, each with a T4 nvidia GPU. Ollama is then deployed as a daemonset on each of the GPU nodes:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ollama-daemonset
  namespace: ollama-ns
spec:
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
      - image: ollama/ollama:latest
        imagePullPolicy: Always
        lifecycle:
          postStart:
            exec:
              command:
              - ollama
              - pull
              - llama3
        name: ollama
        ports:
        - containerPort: 11434
          protocol: TCP

      # A few special selectors / tolerations to schedule on the GPU nodes
      nodeSelector:
        accelerator: nvidia
      tolerations:
      - effect: NoSchedule
        key: sku
        operator: Equal
        value: gpu

These then can be load balanced behind a Kubernetes service with the ClusterIP being exposed internally "as a service" to the rest of the cluster:

apiVersion: v1
kind: Service
metadata:
  name: ollama-service
  namespace: ollama-ns
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 11434
  selector:
    app: ollama
  sessionAffinity: None
  type: ClusterIP

Inside the cluster, I can hit the load balancing service resolved by CoreDNS:

$ curl ollama-service.ollama-ns.svc.cluster.local
Ollama is running

This works pretty well for slower requests but has 2 problems:

The scaling of this is directly tied to the number of nodes (and in this case, expensive GPU nodes) that have daemonsets on them
Kubernetes load balancing isn't perfect and could very easily send a request to a "busy" daemonset pod that already is servicing a request.
- I believe when this happens, the whole pod seems to stale and doesn't print any additional logs, just seems to hang. Might be worth some more experimentation to see if there's a smoking gun when multiple requests come through

It'd be absolutely incredible if multiple requests could be serviced by a single ollama serve: It would take this example daemonset deployment with a pool of GPUs and make it much more scalable.

Thanks for all the amazing work on ollama - I'm happy to test anything out and report back if there's a ollama/ollama:concurrency test image or if there's code reviews needed ❤️

@jpmcb commented on GitHub (Apr 24, 2024): I've had some success load balancing multiple Ollama instances on Kubernetes with a pool of GPU nodes: ``` ❯ kubectl get nodes -A NAME STATUS ROLES AGE VERSION defaultpool-88943984-0 Ready <none> 5d v1.29.2 defaultpool-88943984-1 Ready <none> 5d v1.29.2 gpupool-42074538-0 Ready <none> 41h v1.29.2 gpupool-42074538-1 Ready <none> 41h v1.29.2 gpupool-42074538-2 Ready <none> 41h v1.29.2 gpupool-42074538-3 Ready <none> 41h v1.29.2 gpupool-42074538-4 Ready <none> 41h v1.29.2 ``` Notice the 5 "gpupool" nodes, each with a T4 nvidia GPU. Ollama is then deployed as a daemonset on each of the GPU nodes: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: ollama-daemonset namespace: ollama-ns spec: selector: matchLabels: app: ollama template: metadata: labels: app: ollama spec: containers: - image: ollama/ollama:latest imagePullPolicy: Always lifecycle: postStart: exec: command: - ollama - pull - llama3 name: ollama ports: - containerPort: 11434 protocol: TCP # A few special selectors / tolerations to schedule on the GPU nodes nodeSelector: accelerator: nvidia tolerations: - effect: NoSchedule key: sku operator: Equal value: gpu ``` These then can be load balanced behind a Kubernetes service with the ClusterIP being exposed internally "as a service" to the rest of the cluster: ```yaml apiVersion: v1 kind: Service metadata: name: ollama-service namespace: ollama-ns spec: ports: - port: 80 protocol: TCP targetPort: 11434 selector: app: ollama sessionAffinity: None type: ClusterIP ``` Inside the cluster, I can hit the load balancing service resolved by CoreDNS: ``` $ curl ollama-service.ollama-ns.svc.cluster.local Ollama is running ``` --- This works _pretty well_ for slower requests but has 2 problems: - The scaling of this is directly tied to the number of nodes (and in this case, expensive GPU nodes) that have daemonsets on them - Kubernetes load balancing isn't perfect and could very easily send a request to a "busy" daemonset pod that already is servicing a request. - I believe when this happens, the whole pod seems to stale and doesn't print any additional logs, just seems to hang. Might be worth some more experimentation to see if there's a smoking gun when multiple requests come through It'd be absolutely incredible if multiple requests could be serviced by a single `ollama serve`: It would take this example daemonset deployment with a pool of GPUs and make it much more scalable. Thanks for all the amazing work on ollama - I'm happy to test anything out and report back if there's a `ollama/ollama:concurrency` test image or if there's code reviews needed ❤️

GiteaMirror commented

2026-05-03 07:50:06 -05:00

@skye0402 commented on GitHub (Apr 26, 2024):

I'm with vLLM now for that use case, it's very fast and manages concurrency plus has a big model pool and openAI compatible API.
I'm still with Ollama for trying out new models that's where it really shines and saves time.

@skye0402 commented on GitHub (Apr 26, 2024): I'm with vLLM now for that use case, it's very fast and manages concurrency plus has a big model pool and openAI compatible API. I'm still with Ollama for trying out new models that's where it really shines and saves time.

GiteaMirror commented

2026-05-03 07:50:08 -05:00

@shing100 commented on GitHub (Apr 29, 2024):

I really wish this feature was added.

@shing100 commented on GitHub (Apr 29, 2024): I really wish this feature was added.

GiteaMirror commented

2026-05-03 07:50:09 -05:00

@iakashpaul commented on GitHub (Apr 29, 2024):

For anyone else stumbling onto this thread-

To use webUI with parallel chat generation on multiple clients use TGI or server.cpp/duplicate my HF Space onto a GPU instance directly with parallel flag set as 2 or more basis your VRAM, then run the ollama container with server.cpp acting as an OpenAI replacement as shown below.

Server.cpp

 ./server -m ./models/mistral-7b-instruct-v0.2.Q8_0.gguf -ngl 30 --host 127.0.0.1 --port 8080 --parallel 2

Ollama container

 docker run  -p 3000:8080   -v open-webui:/app/backend/data   -e OPENAI_API_BASE_URLS="http://127.0.0.1:8080/v1"   -e OPENAI_API_KEYS="randomplaceholder"   --restart always   ghcr.io/open-webui/open-webui:main

cc: @ehartford

Latest release has experimental flags,

OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ollama serve

@iakashpaul commented on GitHub (Apr 29, 2024): ## For anyone else stumbling onto this thread- To use webUI with parallel chat generation on multiple clients use [TGI or server.cpp/duplicate my HF Space onto a GPU instance](https://huggingface.co/spaces/iAkashPaul/Ghudsavar) directly with parallel flag set as 2 or more basis your VRAM, then run the ollama container with server.cpp acting as an OpenAI replacement as shown below. > Server.cpp ```bash ./server -m ./models/mistral-7b-instruct-v0.2.Q8_0.gguf -ngl 30 --host 127.0.0.1 --port 8080 --parallel 2 ``` > Ollama container ```bash docker run -p 3000:8080 -v open-webui:/app/backend/data -e OPENAI_API_BASE_URLS="http://127.0.0.1:8080/v1" -e OPENAI_API_KEYS="randomplaceholder" --restart always ghcr.io/open-webui/open-webui:main ``` cc: @ehartford ----- Latest [release](https://github.com/ollama/ollama/releases/tag/v0.1.33-rc5) has experimental flags, ``` OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ollama serve ```

GiteaMirror commented

2026-05-03 07:50:10 -05:00

@mili-tan commented on GitHub (Apr 29, 2024):

https://github.com/ollama/ollama/releases/tag/v0.1.33-rc5

@mili-tan commented on GitHub (Apr 29, 2024): [https://github.com/ollama/ollama/releases/tag/v0.1.33-rc5](https://github.com/ollama/ollama/releases/tag/v0.1.33-rc5)

GiteaMirror commented

2026-05-03 07:50:12 -05:00

@hanzlaramey commented on GitHub (Apr 29, 2024):

game changer!!! Thank you @mili-tan

@hanzlaramey commented on GitHub (Apr 29, 2024): game changer!!! Thank you @mili-tan

GiteaMirror commented

2026-05-03 07:50:12 -05:00

@ehartford commented on GitHub (Apr 30, 2024):

Thank you!

@ehartford commented on GitHub (Apr 30, 2024): Thank you!

GiteaMirror commented

2026-05-03 07:50:14 -05:00

@ehartford commented on GitHub (Apr 30, 2024):

If I don't want to limit the number of loaded models, I just don't set that variable?

@ehartford commented on GitHub (Apr 30, 2024): If I don't want to limit the number of loaded models, I just don't set that variable?

GiteaMirror commented

2026-05-03 07:50:17 -05:00

@BBjie commented on GitHub (Apr 30, 2024):

My question is for the concurrency feature, can I edit and add them into my compose-up file.

ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"

    
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    command: serve
volumes:
  ollama:

or is there other way to pass the value in for
OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ollama serve

@BBjie commented on GitHub (Apr 30, 2024): My question is for the concurrency feature, can I edit and add them into my compose-up file. ``` ollama: image: ollama/ollama:latest container_name: ollama ports: - "11434:11434" volumes: - ollama:/root/.ollama deploy: resources: reservations: devices: - capabilities: [gpu] command: serve volumes: ollama: ``` or is there other way to pass the value in for `OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ollama serve`

GiteaMirror commented

2026-05-03 07:50:20 -05:00

@jammsen commented on GitHub (Apr 30, 2024):

My question is for the concurrency feature, can I edit and add them into my compose-up file.
ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"

    
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]
    command: serve
volumes:
  ollama:
or is there other way to pass the value in for OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ollama serve

I think thoose are 2 different things, the ENVs should be passed to ollama serve when you run the native app on your system. Not sure though how you can use this in a Docker Image of ollama @jmorganca could you clear this up for us please?

@jammsen commented on GitHub (Apr 30, 2024): > My question is for the concurrency feature, can I edit and add them into my compose-up file. > > ``` > ollama: > image: ollama/ollama:latest > container_name: ollama > ports: > - "11434:11434" > > > volumes: > - ollama:/root/.ollama > deploy: > resources: > reservations: > devices: > - capabilities: [gpu] > command: serve > volumes: > ollama: > ``` > > or is there other way to pass the value in for `OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ollama serve` I think thoose are 2 different things, the ENVs should be passed to ollama serve when you run the native app on your system. Not sure though how you can use this in a Docker Image of ollama @jmorganca could you clear this up for us please?

GiteaMirror commented

2026-05-03 07:50:24 -05:00

@BBjie commented on GitHub (Apr 30, 2024):

Have not tested but should work:


ollama:

    image: ollama/ollama:latest

    container_name: ollama
    environment:
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_MAX_LOADED_MODELS=4
...

I tested it one hour ago, it was not working...

@BBjie commented on GitHub (Apr 30, 2024): > Have not tested but should work: > > > ``` > > > > ollama: > > > > image: ollama/ollama:latest > > > > container_name: ollama > > environment: > > - OLLAMA_NUM_PARALLEL=4 > > - OLLAMA_MAX_LOADED_MODELS=4 > > ... > > ``` I tested it one hour ago, it was not working...

GiteaMirror commented

2026-05-03 07:50:26 -05:00

@laktosterror commented on GitHub (Apr 30, 2024):

Sorry you might need to get the pre-release image. Corrected previous answear.

@laktosterror commented on GitHub (Apr 30, 2024): Sorry you might need to get the pre-release image. Corrected previous answear.

GiteaMirror commented

2026-05-03 07:50:27 -05:00

@laktosterror commented on GitHub (Apr 30, 2024):

On mobile. I seem to have messed up my original post.

Try this:

ollama:
    image: ollama/ollama:0.1.33-rc5
    container_name: ollama
    environment:
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_MAX_LOADED_MODELS=4
...

@laktosterror commented on GitHub (Apr 30, 2024): On mobile. I seem to have messed up my original post. Try this: ``` ollama: image: ollama/ollama:0.1.33-rc5 container_name: ollama environment: - OLLAMA_NUM_PARALLEL=4 - OLLAMA_MAX_LOADED_MODELS=4 ... ```

GiteaMirror commented

2026-05-03 07:50:29 -05:00

@BBjie commented on GitHub (Apr 30, 2024):

On mobile. I seem to have messed up my original post.

Try this:

ollama:
    image: ollama/ollama:0.1.33-rc5
    container_name: ollama
    environment:
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_MAX_LOADED_MODELS=4
...

ollama  | {"function":"log_server_request","level":"INFO","line":2734,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"127.0.0.1","remote_port":46394,"status":200,"tid":"140623411642368","timestamp":1714493452}
ollama  | {"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":4,"n_processing_slots":0,"task_id":399,"tid":"140624811528192","timestamp":1714493452}
ollama  | {"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":46410,"status":200,"tid":"140623403249664","timestamp":1714493452}
ollama  | {"function":"log_server_request","level":"INFO","line":2734,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"127.0.0.1","remote_port":46410,"status":200,"tid":"140623403249664","timestamp":1714493452}
ollama  | {"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":4,"n_processing_slots":0,"task_id":400,"tid":"140624811528192","timestamp":1714493452}
ollama  | {"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":46410,"status":200,"tid":"140623403249664","timestamp":1ollama  | {"function":"print_timings","level":"INFO","line":269,"msg":"prompt eval time     =     991.40 ms /   713 tokens (    1.39 ms per token,   719.19 tokens per second)","n_prompt_tokens_processed":713,"n_tokens_second":719.1864417721239,"slot_id":1,"t_prompt_processing":991.398,"t_token":1.3904600280504908,"task_id":402,"tid":"140624811528192","timestamp":1714493454}
ollama  | {"function":"print_timings","level":"INFO","line":283,"msg":"generation eval time =    1251.42 ms /    56 runs   (   22.35 ms per token,    44.75 tokens per second)","n_decoded":56,"n_tokens_second":44.7492364661528,"slot_id":1,"t_token":22.346749999999997,"t_token_generation":1251.418,"task_id":402,"tid":"140624811528192","timestamp":1714493454}
ollama  | {"function":"print_timings","level":"INFO","line":293,"msg":"          total time =    2242.82 ms","slot_id":1,"t_prompt_processing":991.398,"t_token_generation":1251.418,"t_total":2242.816,"task_id":402,"tid":"140624811528192","timestamp":1714493454}
ollama  | {"function":"update_slots","level":"INFO","line":1640,"msg":"slot released","n_cache_tokens":832,"n_ctx":3904,"n_past":831,"n_system_tokens":0,"slot_id":1,"task_id":402,"tid":"140624811528192","timestamp":1714493454,"truncated":false}
ollama  | {"function":"log_server_request","level":"INFO","line":2734,"method":"POST","msg":"request","params":{},"path":"/completion","remote_addr":"127.0.0.1","remote_port":46416,"status":200,"tid":"140623966629888","timestamp":1714493454}
ollama  | [GIN] 2024/04/30 - 16:10:54 | 200 |  2.546651478s |      172.19.0.4 | POST     "/api/chat"

Thhanks, but I believe it still not working as expected..

@BBjie commented on GitHub (Apr 30, 2024): > On mobile. I seem to have messed up my original post. > > Try this: > > ``` > ollama: > image: ollama/ollama:0.1.33-rc5 > container_name: ollama > environment: > - OLLAMA_NUM_PARALLEL=4 > - OLLAMA_MAX_LOADED_MODELS=4 > ... > ``` ``` ollama | {"function":"log_server_request","level":"INFO","line":2734,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"127.0.0.1","remote_port":46394,"status":200,"tid":"140623411642368","timestamp":1714493452} ollama | {"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":4,"n_processing_slots":0,"task_id":399,"tid":"140624811528192","timestamp":1714493452} ollama | {"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":46410,"status":200,"tid":"140623403249664","timestamp":1714493452} ollama | {"function":"log_server_request","level":"INFO","line":2734,"method":"POST","msg":"request","params":{},"path":"/tokenize","remote_addr":"127.0.0.1","remote_port":46410,"status":200,"tid":"140623403249664","timestamp":1714493452} ollama | {"function":"process_single_task","level":"INFO","line":1506,"msg":"slot data","n_idle_slots":4,"n_processing_slots":0,"task_id":400,"tid":"140624811528192","timestamp":1714493452} ollama | {"function":"log_server_request","level":"INFO","line":2734,"method":"GET","msg":"request","params":{},"path":"/health","remote_addr":"127.0.0.1","remote_port":46410,"status":200,"tid":"140623403249664","timestamp":1ollama | {"function":"print_timings","level":"INFO","line":269,"msg":"prompt eval time = 991.40 ms / 713 tokens ( 1.39 ms per token, 719.19 tokens per second)","n_prompt_tokens_processed":713,"n_tokens_second":719.1864417721239,"slot_id":1,"t_prompt_processing":991.398,"t_token":1.3904600280504908,"task_id":402,"tid":"140624811528192","timestamp":1714493454} ollama | {"function":"print_timings","level":"INFO","line":283,"msg":"generation eval time = 1251.42 ms / 56 runs ( 22.35 ms per token, 44.75 tokens per second)","n_decoded":56,"n_tokens_second":44.7492364661528,"slot_id":1,"t_token":22.346749999999997,"t_token_generation":1251.418,"task_id":402,"tid":"140624811528192","timestamp":1714493454} ollama | {"function":"print_timings","level":"INFO","line":293,"msg":" total time = 2242.82 ms","slot_id":1,"t_prompt_processing":991.398,"t_token_generation":1251.418,"t_total":2242.816,"task_id":402,"tid":"140624811528192","timestamp":1714493454} ollama | {"function":"update_slots","level":"INFO","line":1640,"msg":"slot released","n_cache_tokens":832,"n_ctx":3904,"n_past":831,"n_system_tokens":0,"slot_id":1,"task_id":402,"tid":"140624811528192","timestamp":1714493454,"truncated":false} ollama | {"function":"log_server_request","level":"INFO","line":2734,"method":"POST","msg":"request","params":{},"path":"/completion","remote_addr":"127.0.0.1","remote_port":46416,"status":200,"tid":"140623966629888","timestamp":1714493454} ollama | [GIN] 2024/04/30 - 16:10:54 | 200 | 2.546651478s | 172.19.0.4 | POST "/api/chat" ``` Thhanks, but I believe it still not working as expected..

GiteaMirror commented

2026-05-03 07:50:30 -05:00

@dhiltgen commented on GitHub (May 2, 2024):

I'm going to close this as fixed now in 0.1.33. As commenters above have pointed out, it's opt-in for now, but we do intend to eventually do concurrency automatically without requiring the env vars.

To clarify some questions above: There are 2 layers of concurrency. There's OLLAMA_NUM_PARALLEL which controls how many requests can be answered against a single loaded model, and there's OLLAMA_MAX_LOADED_MODELS which controls how many models can be loaded at the same time, up to the limits of VRAM. (note: we do not support loading two or more copies of the same model)

https://github.com/ollama/ollama/releases

@dhiltgen commented on GitHub (May 2, 2024): I'm going to close this as fixed now in 0.1.33. As commenters above have pointed out, it's opt-in for now, but we do intend to eventually do concurrency automatically without requiring the env vars. To clarify some questions above: There are 2 layers of concurrency. There's `OLLAMA_NUM_PARALLEL` which controls how many requests can be answered against a single loaded model, and there's `OLLAMA_MAX_LOADED_MODELS` which controls how many models can be loaded at the same time, up to the limits of VRAM. (note: we do not support loading two or more copies of the same model) https://github.com/ollama/ollama/releases

GiteaMirror commented

2026-05-03 07:50:31 -05:00

@taozhiyuai commented on GitHub (May 3, 2024):

I'm going to close this as fixed now in 0.1.33. As commenters above have pointed out, it's opt-in for now, but we do intend to eventually do concurrency automatically without requiring the env vars.

To clarify some questions above: There are 2 layers of concurrency. There's OLLAMA_NUM_PARALLEL which controls how many requests can be answered against a single loaded model, and there's OLLAMA_MAX_LOADED_MODELS which controls how many models can be loaded at the same time, up to the limits of VRAM. (note: we do not support loading two or more copies of the same model)

https://github.com/ollama/ollama/releases

@dhiltgen
I install ollama on Mac.
are these the default value for this two env?OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4.

the following should work, right? :)
t@603e5f4a42f1 ~ % launchctl setenv OLLAMA_NUM_PARALLEL 5
t@603e5f4a42f1 ~ % launchctl setenv OLLAMA_MAX_LOADED_MODELS 5

to check the value of ENVs :)
launchctl getenv OLLAMA_NUM_PARALLEL
launchctl getenv OLLAMA_MAX_LOADED_MODELS

restart ollama to activate new value of ENVs. :) very nice!

@taozhiyuai commented on GitHub (May 3, 2024): > I'm going to close this as fixed now in 0.1.33. As commenters above have pointed out, it's opt-in for now, but we do intend to eventually do concurrency automatically without requiring the env vars. > > To clarify some questions above: There are 2 layers of concurrency. There's `OLLAMA_NUM_PARALLEL` which controls how many requests can be answered against a single loaded model, and there's `OLLAMA_MAX_LOADED_MODELS` which controls how many models can be loaded at the same time, up to the limits of VRAM. (note: we do not support loading two or more copies of the same model) > > https://github.com/ollama/ollama/releases @dhiltgen I install ollama on Mac. are these the default value for this two env?OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4. the following should work, right? :) t@603e5f4a42f1 ~ % launchctl setenv OLLAMA_NUM_PARALLEL 5 t@603e5f4a42f1 ~ % launchctl setenv OLLAMA_MAX_LOADED_MODELS 5 to check the value of ENVs :) launchctl getenv OLLAMA_NUM_PARALLEL launchctl getenv OLLAMA_MAX_LOADED_MODELS restart ollama to activate new value of ENVs. :) very nice!

GiteaMirror commented

2026-05-03 07:50:32 -05:00

@dhiltgen commented on GitHub (May 4, 2024):

The default values for these settings in 0.1.33 retain existing behavior of only 1 request at a time, and only 1 model at a time. In a future version we plan to adjust the default to enable this automatically, but until then, yes, if you want to use concurrency, you'll have to set these environment variables on the server.

@dhiltgen commented on GitHub (May 4, 2024): The default values for these settings in 0.1.33 retain existing behavior of only 1 request at a time, and only 1 model at a time. In a future version we plan to adjust the default to enable this automatically, but until then, yes, if you want to use concurrency, you'll have to set these environment variables on the server.

GiteaMirror commented

2026-05-03 07:50:33 -05:00

@murillojndem commented on GitHub (May 21, 2024):

How do I set 2 models to run on different ports? I want to use crewAI but I need the ports of each one to use them as agents.

@murillojndem commented on GitHub (May 21, 2024): How do I set 2 models to run on different ports? I want to use crewAI but I need the ports of each one to use them as agents.

GiteaMirror commented

2026-05-03 07:50:35 -05:00

@ITHealer commented on GitHub (May 22, 2024):

How to set env to run parallel in linux?
I had set up env by:

sudo nano ~/.bashrc
export OLLAMA_NUM_PARALLEL=4
source ~/.bashrc

But it doesn't work
Please help me!!!

@ITHealer commented on GitHub (May 22, 2024): How to set env to run parallel in linux? I had set up env by: sudo nano ~/.bashrc export OLLAMA_NUM_PARALLEL=4 source ~/.bashrc But it doesn't work Please help me!!!

GiteaMirror commented

2026-05-03 07:50:36 -05:00

@Miroun commented on GitHub (May 22, 2024):

How does it work ? I mean. what prevents me to do OLLAMA_NUM_PARALLEL=999999 ?
How can I smartly access this number if I want to use my GPU(s) at full capacity ?

@Miroun commented on GitHub (May 22, 2024): How does it work ? I mean. what prevents me to do OLLAMA_NUM_PARALLEL=999999 ? How can I smartly access this number if I want to use my GPU(s) at full capacity ?

GiteaMirror commented

2026-05-03 07:50:37 -05:00

@dhiltgen commented on GitHub (May 23, 2024):

@murillojndem the ollama server only binds to a single port, so in your use-case, you'll want to run multiple servers on different ports using the OLLAMA_HOST environment variable to control which port they bind to. Each Ollama instance wont know about the others, so if you only have a single GPU, you may run into OOM problems if you don't sequence model loading to ensure they're not racing, or control the amount of layers they load by explicitly setting num_gpu. If you do have multiple GPUs, you can use the GPU vendor specific GPU Selection variable to dedicate specific GPUs to specific instances. https://github.com/ollama/ollama/blob/main/docs/gpu.md

@ITHealer this variable must be set on the server, not the client. See https://github.com/ollama/ollama/blob/main/docs/faq.md#setting-environment-variables-on-linux

@Miroun our goal is to make this setting automatic in a future release. Until then, it's your responsibility to test your setup to ensure it meets your needs balancing the model size, context size, and your GPU(s) VRAM capacity. The memory allocation on the GPU is impacted by the parallel setting multiplied by context size, so a large parallel with a large context will result in significant VRAM usage. You can experiment and use the new ollama ps to see where the optimal configuration is for your usecase.

@dhiltgen commented on GitHub (May 23, 2024): @murillojndem the ollama server only binds to a single port, so in your use-case, you'll want to run multiple servers on different ports using the OLLAMA_HOST environment variable to control which port they bind to. Each Ollama instance wont know about the others, so if you only have a single GPU, you may run into OOM problems if you don't sequence model loading to ensure they're not racing, or control the amount of layers they load by explicitly setting `num_gpu`. If you do have multiple GPUs, you can use the GPU vendor specific `GPU Selection` variable to dedicate specific GPUs to specific instances. https://github.com/ollama/ollama/blob/main/docs/gpu.md @ITHealer this variable must be set on the server, not the client. See https://github.com/ollama/ollama/blob/main/docs/faq.md#setting-environment-variables-on-linux @Miroun our goal is to make this setting automatic in a future release. Until then, it's your responsibility to test your setup to ensure it meets your needs balancing the model size, context size, and your GPU(s) VRAM capacity. The memory allocation on the GPU is impacted by the parallel setting multiplied by context size, so a large parallel with a large context will result in significant VRAM usage. You can experiment and use the new `ollama ps` to see where the optimal configuration is for your usecase.

GiteaMirror commented

2026-05-03 07:50:39 -05:00

@murillojndem commented on GitHub (May 24, 2024):

@murillojndem the ollama server only binds to a single port, so in your use-case, you'll want to run multiple servers on different ports using the OLLAMA_HOST environment variable to control which port they bind to. Each Ollama instance wont know about the others, so if you only have a single GPU, you may run into OOM problems if you don't sequence model loading to ensure they're not racing, or control the amount of layers they load by explicitly setting num_gpu. If you do have multiple GPUs, you can use the GPU vendor specific GPU Selection variable to dedicate specific GPUs to specific instances. https://github.com/ollama/ollama/blob/main/docs/gpu.md

@ITHealer this variable must be set on the server, not the client. See https://github.com/ollama/ollama/blob/main/docs/faq.md#setting-environment-variables-on-linux

@Miroun our goal is to make this setting automatic in a future release. Until then, it's your responsibility to test your setup to ensure it meets your needs balancing the model size, context size, and your GPU(s) VRAM capacity. The memory allocation on the GPU is impacted by the parallel setting multiplied by context size, so a large parallel with a large context will result in significant VRAM usage. You can experiment and use the new ollama ps to see where the optimal configuration is for your usecase.

But how do I set the environment variable? Do I set 2 variables each one for one port? It doesn't say in the docs. And they'll be running at the same time but I'll only use one at a time.

@murillojndem commented on GitHub (May 24, 2024): > @murillojndem the ollama server only binds to a single port, so in your use-case, you'll want to run multiple servers on different ports using the OLLAMA_HOST environment variable to control which port they bind to. Each Ollama instance wont know about the others, so if you only have a single GPU, you may run into OOM problems if you don't sequence model loading to ensure they're not racing, or control the amount of layers they load by explicitly setting `num_gpu`. If you do have multiple GPUs, you can use the GPU vendor specific `GPU Selection` variable to dedicate specific GPUs to specific instances. https://github.com/ollama/ollama/blob/main/docs/gpu.md > > @ITHealer this variable must be set on the server, not the client. See https://github.com/ollama/ollama/blob/main/docs/faq.md#setting-environment-variables-on-linux > > @Miroun our goal is to make this setting automatic in a future release. Until then, it's your responsibility to test your setup to ensure it meets your needs balancing the model size, context size, and your GPU(s) VRAM capacity. The memory allocation on the GPU is impacted by the parallel setting multiplied by context size, so a large parallel with a large context will result in significant VRAM usage. You can experiment and use the new `ollama ps` to see where the optimal configuration is for your usecase. But how do I set the environment variable? Do I set 2 variables each one for one port? It doesn't say in the docs. And they'll be running at the same time but I'll only use one at a time.

GiteaMirror commented

2026-05-03 07:50:41 -05:00

@dhiltgen commented on GitHub (May 25, 2024):

@Miroun I'd suggest the easiest path would be to use 2 different containers, but if you want to run on the host, you'll need to set up 2 different systemd services that have the settings you want.

@dhiltgen commented on GitHub (May 25, 2024): @Miroun I'd suggest the easiest path would be to use 2 different [containers](https://github.com/ollama/ollama/blob/main/docs/docker.md), but if you want to run on the host, you'll need to set up 2 different [systemd services](https://github.com/ollama/ollama/blob/main/scripts/install.sh#L100) that have the settings you want.

GiteaMirror commented

2026-05-03 07:50:42 -05:00

@goreng2 commented on GitHub (Jun 10, 2024):

@dhiltgen

How did handle multiple requests?

I want to know the principle.

I've looked at the change log, but I'm not sure.

@goreng2 commented on GitHub (Jun 10, 2024): @dhiltgen How did handle multiple requests? I want to know the principle. I've looked at the change log, but I'm not sure.

GiteaMirror commented

2026-05-03 07:50:43 -05:00

@EthraZa commented on GitHub (Jun 19, 2024):

It took me some minutes to figure it out, but to make things clear to next one searching for it:

1- You can run multiple docker containers in different ports and probably have clear control of the resources for each one.

or

2- You can run multiple LLM in one host:
$ sudo vi /etc/systemd/system/ollama.service

Add env vars to the Service:

Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=4"
Environment="PATH=...

Restart services:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Thank you Ollama guys.

@EthraZa commented on GitHub (Jun 19, 2024): It took me some minutes to figure it out, but to make things clear to next one searching for it: 1- You can run multiple docker containers in different ports and probably have clear control of the resources for each one. or 2- You can run multiple LLM in one host: `$ sudo vi /etc/systemd/system/ollama.service` Add env vars to the Service: ``` Environment="OLLAMA_NUM_PARALLEL=4" Environment="OLLAMA_MAX_LOADED_MODELS=4" Environment="PATH=... ``` Restart services: ``` sudo systemctl daemon-reload sudo systemctl restart ollama ``` Thank you Ollama guys.

GiteaMirror commented

2026-05-03 07:50:44 -05:00

@dhiltgen commented on GitHub (Jun 19, 2024):

@goreng2 while concurrency is in experimental preview, you can enable it with 2 settings
OLLAMA_NUM_PARALLEL sets the number of concurrent requests for each individual loaded model
OLLAMA_MAX_LOADED_MODELS sets the number of models that can be loaded at the same time, provided there is sufficient VRAM to fully load the models. (Partial loads always max at 1 loaded model)

So for example, if you set OLLAMA_NUM_PARALLEL=4 and OLLAMA_MAX_LOADED_MODELS=2 then you could load 2 models, and each model could handle 4 requests in parallel, so a total of 8 concurrent requests can be processed. (assuming you have enough VRAM to load the 2 models with 4x the context size). If more requests are being sent they'll queue in a FIFO order, up to OLLAMA_MAX_QUEUE requests (default 512)

When concurrency is out of experimental status in a future release, if you don't specify these settings, we'll pick reasonable defaults based on the capabilities of your system.

@dhiltgen commented on GitHub (Jun 19, 2024): @goreng2 while concurrency is in experimental preview, you can enable it with 2 settings OLLAMA_NUM_PARALLEL sets the number of concurrent requests for each individual loaded model OLLAMA_MAX_LOADED_MODELS sets the number of models that can be loaded at the same time, provided there is sufficient VRAM to fully load the models. (Partial loads always max at 1 loaded model) So for example, if you set OLLAMA_NUM_PARALLEL=4 and OLLAMA_MAX_LOADED_MODELS=2 then you could load 2 models, and each model could handle 4 requests in parallel, so a total of 8 concurrent requests can be processed. (assuming you have enough VRAM to load the 2 models with 4x the context size). If more requests are being sent they'll queue in a FIFO order, up to OLLAMA_MAX_QUEUE requests (default 512) When concurrency is out of experimental status in a future release, if you don't specify these settings, we'll pick reasonable defaults based on the capabilities of your system.

GiteaMirror commented

2026-05-03 07:50:47 -05:00

@edenbuaa commented on GitHub (Jun 26, 2024):

good idea. @edenbuaa

@edenbuaa commented on GitHub (Jun 26, 2024): good idea. @edenbuaa

GiteaMirror commented

2026-05-03 07:50:50 -05:00

@vincent-herlemont commented on GitHub (Jul 15, 2024):

@dhiltgen Is there a way for an app which use the Ollama API to retrieve the values of OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS in order to know how many parallel requests Ollama can handle?

@vincent-herlemont commented on GitHub (Jul 15, 2024): @dhiltgen Is there a way for an app which use the Ollama API to retrieve the values of OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS in order to know how many parallel requests Ollama can handle?

GiteaMirror commented

2026-05-03 07:50:53 -05:00

@dhiltgen commented on GitHub (Jul 18, 2024):

@vincent-herlemont this isn't exposed via API today.

@dhiltgen commented on GitHub (Jul 18, 2024): @vincent-herlemont this isn't exposed via API today.

GiteaMirror commented

2026-05-03 07:51:01 -05:00

@Omega-Centauri-21 commented on GitHub (Jul 28, 2024):

I install ollama on Mac. are these the default value for this two env?OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4.

the following should work, right? :) t@603e5f4a42f1 ~ % launchctl setenv OLLAMA_NUM_PARALLEL 5 t@603e5f4a42f1 ~ % launchctl setenv OLLAMA_MAX_LOADED_MODELS 5

to check the value of ENVs :) launchctl getenv OLLAMA_NUM_PARALLEL launchctl getenv OLLAMA_MAX_LOADED_MODELS

restart ollama to activate new value of ENVs. :) very nice!

For everyone trying this on Windows, first make sure your Ollama is updated to v3.0+

Then in your command prompt and use

setx OLLAMA_NUM_PARALLEL 5
setx OLLAMA_MAX_LOADED_MODELS 5

And now restart the server

@Omega-Centauri-21 commented on GitHub (Jul 28, 2024): > I install ollama on Mac. are these the default value for this two env?OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4. > > the following should work, right? :) t@603e5f4a42f1 ~ % launchctl setenv OLLAMA_NUM_PARALLEL 5 t@603e5f4a42f1 ~ % launchctl setenv OLLAMA_MAX_LOADED_MODELS 5 > > to check the value of ENVs :) launchctl getenv OLLAMA_NUM_PARALLEL launchctl getenv OLLAMA_MAX_LOADED_MODELS > > restart ollama to activate new value of ENVs. :) very nice! For everyone trying this on Windows, first make sure your Ollama is updated to v3.0+ Then in your command prompt and use ``` setx OLLAMA_NUM_PARALLEL 5 setx OLLAMA_MAX_LOADED_MODELS 5 ``` And now restart the server

GiteaMirror commented

2026-05-03 07:51:03 -05:00

@dhiltgen commented on GitHub (Jul 29, 2024):

Concurrency is no longer experimental as of 0.2.0 and newer. I would recommend leaving these 2 variables unset so the default algorithm is used. If you want more (or less) parallelism per model than 4 (the default) then you can adjust OLLAMA_NUM_PARALLEL. The default max loaded models is 3x GPU count. In our performance testing, we found >3 often leads to degraded performance if all the models are being accessed concurrently, so make sure to run perf tests before settling on an alternative max models setting.

@dhiltgen commented on GitHub (Jul 29, 2024): Concurrency is no longer experimental as of 0.2.0 and newer. I would recommend leaving these 2 variables unset so the default algorithm is used. If you want more (or less) parallelism per model than 4 (the default) then you can adjust OLLAMA_NUM_PARALLEL. The default max loaded models is 3x GPU count. In our performance testing, we found >3 often leads to degraded performance if all the models are being accessed concurrently, so make sure to run perf tests before settling on an alternative max models setting.

GiteaMirror commented

2026-05-03 07:51:05 -05:00

@Omega-Centauri-21 commented on GitHub (Jul 29, 2024):

So Could you please provide the detailed implementation of parallelism on Windows that can be utilized with langserve and API calls?

@Omega-Centauri-21 commented on GitHub (Jul 29, 2024): So Could you please provide the detailed implementation of parallelism on Windows that can be utilized with langserve and API calls?

GiteaMirror commented

2026-05-03 07:51:07 -05:00

@reinhardt-bit commented on GitHub (Oct 11, 2024):

If you cant get it right then use litellm

@reinhardt-bit commented on GitHub (Oct 11, 2024): If you cant get it right then use litellm

GiteaMirror commented

2026-05-03 07:51:08 -05:00

@Omega-Centauri-21 commented on GitHub (Oct 12, 2024):

Actually I finally got it!

I was facing some issues with setting the variables through the terminal so I set those variables via the Edit Environment Variables through system properties in windows and created a new variable for my user.

Pretty simple and worked like a chart when I ran it.

As of today, with the latest running release, the parallelism feature has been delivered and can be closed for good. Anybody still facing any issues can ping me for help.

My heart goes out to the Community for the help and support.
Thankyou.

@Omega-Centauri-21 commented on GitHub (Oct 12, 2024): Actually I finally got it! I was facing some issues with setting the variables through the terminal so I set those variables via the `Edit Environment Variables` through system properties in windows and created a new variable for my user. - Pretty simple and worked like a chart when I ran it. As of today, with the latest running release, the parallelism feature has been `delivered` and can be closed for good. Anybody still facing any issues can ping me for help. My heart goes out to the Community for the help and support. Thankyou.

GiteaMirror commented

2026-05-03 07:51:11 -05:00

@Vimiso commented on GitHub (Oct 13, 2024):

Can I request more information on sensible OLLAMA_NUM_PARALLEL settings? For example, let's say I have one loaded model in memory. How many parallel requests can I throw at it, or rather what system requirements must I know to arrive at an optimal number? I assume increasing OLLAMA_MAX_LOADED_MODELS (given enough VRAM) would increase the throughput. Also, is there a queue built in or will serve http requests timeout while ollama is busy?

@Vimiso commented on GitHub (Oct 13, 2024): Can I request more information on sensible `OLLAMA_NUM_PARALLEL` settings? For example, let's say I have one loaded model in memory. How many parallel requests can I throw at it, or rather what system requirements must I know to arrive at an optimal number? I assume increasing `OLLAMA_MAX_LOADED_MODELS` (given enough VRAM) would increase the throughput. Also, is there a queue built in or will serve http requests timeout while ollama is busy?

GiteaMirror commented

2026-05-03 07:51:13 -05:00

@dhiltgen commented on GitHub (Oct 14, 2024):

@Vimiso the current algorithm is described here It depends on how large the model is and how much VRAM (or system memory) you have - we try to load the model to support 4 concurrent requests, but if there isn't sufficient [V]RAM we fall back to supporting a single request at a time. We don't currently have an API that exposes the final settings, but you can check the server log to see how it was loaded.

@dhiltgen commented on GitHub (Oct 14, 2024): @Vimiso the current algorithm is described [here](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-does-ollama-handle-concurrent-requests) It depends on how large the model is and how much VRAM (or system memory) you have - we try to load the model to support 4 concurrent requests, but if there isn't sufficient [V]RAM we fall back to supporting a single request at a time. We don't currently have an API that exposes the final settings, but you can check the server log to see how it was loaded.

GiteaMirror commented

2026-05-03 07:51:14 -05:00

@phazei commented on GitHub (Nov 12, 2024):

So when using a model in parallel, it uses batching, right? So the two requests HAVE to start at the same time? Is there a setting for how long it waits for the request pool to fill before starting the batch? Suppose I have 2 agents, and they both request at the same time, but it takes 1 second to process the request before the next request. I'd like to use batching, but it would be ideal if it would wait for at least 1 second so be sure I can get both requests started together instead of having to wait.

@phazei commented on GitHub (Nov 12, 2024): So when using a model in parallel, it uses batching, right? So the two requests HAVE to start at the same time? Is there a setting for how long it waits for the request pool to fill before starting the batch? Suppose I have 2 agents, and they both request at the same time, but it takes 1 second to process the request before the next request. I'd like to use batching, but it would be ideal if it would wait for at least 1 second so be sure I can get both requests started together instead of having to wait.

GiteaMirror commented

2026-05-03 07:51:15 -05:00

@jessegross commented on GitHub (Nov 12, 2024):

There's no waiting time for batches, processing will start as soon as something is available with everything that is ready at that time. Generally, it's not helpful to wait because the savings from batching are overwhelmed by the lost cycles while things are sitting idle.

If requests are coming in continuously then new requests will be queued while the previous batch is running. Once that batch finishes, all of the queued requests will form the next batch.

@jessegross commented on GitHub (Nov 12, 2024): There's no waiting time for batches, processing will start as soon as something is available with everything that is ready at that time. Generally, it's not helpful to wait because the savings from batching are overwhelmed by the lost cycles while things are sitting idle. If requests are coming in continuously then new requests will be queued while the previous batch is running. Once that batch finishes, all of the queued requests will form the next batch.

GiteaMirror commented

2026-05-03 07:51:16 -05:00

@phazei commented on GitHub (Nov 13, 2024):

If lots of requests are coming in, then I'd agree, it's not helpful to wait since things will double up often enough. But on a local machine with a not so fast card and only two agents, it seems it would always end up being sequential and not taking advantage of the batching because the next batch starts the moment the previous output is complete which leaves no time for the response to be processed before being turned around for another request. If there was a configurable window of 300ms - 1000ms, then it could save a great deal of time on a GPU poor system.
There's also a thing I recently heard of called continuous batching which can start any request right away though, I don't know if Ollama supports that, but if that's available, then a waiting pool wouldn't have any necessity.

I was thinking if it would be a possibility to create my own waiting pool, but for that to work, there'd need to be a way to poll the server to see if it's busy or not, and once it's no longer busy, wait x seconds before passing through requests. I think there's 2 issues with that though, I looked at the API and didn't see anything that would work for that, I can't even find an output log showing what requests are hitting the API. And second, you said the requests already had to be waiting, so if they're sent x seconds after it's available, there inevitably will be a race condition where one is chosen since there's no pool.

@phazei commented on GitHub (Nov 13, 2024): If lots of requests are coming in, then I'd agree, it's not helpful to wait since things will double up often enough. But on a local machine with a not so fast card and only two agents, it seems it would always end up being sequential and not taking advantage of the batching because the next batch starts the moment the previous output is complete which leaves no time for the response to be processed before being turned around for another request. If there was a configurable window of 300ms - 1000ms, then it could save a great deal of time on a GPU poor system. There's also a thing I recently heard of called continuous batching which can start any request right away though, I don't know if Ollama supports that, but if that's available, then a waiting pool wouldn't have any necessity. I was thinking if it would be a possibility to create my own waiting pool, but for that to work, there'd need to be a way to poll the server to see if it's busy or not, and once it's no longer busy, wait x seconds before passing through requests. I think there's 2 issues with that though, I looked at the API and didn't see anything that would work for that, I can't even find an output log showing what requests are hitting the API. And second, you said the requests already had to be waiting, so if they're sent x seconds after it's available, there inevitably will be a race condition where one is chosen since there's no pool.

GiteaMirror commented

2026-05-03 07:51:18 -05:00

@easp commented on GitHub (Nov 13, 2024):

@phazei I'm pretty sure Ollama uses continuous batching. If I prompt the same model from two different terminal windows in parallel, I get a stream of results in each window simultaneously and the total # of tokens generated per second across the two is higher than it is when generating a single result set. This is what you'd expect from continuous batching.

@easp commented on GitHub (Nov 13, 2024): @phazei I'm pretty sure Ollama uses continuous batching. If I prompt the same model from two different terminal windows in parallel, I get a stream of results in each window simultaneously and the total # of tokens generated per second across the two is higher than it is when generating a single result set. This is what you'd expect from continuous batching.

GiteaMirror commented

2026-05-03 07:51:19 -05:00

@jessegross commented on GitHub (Nov 14, 2024):

Yes, I should have been clearer in what I wrote previously. This is more specifically what will happen:

Ollama will accept up to NUM_PARALLEL requests at a time. For anything beyond that, the requests will get queued. In this case, a request is something like a chat message to the API.
The first batch will be created and start processing as soon as something is available - this will contain a prompt.
Every token that is output needs to be fed back into the LLM as part of a new batch, so batches will be created continuously while a request is being processed.
When additional requests come in, they will be grouped together into one of these batches with tokens from other requests. As soon as the LLM finishes processing a batch, it will start on the next one with anything that is now runnable.

@jessegross commented on GitHub (Nov 14, 2024): Yes, I should have been clearer in what I wrote previously. This is more specifically what will happen: - Ollama will accept up to NUM_PARALLEL requests at a time. For anything beyond that, the requests will get queued. In this case, a request is something like a chat message to the API. - The first batch will be created and start processing as soon as something is available - this will contain a prompt. - Every token that is output needs to be fed back into the LLM as part of a new batch, so batches will be created continuously while a request is being processed. - When additional requests come in, they will be grouped together into one of these batches with tokens from other requests. As soon as the LLM finishes processing a batch, it will start on the next one with anything that is now runnable.

GiteaMirror commented

2026-05-03 07:51:20 -05:00

@Ivanrs297 commented on GitHub (Jan 28, 2025):

Almost year later, any news on this request?

@Ivanrs297 commented on GitHub (Jan 28, 2025): Almost year later, any news on this request?

GiteaMirror commented

2026-05-03 07:51:21 -05:00

@hackthedev commented on GitHub (Jan 28, 2025):

i tried to set the env variables on windows and its being ignored :|

@hackthedev commented on GitHub (Jan 28, 2025): i tried to set the env variables on windows and its being ignored :|

GiteaMirror commented

2026-05-03 07:51:22 -05:00

@jammsen commented on GitHub (Jan 29, 2025):

@Ivanrs297

Almost year later, any news on this request?

You can use it a long time already, have you tried to read the comment 1 person above you before posting?

https://github.com/ollama/ollama/issues/358#issuecomment-2477448229

@jammsen commented on GitHub (Jan 29, 2025): @Ivanrs297 > Almost year later, any news on this request? You can use it a long time already, have you tried to read the comment 1 person above you before posting? https://github.com/ollama/ollama/issues/358#issuecomment-2477448229

GiteaMirror commented

2026-05-03 07:51:23 -05:00

@alirezaghey commented on GitHub (Mar 4, 2025):

Hey folks, I went through this thread and also made some tests on my servers and I'm still not sure if the following scenario is possible:

I have multiple concurrent users
I have 4 GPUs with enough ram such that a model fits completely on one GPU with about 5 Gigs left
When sending prompts to 2 different models, ollama perfectly loads them both on different GPUs and processes them concurrently.
When multiple requests are sent to the same model, ollama handles all of them on a single GPU although there are 3 free GPUs without any resource usage.

Is there a way to load the same model on multiple GPUs and have each of them handle a few of the requests with some configurations and settings?
The next step that comes to mind is to deploy different ollama instances on different ports and handle load balancing in the application.

Any feedback is appreciate. Thank you!

@alirezaghey commented on GitHub (Mar 4, 2025): Hey folks, I went through this thread and also made some tests on my servers and I'm still not sure if the following scenario is possible: - I have multiple concurrent users - I have 4 GPUs with enough ram such that a model fits completely on one GPU with about 5 Gigs left - When sending prompts to 2 different models, ollama perfectly loads them both on different GPUs and processes them concurrently. - When multiple requests are sent to the same model, ollama handles all of them on a single GPU although there are 3 free GPUs without any resource usage. Is there a way to load the same model on multiple GPUs and have each of them handle a few of the requests with some configurations and settings? The next step that comes to mind is to deploy different ollama instances on different ports and handle load balancing in the application. Any feedback is appreciate. Thank you!

GiteaMirror commented

2026-05-03 07:51:24 -05:00

@dhiltgen commented on GitHub (Mar 5, 2025):

@alirezaghey tracked in feature request #3902

@dhiltgen commented on GitHub (Mar 5, 2025): @alirezaghey tracked in feature request #3902

Sign in to join this conversation.

Branches Tags

main

dhiltgen/ci

parth-launch-plan-gating

hoyyeva/anthropic-reference-images-path

parth-anthropic-reference-images-path

brucemacd/download-before-remove

hoyyeva/editor-config-repair

parth-mlx-decode-checkpoints

parth-launch-codex-app

hoyyeva/fix-codex-model-metadata-warning

hoyyeva/qwen

parth/hide-claude-desktop-till-release

hoyyeva/opencode-image-modality

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

hoyyeva/opencode-thinking

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

parth/update-claude-docs

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#62196