feat: Prioritization of Servers (Simple Rules to Control Behavior) #5279

New Issue

GiteaMirror · 2025-11-11T16:16:13-06:00

GiteaMirror commented

2025-11-11 16:16:13 -06:00

Originally created by @rlabusiness on GitHub (May 22, 2025).

Check Existing Issues

I have searched the existing issues and discussions.

Problem Description

My setup includes 1 server that has a powerful GPU that’s not always available (we’ll say 80-90% availability), plus 3 backup servers that are far less powerful but are 100% available.

From the behavior I’ve observed, it seems that if the same model is available on multiple servers (fast and slow), Open-WebUI uses them in a round-robin sequence. Therefore, we never know how fast or slow the response is going to be, and most of the time it’s slow because there are 3x as many slower servers as fast servers.

Desired Solution you'd like

Note: This logic needs to apply to task models as well (full context in the “Alternatives Considered” section below).

GOOD:
Rather than doing round-robin across all servers, I want to be able to somehow set rules that prioritize more performant servers if they are available. This could be accomplished by adding a drop-down labeled “Priority Tier” to select an integer from 1-10. Multiple servers could be assigned to any given Tier, and Priority 1 would always be used when available.

BETTER:
Bonus points if you’re also able to decide whether to fall to a lower priority server based on the higher-tier server’s load (I don’t know if this is exposed or not), then if it’s under heavy load, automatically fall back to the servers in the next-level tier in a round-robin fashion.

BEST:
Alternatively, if you can’t query the server’s load, if you could somehow ask the server “how many requests are you handling”, you could base it on that. For me, server 1 is about 8x as fast as any of the 3 backup servers, so if this # of queries being handled could be known, I could set the threshold to 8, which would cause it to fall back to Tier 2 when the relative performance is equal (or when server 1 is not available).

(A related behavior would be to choose a server within a given tier based on its load/demand, so it’s not just a blind round-robin - Since my Tier 2 servers are all pretty weak, this would ensure consistency for multiple simultaneous user queries).

Extending that thinking, the admin could choose to prioritize performance by assigning powerful servers to tier 1, or prioritize cost by assigning the less powerful servers to tier 1, then still falling back based on demand on the first tier. So a setting to “Always fall back to next tier if number of requests exceeds:” might be useful.

Alternatives Considered

Currently I have created an entire set of duplicate models for each tier. For example, The a model named “Creative Writer” that’s based on an Ollama model that only exists on the primary server has a counterpart on all 3 backup servers called “BACKUP - Creative Writer”. The user is required to select between these two.

Although this works, the experience isn’t very good for the users. For instance, if Server 1 goes offline in the middle of a chat (or the user returns to an existing chat to continue it later), the UI throws an error.

The second problem that I can’t seem to workaround though is the task model. Since I can only select a single task model for local operation, I can’t prefer the model on the Tier 1 server when a Tier 1 model version is being used, so I’ve had to install the exact same task model on all servers. Since Servers in Tier 2 are 8x slower than Tier 1, tasks take 8x as long as they should 75% of the time.

Additional Context

Finally, in case this matters, I’d mention that the way I create and manage models is:

Create Ollama model with the most basic parameters (base model + context size)
Create an Open-WebUI Workspace model based on that model with a specific system prompt and parameters
Create an Open-WebUI Evaluation Arena model based on that Workspace model. THESE ARE THE ONLY MODELS I EXPOSE TO END USERS. The reason is that I want the freedom to switch out or compare different base models behind the scenes without the end user needing to manage anything extra, so the Workspace models are not exposed.

Originally created by @rlabusiness on GitHub (May 22, 2025). ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description My setup includes 1 server that has a powerful GPU that’s not always available (we’ll say 80-90% availability), plus 3 backup servers that are far less powerful but are 100% available. From the behavior I’ve observed, it seems that if the same model is available on multiple servers (fast and slow), Open-WebUI uses them in a round-robin sequence. Therefore, we never know how fast or slow the response is going to be, and most of the time it’s slow because there are 3x as many slower servers as fast servers. ### Desired Solution you'd like Note: This logic needs to apply to task models as well (full context in the “Alternatives Considered” section below). GOOD: Rather than doing round-robin across all servers, I want to be able to somehow set rules that prioritize more performant servers if they are available. This could be accomplished by adding a drop-down labeled “Priority Tier” to select an integer from 1-10. Multiple servers could be assigned to any given Tier, and Priority 1 would always be used when available. BETTER: Bonus points if you’re also able to decide whether to fall to a lower priority server based on the higher-tier server’s load (I don’t know if this is exposed or not), then if it’s under heavy load, automatically fall back to the servers in the next-level tier in a round-robin fashion. BEST: Alternatively, if you can’t query the server’s load, if you could somehow ask the server “how many requests are you handling”, you could base it on that. For me, server 1 is about 8x as fast as any of the 3 backup servers, so if this # of queries being handled could be known, I could set the threshold to 8, which would cause it to fall back to Tier 2 when the relative performance is equal (or when server 1 is not available). (A related behavior would be to choose a server within a given tier based on its load/demand, so it’s not just a blind round-robin - Since my Tier 2 servers are all pretty weak, this would ensure consistency for multiple simultaneous user queries). Extending that thinking, the admin could choose to prioritize performance by assigning powerful servers to tier 1, or prioritize cost by assigning the less powerful servers to tier 1, then still falling back based on demand on the first tier. So a setting to “Always fall back to next tier if number of requests exceeds:” might be useful. ### Alternatives Considered Currently I have created an entire set of duplicate models for each tier. For example, The a model named “Creative Writer” that’s based on an Ollama model that only exists on the primary server has a counterpart on all 3 backup servers called “BACKUP - Creative Writer”. The user is required to select between these two. Although this works, the experience isn’t very good for the users. For instance, if Server 1 goes offline in the middle of a chat (or the user returns to an existing chat to continue it later), the UI throws an error. The second problem that I can’t seem to workaround though is the task model. Since I can only select a single task model for local operation, I can’t prefer the model on the Tier 1 server when a Tier 1 model version is being used, so I’ve had to install the exact same task model on all servers. Since Servers in Tier 2 are 8x slower than Tier 1, tasks take 8x as long as they should 75% of the time. ### Additional Context Finally, in case this matters, I’d mention that the way I create and manage models is: 1. Create Ollama model with the most basic parameters (base model + context size) 2. Create an Open-WebUI Workspace model based on that model with a specific system prompt and parameters 3. Create an Open-WebUI Evaluation Arena model based on that Workspace model. THESE ARE THE ONLY MODELS I EXPOSE TO END USERS. The reason is that I want the freedom to switch out or compare different base models behind the scenes without the end user needing to manage anything extra, so the Workspace models are not exposed.

GiteaMirror closed this issue

2025-11-11 16:16:13 -06:00

GiteaMirror referenced this issue

2025-11-11 17:57:10 -06:00

[PR #5279] [MERGED] feat: Improve `kustomization` usage #8457

GiteaMirror referenced this issue

2026-04-20 03:38:07 -05:00

[PR #5279] [MERGED] feat: Improve `kustomization` usage #21661

GiteaMirror referenced this issue

2026-04-25 10:50:08 -05:00

[PR #5279] [MERGED] feat: Improve `kustomization` usage #37291

GiteaMirror referenced this issue