[GH-ISSUE #23589] feat: Routing ala OpenRouter #58687

New Issue

GiteaMirror · 2026-05-05T23:42:19-05:00

GiteaMirror commented

2026-05-05 23:42:19 -05:00

Originally created by @regulad on GitHub (Apr 11, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/23589

Check Existing Issues

I have searched for all existing open AND closed issues and discussions for similar requests. I have found none that is comparable to my request.

Verify Feature Scope

I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.

Problem Description

Every request incoming to an Open WebUI instance goes to one model, and fails if the upstream provider fails.

Desired Solution you'd like

It would be great if Open WebUI could route incoming requests either via the API or via the web frontend to a model based on some given critera. OpenRouter does this for incoming inference requests based on a variety of heuristics, but for Open WebUI something as simple as a "fallback" model for handling requests when an upstream connection fails would be fantastic.

Alternatives Considered

You could use a service like OpenRouter directly, but then you can't do things like route to local connections. I tried implementing this with a rudamentary Open WebUI function (code below), but it didn't handle tool calls correctly; first class support would be much better.

from pydantic import BaseModel, Field
from fastapi import Request

from open_webui.models.users import Users
from open_webui.utils.chat import generate_chat_completion

FULLFAT_MODELS = [
    "nvidia.moonshotai/kimi-k2.5",
    "openrouter.openrouter/auto",
]

SMALLFAST_MODELS = [
    "openrouter.openrouter/free",
    "llamarpc.gemma-4-E4B-it-UD-Q4_K_X",
]


class Pipe:
    class Valves(BaseModel):
        MODEL_ID: str = Field(default="")

    def __init__(self):
        self.valves = self.Valves()

    def pipes(self):
        return [
            {"id": "fullfat", "name": "router.fullfat"},
            {"id": "smallfast", "name": "router.smallfast"},
        ]

    def _normalize_route(self, route):
        if route.startswith("router."):
            return route.split("router.", 1)[1]
        return route

    def _get_chain(self, route):
        route = self._normalize_route(route)
        if route == "fullfat":
            return FULLFAT_MODELS
        if route == "smallfast":
            return SMALLFAST_MODELS
        raise ValueError(f"Unknown model route: {route!r}")

    async def _run_chain(self, chain, body, user, request: Request):
        last_exc = None

        for model_name in chain:
            try:
                body["model"] = model_name
                response = await generate_chat_completion(request, body, user)
                print(f"Successfully routed {request} to {model_name}")
            except Exception as exc:
                last_exc = exc
                continue

        if last_exc is not None:
            raise last_exc
        raise RuntimeError("No models configured for selected route")

    async def pipe(
        self,
        body: dict,
        __user__: dict,
        __request__: Request,
    ):
        user = Users.get_user_by_id(__user__["id"])

        route = self.valves.MODEL_ID or body.get("model")
        if not route:
            raise ValueError("MODEL_ID or body['model'] must be provided")

        chain = self._get_chain(route)
        return await self._run_chain(chain, body, user, __request__)

Additional Context

No response

Originally created by @regulad on GitHub (Apr 11, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/23589 ### Check Existing Issues - [x] I have searched for all existing **open AND closed** issues and discussions for similar requests. I have found none that is comparable to my request. ### Verify Feature Scope - [x] I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions. ### Problem Description Every request incoming to an Open WebUI instance goes to one model, and fails if the upstream provider fails. ### Desired Solution you'd like It would be great if Open WebUI could route incoming requests either via the API or via the web frontend to a model based on some given critera. OpenRouter does this for incoming inference requests based on a variety of heuristics, but for Open WebUI something as simple as a "fallback" model for handling requests when an upstream connection fails would be fantastic. ### Alternatives Considered You could use a service like OpenRouter directly, but then you can't do things like route to local connections. I tried implementing this with a rudamentary Open WebUI function (code below), but it didn't handle tool calls correctly; first class support would be much better. ```py from pydantic import BaseModel, Field from fastapi import Request from open_webui.models.users import Users from open_webui.utils.chat import generate_chat_completion FULLFAT_MODELS = [ "nvidia.moonshotai/kimi-k2.5", "openrouter.openrouter/auto", ] SMALLFAST_MODELS = [ "openrouter.openrouter/free", "llamarpc.gemma-4-E4B-it-UD-Q4_K_X", ] class Pipe: class Valves(BaseModel): MODEL_ID: str = Field(default="") def __init__(self): self.valves = self.Valves() def pipes(self): return [ {"id": "fullfat", "name": "router.fullfat"}, {"id": "smallfast", "name": "router.smallfast"}, ] def _normalize_route(self, route): if route.startswith("router."): return route.split("router.", 1)[1] return route def _get_chain(self, route): route = self._normalize_route(route) if route == "fullfat": return FULLFAT_MODELS if route == "smallfast": return SMALLFAST_MODELS raise ValueError(f"Unknown model route: {route!r}") async def _run_chain(self, chain, body, user, request: Request): last_exc = None for model_name in chain: try: body["model"] = model_name response = await generate_chat_completion(request, body, user) print(f"Successfully routed {request} to {model_name}") except Exception as exc: last_exc = exc continue if last_exc is not None: raise last_exc raise RuntimeError("No models configured for selected route") async def pipe( self, body: dict, __user__: dict, __request__: Request, ): user = Users.get_user_by_id(__user__["id"]) route = self.valves.MODEL_ID or body.get("model") if not route: raise ValueError("MODEL_ID or body['model'] must be provided") chain = self._get_chain(route) return await self._run_chain(chain, body, user, __request__) ``` ### Additional Context _No response_

GiteaMirror closed this issue

2026-05-05 23:42:19 -05:00

GiteaMirror commented

2026-05-05 23:42:21 -05:00

@Classic298 commented on GitHub (Apr 11, 2026):

Direct support for this is unlikely and should have been opened in discussions because this does not fit this

I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.

And a pipe for this is definitely the best choice. We have a simple pipe as well for routing between models and it works perfectly. Did you make sure to set it to native?

@Classic298 commented on GitHub (Apr 11, 2026): Direct support for this is unlikely and should have been opened in discussions because this does not fit this > I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions. And a pipe for this is definitely the best choice. We have a simple pipe as well for routing between models and it works perfectly. Did you make sure to set it to native?

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#58687