[GH-ISSUE #2205] Support for HuggingFace TGI #51460

New Issue

GiteaMirror · 2026-05-05T12:27:45-05:00

GiteaMirror commented

2026-05-05 12:27:45 -05:00

Originally created by @kengz on GitHub (May 12, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/2205

Is your feature request related to a problem? Please describe.
Existing OpenAI API Endpoints config doesn't work for HuggingFace TGI. TGI serves one model at a time, but has no /models endpoint (see TGI API doc here) that WebUI requires - also it doesn't care what "model" is passed in request param. As a result, configuring with TGI fails.

Describe the solution you'd like
Current config uses OPENAI_API_BASE_URLS, OPENAI_API_KEYS, but for TGI, it needs to explicitly specify which URL maps to which model, e.g. Llama3 -> http://llama3.tgi.my-example.com/v1, Gemma -> http://gemma.tgi.my-example.com/v1, etc.

Originally created by @kengz on GitHub (May 12, 2024). Original GitHub issue: https://github.com/open-webui/open-webui/issues/2205 **Is your feature request related to a problem? Please describe.** Existing [OpenAI API Endpoints](https://docs.openwebui.com/tutorial/openai) config doesn't work for [HuggingFace TGI](https://github.com/huggingface/text-generation-inference). TGI serves one model at a time, but has no `/models` endpoint (see [TGI API doc here](https://huggingface.github.io/text-generation-inference/)) that WebUI requires - also it doesn't care what "model" is passed in request param. As a result, configuring with TGI fails. **Describe the solution you'd like** Current config uses `OPENAI_API_BASE_URLS, OPENAI_API_KEYS`, but for TGI, it needs to explicitly specify which URL maps to which model, e.g. `Llama3 -> http://llama3.tgi.my-example.com/v1`, `Gemma -> http://gemma.tgi.my-example.com/v1`, etc.

GiteaMirror closed this issue

2026-05-05 12:27:45 -05:00

GiteaMirror commented

2026-05-05 12:27:46 -05:00

@cheahjs commented on GitHub (May 12, 2024):

Open WebUI includes LiteLLM as a proxy between various LLM providers and an OpenAI-compatible endpoint. You can add a TGI endpoint to LiteLLM via the UI.

@cheahjs commented on GitHub (May 12, 2024): Open WebUI includes [LiteLLM](https://docs.litellm.ai/docs/providers/huggingface) as a proxy between various LLM providers and an OpenAI-compatible endpoint. You can add a TGI endpoint to LiteLLM via the UI. ![image](https://github.com/open-webui/open-webui/assets/818368/fbb1566f-bbb8-4be7-b335-370dd24fde01)

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#51460