github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-05-06 02:48:13 -05:00
Code Issues 1.1k Packages Projects Releases 126 Wiki Activity

[GH-ISSUE #2205] Support for HuggingFace TGI #12794

New Issue
Closed
opened 2026-04-19 19:40:04 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-19 19:40:04 -05:00
Owner
Copy Link

Originally created by @kengz on GitHub (May 12, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/2205

Is your feature request related to a problem? Please describe.
Existing OpenAI API Endpoints config doesn't work for HuggingFace TGI. TGI serves one model at a time, but has no /models endpoint (see TGI API doc here) that WebUI requires - also it doesn't care what "model" is passed in request param. As a result, configuring with TGI fails.

Describe the solution you'd like
Current config uses OPENAI_API_BASE_URLS, OPENAI_API_KEYS, but for TGI, it needs to explicitly specify which URL maps to which model, e.g. Llama3 -> http://llama3.tgi.my-example.com/v1, Gemma -> http://gemma.tgi.my-example.com/v1, etc.

Originally created by @kengz on GitHub (May 12, 2024). Original GitHub issue: https://github.com/open-webui/open-webui/issues/2205 **Is your feature request related to a problem? Please describe.** Existing [OpenAI API Endpoints](https://docs.openwebui.com/tutorial/openai) config doesn't work for [HuggingFace TGI](https://github.com/huggingface/text-generation-inference). TGI serves one model at a time, but has no `/models` endpoint (see [TGI API doc here](https://huggingface.github.io/text-generation-inference/)) that WebUI requires - also it doesn't care what "model" is passed in request param. As a result, configuring with TGI fails. **Describe the solution you'd like** Current config uses `OPENAI_API_BASE_URLS, OPENAI_API_KEYS`, but for TGI, it needs to explicitly specify which URL maps to which model, e.g. `Llama3 -> http://llama3.tgi.my-example.com/v1`, `Gemma -> http://gemma.tgi.my-example.com/v1`, etc.
GiteaMirror commented 2026-04-19 19:40:05 -05:00
Author
Owner
Copy Link

@cheahjs commented on GitHub (May 12, 2024):

Open WebUI includes LiteLLM as a proxy between various LLM providers and an OpenAI-compatible endpoint. You can add a TGI endpoint to LiteLLM via the UI.

image

<!-- gh-comment-id:2106258289 --> @cheahjs commented on GitHub (May 12, 2024): Open WebUI includes [LiteLLM](https://docs.litellm.ai/docs/providers/huggingface) as a proxy between various LLM providers and an OpenAI-compatible endpoint. You can add a TGI endpoint to LiteLLM via the UI. ![image](https://github.com/open-webui/open-webui/assets/818368/fbb1566f-bbb8-4be7-b335-370dd24fde01)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#12794
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?