Originally created by @kengz on GitHub (May 12, 2024).
Is your feature request related to a problem? Please describe.
Existing OpenAI API Endpoints config doesn't work for HuggingFace TGI. TGI serves one model at a time, but has no /models endpoint (see TGI API doc here) that WebUI requires - also it doesn't care what "model" is passed in request param. As a result, configuring with TGI fails.
Describe the solution you'd like
Current config uses OPENAI_API_BASE_URLS, OPENAI_API_KEYS, but for TGI, it needs to explicitly specify which URL maps to which model, e.g. Llama3 -> http://llama3.tgi.my-example.com/v1, Gemma -> http://gemma.tgi.my-example.com/v1, etc.
Originally created by @kengz on GitHub (May 12, 2024).
**Is your feature request related to a problem? Please describe.**
Existing [OpenAI API Endpoints](https://docs.openwebui.com/tutorial/openai) config doesn't work for [HuggingFace TGI](https://github.com/huggingface/text-generation-inference). TGI serves one model at a time, but has no `/models` endpoint (see [TGI API doc here](https://huggingface.github.io/text-generation-inference/)) that WebUI requires - also it doesn't care what "model" is passed in request param. As a result, configuring with TGI fails.
**Describe the solution you'd like**
Current config uses `OPENAI_API_BASE_URLS, OPENAI_API_KEYS`, but for TGI, it needs to explicitly specify which URL maps to which model, e.g. `Llama3 -> http://llama3.tgi.my-example.com/v1`, `Gemma -> http://gemma.tgi.my-example.com/v1`, etc.
Open WebUI includes LiteLLM as a proxy between various LLM providers and an OpenAI-compatible endpoint. You can add a TGI endpoint to LiteLLM via the UI.
@cheahjs commented on GitHub (May 12, 2024):
Open WebUI includes [LiteLLM](https://docs.litellm.ai/docs/providers/huggingface) as a proxy between various LLM providers and an OpenAI-compatible endpoint. You can add a TGI endpoint to LiteLLM via the UI.

Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @kengz on GitHub (May 12, 2024).
Is your feature request related to a problem? Please describe.
Existing OpenAI API Endpoints config doesn't work for HuggingFace TGI. TGI serves one model at a time, but has no
/modelsendpoint (see TGI API doc here) that WebUI requires - also it doesn't care what "model" is passed in request param. As a result, configuring with TGI fails.Describe the solution you'd like
Current config uses
OPENAI_API_BASE_URLS, OPENAI_API_KEYS, but for TGI, it needs to explicitly specify which URL maps to which model, e.g.Llama3 -> http://llama3.tgi.my-example.com/v1,Gemma -> http://gemma.tgi.my-example.com/v1, etc.@cheahjs commented on GitHub (May 12, 2024):
Open WebUI includes LiteLLM as a proxy between various LLM providers and an OpenAI-compatible endpoint. You can add a TGI endpoint to LiteLLM via the UI.