Is there any idea on how to run vLLM inference engine with Open-webui chat interface instead of ollama?
Originally created by @wolfgangsmdt on GitHub (Nov 14, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6947
Is there any idea on how to run vLLM inference engine with Open-webui chat interface instead of ollama?
I'm pretty sure you can just use the openai compatible server that vllm offers. vllmDocs From there you can just connect it like its an openai server under connections in the admin panel .
<!-- gh-comment-id:2477253540 -->
@jay2jp commented on GitHub (Nov 14, 2024):
I'm pretty sure you can just use the openai compatible server that vllm offers. [vllmDocs](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) From there you can just connect it like its an openai server under connections in the admin panel .
<!-- gh-comment-id:2477263646 -->
@DJJones66 commented on GitHub (Nov 14, 2024):
When I played with it this is what I found and used:
https://github.com/open-webui/open-webui/discussions/508
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @wolfgangsmdt on GitHub (Nov 14, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6947
Is there any idea on how to run vLLM inference engine with Open-webui chat interface instead of ollama?
@jay2jp commented on GitHub (Nov 14, 2024):
I'm pretty sure you can just use the openai compatible server that vllm offers. vllmDocs From there you can just connect it like its an openai server under connections in the admin panel .
@DJJones66 commented on GitHub (Nov 14, 2024):
When I played with it this is what I found and used:
https://github.com/open-webui/open-webui/discussions/508