Originally created by @linhcentrio on GitHub (Mar 1, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/977
How can use AWQ model in open-webui?
vLLM (or any other inference engine with an OpenAI-compatible endpoint).
<!-- gh-comment-id:1972534291 -->
@justinh-rahb commented on GitHub (Mar 1, 2024):
[vLLM](https://github.com/vllm-project/vllm) (or any other inference engine with an OpenAI-compatible endpoint).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @linhcentrio on GitHub (Mar 1, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/977
How can use AWQ model in open-webui?
@justinh-rahb commented on GitHub (Mar 1, 2024):
vLLM (or any other inference engine with an OpenAI-compatible endpoint).