MLC-LLM Quant./Backend Support #526

New Issue

2025-11-11T14:24:18-06:00

GiteaMirror commented

2025-11-11 14:24:18 -06:00

Originally created by @BuildBackBuehler on GitHub (Mar 23, 2024).

Is your feature request related to a problem? Please describe.

I'm always frustrated when...I'm forced to use Ollama as a Mac User 🤪
Just to follow the format hah, but MLC-LLM is an oddly obscure project that provides one with relatively easy quantization (with impressive results) and is platform agnostic -- which is more to say than any other quantization format!

https://github.com/mlc-ai/mlc-llm

On top of that, it uses TVM to provide a backend which can be used as a server. They do have web-based UI but it is barebones as can be.

Describe the solution you'd like

So I'd love to be able to easily run those quants/their backend and plug 'er into Open-WebUI seamlessly. I imagine I can run it via API but...does that have limitations? I've had issues getting that to work in the past, not sure if w/ OWU.

Describe alternatives you've considered

I feel like a dolt who could figure this out...but unsure if API routing would come with consequence, speed-wise and also unable to reap any features/enhancements provided by WebUI to one's LLM. Still, an alternate solution. (Well, looked at the code and it seems Ollama is API-based)

Additional context
Add any other context or screenshots about the feature request here.

Originally created by @BuildBackBuehler on GitHub (Mar 23, 2024). **Is your feature request related to a problem? Please describe.** I'm always frustrated when...I'm forced to use Ollama as a Mac User 🤪 Just to follow the format hah, but MLC-LLM is an oddly obscure project that provides one with relatively easy quantization (with impressive results) and is platform agnostic -- which is more to say than any other quantization format! https://github.com/mlc-ai/mlc-llm On top of that, it uses TVM to provide a backend which can be used as a server. They do have web-based UI but it is barebones as can be. **Describe the solution you'd like** So I'd love to be able to easily run those quants/their backend and plug 'er into Open-WebUI seamlessly. I imagine I can run it via API but...does that have limitations? I've had issues getting that to work in the past, not sure if w/ OWU. **Describe alternatives you've considered** I feel like a dolt who could figure this out...but unsure if API routing would come with consequence, speed-wise and also unable to reap any features/enhancements provided by WebUI to one's LLM. Still, an alternate solution. (Well, looked at the code and it seems Ollama is API-based) **Additional context** Add any other context or screenshots about the feature request here.

GiteaMirror closed this issue

2025-11-11 14:24:18 -06:00

GiteaMirror commented

2025-11-11 14:24:19 -06:00

@justinh-rahb commented on GitHub (Mar 23, 2024):

Uses standard OpenAI API spec, so this is easily added to Open WebUI:
https://llm.mlc.ai/docs/deploy/rest.html
https://docs.openwebui.com/tutorial/openai

@justinh-rahb commented on GitHub (Mar 23, 2024): Uses standard OpenAI API spec, so this is easily added to Open WebUI: https://llm.mlc.ai/docs/deploy/rest.html https://docs.openwebui.com/tutorial/openai