I have searched the existing issues and discussions.
Problem Description
Being able to converse with any backend AI model would be a tremendous next-generation feature for OpenWebUI.
Doing this is complex and new, there isn't a great way to setup an advanced voice agent yet. You have Sesame AI releasing things like their CSM https://github.com/SesameAILabs/csm but the demo code and overall opensource effort is only just starting to kick off.
I believe that OpenWebUI can help bridge the gap with its pipelines project or build something new which brings this technology to openwebui.
Desired Solution you'd like
An advanced voice mode that you can talk to, that is integrated into the traditional UI. This means I want to be able to talk to the LLM via text, then "call" the AI and continue where I left off, with our transcripts conveniently kept in the chat archives to pick up from next.
This would require integrating various models lile:
VoiceActivityDetector by Silero?
an ASR like Whisper
an LLM
some TTS which can be SesameAI's CSM or something different
In a pipeline it could look something like
[VAD] -> [ASR] -> [LLM (whatever selected in ui)] -> [TTS (whatever selected in ui)]
For voice models it'd be nice if the models themselves had a special configuration, which outlines their characteristics but also standardises things like voices which can be selected later in the call.
Its understandable also that the first iterations of an advanced mode will have a bad realtime factor, as a community and as these models start becoming "deepseeked" if you can call it that, I'm certain if OpenWebUI sets the direction, we as the community can contribute further.
Thanks.
Alternatives Considered
No response
Additional Context
No response
Originally created by @Notbici on GitHub (Apr 7, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/12560
### Check Existing Issues
- [x] I have searched the existing issues and discussions.
### Problem Description
Being able to converse with any backend AI model would be a tremendous next-generation feature for OpenWebUI.
Doing this is complex and new, there isn't a great way to setup an advanced voice agent yet. You have Sesame AI releasing things like their CSM https://github.com/SesameAILabs/csm but the demo code and overall opensource effort is only just starting to kick off.
I believe that OpenWebUI can help bridge the gap with its pipelines project or build something new which brings this technology to openwebui.
### Desired Solution you'd like
An advanced voice mode that you can talk to, that is integrated into the traditional UI. This means I want to be able to talk to the LLM via text, then "call" the AI and continue where I left off, with our transcripts conveniently kept in the chat archives to pick up from next.
This would require integrating various models lile:
- VoiceActivityDetector by Silero?
- an ASR like Whisper
- an LLM
- some TTS which can be SesameAI's CSM or something different
In a pipeline it could look something like
[VAD] -> [ASR] -> [LLM (whatever selected in ui)] -> [TTS (whatever selected in ui)]
For voice models it'd be nice if the models themselves had a special configuration, which outlines their characteristics but also standardises things like voices which can be selected later in the call.
Its understandable also that the first iterations of an advanced mode will have a bad realtime factor, as a community and as these models start becoming "deepseeked" if you can call it that, I'm certain if OpenWebUI sets the direction, we as the community can contribute further.
Thanks.
### Alternatives Considered
_No response_
### Additional Context
_No response_
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @Notbici on GitHub (Apr 7, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/12560
Check Existing Issues
Problem Description
Being able to converse with any backend AI model would be a tremendous next-generation feature for OpenWebUI.
Doing this is complex and new, there isn't a great way to setup an advanced voice agent yet. You have Sesame AI releasing things like their CSM https://github.com/SesameAILabs/csm but the demo code and overall opensource effort is only just starting to kick off.
I believe that OpenWebUI can help bridge the gap with its pipelines project or build something new which brings this technology to openwebui.
Desired Solution you'd like
An advanced voice mode that you can talk to, that is integrated into the traditional UI. This means I want to be able to talk to the LLM via text, then "call" the AI and continue where I left off, with our transcripts conveniently kept in the chat archives to pick up from next.
This would require integrating various models lile:
In a pipeline it could look something like
[VAD] -> [ASR] -> [LLM (whatever selected in ui)] -> [TTS (whatever selected in ui)]
For voice models it'd be nice if the models themselves had a special configuration, which outlines their characteristics but also standardises things like voices which can be selected later in the call.
Its understandable also that the first iterations of an advanced mode will have a bad realtime factor, as a community and as these models start becoming "deepseeked" if you can call it that, I'm certain if OpenWebUI sets the direction, we as the community can contribute further.
Thanks.
Alternatives Considered
No response
Additional Context
No response