[GH-ISSUE #12560] feat: Advanced Voice Mode (using CSM from SesameAI) #71244

Closed
opened 2026-05-13 03:12:32 -05:00 by GiteaMirror · 0 comments
Owner

Originally created by @Notbici on GitHub (Apr 7, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/12560

Check Existing Issues

  • I have searched the existing issues and discussions.

Problem Description

Being able to converse with any backend AI model would be a tremendous next-generation feature for OpenWebUI.

Doing this is complex and new, there isn't a great way to setup an advanced voice agent yet. You have Sesame AI releasing things like their CSM https://github.com/SesameAILabs/csm but the demo code and overall opensource effort is only just starting to kick off.

I believe that OpenWebUI can help bridge the gap with its pipelines project or build something new which brings this technology to openwebui.

Desired Solution you'd like

An advanced voice mode that you can talk to, that is integrated into the traditional UI. This means I want to be able to talk to the LLM via text, then "call" the AI and continue where I left off, with our transcripts conveniently kept in the chat archives to pick up from next.

This would require integrating various models lile:

  • VoiceActivityDetector by Silero?
  • an ASR like Whisper
  • an LLM
  • some TTS which can be SesameAI's CSM or something different

In a pipeline it could look something like
[VAD] -> [ASR] -> [LLM (whatever selected in ui)] -> [TTS (whatever selected in ui)]

For voice models it'd be nice if the models themselves had a special configuration, which outlines their characteristics but also standardises things like voices which can be selected later in the call.

Its understandable also that the first iterations of an advanced mode will have a bad realtime factor, as a community and as these models start becoming "deepseeked" if you can call it that, I'm certain if OpenWebUI sets the direction, we as the community can contribute further.

Thanks.

Alternatives Considered

No response

Additional Context

No response

Originally created by @Notbici on GitHub (Apr 7, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/12560 ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description Being able to converse with any backend AI model would be a tremendous next-generation feature for OpenWebUI. Doing this is complex and new, there isn't a great way to setup an advanced voice agent yet. You have Sesame AI releasing things like their CSM https://github.com/SesameAILabs/csm but the demo code and overall opensource effort is only just starting to kick off. I believe that OpenWebUI can help bridge the gap with its pipelines project or build something new which brings this technology to openwebui. ### Desired Solution you'd like An advanced voice mode that you can talk to, that is integrated into the traditional UI. This means I want to be able to talk to the LLM via text, then "call" the AI and continue where I left off, with our transcripts conveniently kept in the chat archives to pick up from next. This would require integrating various models lile: - VoiceActivityDetector by Silero? - an ASR like Whisper - an LLM - some TTS which can be SesameAI's CSM or something different In a pipeline it could look something like [VAD] -> [ASR] -> [LLM (whatever selected in ui)] -> [TTS (whatever selected in ui)] For voice models it'd be nice if the models themselves had a special configuration, which outlines their characteristics but also standardises things like voices which can be selected later in the call. Its understandable also that the first iterations of an advanced mode will have a bad realtime factor, as a community and as these models start becoming "deepseeked" if you can call it that, I'm certain if OpenWebUI sets the direction, we as the community can contribute further. Thanks. ### Alternatives Considered _No response_ ### Additional Context _No response_
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#71244