I have searched for all existing open AND closed issues and discussions for similar requests. I have found none that is comparable to my request.
Verify Feature Scope
I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.
Problem Description
gemma4 just released - with combined audio and vision support
Im sure there will be multiple changes required in OpenWebUI for taking fully advantage.
Desired Solution you'd like
I guess there will be multiple changes required in OpenWebUI for taking fully advantage of the new gemma4:
Another seperate way to interact with audio directly - this would make things probably easier and more intuitive as the the whisper dictate is a replacement for the keyboard and can stay as such. And probably users wat to continue to use speech recognition instead of typing but still prefer text to be sent to gemma4 via chat.
OR (but i think this is worse:
When a model with native audio capability is detected ar configured as such in the Admin UI audio should not be sent to whisper.
Alternatives Considered
No response
Additional Context
No response
Originally created by @TomTheWise on GitHub (Apr 2, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/23357
### Check Existing Issues
- [x] I have searched for all existing **open AND closed** issues and discussions for similar requests. I have found none that is comparable to my request.
### Verify Feature Scope
- [x] I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.
### Problem Description
gemma4 just released - with combined audio and vision support
Im sure there will be multiple changes required in OpenWebUI for taking fully advantage.
### Desired Solution you'd like
I guess there will be multiple changes required in OpenWebUI for taking fully advantage of the new gemma4:
- Another seperate way to interact with audio directly - this would make things probably easier and more intuitive as the the whisper dictate is a replacement for the keyboard and can stay as such. And probably users wat to continue to use speech recognition instead of typing but still prefer text to be sent to gemma4 via chat.
OR (but i think this is worse:
- When a model with native audio capability is detected ar configured as such in the Admin UI audio should not be sent to whisper.
### Alternatives Considered
_No response_
### Additional Context
_No response_
Llama.cpp is currently only supporting text+image for day0 support.
I guess the audio support will maybe more like comparable to supporting Mistral AIs Voxtral realtime and maybe less like whisper?
<!-- gh-comment-id:4182124916 -->
@TomTheWise commented on GitHub (Apr 3, 2026):
The smaller ones can understand audio, the larger ones can even understand / „see“ video without audio.
https://ai.google.dev/gemma/docs/core/model_card_4
https://youtu.be/jZVBoFOJK-Q?t=117
Llama.cpp is currently only supporting text+image for day0 support.
I guess the audio support will maybe more like comparable to supporting Mistral AIs Voxtral realtime and maybe less like whisper?
We use Open WebUI in as a tool for our university students and we have many use cases in teaching and research where interacting with audio / video is common demand (i.e. transcribe a recorded interview).
Thus, I strongly support the idea of having a dedicated way to deal with audio / video input. Users want to be able to deliberately send an audio to a transcription endpoint together with a prompt or pass along a video to "see". Right now videos are treated in the same way as PDF files an audio is only transcribed implicitly.
<!-- gh-comment-id:4212349329 -->
@mglaubitz commented on GitHub (Apr 9, 2026):
We use Open WebUI in as a tool for our university students and we have many use cases in teaching and research where interacting with audio / video is common demand (i.e. transcribe a recorded interview).
Thus, I strongly support the idea of having a dedicated way to deal with audio / video input. Users want to be able to deliberately send an audio to a transcription endpoint together with a prompt or pass along a video to "see". Right now videos are treated in the same way as PDF files an audio is only transcribed implicitly.
Google finally uploaded jinja chat templates that apparently now have a good quality.
IF (and I have no idea how it works in OWUI) OWUI does indeed need updates for the gemma4 control tokens, now they are official clear. Here is the google documentation
<!-- gh-comment-id:4226004777 -->
@TomTheWise commented on GitHub (Apr 10, 2026):
Google finally uploaded jinja chat templates that apparently now have a good quality.
IF (and I have no idea how it works in OWUI) OWUI does indeed need updates for the gemma4 control tokens, now they are official clear.
[Here is the google documentation](https://ai.google.dev/gemma/docs/core/prompt-formatting-gemma4?hl=de)
llama.cpp now has initial multimodal audio support for gemma4 E4B / E2B
<!-- gh-comment-id:4231609860 -->
@TomTheWise commented on GitHub (Apr 12, 2026):
https://github.com/ggml-org/llama.cpp/pull/21421
llama.cpp now has initial multimodal audio support for gemma4 E4B / E2B
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @TomTheWise on GitHub (Apr 2, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/23357
Check Existing Issues
Verify Feature Scope
Problem Description
gemma4 just released - with combined audio and vision support
Im sure there will be multiple changes required in OpenWebUI for taking fully advantage.
Desired Solution you'd like
I guess there will be multiple changes required in OpenWebUI for taking fully advantage of the new gemma4:
OR (but i think this is worse:
Alternatives Considered
No response
Additional Context
No response
@Classic298 commented on GitHub (Apr 2, 2026):
Audio input you say?
@tjbck strongly related: https://github.com/open-webui/open-webui/issues/5894
@TomTheWise commented on GitHub (Apr 3, 2026):
The smaller ones can understand audio, the larger ones can even understand / „see“ video without audio.
https://ai.google.dev/gemma/docs/core/model_card_4
https://youtu.be/jZVBoFOJK-Q?t=117
Llama.cpp is currently only supporting text+image for day0 support.
I guess the audio support will maybe more like comparable to supporting Mistral AIs Voxtral realtime and maybe less like whisper?
@Classic298 commented on GitHub (Apr 7, 2026):
Related: https://github.com/open-webui/open-webui/issues/23471
@mglaubitz commented on GitHub (Apr 9, 2026):
We use Open WebUI in as a tool for our university students and we have many use cases in teaching and research where interacting with audio / video is common demand (i.e. transcribe a recorded interview).
Thus, I strongly support the idea of having a dedicated way to deal with audio / video input. Users want to be able to deliberately send an audio to a transcription endpoint together with a prompt or pass along a video to "see". Right now videos are treated in the same way as PDF files an audio is only transcribed implicitly.
@Classic298 commented on GitHub (Apr 9, 2026):
https://github.com/open-webui/open-webui/issues/17217
@Classic298 commented on GitHub (Apr 9, 2026):
https://github.com/open-webui/open-webui/issues/23538
@TomTheWise commented on GitHub (Apr 10, 2026):
Google finally uploaded jinja chat templates that apparently now have a good quality.
IF (and I have no idea how it works in OWUI) OWUI does indeed need updates for the gemma4 control tokens, now they are official clear.
Here is the google documentation
@TomTheWise commented on GitHub (Apr 12, 2026):
https://github.com/ggml-org/llama.cpp/pull/21421
llama.cpp now has initial multimodal audio support for gemma4 E4B / E2B