github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-16 06:03:26 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #23357] feat: gemma4 changes required, especially for audio #106954

New Issue
Open
opened 2026-05-18 05:27:56 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-05-18 05:27:56 -05:00
Owner
Copy Link

Originally created by @TomTheWise on GitHub (Apr 2, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/23357

Check Existing Issues

  • I have searched for all existing open AND closed issues and discussions for similar requests. I have found none that is comparable to my request.

Verify Feature Scope

  • I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.

Problem Description

gemma4 just released - with combined audio and vision support

Im sure there will be multiple changes required in OpenWebUI for taking fully advantage.

Desired Solution you'd like

I guess there will be multiple changes required in OpenWebUI for taking fully advantage of the new gemma4:

  • Another seperate way to interact with audio directly - this would make things probably easier and more intuitive as the the whisper dictate is a replacement for the keyboard and can stay as such. And probably users wat to continue to use speech recognition instead of typing but still prefer text to be sent to gemma4 via chat.
    OR (but i think this is worse:
  • When a model with native audio capability is detected ar configured as such in the Admin UI audio should not be sent to whisper.

Alternatives Considered

No response

Additional Context

No response

Originally created by @TomTheWise on GitHub (Apr 2, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/23357 ### Check Existing Issues - [x] I have searched for all existing **open AND closed** issues and discussions for similar requests. I have found none that is comparable to my request. ### Verify Feature Scope - [x] I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions. ### Problem Description gemma4 just released - with combined audio and vision support Im sure there will be multiple changes required in OpenWebUI for taking fully advantage. ### Desired Solution you'd like I guess there will be multiple changes required in OpenWebUI for taking fully advantage of the new gemma4: - Another seperate way to interact with audio directly - this would make things probably easier and more intuitive as the the whisper dictate is a replacement for the keyboard and can stay as such. And probably users wat to continue to use speech recognition instead of typing but still prefer text to be sent to gemma4 via chat. OR (but i think this is worse: - When a model with native audio capability is detected ar configured as such in the Admin UI audio should not be sent to whisper. ### Alternatives Considered _No response_ ### Additional Context _No response_
GiteaMirror commented 2026-05-18 05:27:57 -05:00
Author
Owner
Copy Link

@Classic298 commented on GitHub (Apr 2, 2026):

Audio input you say?

@tjbck strongly related: https://github.com/open-webui/open-webui/issues/5894

<!-- gh-comment-id:4180376363 --> @Classic298 commented on GitHub (Apr 2, 2026): Audio input you say? @tjbck strongly related: https://github.com/open-webui/open-webui/issues/5894
GiteaMirror commented 2026-05-18 05:27:58 -05:00
Author
Owner
Copy Link

@TomTheWise commented on GitHub (Apr 3, 2026):

The smaller ones can understand audio, the larger ones can even understand / „see“ video without audio.

https://ai.google.dev/gemma/docs/core/model_card_4

https://youtu.be/jZVBoFOJK-Q?t=117

Llama.cpp is currently only supporting text+image for day0 support.

I guess the audio support will maybe more like comparable to supporting Mistral AIs Voxtral realtime and maybe less like whisper?

<!-- gh-comment-id:4182124916 --> @TomTheWise commented on GitHub (Apr 3, 2026): The smaller ones can understand audio, the larger ones can even understand / „see“ video without audio. https://ai.google.dev/gemma/docs/core/model_card_4 https://youtu.be/jZVBoFOJK-Q?t=117 Llama.cpp is currently only supporting text+image for day0 support. I guess the audio support will maybe more like comparable to supporting Mistral AIs Voxtral realtime and maybe less like whisper?
GiteaMirror commented 2026-05-18 05:27:59 -05:00
Author
Owner
Copy Link

@Classic298 commented on GitHub (Apr 7, 2026):

Related: https://github.com/open-webui/open-webui/issues/23471

<!-- gh-comment-id:4199525599 --> @Classic298 commented on GitHub (Apr 7, 2026): Related: https://github.com/open-webui/open-webui/issues/23471
GiteaMirror commented 2026-05-18 05:28:00 -05:00
Author
Owner
Copy Link

@mglaubitz commented on GitHub (Apr 9, 2026):

We use Open WebUI in as a tool for our university students and we have many use cases in teaching and research where interacting with audio / video is common demand (i.e. transcribe a recorded interview).
Thus, I strongly support the idea of having a dedicated way to deal with audio / video input. Users want to be able to deliberately send an audio to a transcription endpoint together with a prompt or pass along a video to "see". Right now videos are treated in the same way as PDF files an audio is only transcribed implicitly.

<!-- gh-comment-id:4212349329 --> @mglaubitz commented on GitHub (Apr 9, 2026): We use Open WebUI in as a tool for our university students and we have many use cases in teaching and research where interacting with audio / video is common demand (i.e. transcribe a recorded interview). Thus, I strongly support the idea of having a dedicated way to deal with audio / video input. Users want to be able to deliberately send an audio to a transcription endpoint together with a prompt or pass along a video to "see". Right now videos are treated in the same way as PDF files an audio is only transcribed implicitly.
GiteaMirror commented 2026-05-18 05:28:01 -05:00
Author
Owner
Copy Link

@Classic298 commented on GitHub (Apr 9, 2026):

https://github.com/open-webui/open-webui/issues/17217

<!-- gh-comment-id:4212359297 --> @Classic298 commented on GitHub (Apr 9, 2026): https://github.com/open-webui/open-webui/issues/17217
GiteaMirror commented 2026-05-18 05:28:01 -05:00
Author
Owner
Copy Link

@Classic298 commented on GitHub (Apr 9, 2026):

https://github.com/open-webui/open-webui/issues/23538

<!-- gh-comment-id:4213615518 --> @Classic298 commented on GitHub (Apr 9, 2026): https://github.com/open-webui/open-webui/issues/23538
GiteaMirror commented 2026-05-18 05:28:03 -05:00
Author
Owner
Copy Link

@TomTheWise commented on GitHub (Apr 10, 2026):

Google finally uploaded jinja chat templates that apparently now have a good quality.

IF (and I have no idea how it works in OWUI) OWUI does indeed need updates for the gemma4 control tokens, now they are official clear.
Here is the google documentation

<!-- gh-comment-id:4226004777 --> @TomTheWise commented on GitHub (Apr 10, 2026): Google finally uploaded jinja chat templates that apparently now have a good quality. IF (and I have no idea how it works in OWUI) OWUI does indeed need updates for the gemma4 control tokens, now they are official clear. [Here is the google documentation](https://ai.google.dev/gemma/docs/core/prompt-formatting-gemma4?hl=de)
GiteaMirror commented 2026-05-18 05:28:03 -05:00
Author
Owner
Copy Link

@TomTheWise commented on GitHub (Apr 12, 2026):

https://github.com/ggml-org/llama.cpp/pull/21421

llama.cpp now has initial multimodal audio support for gemma4 E4B / E2B

<!-- gh-comment-id:4231609860 --> @TomTheWise commented on GitHub (Apr 12, 2026): https://github.com/ggml-org/llama.cpp/pull/21421 llama.cpp now has initial multimodal audio support for gemma4 E4B / E2B
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#106954
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?