feat: whisper integration #145

New Issue

Closed

opened 2025-11-11 14:08:04 -06:00 by GiteaMirror · 16 comments

GiteaMirror commented 2025-11-11 14:08:04 -06:00

Owner

Copy Link

Originally created by @adan89lion on GitHub (Jan 3, 2024).

Is your feature request related to a problem? Please describe.
The current voice input implementation is only supported by Safari. Also, it is highly unreliable due to constant typos and lack of bilingual or multilingual support. For instance, I'm studying foreign languages with the help of LLMs, and I'm unable to ask questions with both English and a second language. OpenAI has the ability to do that with Whisper model and it has been extremely helpful.

Describe the solution you'd like

1. The ability to download and run Whisper models (different size, e.g. base, medium, large)
   • or separate Whisper as another Docker container
2. Access microphone with browser's built-in API
3. Transcribe the voice via Whisper
4. Submit the transcription to the input box, or directly send the text to ollama for inference

Describe alternatives you've considered
N/A

Additional context
Whisper definitely outperforms OS's built-in voice input (with automatic punctuation, seamless multi-language support, great coverage of rare words).

Currently, only Safari integrate speech-to-text at OS level, making the voice input button useless on non-Apple devices or Chromium-based/Gecko-based browsers.

This is an extension to the feature request #49

Originally created by @adan89lion on GitHub (Jan 3, 2024). **Is your feature request related to a problem? Please describe.** The current voice input implementation is only supported by Safari. Also, it is highly unreliable due to constant typos and lack of bilingual or multilingual support. For instance, I'm studying foreign languages with the help of LLMs, and I'm unable to ask questions with _both_ English and a second language. OpenAI has the ability to do that with Whisper model and it has been extremely helpful. **Describe the solution you'd like** 1. The ability to download and run Whisper models (different size, e.g. base, medium, large) * or separate Whisper as another Docker container 2. Access microphone with browser's built-in API 3. Transcribe the voice via Whisper 4. Submit the transcription to the input box, or directly send the text to ollama for inference **Describe alternatives you've considered** N/A **Additional context** Whisper definitely outperforms OS's built-in voice input (with automatic punctuation, seamless multi-language support, great coverage of rare words). Currently, only Safari integrate speech-to-text at OS level, making the voice input button useless on non-Apple devices or Chromium-based/Gecko-based browsers. This is an extension to the feature request #49

GiteaMirror closed this issue 2025-11-11 14:08:04 -06:00

GiteaMirror commented 2025-11-11 14:08:05 -06:00

Author

Owner

Copy Link

@tjbck commented on GitHub (Jan 4, 2024):

Hi, Thanks for the feature request! FYI, voice input feature should work with chrome on non-apple devices as well, so if you're facing issues with chromium based browsers, please let us know! As for the feature request, I'll take a look in the near future and assess it's usability/feasibility. Thanks!

@tjbck commented on GitHub (Jan 4, 2024): Hi, Thanks for the feature request! FYI, voice input feature should work with chrome on non-apple devices as well, so if you're facing issues with chromium based browsers, please let us know! As for the feature request, I'll take a look in the near future and assess it's usability/feasibility. Thanks!

GiteaMirror commented 2025-11-11 14:08:06 -06:00

Author

Owner

Copy Link

@ThatOneCalculator commented on GitHub (Jan 4, 2024):

For me it works on Chromium and Firefox on Linux. No clue what OP is talking about with the whole "only safari" nonsense.

@ThatOneCalculator commented on GitHub (Jan 4, 2024): For me it works on Chromium and Firefox on Linux. No clue what OP is talking about with the whole "only safari" nonsense.

GiteaMirror commented 2025-11-11 14:08:06 -06:00

Author

Owner

Copy Link

@adan89lion commented on GitHub (Jan 4, 2024):

@ThatOneCalculator I just confirmed that voice input works on Chrome now. The reason it wasn't working is because the lack of a valid SSL certificate (I'm running on localhost all the time). Using an HTTPS connection successfully let Chrome prompt me for microphone permission.

As for Firefox, I'm not sure why it still don't work. The browser didn't prompt me to enable microphone access. I've:

1. Checked Firefox has microphone access in macOS Settings
2. Tested on ther websites (like Google Voice Search) can invoke Firefox microphone permission dialogue

@adan89lion commented on GitHub (Jan 4, 2024): @ThatOneCalculator I just confirmed that voice input works on Chrome now. The reason it wasn't working is because the lack of a valid SSL certificate (I'm running on localhost all the time). Using an HTTPS connection successfully let Chrome prompt me for microphone permission. As for Firefox, I'm not sure why it still don't work. The browser didn't prompt me to enable microphone access. I've: 1. Checked Firefox has microphone access in macOS Settings 2. Tested on ther websites (like Google Voice Search) can invoke Firefox microphone permission dialogue 

GiteaMirror commented 2025-11-11 14:08:06 -06:00

Author

Owner

Copy Link

@tarbard commented on GitHub (Jan 4, 2024):

Chrome has a default security policy where it doesn't allow microphone usage for non HTTPS sites, another workaround is that you can whitelist insecure sites within chrome settings.

@tarbard commented on GitHub (Jan 4, 2024): Chrome has a default security policy where it doesn't allow microphone usage for non HTTPS sites, another workaround is that you can whitelist insecure sites within chrome settings.

GiteaMirror commented 2025-11-11 14:08:07 -06:00

Author

Owner

Copy Link

@coder543 commented on GitHub (Jan 9, 2024):

I would love for this option to exist just because Whisper is so much more accurate.

@coder543 commented on GitHub (Jan 9, 2024): I would love for this option to exist just because Whisper is so much more accurate.

GiteaMirror commented 2025-11-11 14:08:07 -06:00

Author

Owner

Copy Link

@justinh-rahb commented on GitHub (Jan 18, 2024):

Related: #126

@justinh-rahb commented on GitHub (Jan 18, 2024): Related: #126

GiteaMirror commented 2025-11-11 14:08:08 -06:00

Author

Owner

Copy Link

@tjbck commented on GitHub (Jan 21, 2024):

Personal major blocker for this at the moment is how convoluted the whisper installation process can be to have it run locally, so I'm hoping Ollama team would include whisper.cpp to their project. It would streamline the whole installation process for everyone, and would align better with our project ethos of local first approach.

@tjbck commented on GitHub (Jan 21, 2024): Personal major blocker for this at the moment is how convoluted the whisper installation process can be to have it run locally, so I'm hoping Ollama team would include whisper.cpp to their project. It would streamline the whole installation process for everyone, and would align better with our project ethos of local first approach.

GiteaMirror commented 2025-11-11 14:08:08 -06:00

Author

Owner

Copy Link

@Collected5353 commented on GitHub (Jan 24, 2024):

https://github.com/oobabooga/text-generation-webui/blob/main/extensions/whisper_stt/script.py

We would need to add some packages to the docker image as well or your local system. I still think this is the way to go for audio input: webkitSpeechRecognition

@tjbck Looks like whispper.cpp has a WASM available: https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.wasm

Might need to add a line to the documentation that it requires SSL for most browser security as others have mentioned here. Something like ngrok free plan can work.

https://github.com/oobabooga/text-generation-webui/blob/main/extensions/coqui_tts/script.py

Is using Coqui, does anyone hav experience using Coqui_tts or silero_tts?

We could pull down and container in the compose also to just use a tts from the api.
https://docs.coqui.ai/en/latest/marytts.html

@Collected5353 commented on GitHub (Jan 24, 2024): https://github.com/oobabooga/text-generation-webui/blob/main/extensions/whisper_stt/script.py We would need to add some packages to the docker image as well or your local system. I still think this is the way to go for audio input: webkitSpeechRecognition @tjbck Looks like whispper.cpp has a WASM available: https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.wasm Might need to add a line to the documentation that it requires SSL for most browser security as others have mentioned here. Something like ngrok free plan can work. https://github.com/oobabooga/text-generation-webui/blob/main/extensions/coqui_tts/script.py Is using Coqui, does anyone hav experience using Coqui_tts or silero_tts? We could pull down and container in the compose also to just use a tts from the api. https://docs.coqui.ai/en/latest/marytts.html

GiteaMirror commented 2025-11-11 14:08:08 -06:00

Author

Owner

Copy Link

@tjbck commented on GitHub (Feb 11, 2024):

Whisper support has been added with #707! FYI, whisper STT will only work over https and It'll take some time to download the whisper model when you first use it so please give it a minute or two to finish downloading! Let me know if you guys encounter any issues, thanks!

@tjbck commented on GitHub (Feb 11, 2024):  Whisper support has been added with #707! FYI, whisper STT will only work over `https` and It'll take some time to download the whisper model when you first use it so please give it a minute or two to finish downloading! Let me know if you guys encounter any issues, thanks!

GiteaMirror commented 2025-11-11 14:08:09 -06:00

Author

Owner

Copy Link

@tino926 commented on GitHub (Mar 4, 2024):

@tjbck
how can i use open webui over https? I follow the instruction at this repo to install the ollama and open-webui docker on a computer. If i connect to open-webui from another computer with https, is always show message like:

Secure Connection Failed

An error occurred during a connection to xxx.xxx.xxx.xxx:3000. SSL received a record that exceeded the maximum permissible length.

Error code: SSL_ERROR_RX_RECORD_TOO_LONG

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
Please contact the website owners to inform them of this problem.

@tino926 commented on GitHub (Mar 4, 2024): @tjbck how can i use open webui over https? I follow the instruction at this repo to install the ollama and open-webui docker on a computer. If i connect to open-webui from another computer with https, is always show message like: Secure Connection Failed An error occurred during a connection to xxx.xxx.xxx.xxx:3000. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. Please contact the website owners to inform them of this problem.

GiteaMirror commented 2025-11-11 14:08:10 -06:00

Author

Owner

Copy Link

@justinh-rahb commented on GitHub (Mar 4, 2024):

The truth is that there is no simple way to set up HTTPS for a web application that does not require careful attention and expertise of the user. "Easy" HTTPS requires exposure to the Internet in most cases. Running a web application exposed to the internet is a serious responsibility, and it is essential to take all necessary precautions to ensure the integrity and confidentiality of your data and systems. Providing simplified instructions for setting up HTTPS can be irresponsible in the hands of those that don't fully understand the implications of such.

@justinh-rahb commented on GitHub (Mar 4, 2024): The truth is that there is no simple way to set up HTTPS for a web application that does not require careful attention and expertise of the user. "Easy" HTTPS requires exposure to the Internet in most cases. Running a web application exposed to the internet is a serious responsibility, and it is essential to take all necessary precautions to ensure the integrity and confidentiality of your data and systems. Providing simplified instructions for setting up HTTPS can be irresponsible in the hands of those that don't fully understand the implications of such.

GiteaMirror commented 2025-11-11 14:08:10 -06:00

Author

Owner

Copy Link

@phyzical commented on GitHub (May 19, 2024):

Just curious if support for using a external hosted whisper container will be added one day? just as i run one for other stuff, feels redundant to also have the model in webui, maybe inversely if webui exposed whisper to be used by other services?

@phyzical commented on GitHub (May 19, 2024): Just curious if support for using a external hosted whisper container will be added one day? just as i run one for other stuff, feels redundant to also have the model in webui, maybe inversely if webui exposed whisper to be used by other services?

GiteaMirror commented 2025-11-11 14:08:10 -06:00

Author

Owner

Copy Link

@coder543 commented on GitHub (May 19, 2024):

@phyzical out of curiosity, which whisper container do you use (to be clear, I have not contributed to open-webui, but I am curious about a whisper server)

@coder543 commented on GitHub (May 19, 2024): @phyzical out of curiosity, which whisper container do you use (to be clear, I have not contributed to open-webui, but I am curious about a whisper server)

GiteaMirror commented 2025-11-11 14:08:11 -06:00

Author

Owner

Copy Link

@malteneuss commented on GitHub (May 19, 2024):

@justinh-rahb Just wanted to further promote Let's encrypt "DNS-01" challenges to get browser-trusted https certificates for local self-host services without internet exposure. The nice overview can be found by "Wolfgangs channel" at https://www.youtube.com/watch?v=qlcVx-k-02E.

Also i would like to promote NixOS, which is gaining a lot of traction and has one of the easiest ways to setup Let's encrypt using a Nginx reverse proxy i found so far. The rough amount of config code, once you have some familiarity, is:

{
  services.nginx = {
    enable = true;

    virtualHosts."subdomain.example.org" = {
      # use Let's encrypt https certificate defined below 
      useACMEHost = "example.org";
      forceSSL = true;
      locations."/" = {
         # usual nginx forwarding.
      };
    };
  };

  security.acme.acceptTerms = true;
  security.acme.defaults.email = "info@example.org";

  # Let NixOS generate a Let's Encrypt certificate that we can reuse
  # above for several virtualhosts above.
  security.acme.certs."example.org" = {
    domain = "example.org";
    extraDomainNames = [ "subdomain.example.org" ];
    # Nix uses Go library LEGO DNS to perform the DNS challence
    dnsProvider = "<your-dns-provider>";
    dnsPropagationCheck = true;
  };
}

See "Minimal Private Local LAN Server Example" at https://wiki.nixos.org/wiki/Nginx for more details.

Furthermore NixOS already built-in Ollama as a web service support. The amount of code to enable an Ollama service is:

 services.ollama = {
    enable = true;
    # Make it available to the local network.
    listenAddress = "0.0.0.0:11434";
  };

@malteneuss commented on GitHub (May 19, 2024): @justinh-rahb Just wanted to further promote Let's encrypt "DNS-01" challenges to get browser-trusted https certificates for local self-host services without internet exposure. The nice overview can be found by "Wolfgangs channel" at https://www.youtube.com/watch?v=qlcVx-k-02E. Also i would like to promote NixOS, which is gaining a lot of traction and has one of the easiest ways to setup Let's encrypt using a Nginx reverse proxy i found so far. The rough amount of config code, once you have some familiarity, is: ```nix { services.nginx = { enable = true; virtualHosts."subdomain.example.org" = { # use Let's encrypt https certificate defined below useACMEHost = "example.org"; forceSSL = true; locations."/" = { # usual nginx forwarding. }; }; }; security.acme.acceptTerms = true; security.acme.defaults.email = "info@example.org"; # Let NixOS generate a Let's Encrypt certificate that we can reuse # above for several virtualhosts above. security.acme.certs."example.org" = { domain = "example.org"; extraDomainNames = [ "subdomain.example.org" ]; # Nix uses Go library LEGO DNS to perform the DNS challence dnsProvider = "<your-dns-provider>"; dnsPropagationCheck = true; }; } ``` See "Minimal Private Local LAN Server Example" at https://wiki.nixos.org/wiki/Nginx for more details. Furthermore NixOS already built-in Ollama as a web service support. The amount of code to enable an Ollama service is: ```nix services.ollama = { enable = true; # Make it available to the local network. listenAddress = "0.0.0.0:11434"; }; ```

GiteaMirror commented 2025-11-11 14:08:11 -06:00

Author

Owner

Copy Link

@phyzical commented on GitHub (May 20, 2024):

@coder543 https://github.com/ahmetoner/whisper-asr-webservice

@phyzical commented on GitHub (May 20, 2024): @coder543 https://github.com/ahmetoner/whisper-asr-webservice

GiteaMirror commented 2025-11-11 14:08:12 -06:00

Author

Owner

Copy Link

@mikael1234 commented on GitHub (Sep 25, 2024):

Is there some settings for Whisper? Its producing random garbage in random languages in Chrome. Same with Web API

@mikael1234 commented on GitHub (Sep 25, 2024): Is there some settings for Whisper? Its producing random garbage in random languages in Chrome. Same with Web API

GiteaMirror referenced this issue 2025-11-11 17:13:16 -06:00

[PR #145] [MERGED] fix: user profile image for disabled auth #6973

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dev

backend-outlet

v0.8.10

v0.8.9

v0.8.8

v0.8.7

v0.8.6

v0.8.5

v0.8.4

v0.8.3

v0.8.2

v0.8.1

v0.8.0

v0.7.2

v0.7.1

v0.7.0

v0.6.43

v0.6.42

v0.6.41

v0.6.40

v0.6.39

v0.6.38

v0.6.37

v0.6.36

v0.6.35

v0.6.34

v0.6.33

v0.6.32

v0.6.31

v0.6.30

v0.6.29

v0.6.28

v0.6.27

v0.6.26

v0.6.25

v0.6.24

v0.6.23

v0.6.22

v0.6.21

v0.6.20

v0.6.19

v0.6.18

v0.6.17

v0.6.16

v0.6.15

v0.6.14

v0.6.13

v0.6.12

v0.6.11

v0.6.10

v0.6.9

v0.6.8

v0.6.7

v0.6.6

v0.6.5

v0.6.4

v0.6.3

v0.6.2

v0.6.1

v0.6.0

v0.5.20

v0.5.19

v0.5.18

v0.5.17

v0.5.16

v0.5.15

v0.5.14

v0.5.13

v0.5.12

v0.5.11

v0.5.10

v0.5.9

v0.5.8

v0.5.7

v0.5.6

v0.5.5

v0.5.4

v0.5.3

v0.5.2

v0.5.1

v0.5.0

v0.4.8

v0.4.7

v0.4.6

v0.4.5

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.4.0

v0.3.35

v0.3.34

v0.3.33

v0.3.32

v0.3.31

v0.3.30

v0.3.29

v0.3.28

v0.3.27

v0.3.26

v0.3.25

v0.3.24

v0.3.23

v0.3.22

v0.3.21

v0.3.20

v0.3.19

v0.3.18

v0.3.17

v0.3.16

v0.3.15

v0.3.14

v0.3.13

v0.3.12

v0.3.11

v0.3.10

v0.3.9

v0.3.8

v0.3.7

v0.3.6

v0.3.5

v0.3.4

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.2.5

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.125

v0.1.124

v0.1.123

v0.1.122

v0.1.121

v0.1.120

v0.1.119

v0.1.118

v0.1.117

v0.1.116

v0.1.115

v0.1.114

v0.1.113

v0.1.112

v0.1.111

v0.1.110

v0.1.109

v0.1.108

v0.1.107

v0.1.106

v0.1.105

v0.1.104

v0.1.103

v0.1.102

Labels

Clear labels

bug

core

documentation

enhancement

good first issue

help wanted

non-core

pull-request

Mirrored from GitHub Pull Request

python

question

testing wanted

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

GiteaMirror ninjasurge

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#145