[GH-ISSUE #6953] Removing Silence from Audio Files for Local OpenAI/Whisper Models to Prevent Hallucinations #85286

New Issue

GiteaMirror · 2026-05-15T09:58:35-05:00

GiteaMirror commented

2026-05-15 09:58:35 -05:00

Originally created by @ghost on GitHub (Nov 14, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/6953

Proposal for Enhancement in Open WebUI: Silence Removal in Audio Files Before Processing with Whisper Model to Improve Speech Recognition Quality

Dear Developers,

In the current version of Open WebUI (v0.3.35), using local Whisper models for continuous real-time communication can lead to issues. When the Call mode (headphones icon to the right of the microphone) is activated and the user steps away from the computer, the model often listens to extended periods of silence. This situation results in Whisper generating random, nonsensical output when interaction resumes, instead of a meaningful response.

To address this issue, I developed a code that removes silence from audio files before they are processed by the model. This solution avoids "hallucinations" and greatly improves speech recognition quality. It allows for smooth and meaningful interaction with the Open WebUI voice assistant, eliminating unwanted noise and random text. This finally allowed me to communicate in voice assistant mode without the Whisper hallucinations!
Solution Overview:

Solution Overview:

Silence Removal. The code segments the audio file and removes silent sections in each fragment. It applies a volume threshold and a minimum silence duration to avoid cutting out essential pauses and to improve processing efficiency.
Parallel Processing. To speed up the process, it performs parallel processing on the audio chunks, allowing for efficient handling of longer recordings.
Result Generation. The silence-free audio is then fed into the model, generating a JSON file with the transcription, which delivers highly accurate speech recognition and clear transcription output.

My code was developed specifically for Open WebUI v0.3.35, and I cannot directly submit it to the main development branch, as there are significant differences in code structure that would require adaptation. However, implementing a similar solution in the latest version of WebUI would be extremely beneficial.

Optimization Recommendation: Silence removal could be accelerated using GPU processing, which would offer a significant boost in real-time applications. GPU-based parallel processing would enhance the speed of audio filtering, delivering high-quality and efficient user interaction.

Thank you for your hard work!

This is a repeat request to add audio cleanup, I originally did it here but was told to go here on the development branch. I hope that you will not reject my offer. Perhaps you can do much better sound cleaning than I suggest. I believe in your professionalism. I just want to make Open WebUI better.
https://github.com/open-webui/open-webui/pull/6894

The file and the code in it that I propose to change
open-webui/backend/open_webui/apps/audio /main.py

main.py.txt

import multiprocessing

def remove_silence_from_chunk(audio_chunk):

Removes silence from an audio chunk.

return split_on_silence(
    audio_chunk,
    min_silence_len=5000,  # minimum silence length for splitting
    silence_thresh=-40  # silence volume threshold
)

def remove_silence(audio):
# Removes silence from an audio file by splitting it into chunks and processing them in parallel.
chunk_size = 5000 # chunk size in milliseconds
chunks = [audio[i:i+chunk_size] for i in range(0, len(audio), chunk_size)]

# Parallel processing of audio chunks
with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
    results = pool.map(remove_silence_from_chunk, chunks)

# Combining non-silent chunks
non_silent_segments = [segment for chunk in results for segment in chunk]
return sum(non_silent_segments) if non_silent_segments else audio

def transcribe(file_path):
# Processes an audio file: removes silence and performs transcription.
print("Transcribing", file_path)

filename = os.path.basename(file_path)
file_dir = os.path.dirname(file_path)
file_id = filename.split(".")[0]  # File identifier without extension

# Check for speech recognition engine selection
if app.state.config.STT_ENGINE == "":
    if app.state.faster_whisper_model is None:
        set_faster_whisper_model(app.state.config.WHISPER_MODEL)

    model = app.state.faster_whisper_model

    # Loading and processing the audio file
    audio = AudioSegment.from_file(file_path)
    processed_audio = remove_silence(audio)  # Removing silence from audio
    
    # Saving processed audio to a temporary file
    temp_path = os.path.join(file_dir, f"{file_id}_processed.wav")
    processed_audio.export(temp_path, format="wav")
    
    # Transcription using the model
    segments, info = model.transcribe(temp_path, beam_size=5)
    log.info(
        "Detected language '%s' with probability %f" % (info.language, info.language_probability)
    )

    # Constructing the transcript from segments
    transcript = "".join([segment.text for segment in segments])
    data = {"text": transcript.strip()}

    # Saving the transcription to a JSON file
    transcript_file = os.path.join(file_dir, f"{file_id}.json")
    with open(transcript_file, "w") as f:
        json.dump(data, f)

    log.debug(data)
    return data

Originally created by @ghost on GitHub (Nov 14, 2024). Original GitHub issue: https://github.com/open-webui/open-webui/issues/6953 Proposal for Enhancement in Open WebUI: Silence Removal in Audio Files Before Processing with Whisper Model to Improve Speech Recognition Quality Dear Developers, In the current version of Open WebUI (v0.3.35), using local Whisper models for continuous real-time communication can lead to issues. When the Call mode (headphones icon to the right of the microphone) is activated and the user steps away from the computer, the model often listens to extended periods of silence. This situation results in Whisper generating random, nonsensical output when interaction resumes, instead of a meaningful response. To address this issue, I developed a code that removes silence from audio files before they are processed by the model. This solution avoids "hallucinations" and greatly improves speech recognition quality. It allows for smooth and meaningful interaction with the Open WebUI voice assistant, eliminating unwanted noise and random text. This finally allowed me to communicate in voice assistant mode without the Whisper hallucinations! Solution Overview: ### Solution Overview: 1. **Silence Removal**. The code segments the audio file and removes silent sections in each fragment. It applies a volume threshold and a minimum silence duration to avoid cutting out essential pauses and to improve processing efficiency. 2. **Parallel Processing**. To speed up the process, it performs parallel processing on the audio chunks, allowing for efficient handling of longer recordings. 3. **Result Generation**. The silence-free audio is then fed into the model, generating a JSON file with the transcription, which delivers highly accurate speech recognition and clear transcription output. My code was developed specifically for Open WebUI v0.3.35, and I cannot directly submit it to the main development branch, as there are significant differences in code structure that would require adaptation. However, implementing a similar solution in the latest version of WebUI would be extremely beneficial. **Optimization Recommendation**: Silence removal could be accelerated using GPU processing, which would offer a significant boost in real-time applications. GPU-based parallel processing would enhance the speed of audio filtering, delivering high-quality and efficient user interaction. Thank you for your hard work! This is a repeat request to add audio cleanup, I originally did it here but was told to go here on the development branch. I hope that you will not reject my offer. Perhaps you can do much better sound cleaning than I suggest. I believe in your professionalism. I just want to make Open WebUI better. https://github.com/open-webui/open-webui/pull/6894 The file and the code in it that I propose to change open-webui/backend/open_webui/apps/audio /main.py [main.py.txt](https://github.com/user-attachments/files/17754280/main.py.txt) import multiprocessing def remove_silence_from_chunk(audio_chunk): # Removes silence from an audio chunk. return split_on_silence( audio_chunk, min_silence_len=5000, # minimum silence length for splitting silence_thresh=-40 # silence volume threshold ) def remove_silence(audio): # Removes silence from an audio file by splitting it into chunks and processing them in parallel. chunk_size = 5000 # chunk size in milliseconds chunks = [audio[i:i+chunk_size] for i in range(0, len(audio), chunk_size)] # Parallel processing of audio chunks with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool: results = pool.map(remove_silence_from_chunk, chunks) # Combining non-silent chunks non_silent_segments = [segment for chunk in results for segment in chunk] return sum(non_silent_segments) if non_silent_segments else audio def transcribe(file_path): # Processes an audio file: removes silence and performs transcription. print("Transcribing", file_path) filename = os.path.basename(file_path) file_dir = os.path.dirname(file_path) file_id = filename.split(".")[0] # File identifier without extension # Check for speech recognition engine selection if app.state.config.STT_ENGINE == "": if app.state.faster_whisper_model is None: set_faster_whisper_model(app.state.config.WHISPER_MODEL) model = app.state.faster_whisper_model # Loading and processing the audio file audio = AudioSegment.from_file(file_path) processed_audio = remove_silence(audio) # Removing silence from audio # Saving processed audio to a temporary file temp_path = os.path.join(file_dir, f"{file_id}_processed.wav") processed_audio.export(temp_path, format="wav") # Transcription using the model segments, info = model.transcribe(temp_path, beam_size=5) log.info( "Detected language '%s' with probability %f" % (info.language, info.language_probability) ) # Constructing the transcript from segments transcript = "".join([segment.text for segment in segments]) data = {"text": transcript.strip()} # Saving the transcription to a JSON file transcript_file = os.path.join(file_dir, f"{file_id}.json") with open(transcript_file, "w") as f: json.dump(data, f) log.debug(data) return data

GiteaMirror closed this issue

2026-05-15 09:58:35 -05:00

GiteaMirror commented

2026-05-15 09:58:37 -05:00

@nphalem commented on GitHub (Dec 11, 2024):

Why don't you make a pull request from this instead of opening an issue?

@nphalem commented on GitHub (Dec 11, 2024): Why don't you make a pull request from this instead of opening an issue?

GiteaMirror commented

2026-05-15 09:58:37 -05:00

@SirBadfish commented on GitHub (Dec 20, 2024):

I'm just copy-pasting my comment from another thread here about this same issue:

Yeah, I just can't get the Whisper hallucinations to go away no matter what I do. It ruins OWUI entirely for me and makes it unusable for my purposes. "Thank you." gets added to everything I say. Sometimes, "Thank you. Thank you. Thank you." gets added. I thought it might be because I was using OWUI within Pinkokio, so I switched to a Docker install. I thought maybe it was my mic, my device, a bad wire, the Whisper model I was using, etc. Nothing I do gets it to go away. Happens on my Macbook Pro M1 Max 64gb, and on my PC w a 4090 as well. I've spent a good 8 hours troubleshooting this issue at this point, but it very much seems to be the way it's implemented into OWUI and not something on my end.

@SirBadfish commented on GitHub (Dec 20, 2024): I'm just copy-pasting my comment from another thread here about this same issue: Yeah, I just can't get the Whisper hallucinations to go away no matter what I do. It ruins OWUI entirely for me and makes it unusable for my purposes. "Thank you." gets added to everything I say. Sometimes, "Thank you. Thank you. Thank you." gets added. I thought it might be because I was using OWUI within Pinkokio, so I switched to a Docker install. I thought maybe it was my mic, my device, a bad wire, the Whisper model I was using, etc. Nothing I do gets it to go away. Happens on my Macbook Pro M1 Max 64gb, and on my PC w a 4090 as well. I've spent a good 8 hours troubleshooting this issue at this point, but it very much seems to be the way it's implemented into OWUI and not something on my end.

GiteaMirror commented

2026-05-15 09:58:37 -05:00

@nengoxx commented on GitHub (Dec 30, 2024):

I used a workaround for the hallucinations, in the audio.py you can add the VAD filter to the transcribe function call: segments, info = model.transcribe(file_path, language="en", beam_size=5, vad_filter=True).
Another solution would be to use faster-whisper-server instead of local whisper, it seems to have that silence removal functionality implemented, but still gets a bit too many hallucinations without adding the VAD parameter.

@nengoxx commented on GitHub (Dec 30, 2024): I used a workaround for the hallucinations, in the audio.py you can add the VAD filter to the transcribe function call: `segments, info = model.transcribe(file_path, language="en", beam_size=5, vad_filter=True)`. Another solution would be to use faster-whisper-server instead of local whisper, it seems to have that silence removal functionality implemented, but still gets a bit too many hallucinations without adding the VAD parameter.

GiteaMirror commented

2026-05-15 09:58:38 -05:00

@SirBadfish commented on GitHub (Dec 31, 2024):

@nengoxx yeah this:
I used a workaround for the hallucinations, in the audio.py you can add the VAD filter to the transcribe function call: segments, info = model.transcribe(file_path, language="en", beam_size=5, vad_filter=True)
... you mentioned this to me on Reddit also and it works great... Like it 100% resolved the hallucinations for me. Anyone that hasn't made this quick edit needs to try it. Just find the OWUI file named audio.py inside of the backend > open_webui > router folder and edit line 468 to the above and you are golden.

@SirBadfish commented on GitHub (Dec 31, 2024): @nengoxx yeah this: _I used a workaround for the hallucinations, in the audio.py you can add the VAD filter to the transcribe function call: `segments, info = model.transcribe(file_path, language="en", beam_size=5, vad_filter=True)`_ ... you mentioned this to me on Reddit also and it works great... Like it 100% resolved the hallucinations for me. Anyone that hasn't made this quick edit needs to try it. Just find the OWUI file named `audio.py` inside of the `backend > open_webui > router` folder and edit line 468 to the above and you are golden.

GiteaMirror commented

2026-05-15 09:58:39 -05:00

@TomBayne commented on GitHub (Apr 13, 2025):

I used a workaround for the hallucinations, in the audio.py you can add the VAD filter to the transcribe function call: segments, info = model.transcribe(file_path, language="en", beam_size=5, vad_filter=True). Another solution would be to use faster-whisper-server instead of local whisper, it seems to have that silence removal functionality implemented, but still gets a bit too many hallucinations without adding the VAD parameter.

I created a PR for this change. Seems like a no-brainer with no real downsides.

@TomBayne commented on GitHub (Apr 13, 2025): > I used a workaround for the hallucinations, in the audio.py you can add the VAD filter to the transcribe function call: `segments, info = model.transcribe(file_path, language="en", beam_size=5, vad_filter=True)`. Another solution would be to use faster-whisper-server instead of local whisper, it seems to have that silence removal functionality implemented, but still gets a bit too many hallucinations without adding the VAD parameter. I created a PR for this change. Seems like a no-brainer with no real downsides.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#85286