[GH-ISSUE #7779] bug: Changing CMU Arctic speakers doesn't work #118198

Closed
opened 2026-05-20 19:38:20 -05:00 by GiteaMirror · 4 comments
Owner

Originally created by @nphalem on GitHub (Dec 11, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/7779

Bug Report

Installation Method

Docker compose with searxng

Environment

  • Open WebUI Version: v0.4.8

  • Ollama (if applicable): v.0.5.1

  • Operating System: Arch Linux 6.12.4

Confirmation:

  • I have read and followed all the instructions provided in the README.md.
  • I am on the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.

Expected Behavior:

OpenWebUI changes the voice model when changed in settings.

Actual Behavior:

OpenWebUI doesn't change the voice model when changed in settings.

Description

Bug Summary:
When changing the CMU Artic speaker model the voice stays the same.

Reproduction Details

Steps to Reproduce:

  1. In Settings > Audio set TTS Settings to Transformers (Local)
  2. Set TTS Model to slt (US female)
  3. Open new chat and test the voice
  4. Change the TTS Model to bdl (US male)

Logs and Screenshots

Docker Container Logs:

  ___                    __        __   _     _   _ ___
 / _ \ _ __   ___ _ __   \ \      / /__| |__ | | | |_ _|
| | | | '_ \ / _ \ '_ \   \ \ /\ / / _ \ '_ \| | | || |
| |_| | |_) |  __/ | | |   \ V  V /  __/ |_) | |_| || |
 \___/| .__/ \___|_| |_|    \_/\_/ \___|_.__/ \___/|___|
      |_|


v0.4.8 - building the best open-source AI user interface.

https://github.com/open-webui/open-webui

INFO:     192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprMH- HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=websocket&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 400 Bad Request
INFO:     192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprMIA&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprMIA.0&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprMIR&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprSOo&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /static/favicon.png HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/v1/users/ HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/v1/users/ HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprSOo.0&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprYVZ&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprYVZ.0&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprecL&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprecM&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprkj8&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprkj8.0&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprqpv&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /static/favicon.png HTTP/1.0" 304 Not Modified
INFO:     192.168.1.13:0 - "GET /api/v1/auths/admin/config HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/webhook HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/v1/auths/admin/config/ldap/server HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/v1/auths/admin/config/ldap HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /audio/api/v1/config HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /audio/api/v1/voices HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /audio/api/v1/models HTTP/1.0" 200 OK
INFO  [open_webui.env] Saving 'AUDIO_TTS_OPENAI_API_BASE_URL' to the database
INFO  [open_webui.env] Saving 'AUDIO_TTS_OPENAI_API_KEY' to the database
INFO  [open_webui.env] Saving 'AUDIO_TTS_API_KEY' to the database
INFO  [open_webui.env] Saving 'AUDIO_TTS_ENGINE' to the database
INFO  [open_webui.env] Saving 'AUDIO_TTS_MODEL' to the database
INFO  [open_webui.env] Saving 'AUDIO_TTS_VOICE' to the database
INFO  [open_webui.env] Saving 'AUDIO_TTS_SPLIT_ON' to the database
INFO  [open_webui.env] Saving 'AUDIO_TTS_AZURE_SPEECH_REGION' to the database
INFO  [open_webui.env] Saving 'AUDIO_TTS_AZURE_SPEECH_OUTPUT_FORMAT' to the database
INFO  [open_webui.env] Saving 'AUDIO_STT_OPENAI_API_BASE_URL' to the database
INFO  [open_webui.env] Saving 'AUDIO_STT_OPENAI_API_KEY' to the database
INFO  [open_webui.env] Saving 'AUDIO_STT_ENGINE' to the database
INFO  [open_webui.env] Saving 'AUDIO_STT_MODEL' to the database
INFO  [open_webui.env] Saving 'WHISPER_MODEL' to the database
INFO:     192.168.1.13:0 - "POST /audio/api/v1/config/update HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/config HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /static/favicon.png HTTP/1.0" 304 Not Modified
INFO:     192.168.1.13:0 - "GET /api/v1/chats/00381707-0a57-4ce0-b336-16ff7c6557a8 HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/v1/chats/00381707-0a57-4ce0-b336-16ff7c6557a8/tags HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/v1/users/user/settings HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /ollama/api/version HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/v1/chats/002cca7f-dfbb-40c2-8238-328e4e02befe HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/v1/chats/002cca7f-dfbb-40c2-8238-328e4e02befe/tags HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /api/v1/users/user/settings HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprqpv.0&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprwwh&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK
INFO:     192.168.1.13:0 - "POST /audio/api/v1/speech HTTP/1.0" 200 OK
Originally created by @nphalem on GitHub (Dec 11, 2024). Original GitHub issue: https://github.com/open-webui/open-webui/issues/7779 # Bug Report ## Installation Method Docker compose with searxng ## Environment - **Open WebUI Version:** v0.4.8 - **Ollama (if applicable):** v.0.5.1 - **Operating System:** Arch Linux 6.12.4 **Confirmation:** - [x] I have read and followed all the instructions provided in the README.md. - [x] I am on the latest version of both Open WebUI and Ollama. - [ ] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below. ## Expected Behavior: OpenWebUI changes the voice model when changed in settings. ## Actual Behavior: OpenWebUI doesn't change the voice model when changed in settings. ## Description **Bug Summary:** When changing the CMU Artic speaker model the voice stays the same. ## Reproduction Details **Steps to Reproduce:** 1. In **Settings** > **Audio** set **TTS Settings** to **Transformers (Local)** 2. Set **TTS Model** to `slt` (US female) 3. Open new chat and test the voice 4. Change the **TTS Model** to `bdl` (US male) ## Logs and Screenshots **Docker Container Logs:** ``` ___ __ __ _ _ _ ___ / _ \ _ __ ___ _ __ \ \ / /__| |__ | | | |_ _| | | | | '_ \ / _ \ '_ \ \ \ /\ / / _ \ '_ \| | | || | | |_| | |_) | __/ | | | \ V V / __/ |_) | |_| || | \___/| .__/ \___|_| |_| \_/\_/ \___|_.__/ \___/|___| |_| v0.4.8 - building the best open-source AI user interface. https://github.com/open-webui/open-webui INFO: 192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprMH- HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=websocket&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 400 Bad Request INFO: 192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprMIA&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprMIA.0&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprMIR&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprSOo&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /static/favicon.png HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/v1/users/ HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/v1/users/ HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprSOo.0&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprYVZ&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprYVZ.0&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprecL&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprecM&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprkj8&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprkj8.0&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprqpv&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /static/favicon.png HTTP/1.0" 304 Not Modified INFO: 192.168.1.13:0 - "GET /api/v1/auths/admin/config HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/webhook HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/v1/auths/admin/config/ldap/server HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/v1/auths/admin/config/ldap HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /audio/api/v1/config HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /audio/api/v1/voices HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /audio/api/v1/models HTTP/1.0" 200 OK INFO [open_webui.env] Saving 'AUDIO_TTS_OPENAI_API_BASE_URL' to the database INFO [open_webui.env] Saving 'AUDIO_TTS_OPENAI_API_KEY' to the database INFO [open_webui.env] Saving 'AUDIO_TTS_API_KEY' to the database INFO [open_webui.env] Saving 'AUDIO_TTS_ENGINE' to the database INFO [open_webui.env] Saving 'AUDIO_TTS_MODEL' to the database INFO [open_webui.env] Saving 'AUDIO_TTS_VOICE' to the database INFO [open_webui.env] Saving 'AUDIO_TTS_SPLIT_ON' to the database INFO [open_webui.env] Saving 'AUDIO_TTS_AZURE_SPEECH_REGION' to the database INFO [open_webui.env] Saving 'AUDIO_TTS_AZURE_SPEECH_OUTPUT_FORMAT' to the database INFO [open_webui.env] Saving 'AUDIO_STT_OPENAI_API_BASE_URL' to the database INFO [open_webui.env] Saving 'AUDIO_STT_OPENAI_API_KEY' to the database INFO [open_webui.env] Saving 'AUDIO_STT_ENGINE' to the database INFO [open_webui.env] Saving 'AUDIO_STT_MODEL' to the database INFO [open_webui.env] Saving 'WHISPER_MODEL' to the database INFO: 192.168.1.13:0 - "POST /audio/api/v1/config/update HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/config HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /static/favicon.png HTTP/1.0" 304 Not Modified INFO: 192.168.1.13:0 - "GET /api/v1/chats/00381707-0a57-4ce0-b336-16ff7c6557a8 HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/v1/chats/00381707-0a57-4ce0-b336-16ff7c6557a8/tags HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/v1/users/user/settings HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /ollama/api/version HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/v1/chats/002cca7f-dfbb-40c2-8238-328e4e02befe HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/v1/chats/002cca7f-dfbb-40c2-8238-328e4e02befe/tags HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /api/v1/users/user/settings HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "GET /ws/socket.io/?EIO=4&transport=polling&t=PEprqpv.0&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "POST /ws/socket.io/?EIO=4&transport=polling&t=PEprwwh&sid=ZgnRn8YsAqIms3UoAAAA HTTP/1.0" 200 OK INFO: 192.168.1.13:0 - "POST /audio/api/v1/speech HTTP/1.0" 200 OK ```
Author
Owner

@tjbck commented on GitHub (Dec 11, 2024):

Incorrect format.

<!-- gh-comment-id:2537223056 --> @tjbck commented on GitHub (Dec 11, 2024): Incorrect format.
Author
Owner

@nphalem commented on GitHub (Dec 12, 2024):

Apologies @tjbck but could you clarify what I did wrong so I can correct it? Or are you saying the TTS model has incorrect formatting?

<!-- gh-comment-id:2538789098 --> @nphalem commented on GitHub (Dec 12, 2024): Apologies @tjbck but could you clarify what I did wrong so I can correct it? Or are you saying the TTS model has incorrect formatting?
Author
Owner

@dmabry commented on GitHub (Dec 20, 2024):

@nphalem I had the same issue and I dove into the code to figure it out.

tldr; use cmu_us_bdl_arctic-wav-arctic_a0001 as the TTS_MODEL and it should work.

I wrote a little python script that pulls down the embeddings and lists out the files in the dataset. The open-webui backend code basically does a lookup for that file name and uses the index to "unsqueeze". Here is the specific code that should help you understand what is going on.

    elif app.state.config.TTS_ENGINE == "transformers":
        payload = None
        try:
            payload = json.loads(body.decode("utf-8"))
        except Exception as e:
            log.exception(e)
            raise HTTPException(status_code=400, detail="Invalid JSON payload")

        import torch
        import soundfile as sf

        load_speech_pipeline()

        embeddings_dataset = app.state.speech_speaker_embeddings_dataset

        speaker_index = 6799
        try:
            speaker_index = embeddings_dataset["filename"].index(
                app.state.config.TTS_MODEL
            )
        except Exception:
            pass

        speaker_embedding = torch.tensor(
            embeddings_dataset[speaker_index]["xvector"]
        ).unsqueeze(0)

        speech = app.state.speech_synthesiser(
            payload["input"],
            forward_params={"speaker_embeddings": speaker_embedding},
        )

        sf.write(file_path, speech["audio"], samplerate=speech["sampling_rate"])
        with open(file_body_path, "w") as f:
            json.dump(json.loads(body.decode("utf-8")), f)

        return FileResponse(file_path)
        ```
        
        So, if you put it something that isn't found like "bdl" then it defaults to index 6799 which is why you couldn't get the voice to change.
<!-- gh-comment-id:2557254631 --> @dmabry commented on GitHub (Dec 20, 2024): @nphalem I had the same issue and I dove into the code to figure it out. tldr; use cmu_us_bdl_arctic-wav-arctic_a0001 as the TTS_MODEL and it should work. I wrote a little python script that pulls down the embeddings and lists out the files in the dataset. The open-webui backend code basically does a lookup for that file name and uses the index to "unsqueeze". Here is the specific code that should help you understand what is going on. ```python elif app.state.config.TTS_ENGINE == "transformers": payload = None try: payload = json.loads(body.decode("utf-8")) except Exception as e: log.exception(e) raise HTTPException(status_code=400, detail="Invalid JSON payload") import torch import soundfile as sf load_speech_pipeline() embeddings_dataset = app.state.speech_speaker_embeddings_dataset speaker_index = 6799 try: speaker_index = embeddings_dataset["filename"].index( app.state.config.TTS_MODEL ) except Exception: pass speaker_embedding = torch.tensor( embeddings_dataset[speaker_index]["xvector"] ).unsqueeze(0) speech = app.state.speech_synthesiser( payload["input"], forward_params={"speaker_embeddings": speaker_embedding}, ) sf.write(file_path, speech["audio"], samplerate=speech["sampling_rate"]) with open(file_body_path, "w") as f: json.dump(json.loads(body.decode("utf-8")), f) return FileResponse(file_path) ``` So, if you put it something that isn't found like "bdl" then it defaults to index 6799 which is why you couldn't get the voice to change.
Author
Owner

@dmabry commented on GitHub (Dec 20, 2024):

And to just put it out there. Here is the list of relevant speakers with their indexes and names:

0: cmu_us_awb_arctic-wav-arctic_a0001
1138: cmu_us_bdl_arctic-wav-arctic_a0001
2271: cmu_us_clb_arctic-wav-arctic_a0001
3403: cmu_us_jmk_arctic-wav-arctic_a0001
4535: cmu_us_ksp_arctic-wav-arctic_a0001
5667: cmu_us_rms_arctic-wav-arctic_a0001
6799: cmu_us_slt_arctic-wav-arctic_a0001

The below code lists all files in the dataset, but you only need to reference the first one for each speaker in order for it to work.

list_speakers.py

#!/usr/bin/env python3
from datasets import load_dataset


def list_available_speakers():
    # Load the speaker embeddings dataset
    embeddings_dataset = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation")

    # Get a list of all available filenames (speaker model identifiers)
    speaker_names = embeddings_dataset["filename"]
    return speaker_names


if __name__ == "__main__":
    # Fetch and print all valid speaker names
    speakers = list_available_speakers()
    print("Available Speaker Names:")
    for i, speaker in enumerate(speakers):
        print(f"{i}: {speaker}")
<!-- gh-comment-id:2557263850 --> @dmabry commented on GitHub (Dec 20, 2024): And to just put it out there. Here is the list of relevant speakers with their indexes and names: ``` 0: cmu_us_awb_arctic-wav-arctic_a0001 1138: cmu_us_bdl_arctic-wav-arctic_a0001 2271: cmu_us_clb_arctic-wav-arctic_a0001 3403: cmu_us_jmk_arctic-wav-arctic_a0001 4535: cmu_us_ksp_arctic-wav-arctic_a0001 5667: cmu_us_rms_arctic-wav-arctic_a0001 6799: cmu_us_slt_arctic-wav-arctic_a0001 ``` The below code lists all files in the dataset, but you only need to reference the first one for each speaker in order for it to work. list_speakers.py ```python #!/usr/bin/env python3 from datasets import load_dataset def list_available_speakers(): # Load the speaker embeddings dataset embeddings_dataset = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation") # Get a list of all available filenames (speaker model identifiers) speaker_names = embeddings_dataset["filename"] return speaker_names if __name__ == "__main__": # Fetch and print all valid speaker names speakers = list_available_speakers() print("Available Speaker Names:") for i, speaker in enumerate(speakers): print(f"{i}: {speaker}") ```
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#118198