[GH-ISSUE #21875] issue: OUI 0.8.5 faster-whisper/ctranslate2 problem with Tesla V100S GPU

Originally created by @dohabandit on GitHub (Feb 25, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/21875 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Pip Install ### Open WebUI Version 0.8.5 ### Ollama Version (if applicable) 0.15.2 ### Operating System Ubuntu 24.04 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior STT via local whisper should have used the cuda library to process the audio chunks. ### Actual Behavior STT whisper/ctranslate2 are throwing CUDNN errors when using cuda regardless of the format being set to int8, float16, float32, or int8-float16. ### Steps to Reproduce Setting .env vars DEVICE_TYPE=cuda USE_CUDA_DOCKER=true forces OUI to use cuda for LLM inference as well as STT via local whisper/ctranslate2 modules are using DEVICE_TYPE setting. This throws errors during processing any audio input from the browser chat interface. ### Logs & Screenshots > fastapi.exceptions.HTTPException: 500: Error transcribing chunk: cuDNN failed with status CUDNN_STATUS_EXECUTION_FAILED_CUDART ### Additional Information I have an issue when using cuda for the local faster-whisper/ctranslate2 STT function. Switching the model type to float16 or float32 didn't work with my Tesla V100S for some reason. I get the same errors when it is attempting to process chunks of audio: > fastapi.exceptions.HTTPException: 500: Error transcribing chunk: cuDNN failed with status CUDNN_STATUS_EXECUTION_FAILED_CUDART I decided to switch the whisper STT to CPU, but there wasn't an easy way to do that with the OUI configuration. I made changes to audio.py and env.py to add an env variable "WHISPER_DEVICE_TYPE", which I set to "cpu". `# whisper is broken with V100S and CUDA12 libraries, use CPU WHISPER_DEVICE_TYPE=cpu WHISPER_COMPUTE_TYPE=int8 WHISPER_MODEL_AUTO_UPDATE=false ` Also I am running OUI locally in a venv and not running in a docker container. The env var to tell OUI to use cuda is counterintuitive and I think it needs to be changed because USE_CUDA_DOCKER is required to be set to true to use cuda even though you are not using docker!: `# use the V100S DEVICE_TYPE=cuda USE_CUDA_DOCKER=true CUDA_VISIBLE_DEVICES=0` Change to env.py: `# device type embedding models - "cpu" (default), "cuda" (nvidia gpu required) or "mps" (apple silicon) - choosing this right can lead to better performance USE_CUDA = os.environ.get("USE_CUDA_DOCKER", "false") WHISPER_DEVICE_TYPE = os.environ.get("WHISPER_DEVICE_TYPE") ` Change to audio.py: `faster_whisper_kwargs = { "model_size_or_path": model, "device": WHISPER_DEVICE_TYPE if WHISPER_DEVICE_TYPE and WHISPER_DEVICE_TYPE == "cuda" else "cpu", "compute_type": WHISPER_COMPUTE_TYPE, "download_root": WHISPER_MODEL_DIR, "local_files_only": not auto_update, } ` This lets me set DEVICE_TYPE to cuda for everything else tha OUI does, but have whisper/ctranslate2 using cpu instead via WHISPER_DEVICE_TYPE. My GPU resources are very limited already and having STT run via cpu is acceptable in my use case, albeit quite a bit slower. From my venv: faster-whisper==1.2.1 ctranslate2==4.6.2 nvidia-cudnn-cu12==9.10.2.21 PyTorch: 2.9.1+cu128 CUDA available: True CUDA version: 12.8 cuDNN version: 91002 ctranslate2: 4.6.2 GPU: Tesla V100S-PCIE-32GB

GiteaMirror commented

2026-04-25 09:20:42 -05:00

Owner

Originally created by @dohabandit on GitHub (Feb 25, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/21875

Check Existing Issues

I have searched for any existing and/or related issues.
I have searched for any existing and/or related discussions.
I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
I am using the latest version of Open WebUI.

Installation Method

Pip Install

Open WebUI Version

0.8.5

Ollama Version (if applicable)

0.15.2

Operating System

Ubuntu 24.04

Browser (if applicable)

No response

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

STT via local whisper should have used the cuda library to process the audio chunks.

Actual Behavior

STT whisper/ctranslate2 are throwing CUDNN errors when using cuda regardless of the format being set to int8, float16, float32, or int8-float16.

Steps to Reproduce

Setting .env vars DEVICE_TYPE=cuda USE_CUDA_DOCKER=true forces OUI to use cuda for LLM inference as well as STT via local whisper/ctranslate2 modules are using DEVICE_TYPE setting.
This throws errors during processing any audio input from the browser chat interface.

Logs & Screenshots

fastapi.exceptions.HTTPException: 500: Error transcribing chunk: cuDNN failed with status CUDNN_STATUS_EXECUTION_FAILED_CUDART

Additional Information

I have an issue when using cuda for the local faster-whisper/ctranslate2 STT function.
Switching the model type to float16 or float32 didn't work with my Tesla V100S for some reason. I get the same errors when it is attempting to process chunks of audio:

I decided to switch the whisper STT to CPU, but there wasn't an easy way to do that with the OUI configuration.
I made changes to audio.py and env.py to add an env variable "WHISPER_DEVICE_TYPE", which I set to "cpu".
# whisper is broken with V100S and CUDA12 libraries, use CPU WHISPER_DEVICE_TYPE=cpu WHISPER_COMPUTE_TYPE=int8 WHISPER_MODEL_AUTO_UPDATE=false

Also I am running OUI locally in a venv and not running in a docker container. The env var to tell OUI to use cuda is counterintuitive and I think it needs to be changed because USE_CUDA_DOCKER is required to be set to true to use cuda even though you are not using docker!:
# use the V100S DEVICE_TYPE=cuda USE_CUDA_DOCKER=true CUDA_VISIBLE_DEVICES=0

Change to env.py:
# device type embedding models - "cpu" (default), "cuda" (nvidia gpu required) or "mps" (apple silicon) - choosing this right can lead to better performance USE_CUDA = os.environ.get("USE_CUDA_DOCKER", "false") WHISPER_DEVICE_TYPE = os.environ.get("WHISPER_DEVICE_TYPE")

Change to audio.py:
faster_whisper_kwargs = { "model_size_or_path": model, "device": WHISPER_DEVICE_TYPE if WHISPER_DEVICE_TYPE and WHISPER_DEVICE_TYPE == "cuda" else "cpu", "compute_type": WHISPER_COMPUTE_TYPE, "download_root": WHISPER_MODEL_DIR, "local_files_only": not auto_update, }
This lets me set DEVICE_TYPE to cuda for everything else tha OUI does, but have whisper/ctranslate2 using cpu instead via WHISPER_DEVICE_TYPE. My GPU resources are very limited already and having STT run via cpu is acceptable in my use case, albeit quite a bit slower.

From my venv:
faster-whisper==1.2.1
ctranslate2==4.6.2
nvidia-cudnn-cu12==9.10.2.21
PyTorch: 2.9.1+cu128
CUDA available: True
CUDA version: 12.8
cuDNN version: 91002
ctranslate2: 4.6.2
GPU: Tesla V100S-PCIE-32GB

GiteaMirror added the bug label 2026-04-25 09:20:42 -05:00

GiteaMirror closed this issue

2026-04-25 09:20:43 -05:00

[GH-ISSUE #21875] issue: OUI 0.8.5 faster-whisper/ctranslate2 problem with Tesla V100S GPU #35130