[PR #20259] [CLOSED] fix: multi-worker direct connection chat streaming timeout #25524

New Issue

GiteaMirror · 2026-04-20T05:58:34-05:00

GiteaMirror commented

2026-04-20 05:58:34 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/20259
Author: @Classic298
Created: 12/30/2025
Status: ❌ Closed

Base: dev ← Head: redis-queue

📝 Commits (10+)

710003b init
8fa83a0 Update main.py
5a61241 Update chat.py
a0849d0 Update +layout.svelte
2cfd08b Update main.py
44450e8 Update main.py
3950e22 Update main.py
296686d Update main.py
5107f34 Update main.py
7db6d6c Update chat.py

📊 Changes

3 files changed (+98 additions, -7 deletions)

View changed files

📝 backend/open_webui/main.py (+14 -0)
📝 backend/open_webui/socket/main.py (+69 -0)
📝 backend/open_webui/utils/chat.py (+15 -7)

📄 Description

Problem

When using direct connection mode with multiple backend workers (uvicorn workers > 1 or multiple k8s pods), streaming chat requests intermittently timeout and never complete.

Root Cause: The generate_direct_chat_completion function dynamically registers a Socket.IO event handler using sio.on(channel, handler) for each streaming request. This handler only exists in the local worker's memory. When the browser's WebSocket connection lives on a different worker, streaming chunks arrive at that worker but no handler exists there - causing messages to be silently dropped and the original request to wait forever.

Solution

Replace per-request dynamic handler registration with a global static handler and Redis pub/sub routing:

Global Queue Registry: Add DIRECT_CHAT_QUEUES dictionary to track channel-to-queue mappings per worker
Static Global Handler: Add a sio.on("direct-chat-stream") handler that exists in all workers and routes messages to local queues or publishes via Redis
Redis Pub/Sub Listener: Each worker subscribes to direct chat channel patterns and delivers messages to local queues when present
Frontend Update: Changed socket.emit to use the global event name with channel metadata instead of dynamic channel names

Flow After Fix

HTTP request arrives at Worker A, registers queue in DIRECT_CHAT_QUEUES
Browser emits streaming chunks to "direct-chat-stream" global event
Worker B (holding WebSocket) receives event via global handler
Worker B publishes to Redis since queue is not local
Worker A receives Redis message, finds local queue, delivers data
Streaming completes successfully

Files Changed

backend/open_webui/socket/main.py - Added global handler and Redis pub/sub listener
backend/open_webui/utils/chat.py - Use queue registry instead of dynamic sio.on()
backend/open_webui/main.py - Start Redis listener on startup
src/routes/+layout.svelte - Emit to global event with channel metadata

Testing

Deploy with WEBSOCKET_MANAGER=redis and multiple workers
Enable direct connection mode in user settings
Send concurrent streaming chat requests
Verify all requests complete without timeout

Closes #15162

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/20259 **Author:** [@Classic298](https://github.com/Classic298) **Created:** 12/30/2025 **Status:** ❌ Closed **Base:** `dev` ← **Head:** `redis-queue` --- ### 📝 Commits (10+) - [`710003b`](https://github.com/open-webui/open-webui/commit/710003b2f8d3a4796cd71f5c1c9f5bcfa0103e60) init - [`8fa83a0`](https://github.com/open-webui/open-webui/commit/8fa83a0f1f877c49d3bd74cb02454e3673a75d04) Update main.py - [`5a61241`](https://github.com/open-webui/open-webui/commit/5a61241900999c1e957c62f49a33b95422a9615f) Update chat.py - [`a0849d0`](https://github.com/open-webui/open-webui/commit/a0849d00f98ccc3ba82ff4b2851f4fd8af9da399) Update +layout.svelte - [`2cfd08b`](https://github.com/open-webui/open-webui/commit/2cfd08b2c2b5339315fca787b3bfdc118274067e) Update main.py - [`44450e8`](https://github.com/open-webui/open-webui/commit/44450e8a6dc75754daf013bb6c91bf8c3039a168) Update main.py - [`3950e22`](https://github.com/open-webui/open-webui/commit/3950e2222fa5ec9a6b5888cf48fcd72c9d943f22) Update main.py - [`296686d`](https://github.com/open-webui/open-webui/commit/296686da4081cb52704b45b05080a4553154c2a8) Update main.py - [`5107f34`](https://github.com/open-webui/open-webui/commit/5107f34daa75869728bb65fdddbe3921ca5b40e0) Update main.py - [`7db6d6c`](https://github.com/open-webui/open-webui/commit/7db6d6cbfc71fc99e1338be16b23792e6260032b) Update chat.py ### 📊 Changes **3 files changed** (+98 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/main.py` (+14 -0) 📝 `backend/open_webui/socket/main.py` (+69 -0) 📝 `backend/open_webui/utils/chat.py` (+15 -7) </details> ### 📄 Description ## Problem When using direct connection mode with multiple backend workers (uvicorn workers > 1 or multiple k8s pods), streaming chat requests intermittently timeout and never complete. **Root Cause:** The generate_direct_chat_completion function dynamically registers a Socket.IO event handler using sio.on(channel, handler) for each streaming request. This handler only exists in the local worker's memory. When the browser's WebSocket connection lives on a different worker, streaming chunks arrive at that worker but no handler exists there - causing messages to be silently dropped and the original request to wait forever. ## Solution Replace per-request dynamic handler registration with a global static handler and Redis pub/sub routing: 1. **Global Queue Registry**: Add DIRECT_CHAT_QUEUES dictionary to track channel-to-queue mappings per worker 2. **Static Global Handler**: Add a sio.on("direct-chat-stream") handler that exists in all workers and routes messages to local queues or publishes via Redis 3. **Redis Pub/Sub Listener**: Each worker subscribes to direct chat channel patterns and delivers messages to local queues when present 4. **Frontend Update**: Changed socket.emit to use the global event name with channel metadata instead of dynamic channel names ## Flow After Fix 1. HTTP request arrives at Worker A, registers queue in DIRECT_CHAT_QUEUES 2. Browser emits streaming chunks to "direct-chat-stream" global event 3. Worker B (holding WebSocket) receives event via global handler 4. Worker B publishes to Redis since queue is not local 5. Worker A receives Redis message, finds local queue, delivers data 6. Streaming completes successfully ## Files Changed - backend/open_webui/socket/main.py - Added global handler and Redis pub/sub listener - backend/open_webui/utils/chat.py - Use queue registry instead of dynamic sio.on() - backend/open_webui/main.py - Start Redis listener on startup - src/routes/+layout.svelte - Emit to global event with channel metadata ## Testing - Deploy with WEBSOCKET_MANAGER=redis and multiple workers - Enable direct connection mode in user settings - Send concurrent streaming chat requests - Verify all requests complete without timeout Closes #15162 ### Contributor License Agreement By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-04-20 05:58:34 -05:00

GiteaMirror closed this issue

2026-04-20 05:58:35 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#25524