[PR #22839] [CLOSED] fix: make RedisDict.set() atomic to prevent intermittent model lookup failures #130543

New Issue

GiteaMirror · 2026-05-21T14:50:06-05:00

GiteaMirror commented

2026-05-21 14:50:06 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/22839
Author: @themavik
Created: 3/19/2026
Status: ❌ Closed

Base: main ← Head: fix/22734-redisdict-atomic-set

📝 Commits (1)

6eecb6f fix: make RedisDict.set() atomic to prevent intermittent model lookup failures

📊 Changes

1 file changed (+10 additions, -6 deletions)

View changed files

📝 backend/open_webui/socket/utils.py (+10 -6)

📄 Description

Summary

Fixes #22734.

Root cause: RedisDict.set() used DELETE followed by HSET in a non-transactional pipeline. While Redis pipelines batch commands for network efficiency, they do not guarantee atomicity — a concurrent HGET/HGETALL between the DELETE and HSET sees an empty hash, causing KeyError / "Model not found" errors in generate_chat_completion and generate_follow_ups under high concurrency (multiple Kubernetes replicas).

Fix: Replace the destructive DELETE + HSET pattern with a diff-based approach:

Read existing keys from the hash
Compute which keys were removed
Use a transactional pipeline (MULTI/EXEC) to HDEL only removed keys and HSET new/changed keys

This ensures the hash is never empty during updates. Concurrent readers always see either the old state or the new state, never an empty intermediate state.

Changes

backend/open_webui/socket/utils.py: Rewrote RedisDict.set() to use diff-based updates with a transactional pipeline instead of DELETE + HSET.

Testing

Verified fix addresses the reported race condition scenario
Change preserves the same external behavior (full replacement of hash contents)
Transactional pipeline (MULTI/EXEC) guarantees atomicity of the HDEL + HSET operations
No behavioral change for single-replica deployments

Made with Cursor

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/22839 **Author:** [@themavik](https://github.com/themavik) **Created:** 3/19/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/22734-redisdict-atomic-set` --- ### 📝 Commits (1) - [`6eecb6f`](https://github.com/open-webui/open-webui/commit/6eecb6f87e0b6740788b2e34255fd5f830c175bb) fix: make RedisDict.set() atomic to prevent intermittent model lookup failures ### 📊 Changes **1 file changed** (+10 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/socket/utils.py` (+10 -6) </details> ### 📄 Description ## Summary Fixes #22734. **Root cause:** `RedisDict.set()` used `DELETE` followed by `HSET` in a non-transactional pipeline. While Redis pipelines batch commands for network efficiency, they do not guarantee atomicity — a concurrent `HGET`/`HGETALL` between the `DELETE` and `HSET` sees an empty hash, causing `KeyError` / "Model not found" errors in `generate_chat_completion` and `generate_follow_ups` under high concurrency (multiple Kubernetes replicas). **Fix:** Replace the destructive `DELETE` + `HSET` pattern with a diff-based approach: 1. Read existing keys from the hash 2. Compute which keys were removed 3. Use a transactional pipeline (`MULTI`/`EXEC`) to `HDEL` only removed keys and `HSET` new/changed keys This ensures the hash is never empty during updates. Concurrent readers always see either the old state or the new state, never an empty intermediate state. ## Changes - `backend/open_webui/socket/utils.py`: Rewrote `RedisDict.set()` to use diff-based updates with a transactional pipeline instead of `DELETE` + `HSET`. ## Testing - Verified fix addresses the reported race condition scenario - Change preserves the same external behavior (full replacement of hash contents) - Transactional pipeline (`MULTI`/`EXEC`) guarantees atomicity of the `HDEL` + `HSET` operations - No behavioral change for single-replica deployments Made with [Cursor](https://cursor.com) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-05-21 14:50:06 -05:00

GiteaMirror closed this issue

2026-05-21 14:50:12 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#130543