[PR #23573] [MERGED] fix: add health_check_interval to detect stale pooled connections #98296

New Issue

GiteaMirror · 2026-05-16T01:05:36-05:00

GiteaMirror commented

2026-05-16 01:05:36 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/23573
Author: @Classic298
Created: 4/10/2026
Status: ✅ Merged
Merged: 4/11/2026
Merged by: @tjbck

Base: dev ← Head: claude/redis-health-check-interval

📝 Commits (1)

853c901 fix(redis): add opt-in health_check_interval for stale pooled connections

📊 Changes

2 files changed (+27 additions, -0 deletions)

View changed files

📝 backend/open_webui/env.py (+14 -0)
📝 backend/open_webui/utils/redis.py (+13 -0)

📄 Description

Introduces REDIS_HEALTH_CHECK_INTERVAL and wires it through to every Redis client created by get_redis_connection (plain, cluster and sentinel paths, sync and async). When set to a positive integer, redis-py will PING any pooled connection idle longer than the interval on checkout, so dead sockets are surfaced as reconnectable errors before a real command lands on them.
Backward compatible / opt-in. Defaults to unset (empty string), which is parsed to None and skipped at the call site — so existing deployments see zero behavioural change after merging this PR. Operators who want the protection set REDIS_HEALTH_CHECK_INTERVAL in their environment (e.g. 60), and must pick a value shorter than both the Redis server timeout setting and any firewall/LB idle timeout on the path to Redis.
Why we need this
Our Redis server reaps stale connections via timeout 1800. Open WebUI doesn't currently check/ping pooled Redis connections before reusing them, so the first request that grabs a reaped socket out of the pool blows up with redis.exceptions.ConnectionError: Connection reset by peer, which bubbles up through AppConfig.getattr / the request middleware and surfaces as user-visible 500s and unexpected runtime errors.
Server-side reaping is non-negotiable for us — without it we run out of pool slots because Open WebUI doesn't appear to close Redis connections deterministically, which is why we originally raised maxclients from 1000 to 50000 and set timeout 1800 in the first place.
With this PR merged and REDIS_HEALTH_CHECK_INTERVAL=60 set, redis-py PINGs any idle connection before handing it back to a caller, which both (a) validates the socket and (b) resets Redis's idle-timeout counter for that connection. Connections still in active rotation stay warm indefinitely; orphaned/leaked ones still get reaped by the server timeout as a safety net. Two different failure modes, covered by two different mechanisms, no conflict between them.

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/23573 **Author:** [@Classic298](https://github.com/Classic298) **Created:** 4/10/2026 **Status:** ✅ Merged **Merged:** 4/11/2026 **Merged by:** [@tjbck](https://github.com/tjbck) **Base:** `dev` ← **Head:** `claude/redis-health-check-interval` --- ### 📝 Commits (1) - [`853c901`](https://github.com/open-webui/open-webui/commit/853c9010bc64c07ea5f6719966b66b5fcc63f4a1) fix(redis): add opt-in health_check_interval for stale pooled connections ### 📊 Changes **2 files changed** (+27 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/env.py` (+14 -0) 📝 `backend/open_webui/utils/redis.py` (+13 -0) </details> ### 📄 Description Introduces REDIS_HEALTH_CHECK_INTERVAL and wires it through to every Redis client created by get_redis_connection (plain, cluster and sentinel paths, sync and async). When set to a positive integer, redis-py will PING any pooled connection idle longer than the interval on checkout, so dead sockets are surfaced as reconnectable errors before a real command lands on them. Backward compatible / opt-in. Defaults to unset (empty string), which is parsed to None and skipped at the call site — so existing deployments see zero behavioural change after merging this PR. Operators who want the protection set REDIS_HEALTH_CHECK_INTERVAL in their environment (e.g. 60), and must pick a value shorter than both the Redis server timeout setting and any firewall/LB idle timeout on the path to Redis. Why we need this Our Redis server reaps stale connections via timeout 1800. Open WebUI doesn't currently check/ping pooled Redis connections before reusing them, so the first request that grabs a reaped socket out of the pool blows up with redis.exceptions.ConnectionError: Connection reset by peer, which bubbles up through AppConfig.__getattr__ / the request middleware and surfaces as user-visible 500s and unexpected runtime errors. Server-side reaping is non-negotiable for us — without it we run out of pool slots because Open WebUI doesn't appear to close Redis connections deterministically, which is why we originally raised maxclients from 1000 to 50000 and set timeout 1800 in the first place. With this PR merged and REDIS_HEALTH_CHECK_INTERVAL=60 set, redis-py PINGs any idle connection before handing it back to a caller, which both (a) validates the socket and (b) resets Redis's idle-timeout counter for that connection. Connections still in active rotation stay warm indefinitely; orphaned/leaked ones still get reaped by the server timeout as a safety net. Two different failure modes, covered by two different mechanisms, no conflict between them. ### Contributor License Agreement  - [X] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-05-16 01:05:37 -05:00

GiteaMirror closed this issue

2026-05-16 01:05:39 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#98296