[PR #20542] [MERGED] fix: use efficient COUNT queries in telemetry metrics to prevent connection pool exhaustion #64519

New Issue

GiteaMirror · 2026-05-06T10:07:43-05:00

GiteaMirror commented

2026-05-06 10:07:43 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/20542
Author: @Classic298
Created: 1/10/2026
Status: ✅ Merged
Merged: 1/10/2026
Merged by: @tjbck

Base: dev ← Head: db

📝 Commits (1)

e2aa65d fix: use efficient COUNT queries in telemetry metrics to prevent connection pool exhaustion

📊 Changes

1 file changed (+4 additions, -1 deletions)

View changed files

📝 backend/open_webui/utils/telemetry/metrics.py (+4 -1)

📄 Description

fix: use efficient COUNT queries in telemetry metrics to prevent connection pool exhaustion

This fixes database connection pool exhaustion issues reported after v0.7.0, particularly affecting PostgreSQL deployments on high-latency networks (e.g., AWS Aurora).

The Problem

The telemetry metrics callbacks (running every 10 seconds via OpenTelemetry's PeriodicExportingMetricReader) were using inefficient queries that loaded entire database tables into memory just to count records:

len(Users.get_users()["users"])  # Loads ALL user records to count them

On high-latency network-attached databases like AWS Aurora, this would:

Hold database connections for hundreds of milliseconds while transferring data
Deserialize all records into Python objects
Only then count the list length

Under concurrent load, these long-held connections would stack up and drain the connection pool, resulting in:

sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached,
connection timed out, timeout 30.00

The Fix

Replace inefficient full-table loads with efficient COUNT(*) queries using methods that already exist in the codebase:

len(Users.get_users()["users"]) → Users.get_num_users()
Similar changes for other telemetry callbacks as needed

COUNT(*) queries use database indexes and return a single integer, completing in ~5-10ms even on Aurora, versus potentially 500ms+ for loading all records.

The v0.7.1 change to disable DATABASE_ENABLE_SESSION_SHARING by default appeared to fix the issue, but it was masking the root cause. Disabling session sharing causes connections to be returned to the pool faster (more connection churn), which reduced the window for pool exhaustion but didn't address the underlying inefficient queries.

With this fix, session sharing can be safely re-enabled for deployments that benefit from it (especially PostgreSQL), as telemetry will no longer hold connections for extended periods.

Impact

Telemetry connection usage drops from potentially seconds to ~30ms total per collection cycle
Connection pool pressure from telemetry becomes negligible (~0.3% utilization)
Enterprise PostgreSQL deployments (Aurora, RDS, etc.) should no longer experience pool exhaustion under normal load

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/20542 **Author:** [@Classic298](https://github.com/Classic298) **Created:** 1/10/2026 **Status:** ✅ Merged **Merged:** 1/10/2026 **Merged by:** [@tjbck](https://github.com/tjbck) **Base:** `dev` ← **Head:** `db` --- ### 📝 Commits (1) - [`e2aa65d`](https://github.com/open-webui/open-webui/commit/e2aa65dcde1949e1cd545eb82c0c4c211b4c3a55) fix: use efficient COUNT queries in telemetry metrics to prevent connection pool exhaustion ### 📊 Changes **1 file changed** (+4 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/utils/telemetry/metrics.py` (+4 -1) </details> ### 📄 Description fix: use efficient COUNT queries in telemetry metrics to prevent connection pool exhaustion This fixes database connection pool exhaustion issues reported after v0.7.0, particularly affecting PostgreSQL deployments on high-latency networks (e.g., AWS Aurora). ## The Problem The telemetry metrics callbacks (running every 10 seconds via OpenTelemetry's PeriodicExportingMetricReader) were using inefficient queries that loaded entire database tables into memory just to count records: len(Users.get_users()["users"]) # Loads ALL user records to count them On high-latency network-attached databases like AWS Aurora, this would: 1. Hold database connections for hundreds of milliseconds while transferring data 2. Deserialize all records into Python objects 3. Only then count the list length Under concurrent load, these long-held connections would stack up and drain the connection pool, resulting in: sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 ## The Fix Replace inefficient full-table loads with efficient COUNT(*) queries using methods that already exist in the codebase: - `len(Users.get_users()["users"])` → `Users.get_num_users()` - Similar changes for other telemetry callbacks as needed COUNT(*) queries use database indexes and return a single integer, completing in ~5-10ms even on Aurora, versus potentially 500ms+ for loading all records. ## Why v0.7.1's Session Sharing Disable "Helped" The v0.7.1 change to disable DATABASE_ENABLE_SESSION_SHARING by default appeared to fix the issue, but it was masking the root cause. Disabling session sharing causes connections to be returned to the pool faster (more connection churn), which reduced the window for pool exhaustion but didn't address the underlying inefficient queries. With this fix, session sharing can be safely re-enabled for deployments that benefit from it (especially PostgreSQL), as telemetry will no longer hold connections for extended periods. ## Impact - Telemetry connection usage drops from potentially seconds to ~30ms total per collection cycle - Connection pool pressure from telemetry becomes negligible (~0.3% utilization) - Enterprise PostgreSQL deployments (Aurora, RDS, etc.) should no longer experience pool exhaustion under normal load ### Contributor License Agreement By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-05-06 10:07:43 -05:00

GiteaMirror closed this issue

2026-05-06 10:07:45 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#64519