[PR #20573] [CLOSED] fix(db): release connection before embedding in process_file #25678

New Issue

GiteaMirror · 2026-04-20T06:04:15-05:00

GiteaMirror commented

2026-04-20 06:04:15 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/20573
Author: @Classic298
Created: 1/11/2026
Status: ❌ Closed

Base: dev ← Head: fix/db-pool-process-file

📝 Commits (1)

1038bb0 fix(db): release connection before embedding in process_file

📊 Changes

1 file changed (+99 additions, -74 deletions)

View changed files

📝 backend/open_webui/routers/retrieval.py (+99 -74)

📄 Description

Implement granular session management in process_file() to prevent database connection pool exhaustion during slow embedding operations.

Previously, the database session was acquired at request start and held until the entire file processing completed, including external embedding API calls (5-60+ seconds). Under concurrent file processing, this exhausted the connection pool, causing QueuePool timeout errors for other operations.

Changes:

Extract primitive values (file_id, file_filename, etc.) before any session commits
Commit and release the session before save_docs_to_vector_db() which makes slow external embedding API calls
Use get_db() context manager for final status updates after embedding completes
Use get_db() context manager for error status updates in exception handler

This fix applies to 6 endpoints that rely on process_file():

POST /api/v1/files/ (file upload with processing)
POST /api/v1/files/{id}/data/content/update
POST /api/v1/knowledge/{id}/file/add
POST /api/v1/knowledge/{id}/file/update
POST /api/v1/knowledge/reindex
POST /api/v1/retrieval/process/file

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/20573 **Author:** [@Classic298](https://github.com/Classic298) **Created:** 1/11/2026 **Status:** ❌ Closed **Base:** `dev` ← **Head:** `fix/db-pool-process-file` --- ### 📝 Commits (1) - [`1038bb0`](https://github.com/open-webui/open-webui/commit/1038bb080a20bc506a849b49597544143fa44afb) fix(db): release connection before embedding in process_file ### 📊 Changes **1 file changed** (+99 additions, -74 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/routers/retrieval.py` (+99 -74) </details> ### 📄 Description Implement granular session management in process_file() to prevent database connection pool exhaustion during slow embedding operations. Previously, the database session was acquired at request start and held until the entire file processing completed, including external embedding API calls (5-60+ seconds). Under concurrent file processing, this exhausted the connection pool, causing QueuePool timeout errors for other operations. Changes: - Extract primitive values (file_id, file_filename, etc.) before any session commits - Commit and release the session before save_docs_to_vector_db() which makes slow external embedding API calls - Use get_db() context manager for final status updates after embedding completes - Use get_db() context manager for error status updates in exception handler This fix applies to 6 endpoints that rely on process_file(): - POST /api/v1/files/ (file upload with processing) - POST /api/v1/files/{id}/data/content/update - POST /api/v1/knowledge/{id}/file/add - POST /api/v1/knowledge/{id}/file/update - POST /api/v1/knowledge/reindex - POST /api/v1/retrieval/process/file ### Contributor License Agreement By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-04-20 06:04:15 -05:00

GiteaMirror closed this issue

2026-04-20 06:04:15 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#25678