[GH-ISSUE #24566] feat: Improve Knowledge file processing reliability for slow external parsers: bounded concurrency and durable queue #107332

New Issue

GiteaMirror · 2026-05-18T06:07:34-05:00

GiteaMirror commented

2026-05-18 06:07:34 -05:00

Originally created by @KingsleyOWO on GitHub (May 11, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/24566

Check Existing Issues

I have searched for all existing open AND closed issues and discussions for similar requests. I have found none that is comparable to my request.

Verify Feature Scope

I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.

Problem Description

OpenWebUI currently relies on FastAPI BackgroundTasks for Knowledge file parsing and ingestion. The upload request can return before parsing is actually finished, and the file lifecycle is tracked mainly through file.data.status rather than a durable processing job.

This becomes problematic when OpenWebUI is configured with a slow external document parser such as MinerU, Tika, or Docling.

In multi-file Knowledge uploads, several files can be processed at the same time. Each background task can call the external parser concurrently, but there is no durable queue, retry counter, worker ID, lease, or configurable concurrency limit on the OpenWebUI side.

This can lead to:

external parser overload
transient timeout / connection reset failures
files being marked as failed even when a retry may succeed
files staying pending after an OpenWebUI restart
Knowledge files being linked while parsing is not fully completed
limited admin visibility into queued/running/failed file processing work

This is not specific to one deployment. It can be reproduced with a mock external parser that sleeps for several seconds per request and then uploading multiple files quickly.

The core issue is that file processing is currently in-process and non-durable, while external document parsers may be slow or resource-constrained.

Desired Solution you'd like

I would like OpenWebUI to support safer file processing for slow external parsers.

A minimal first step could be a configurable concurrency limit around RAG file processing / external parser calls, for example:

RAG_FILE_PROCESSING_MAX_CONCURRENCY=1

### Alternatives Considered

1. Add a reverse proxy in front of the external parser

This can limit concurrency, but it only protects the parser. It does not solve OpenWebUI-side durability, stuck pending files, retry tracking, or Knowledge linking race conditions.

2. Make the external parser more robust

This is useful, but OpenWebUI still needs backpressure when calling slow external parsers. Not every parser service will implement its own queue correctly.

3. Increase parser timeout

This helps large files but does not solve concurrent fan-out. If many files are uploaded at once, the parser can still be overloaded.

4. Implement a full persistent queue immediately

This is likely the most robust long-term design, but it is a large change involving database schema, worker lifecycle, status handling, retry/reaper logic, and UI/admin visibility. A smaller concurrency-limit PR may be easier to review and merge first.

### Additional Context

A local reproducer can be built with a small FastAPI mock parser:

```python
from fastapi import FastAPI, UploadFile, File
import asyncio
import os

app = FastAPI()
ACTIVE = 0
MAX_ACTIVE_SEEN = 0

@app.get("/health")
async def health():
    return {"ok": True, "active": ACTIVE, "max_active_seen": MAX_ACTIVE_SEEN}

@app.post("/file_parse")
async def file_parse(files: UploadFile = File(...)):
    global ACTIVE, MAX_ACTIVE_SEEN
    ACTIVE += 1
    MAX_ACTIVE_SEEN = max(MAX_ACTIVE_SEEN, ACTIVE)
    try:
        await asyncio.sleep(float(os.getenv("MOCK_PARSE_SECONDS", "30")))
        return {"md_content": f"# Parsed {files.filename}\n\nmock content"}
    finally:
        ACTIVE -= 1

Originally created by @KingsleyOWO on GitHub (May 11, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/24566 ### Check Existing Issues - [x] I have searched for all existing **open AND closed** issues and discussions for similar requests. I have found none that is comparable to my request. ### Verify Feature Scope - [x] I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions. ### Problem Description OpenWebUI currently relies on FastAPI `BackgroundTasks` for Knowledge file parsing and ingestion. The upload request can return before parsing is actually finished, and the file lifecycle is tracked mainly through `file.data.status` rather than a durable processing job. This becomes problematic when OpenWebUI is configured with a slow external document parser such as MinerU, Tika, or Docling. In multi-file Knowledge uploads, several files can be processed at the same time. Each background task can call the external parser concurrently, but there is no durable queue, retry counter, worker ID, lease, or configurable concurrency limit on the OpenWebUI side. This can lead to: - external parser overload - transient timeout / connection reset failures - files being marked as failed even when a retry may succeed - files staying pending after an OpenWebUI restart - Knowledge files being linked while parsing is not fully completed - limited admin visibility into queued/running/failed file processing work This is not specific to one deployment. It can be reproduced with a mock external parser that sleeps for several seconds per request and then uploading multiple files quickly. The core issue is that file processing is currently in-process and non-durable, while external document parsers may be slow or resource-constrained. ### Desired Solution you'd like I would like OpenWebUI to support safer file processing for slow external parsers. A minimal first step could be a configurable concurrency limit around RAG file processing / external parser calls, for example: ```env RAG_FILE_PROCESSING_MAX_CONCURRENCY=1 ### Alternatives Considered 1. Add a reverse proxy in front of the external parser This can limit concurrency, but it only protects the parser. It does not solve OpenWebUI-side durability, stuck pending files, retry tracking, or Knowledge linking race conditions. 2. Make the external parser more robust This is useful, but OpenWebUI still needs backpressure when calling slow external parsers. Not every parser service will implement its own queue correctly. 3. Increase parser timeout This helps large files but does not solve concurrent fan-out. If many files are uploaded at once, the parser can still be overloaded. 4. Implement a full persistent queue immediately This is likely the most robust long-term design, but it is a large change involving database schema, worker lifecycle, status handling, retry/reaper logic, and UI/admin visibility. A smaller concurrency-limit PR may be easier to review and merge first. ### Additional Context A local reproducer can be built with a small FastAPI mock parser: ```python from fastapi import FastAPI, UploadFile, File import asyncio import os app = FastAPI() ACTIVE = 0 MAX_ACTIVE_SEEN = 0 @app.get("/health") async def health(): return {"ok": True, "active": ACTIVE, "max_active_seen": MAX_ACTIVE_SEEN} @app.post("/file_parse") async def file_parse(files: UploadFile = File(...)): global ACTIVE, MAX_ACTIVE_SEEN ACTIVE += 1 MAX_ACTIVE_SEEN = max(MAX_ACTIVE_SEEN, ACTIVE) try: await asyncio.sleep(float(os.getenv("MOCK_PARSE_SECONDS", "30"))) return {"md_content": f"# Parsed {files.filename}\n\nmock content"} finally: ACTIVE -= 1

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#107332