[GH-ISSUE #17755] feat: Add batch upload endpoint and multi-threaded support for "Upload Directory" #33914

New Issue

GiteaMirror · 2026-04-25T07:47:26-05:00

GiteaMirror commented

2026-04-25 07:47:26 -05:00

Originally created by @shgana on GitHub (Sep 25, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/17755

Check Existing Issues

I have searched the existing issues and discussions.

Problem Description

Currently, when a user selects “Upload Directory” in the Knowledge Base UI, the files are uploaded sequentially (one POST /files call at a time).

This creates several issues:

Uploading many files is slow because the process is single-threaded.
Race conditions appear when trying to associate files with knowledge bases while background processing is still running.
Users expect directory uploads to behave like other modern systems (browser-based multi-stream uploads, e.g., Chrome/Edge).

In short: the existing “Upload Directory” looks like a batch feature but is effectively single-threaded, which limits usability at scale.

Desired Solution you'd like

Introduce a new batch upload endpoint (POST /files/batch) that accepts multiple files in one request and processes them concurrently.

Key details:

Concurrency: Implemented on the backend with configurable max workers.
Default behavior: MAX_CONCURRENT_UPLOADS = 1, so the system behaves exactly as it does today (sequential, single-threaded).
Opt-in performance: Admins can raise MAX_CONCURRENT_UPLOADS to 5 (or more) to enable parallelism.
Frontend change: Modify “Upload Directory” so that when multiple files are selected, it calls /files/batch instead of looping through /files N times.
Error handling: Response should include per-file success/failure, so partial uploads don’t block others.

This preserves backwards compatibility while offering major performance improvements for those who want them.

Alternatives Considered

Standalone tool

Already possible to write external tools that parallelize multiple POST /files requests
Downsides: no UI integration, separate maintenance burden, different stack.

Pipeline/OpenAPI Tool

Wrapping the existing API in another service.
Adds deployment/maintenance complexity, unintuitive for users.

Status quo (sequential uploads)

Simple and safe, but too slow for larger directories.
The upstream batch upload endpoint is the cleanest long-term solution: seamless for users, opt-in concurrency, and consistent with the existing codebase.

Additional Context

The new process_in_background flag in the upload API is very helpful: with process_in_background=false, files are guaranteed usable as soon as the request returns.

However, even with synchronous mode, the current “Upload Directory” UI still uploads one file at a time.

By combining process_in_background=false with a batch/multi-threaded upload endpoint, we can resolve both performance and race-condition issues.

Proposed implementation impact:

Backend: add /files/batch endpoint, configurable concurrency, per-file error reporting.
Frontend: adjust “Upload Directory” UI to use the new endpoint.
Compatibility: safe by default, no regressions.

Originally created by @shgana on GitHub (Sep 25, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/17755 ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description Currently, when a user selects “Upload Directory” in the Knowledge Base UI, the files are uploaded sequentially (one POST /files call at a time). This creates several issues: 1. Uploading many files is slow because the process is single-threaded. 2. Race conditions appear when trying to associate files with knowledge bases while background processing is still running. 3. Users expect directory uploads to behave like other modern systems (browser-based multi-stream uploads, e.g., Chrome/Edge). In short: the existing “Upload Directory” looks like a batch feature but is effectively single-threaded, which limits usability at scale. ### Desired Solution you'd like Introduce a new batch upload endpoint (POST /files/batch) that accepts multiple files in one request and processes them concurrently. Key details: 1. Concurrency: Implemented on the backend with configurable max workers. 2. Default behavior: MAX_CONCURRENT_UPLOADS = 1, so the system behaves exactly as it does today (sequential, single-threaded). 3. Opt-in performance: Admins can raise MAX_CONCURRENT_UPLOADS to 5 (or more) to enable parallelism. 4. Frontend change: Modify “Upload Directory” so that when multiple files are selected, it calls /files/batch instead of looping through /files N times. 5. Error handling: Response should include per-file success/failure, so partial uploads don’t block others. 6. This preserves backwards compatibility while offering major performance improvements for those who want them. ### Alternatives Considered 1. Standalone tool Already possible to write external tools that parallelize multiple POST /files requests Downsides: no UI integration, separate maintenance burden, different stack. 2. Pipeline/OpenAPI Tool Wrapping the existing API in another service. Adds deployment/maintenance complexity, unintuitive for users. 3. Status quo (sequential uploads) Simple and safe, but too slow for larger directories. The upstream batch upload endpoint is the cleanest long-term solution: seamless for users, opt-in concurrency, and consistent with the existing codebase. ### Additional Context The new process_in_background flag in the upload API is very helpful: with process_in_background=false, files are guaranteed usable as soon as the request returns. However, even with synchronous mode, the current “Upload Directory” UI still uploads one file at a time. By combining process_in_background=false with a batch/multi-threaded upload endpoint, we can resolve both performance and race-condition issues. Proposed implementation impact: Backend: add /files/batch endpoint, configurable concurrency, per-file error reporting. Frontend: adjust “Upload Directory” UI to use the new endpoint. Compatibility: safe by default, no regressions.

GiteaMirror closed this issue

2026-04-25 07:47:27 -05:00

GiteaMirror commented

2026-04-25 07:47:28 -05:00

@tjbck commented on GitHub (Sep 25, 2025):

Open to PRs!

@tjbck commented on GitHub (Sep 25, 2025): Open to PRs!

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#33914