github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-03-10 07:43:10 -05:00
Code Issues 308 Packages Projects Releases 100 Wiki Activity

feat: Why are files uploaded to Knowledge Base processed one by one, even during bulk uploads? #5958

New Issue
Closed
opened 2025-11-11 16:40:09 -06:00 by GiteaMirror · 1 comment
GiteaMirror commented 2025-11-11 16:40:09 -06:00
Owner
Copy Link

Originally created by @wolfplay2K19 on GitHub (Aug 4, 2025).

Check Existing Issues

  • I have searched the existing issues and discussions.

Problem Description

I’m currently uploading a large number of documents into the Knowledge Base using a local embedding model. However, I noticed that even when I upload multiple files at once, they are still processed sequentially — one file at a time — which results in a very slow overall import time.

Could you please clarify:

  1. Is there currently a way to parallelize file processing or embedding in the Knowledge Base?
  2. Does the API endpoint /knowledge/{id}/files/batch/add actually process files in parallel, or does it just enqueue them for sequential embedding?
  3. Are there any settings I can adjust (e.g., batch size, concurrency limits) to speed this up when using a local embedding model?
  4. Would switching to an external embedding provider (e.g., OpenAI, Cohere) improve performance for bulk file ingestion?

Any insights or recommendations for optimizing this workflow would be greatly appreciated.

Desired Solution you'd like

Ideally, I would like the Knowledge Base to support true parallel processing of multiple files during upload — especially when using a local embedding model. This could mean:

Allowing multiple documents to be embedded concurrently;

Increasing throughput by batching chunks across files instead of per file;

Providing configuration options (e.g., concurrency limits, batch size) to control parallelism;

Or even a background queue system that handles embedding jobs in parallel threads or workers.

This would significantly reduce total upload time and make bulk document ingestion more scalable.

Alternatives Considered

No response

Additional Context

No response

Originally created by @wolfplay2K19 on GitHub (Aug 4, 2025). ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description I’m currently uploading a large number of documents into the Knowledge Base using a local embedding model. However, I noticed that even when I upload multiple files at once, they are still processed sequentially — one file at a time — which results in a very slow overall import time. Could you please clarify: 1. Is there currently a way to parallelize file processing or embedding in the Knowledge Base? 2. Does the API endpoint /knowledge/{id}/files/batch/add actually process files in parallel, or does it just enqueue them for sequential embedding? 3. Are there any settings I can adjust (e.g., batch size, concurrency limits) to speed this up when using a local embedding model? 4. Would switching to an external embedding provider (e.g., OpenAI, Cohere) improve performance for bulk file ingestion? Any insights or recommendations for optimizing this workflow would be greatly appreciated. ### Desired Solution you'd like Ideally, I would like the Knowledge Base to support true parallel processing of multiple files during upload — especially when using a local embedding model. This could mean: Allowing multiple documents to be embedded concurrently; Increasing throughput by batching chunks across files instead of per file; Providing configuration options (e.g., concurrency limits, batch size) to control parallelism; Or even a background queue system that handles embedding jobs in parallel threads or workers. This would significantly reduce total upload time and make bulk document ingestion more scalable. ### Alternatives Considered _No response_ ### Additional Context _No response_
GiteaMirror commented 2025-11-11 16:40:10 -06:00
Author
Owner
Copy Link

@tjbck commented on GitHub (Aug 4, 2025):

#15023

@tjbck commented on GitHub (Aug 4, 2025): #15023
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#5958
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?