feat: Why are files uploaded to Knowledge Base processed one by one, even during bulk uploads? #5958

New Issue

2025-11-11T16:40:09-06:00

GiteaMirror commented

2025-11-11 16:40:09 -06:00

Originally created by @wolfplay2K19 on GitHub (Aug 4, 2025).

Check Existing Issues

I have searched the existing issues and discussions.

Problem Description

I’m currently uploading a large number of documents into the Knowledge Base using a local embedding model. However, I noticed that even when I upload multiple files at once, they are still processed sequentially — one file at a time — which results in a very slow overall import time.

Could you please clarify:

Is there currently a way to parallelize file processing or embedding in the Knowledge Base?
Does the API endpoint /knowledge/{id}/files/batch/add actually process files in parallel, or does it just enqueue them for sequential embedding?
Are there any settings I can adjust (e.g., batch size, concurrency limits) to speed this up when using a local embedding model?
Would switching to an external embedding provider (e.g., OpenAI, Cohere) improve performance for bulk file ingestion?

Any insights or recommendations for optimizing this workflow would be greatly appreciated.

Desired Solution you'd like

Ideally, I would like the Knowledge Base to support true parallel processing of multiple files during upload — especially when using a local embedding model. This could mean:

Allowing multiple documents to be embedded concurrently;

Increasing throughput by batching chunks across files instead of per file;

Providing configuration options (e.g., concurrency limits, batch size) to control parallelism;

Or even a background queue system that handles embedding jobs in parallel threads or workers.

This would significantly reduce total upload time and make bulk document ingestion more scalable.

Alternatives Considered

No response

Additional Context

No response

Originally created by @wolfplay2K19 on GitHub (Aug 4, 2025). ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description I’m currently uploading a large number of documents into the Knowledge Base using a local embedding model. However, I noticed that even when I upload multiple files at once, they are still processed sequentially — one file at a time — which results in a very slow overall import time. Could you please clarify: 1. Is there currently a way to parallelize file processing or embedding in the Knowledge Base? 2. Does the API endpoint /knowledge/{id}/files/batch/add actually process files in parallel, or does it just enqueue them for sequential embedding? 3. Are there any settings I can adjust (e.g., batch size, concurrency limits) to speed this up when using a local embedding model? 4. Would switching to an external embedding provider (e.g., OpenAI, Cohere) improve performance for bulk file ingestion? Any insights or recommendations for optimizing this workflow would be greatly appreciated. ### Desired Solution you'd like Ideally, I would like the Knowledge Base to support true parallel processing of multiple files during upload — especially when using a local embedding model. This could mean: Allowing multiple documents to be embedded concurrently; Increasing throughput by batching chunks across files instead of per file; Providing configuration options (e.g., concurrency limits, batch size) to control parallelism; Or even a background queue system that handles embedding jobs in parallel threads or workers. This would significantly reduce total upload time and make bulk document ingestion more scalable. ### Alternatives Considered _No response_ ### Additional Context _No response_

GiteaMirror closed this issue

2025-11-11 16:40:09 -06:00

GiteaMirror commented

2025-11-11 16:40:10 -06:00

@tjbck commented on GitHub (Aug 4, 2025):

#15023

@tjbck commented on GitHub (Aug 4, 2025): #15023