I have searched the existing issues and discussions.
Problem Description
The current RAG (Retrieval-Augmented Generation) pipeline implementation in Open WebUI is significantly inefficient, even for straightforward, single-document use cases. The pipeline handles its core stages—document loading, splitting, embedding, and vector database insertion—in a fully synchronous series of steps. This sequential execution introduces substantial delays that can exceed typical web response timeouts or appear unresponsive to users.
Additionally, the synchronous design may introduce broader performance issues on the server side. I'm new to Python and FastAPI development, but I could see a long running API call being troublesome, especially in a single-worker environment (which is the Uvicorn default).
Desired Solution you'd like
I've prototyped a more efficient version of the RAG pipeline that introduces asynchronous and parallel processing across its four main stages: document loading, splitting, embedding, and vector database insertion. This redesign leverages Python’s async features and concurrency tools to eliminate unnecessary blocking, especially during I/O-heavy operations.
Through a series of targeted tests and time profiling in our production environment, I found that the embedding step consistently consumes over 93% of the total pipeline duration. This stage involves outbound requests to an external embedding model and is therefore primarily I/O-bound. Because of this, it presents a strong opportunity for performance optimization through asynchronous execution, parallelism, and batching. By overlapping requests and batching inputs where appropriate, the redesigned pipeline significantly reduces overall latency while making more efficient use of resources.
In its first iteration, this approach has already led to a 60%+ reduction in execution time—with further improvements likely. I'm continuing to tweak and refine the implementation and plan to submit a pull request soon. I’m fully open to feedback and eager to collaborate on integrating this into Open WebUI in a way that aligns with the project’s goals and standards.
Alternatives Considered
I initially explored using Open WebUI's built-in pipelines to introduce concurrency within the RAG flow. While this provided some degree of parallelism, it did not allow for true asynchronous execution across the full lifecycle of an API request. As a result, key bottlenecks—particularly in the I/O-bound embedding stage—remained largely unaddressed.
I also evaluated the Functions and Tools frameworks within Open WebUI, but these are not currently designed to fully replace or control the internal RAG pipeline.
Given these constraints, I pursued a source code change. The changes needed to support this effort require finer control over the pipeline stages in hopes to unlock meaningful performance gains.
Additional Context
We currently support approximately 300 users on our production deployment of Open WebUI, hosted in AWS. Until recently, the system was running with a single Uvicorn worker, which likely exacerbated the responsiveness issues described above. While we've since scaled up to four workers, the problem persists—indicating that the bottleneck is more architectural than purely resource-based.
This unresponsiveness in the RAG pipeline has become a significant barrier to adoption. Users are increasingly discouraged from relying on our self-hosted AI assistant because the latency creates the impression that the system is unreliable or broken. Improving the efficiency and concurrency of the pipeline is critical to enabling broader and more confident usage.
Originally created by @samuelbailor on GitHub (May 15, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/13936
### Check Existing Issues
- [x] I have searched the existing issues and discussions.
### Problem Description
The current RAG (Retrieval-Augmented Generation) pipeline implementation in Open WebUI is significantly inefficient, even for straightforward, single-document use cases. The pipeline handles its core stages—document loading, splitting, embedding, and vector database insertion—in a fully synchronous series of steps. This sequential execution introduces substantial delays that can exceed typical web response timeouts or appear unresponsive to users.
Additionally, the synchronous design may introduce broader performance issues on the server side. I'm new to Python and FastAPI development, but I could see a long running API call being troublesome, especially in a single-worker environment (which is the Uvicorn default).
### Desired Solution you'd like
I've prototyped a more efficient version of the RAG pipeline that introduces asynchronous and parallel processing across its four main stages: document loading, splitting, embedding, and vector database insertion. This redesign leverages Python’s async features and concurrency tools to eliminate unnecessary blocking, especially during I/O-heavy operations.
Through a series of targeted tests and time profiling in our production environment, I found that the embedding step consistently consumes over 93% of the total pipeline duration. This stage involves outbound requests to an external embedding model and is therefore primarily I/O-bound. Because of this, it presents a strong opportunity for performance optimization through asynchronous execution, parallelism, and batching. By overlapping requests and batching inputs where appropriate, the redesigned pipeline significantly reduces overall latency while making more efficient use of resources.
In its first iteration, this approach has already led to a 60%+ reduction in execution time—with further improvements likely. I'm continuing to tweak and refine the implementation and plan to submit a pull request soon. I’m fully open to feedback and eager to collaborate on integrating this into Open WebUI in a way that aligns with the project’s goals and standards.
### Alternatives Considered
I initially explored using Open WebUI's built-in pipelines to introduce concurrency within the RAG flow. While this provided some degree of parallelism, it did not allow for true asynchronous execution across the full lifecycle of an API request. As a result, key bottlenecks—particularly in the I/O-bound embedding stage—remained largely unaddressed.
I also evaluated the Functions and Tools frameworks within Open WebUI, but these are not currently designed to fully replace or control the internal RAG pipeline.
Given these constraints, I pursued a source code change. The changes needed to support this effort require finer control over the pipeline stages in hopes to unlock meaningful performance gains.
### Additional Context
We currently support approximately 300 users on our production deployment of Open WebUI, hosted in AWS. Until recently, the system was running with a single Uvicorn worker, which likely exacerbated the responsiveness issues described above. While we've since scaled up to four workers, the problem persists—indicating that the bottleneck is more architectural than purely resource-based.
This unresponsiveness in the RAG pipeline has become a significant barrier to adoption. Users are increasingly discouraged from relying on our self-hosted AI assistant because the latency creates the impression that the system is unreliable or broken. Improving the efficiency and concurrency of the pipeline is critical to enabling broader and more confident usage.
As part of this prototype, I also introduced a simple in-memory cache to store previously computed embeddings. If a document’s content has already been embedded, this cache allows the pipeline to skip the embedding step entirely, avoiding the longest part of the pipeline (which accounts for over 93% of total time). This approach has been extremely effective in practice, especially for repeated or similar queries.
That said, I recognize that an in-memory cache is a rudimentary solution and may not be ideal for all deployments—especially in horizontally scaled environments or long-running services. A more robust alternative (e.g., Redis or disk-based caching) may be preferable for production use, but I included the in-memory version to demonstrate the viability and impact of this optimization pattern.
<!-- gh-comment-id:2885160556 -->
@samuelbailor commented on GitHub (May 15, 2025):
As part of this prototype, I also introduced a simple in-memory cache to store previously computed embeddings. If a document’s content has already been embedded, this cache allows the pipeline to skip the embedding step entirely, avoiding the longest part of the pipeline (which accounts for over 93% of total time). This approach has been extremely effective in practice, especially for repeated or similar queries.
That said, I recognize that an in-memory cache is a rudimentary solution and may not be ideal for all deployments—especially in horizontally scaled environments or long-running services. A more robust alternative (e.g., Redis or disk-based caching) may be preferable for production use, but I included the in-memory version to demonstrate the viability and impact of this optimization pattern.
@samuelbailor This would be great. I notice when doing RAG with +10 files, it really slows down. Doesn't matter how fast my server is
<!-- gh-comment-id:2885615227 -->
@gaby commented on GitHub (May 16, 2025):
@samuelbailor This would be great. I notice when doing RAG with +10 files, it really slows down. Doesn't matter how fast my server is
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @samuelbailor on GitHub (May 15, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/13936
Check Existing Issues
Problem Description
The current RAG (Retrieval-Augmented Generation) pipeline implementation in Open WebUI is significantly inefficient, even for straightforward, single-document use cases. The pipeline handles its core stages—document loading, splitting, embedding, and vector database insertion—in a fully synchronous series of steps. This sequential execution introduces substantial delays that can exceed typical web response timeouts or appear unresponsive to users.
Additionally, the synchronous design may introduce broader performance issues on the server side. I'm new to Python and FastAPI development, but I could see a long running API call being troublesome, especially in a single-worker environment (which is the Uvicorn default).
Desired Solution you'd like
I've prototyped a more efficient version of the RAG pipeline that introduces asynchronous and parallel processing across its four main stages: document loading, splitting, embedding, and vector database insertion. This redesign leverages Python’s async features and concurrency tools to eliminate unnecessary blocking, especially during I/O-heavy operations.
Through a series of targeted tests and time profiling in our production environment, I found that the embedding step consistently consumes over 93% of the total pipeline duration. This stage involves outbound requests to an external embedding model and is therefore primarily I/O-bound. Because of this, it presents a strong opportunity for performance optimization through asynchronous execution, parallelism, and batching. By overlapping requests and batching inputs where appropriate, the redesigned pipeline significantly reduces overall latency while making more efficient use of resources.
In its first iteration, this approach has already led to a 60%+ reduction in execution time—with further improvements likely. I'm continuing to tweak and refine the implementation and plan to submit a pull request soon. I’m fully open to feedback and eager to collaborate on integrating this into Open WebUI in a way that aligns with the project’s goals and standards.
Alternatives Considered
I initially explored using Open WebUI's built-in pipelines to introduce concurrency within the RAG flow. While this provided some degree of parallelism, it did not allow for true asynchronous execution across the full lifecycle of an API request. As a result, key bottlenecks—particularly in the I/O-bound embedding stage—remained largely unaddressed.
I also evaluated the Functions and Tools frameworks within Open WebUI, but these are not currently designed to fully replace or control the internal RAG pipeline.
Given these constraints, I pursued a source code change. The changes needed to support this effort require finer control over the pipeline stages in hopes to unlock meaningful performance gains.
Additional Context
We currently support approximately 300 users on our production deployment of Open WebUI, hosted in AWS. Until recently, the system was running with a single Uvicorn worker, which likely exacerbated the responsiveness issues described above. While we've since scaled up to four workers, the problem persists—indicating that the bottleneck is more architectural than purely resource-based.
This unresponsiveness in the RAG pipeline has become a significant barrier to adoption. Users are increasingly discouraged from relying on our self-hosted AI assistant because the latency creates the impression that the system is unreliable or broken. Improving the efficiency and concurrency of the pipeline is critical to enabling broader and more confident usage.
@samuelbailor commented on GitHub (May 15, 2025):
As part of this prototype, I also introduced a simple in-memory cache to store previously computed embeddings. If a document’s content has already been embedded, this cache allows the pipeline to skip the embedding step entirely, avoiding the longest part of the pipeline (which accounts for over 93% of total time). This approach has been extremely effective in practice, especially for repeated or similar queries.
That said, I recognize that an in-memory cache is a rudimentary solution and may not be ideal for all deployments—especially in horizontally scaled environments or long-running services. A more robust alternative (e.g., Redis or disk-based caching) may be preferable for production use, but I included the in-memory version to demonstrate the viability and impact of this optimization pattern.
@gaby commented on GitHub (May 16, 2025):
@samuelbailor This would be great. I notice when doing RAG with +10 files, it really slows down. Doesn't matter how fast my server is