[GH-ISSUE #20510] feat: Knowledge Base taking hours to load files #34736

Closed
opened 2026-04-25 08:51:09 -05:00 by GiteaMirror · 0 comments
Owner

Originally created by @Code-With-Samuel on GitHub (Jan 9, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/20510

Check Existing Issues

  • I have searched for all existing open AND closed issues and discussions for similar requests. I have found none that is comparable to my request.

Verify Feature Scope

  • I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.

Problem Description

Image Image Image Image Image Image

This thing has been troubling me for so long. I'm working to build a RAG project to write Autodesk Maya's python commands and have around 5212 .md files after data acquisition. I'm trying to use it for my knowledge base. The total size of those 5212 files all together is just 20.3 MB to be precise. I'm working with the latest version of open Web UI and my general settings configuration for Documents are attached. Now the problem is:

  1. My GPU is NVIDIA GeForce GTX 1070 with dedicated GPU of 8 GB. I ran "Open WebUI with Nvidia GPU support" as a docker container i.e. : docker run -d -p 3000:8080 --gpus all --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda . Matter of fact, my GPU is not consumed at all while uploading the files to Knowledge section. Is my GPU only supposed to be used while generating response? Is this getting slow because my GPU is not getting consumed? Or am i missing something out? Please enlighten me.

  2. The more file I'm uploading to Knowledge base, the more slower it is processing. Is it an expected behaviour or I'm supposed to do something else to speed this up (like making the use of gpu, configure with a faster content extraction engine and embedding model)? So far, only 1291 files are uploaded successfully and I've got around 5000 files in total!! However the files aren't too big, almost all of them are in bytes. I've almost dedicated 10 hours already thats insane. I've got a deadline to finish this project quick so i want help on how can this be done faster.

Desired Solution you'd like

-> Get the gpu consumed if its supposed to be!
-> How can i make this operation faster?

Alternatives Considered

No response

Additional Context

No response

Originally created by @Code-With-Samuel on GitHub (Jan 9, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/20510 ### Check Existing Issues - [x] I have searched for all existing **open AND closed** issues and discussions for similar requests. I have found none that is comparable to my request. ### Verify Feature Scope - [x] I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions. ### Problem Description <img width="1854" height="903" alt="Image" src="https://github.com/user-attachments/assets/be86b1a4-0d71-42d2-813a-1749848c1e49" /> <img width="1691" height="870" alt="Image" src="https://github.com/user-attachments/assets/39710ff5-43f4-4f5e-8304-56e6c2bdf69f" /> <img width="1678" height="950" alt="Image" src="https://github.com/user-attachments/assets/b486ddca-8c66-4f83-b94e-8aa7707f9dab" /> <img width="1476" height="817" alt="Image" src="https://github.com/user-attachments/assets/3b4fb11a-06a4-4e12-87d3-6f24ec17e634" /> <img width="1436" height="132" alt="Image" src="https://github.com/user-attachments/assets/61b43051-3c08-4b25-9cff-874ac62588ab" /> <img width="1091" height="75" alt="Image" src="https://github.com/user-attachments/assets/41408bcc-b385-40db-b9aa-53a9abd580c4" /> This thing has been troubling me for so long. I'm working to build a RAG project to write Autodesk Maya's python commands and have around 5212 .md files after data acquisition. I'm trying to use it for my knowledge base. The total size of those 5212 files all together is just 20.3 MB to be precise. I'm working with the latest version of open Web UI and my general settings configuration for Documents are attached. Now the problem is: 1. My GPU is NVIDIA GeForce GTX 1070 with dedicated GPU of 8 GB. I ran "Open WebUI with Nvidia GPU support" as a docker container i.e. : docker run -d -p 3000:8080 --gpus all --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda . Matter of fact, my GPU is not consumed at all while uploading the files to Knowledge section. Is my GPU only supposed to be used while generating response? Is this getting slow because my GPU is not getting consumed? Or am i missing something out? Please enlighten me. 2. The more file I'm uploading to Knowledge base, the more slower it is processing. Is it an expected behaviour or I'm supposed to do something else to speed this up (like making the use of gpu, configure with a faster content extraction engine and embedding model)? So far, only 1291 files are uploaded successfully and I've got around 5000 files in total!! However the files aren't too big, almost all of them are in bytes. I've almost dedicated 10 hours already thats insane. I've got a deadline to finish this project quick so i want help on how can this be done faster. ### Desired Solution you'd like -> Get the gpu consumed if its supposed to be! -> How can i make this operation faster? ### Alternatives Considered _No response_ ### Additional Context _No response_
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#34736