I have searched the existing issues and discussions.
I am using the latest version of Open WebUI.
Installation Method
Kubernetes Containers
Open WebUI Version
v0.5.20
Ollama Version (if applicable)
No response
Operating System
CentOS 7
Browser (if applicable)
No response
Confirmation
I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have listed steps to reproduce the bug in detail.
Actual Behavior
I use the Milvus vector database in the backend to store RAG vector data.
When I upload a directory to the knowledge base, I can see a Collection in Milvus corresponding to the knowledge base ID (named open_webui_<knowledge_id>), as well as separate Collections for each file in the directory (named open_webui_file_<file_id>).
When I upload the same directory to the knowledge base a second time, I notice that no new data is added to the knowledge base's Collection. However, new Collections are recreated for the files in the directory because the file_id has changed.
After reviewing the Open WebUI source code, I found that when uploading documents to the knowledge base's Collection, a hash value is computed based on the text content. The system then checks for this hash in the knowledge base's Collection. If it exists, it means the document already exists and the content hasn't been updated, so an exception for duplicate uploads is thrown.
Since the file_id is regenerated using UUID each time a file is uploaded, the name of the file's corresponding Collection changes. As a result, the system can't find the existing data by hash value and Collection name, leading to the creation of a new Collection.
Replace generating file IDs using UUID with calculating the hash value of the file content as the file ID?
Originally created by @qiaozhi199 on GitHub (Apr 28, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/13266
### Check Existing Issues
- [x] I have searched the existing issues and discussions.
- [x] I am using the latest version of Open WebUI.
### Installation Method
Kubernetes Containers
### Open WebUI Version
v0.5.20
### Ollama Version (if applicable)
_No response_
### Operating System
CentOS 7
### Browser (if applicable)
_No response_
### Confirmation
- [x] I have read and followed all instructions in `README.md`.
- [x] I am using the latest version of **both** Open WebUI and Ollama.
- [x] I have included the browser console logs.
- [x] I have included the Docker container logs.
- [x] I have listed steps to reproduce the bug in detail.
### Actual Behavior
I use the Milvus vector database in the backend to store RAG vector data.
When I upload a directory to the knowledge base, I can see a Collection in Milvus corresponding to the knowledge base ID (named open_webui_<knowledge_id>), as well as separate Collections for each file in the directory (named open_webui_file_<file_id>).
When I upload the same directory to the knowledge base a second time, I notice that no new data is added to the knowledge base's Collection. However, new Collections are recreated for the files in the directory because the file_id has changed.
After reviewing the Open WebUI source code, I found that when uploading documents to the knowledge base's Collection, a hash value is computed based on the text content. The system then checks for this hash in the knowledge base's Collection. If it exists, it means the document already exists and the content hasn't been updated, so an exception for duplicate uploads is thrown.
Since the file_id is regenerated using UUID each time a file is uploaded, the name of the file's corresponding Collection changes. As a result, the system can't find the existing data by hash value and Collection name, leading to the creation of a new Collection.
Replace generating file IDs using UUID with calculating the hash value of the file content as the file ID?
### Logs & Screenshots
https://github.com/open-webui/open-webui/blob/main/backend/open_webui/routers/files.py#L94-L95
https://github.com/open-webui/open-webui/blob/main/backend/open_webui/routers/retrieval.py#L825-L836
### Additional Information
_No response_
GiteaMirror
added the bug label 2026-05-20 21:46:01 -05:00
@tjbck Doesn't this issue need to be fixed?
When I was uploading a directory containing thousands of documents, I found that some documents failed to upload, so I had to upload the directory again. However, I found that the Collection corresponding to a single document would be created again (named open_webui_file_<file_id>). As a result, there are many duplicate Collections in my Milvus vector database.
I think calculating the hash value of the document content as the file ID could solve this problem.
<!-- gh-comment-id:2837609271 -->
@qiaozhi199 commented on GitHub (Apr 29, 2025):
@tjbck Doesn't this issue need to be fixed?
When I was uploading a directory containing thousands of documents, I found that some documents failed to upload, so I had to upload the directory again. However, I found that the Collection corresponding to a single document would be created again (named open_webui_file_<file_id>). As a result, there are many duplicate Collections in my Milvus vector database.
I think calculating the hash value of the document content as the file ID could solve this problem.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @qiaozhi199 on GitHub (Apr 28, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/13266
Check Existing Issues
Installation Method
Kubernetes Containers
Open WebUI Version
v0.5.20
Ollama Version (if applicable)
No response
Operating System
CentOS 7
Browser (if applicable)
No response
Confirmation
README.md.Actual Behavior
I use the Milvus vector database in the backend to store RAG vector data.
When I upload a directory to the knowledge base, I can see a Collection in Milvus corresponding to the knowledge base ID (named open_webui_<knowledge_id>), as well as separate Collections for each file in the directory (named open_webui_file_<file_id>).
When I upload the same directory to the knowledge base a second time, I notice that no new data is added to the knowledge base's Collection. However, new Collections are recreated for the files in the directory because the file_id has changed.
After reviewing the Open WebUI source code, I found that when uploading documents to the knowledge base's Collection, a hash value is computed based on the text content. The system then checks for this hash in the knowledge base's Collection. If it exists, it means the document already exists and the content hasn't been updated, so an exception for duplicate uploads is thrown.
Since the file_id is regenerated using UUID each time a file is uploaded, the name of the file's corresponding Collection changes. As a result, the system can't find the existing data by hash value and Collection name, leading to the creation of a new Collection.
Replace generating file IDs using UUID with calculating the hash value of the file content as the file ID?
Logs & Screenshots
https://github.com/open-webui/open-webui/blob/main/backend/open_webui/routers/files.py#L94-L95
https://github.com/open-webui/open-webui/blob/main/backend/open_webui/routers/retrieval.py#L825-L836
Additional Information
No response
@qiaozhi199 commented on GitHub (Apr 29, 2025):
@tjbck Doesn't this issue need to be fixed?
When I was uploading a directory containing thousands of documents, I found that some documents failed to upload, so I had to upload the directory again. However, I found that the Collection corresponding to a single document would be created again (named open_webui_file_<file_id>). As a result, there are many duplicate Collections in my Milvus vector database.
I think calculating the hash value of the document content as the file ID could solve this problem.