[GH-ISSUE #20853] [BUG] RAG reindexing fails with "Duplicate content detected" - cannot update existing documents #122625

New Issue

GiteaMirror · 2026-05-21T01:31:54-05:00

GiteaMirror commented

2026-05-21 01:31:54 -05:00

Originally created by @dimzon on GitHub (Jan 21, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/20853

## Description
The "Reindex" functionality in Knowledge Bases is completely broken due to overly aggressive duplicate content detection. When attempting to reindex documents or upload the same document again (even with different chunking settings), the system throws an error and refuses to proceed.

## Steps to Reproduce

Upload a document to a knowledge base
Click "Reindex" button for that knowledge base
Observe the error: ValueError: Duplicate content detected. Please provide unique content to proceed.
Try to upload the same document again
Get the same error

## Expected Behavior

Reindexing should update existing embeddings
Uploading the same document should either update or skip with a notification
Users should be able to force reindexing/updating when needed
System should provide options: "Update existing", "Skip", or "Replace"

## Current Behavior

System crashes with an error when detecting duplicate content
No way to update documents except deleting the entire knowledge base
No user-friendly options for handling duplicates
Basic document management workflow is broken

## Error Logs

2026-01-22 01:30:58.476 | ERROR | open_webui.routers.knowledge:reindex_knowledge_files:317 - Error processing file document.docx: 400: Duplicate content detected. Please provide unique content to proceed.
2026-01-22 01:30:58.518 | INFO | open_webui.routers.retrieval:save_docs_to_vector_db:1420 - Document with hash b3fb86702655a887c4bf9b807484bdf02bd7a24e0124e84491c331dd3c3598ab already exists

## Root Cause
The issue appears to be in open_webui/routers/retrieval.py around line 1420 where duplicate detection raises an exception instead of providing update options:

# Current problematic code:
if existing_hash:
    raise ValueError(ERROR_MESSAGES.DUPLICATE_CONTENT)  # This breaks reindexing

## Impact

Cannot update documents - When documents are revised, users cannot reindex them
Cannot test chunking settings - Cannot experiment with different chunk sizes/overlaps
Broken workflow - Basic document management operations are impossible
Forces workarounds - Users must rename files or delete entire knowledge bases

## Critical Design Flaw: No Logical Reason to Ban Identical Content

There is no logical reason to prohibit identical content fragments across different documents:

Normal document scenarios:
- Multiple documents may have identical legal clauses
- Technical documentation often repeats standard procedures
- Training materials contain repeated examples
- API documentation has identical parameter descriptions across endpoints
RAG systems are designed to handle this:
- Vector search returns multiple results with similar content
- LLMs can synthesize information from multiple sources
- Metadata (source document, position) distinguishes identical fragments
Current implementation harms usability:
- Prevents document versioning
- Blocks document updates
- Creates artificial constraints
- Goes against real-world document management needs

## Suggested Fix

For reindexing: Always allow reindexing of existing documents
For new uploads: Provide duplicate handling options:
- Update existing
- Skip
- Replace
Add UI controls: Checkbox options like "Update existing documents" or "Force reindex"
Remove duplicate content prohibition: Allow identical content with proper metadata tracking

Proposed code change:

# In save_docs_to_vector_db function
if existing_hash:
    if is_reindexing or force_update:
        # Delete existing chunks and re-add
        delete_chunks_by_doc_hash(existing_hash)
        logger.info(f"Updated existing document: {existing_hash}")
    else:
        # Return a proper response instead of throwing error
        return {
            "status": "duplicate",
            "message": "Document already exists",
            "options": ["update", "skip"]
        }

## Workarounds Users Are Forced to Use

Rename all files (add _v1, _v2 suffixes)
Delete entire knowledge base and recreate
Manually edit files to change content hash
Use external scripts to bypass the check

## Environment

Open WebUI version: Latest (2026-01-22)
OS: All (Windows, Linux confirmed)
Installation: Docker and native

## Additional Context
This bug completely breaks the intended use case where documents need periodic updates or reindexing with new settings. The current implementation treats all duplicates as errors rather than providing management options, which goes against standard document management practices in other RAG systems (ChromaDB, Pinecone, etc. all support upsert operations).

Most importantly, there's no valid reason to prohibit identical content across documents. RAG systems should handle this naturally through metadata and search ranking, not by blocking document management operations.

## Priority
This is a BLOCKER bug for any serious use of the Knowledge Base feature, as it prevents basic document maintenance and optimization.

Originally created by @dimzon on GitHub (Jan 21, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/20853 **## Description** The "Reindex" functionality in Knowledge Bases is completely broken due to overly aggressive duplicate content detection. When attempting to reindex documents or upload the same document again (even with different chunking settings), the system throws an error and refuses to proceed. **## Steps to Reproduce** 1. Upload a document to a knowledge base 2. Click "Reindex" button for that knowledge base 3. Observe the error: `ValueError: Duplicate content detected. Please provide unique content to proceed.` 4. Try to upload the same document again 5. Get the same error **## Expected Behavior** - Reindexing should update existing embeddings - Uploading the same document should either update or skip with a notification - Users should be able to force reindexing/updating when needed - System should provide options: "Update existing", "Skip", or "Replace" **## Current Behavior** - System crashes with an error when detecting duplicate content - No way to update documents except deleting the entire knowledge base - No user-friendly options for handling duplicates - Basic document management workflow is broken **## Error Logs** ``` 2026-01-22 01:30:58.476 | ERROR | open_webui.routers.knowledge:reindex_knowledge_files:317 - Error processing file document.docx: 400: Duplicate content detected. Please provide unique content to proceed. 2026-01-22 01:30:58.518 | INFO | open_webui.routers.retrieval:save_docs_to_vector_db:1420 - Document with hash b3fb86702655a887c4bf9b807484bdf02bd7a24e0124e84491c331dd3c3598ab already exists ``` **## Root Cause** The issue appears to be in `open_webui/routers/retrieval.py` around line 1420 where duplicate detection raises an exception instead of providing update options: ```python # Current problematic code: if existing_hash: raise ValueError(ERROR_MESSAGES.DUPLICATE_CONTENT) # This breaks reindexing ``` **## Impact** 1. **Cannot update documents** - When documents are revised, users cannot reindex them 2. **Cannot test chunking settings** - Cannot experiment with different chunk sizes/overlaps 3. **Broken workflow** - Basic document management operations are impossible 4. **Forces workarounds** - Users must rename files or delete entire knowledge bases **## Critical Design Flaw: No Logical Reason to Ban Identical Content** There is **no logical reason** to prohibit identical content fragments across different documents: 1. **Normal document scenarios:** - Multiple documents may have identical legal clauses - Technical documentation often repeats standard procedures - Training materials contain repeated examples - API documentation has identical parameter descriptions across endpoints 2. **RAG systems are designed to handle this:** - Vector search returns multiple results with similar content - LLMs can synthesize information from multiple sources - Metadata (source document, position) distinguishes identical fragments 3. **Current implementation harms usability:** - Prevents document versioning - Blocks document updates - Creates artificial constraints - Goes against real-world document management needs **## Suggested Fix** 1. **For reindexing**: Always allow reindexing of existing documents 2. **For new uploads**: Provide duplicate handling options: - Update existing - Skip - Replace 3. **Add UI controls**: Checkbox options like "Update existing documents" or "Force reindex" 4. **Remove duplicate content prohibition**: Allow identical content with proper metadata tracking **Proposed code change:** ```python # In save_docs_to_vector_db function if existing_hash: if is_reindexing or force_update: # Delete existing chunks and re-add delete_chunks_by_doc_hash(existing_hash) logger.info(f"Updated existing document: {existing_hash}") else: # Return a proper response instead of throwing error return { "status": "duplicate", "message": "Document already exists", "options": ["update", "skip"] } ``` **## Workarounds Users Are Forced to Use** - Rename all files (add _v1, _v2 suffixes) - Delete entire knowledge base and recreate - Manually edit files to change content hash - Use external scripts to bypass the check **## Environment** - Open WebUI version: Latest (2026-01-22) - OS: All (Windows, Linux confirmed) - Installation: Docker and native **## Additional Context** This bug completely breaks the intended use case where documents need periodic updates or reindexing with new settings. The current implementation treats all duplicates as errors rather than providing management options, which goes against standard document management practices in other RAG systems (ChromaDB, Pinecone, etc. all support upsert operations). **Most importantly, there's no valid reason to prohibit identical content across documents. RAG systems should handle this naturally through metadata and search ranking, not by blocking document management operations.** **## Priority** This is a **BLOCKER** bug for any serious use of the Knowledge Base feature, as it prevents basic document maintenance and optimization.

GiteaMirror closed this issue

2026-05-21 01:31:55 -05:00

GiteaMirror commented

2026-05-21 01:31:56 -05:00

@Classic298 commented on GitHub (Jan 21, 2026):

ah the issue is not that the check is there, it correctly prevents duplicate files but that the check is being done when reindexing.

Will need to see first if I am able to reproduce.

This is a BLOCKER bug for any serious use of the Knowledge Base feature, as it prevents basic document maintenance and optimization.

No. Simply no.

Most importantly, there's no valid reason to prohibit identical content across documents.

Yes there is.
It's to prevent duplicate file uploads by users who don't know any better and spam your backend with many duplicate files. Nobody wants that.

And it sure as hell does not IMPROVE the rag performance if you have many duplicates, even if reranking should catch it.

@Classic298 commented on GitHub (Jan 21, 2026): ah the issue is not that the check is there, it correctly prevents duplicate files but that the check is being done when reindexing. Will need to see first if I am able to reproduce. > > This is a BLOCKER bug for any serious use of the Knowledge Base feature, as it prevents basic document maintenance and optimization. > No. Simply no. > `Most importantly, there's no valid reason to prohibit identical content across documents.` Yes there is. It's to prevent duplicate file uploads by users who don't know any better and spam your backend with many duplicate files. Nobody wants that. And it sure as hell does not IMPROVE the rag performance if you have many duplicates, even if reranking should catch it.

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#122625