[GH-ISSUE #18115] issue: ERROR_MESSAGES.DUPLICATE_CONTENT error for different files #138224

Closed
opened 2026-05-25 09:48:05 -05:00 by GiteaMirror · 1 comment
Owner

Originally created by @hasanbozok on GitHub (Oct 7, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/18115

Check Existing Issues

  • I have searched for any existing and/or related issues.
  • I have searched for any existing and/or related discussions.
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.6.32

Ollama Version (if applicable)

No response

Operating System

Ubuntu 24.04.3

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

store vectors to milvusdb

Actual Behavior

Latest version of open-webui (v0.6.32) is running on official docker image.
milvus is running with milvus provided docker-compose file. I can only upload 1 document. After 1 succesful upload, for every other document, owui logs error below.

Steps to Reproduce

1- create knowledge base
2- upload 2 pdf files or
3- upload 1 file than upload another file

Logs & Screenshots

Gives below error for different files while uploading them into newly created knowledge base.

Traceback (most recent call last):

File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
│ └ <function Thread._bootstrap_inner at 0x7c3c0e0909a0>
└ <WorkerThread(AnyIO worker thread, started 136593462507200)>
File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
│ └ <function WorkerThread.run at 0x7c3b67fcd6c0>
└ <WorkerThread(AnyIO worker thread, started 136593462507200)>
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 976, in run
result = context.run(func, *args)
│ │ │ └ ()
│ │ └ functools.partial(<function add_file_to_knowledge_by_id at 0x7c3b748c3ba0>, user=UserModel(id='54c8dc24-1d55-417b-9f2b-11ac44...
│ └ <method 'run' of '_contextvars.Context' objects>
└ <_contextvars.Context object at 0x7c3b24e21000>

File "/app/backend/open_webui/routers/knowledge.py", line 398, in add_file_to_knowledge_by_id
process_file(
└ <function process_file at 0x7c3b749f6b60>

File "/app/backend/open_webui/routers/retrieval.py", line 1617, in process_file
raise e
└ ValueError(<ERROR_MESSAGES.DUPLICATE_CONTENT: 'Duplicate content detected. Please provide unique content to proceed.'>)

File "/app/backend/open_webui/routers/retrieval.py", line 1581, in process_file
result = save_docs_to_vector_db(
└ <function save_docs_to_vector_db at 0x7c3b749f5d00>

File "/app/backend/open_webui/routers/retrieval.py", line 1245, in save_docs_to_vector_db
raise ValueError(ERROR_MESSAGES.DUPLICATE_CONTENT)
│ └ <ERROR_MESSAGES.DUPLICATE_CONTENT: 'Duplicate content detected. Please provide unique content to proceed.'>
└ <enum 'ERROR_MESSAGES'>

ValueError: Duplicate content detected. Please provide unique content to proceed.

Additional Information

Document flow:

Select files to upload -> extract with Apache Tika -> Calculate embeddings with gemma-embedding-300m running on vllm -> store vectors to milvus -> owui error

Originally created by @hasanbozok on GitHub (Oct 7, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/18115 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.6.32 ### Ollama Version (if applicable) _No response_ ### Operating System Ubuntu 24.04.3 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior store vectors to milvusdb ### Actual Behavior Latest version of open-webui (v0.6.32) is running on official docker image. milvus is running with milvus provided docker-compose file. I can only upload 1 document. After 1 succesful upload, for every other document, owui logs error below. ### Steps to Reproduce 1- create knowledge base 2- upload 2 pdf files or 3- upload 1 file than upload another file ### Logs & Screenshots Gives below error for different files while uploading them into newly created knowledge base. Traceback (most recent call last): File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap self._bootstrap_inner() │ └ <function Thread._bootstrap_inner at 0x7c3c0e0909a0> └ <WorkerThread(AnyIO worker thread, started 136593462507200)> File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner self.run() │ └ <function WorkerThread.run at 0x7c3b67fcd6c0> └ <WorkerThread(AnyIO worker thread, started 136593462507200)> File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 976, in run result = context.run(func, *args) │ │ │ └ () │ │ └ functools.partial(<function add_file_to_knowledge_by_id at 0x7c3b748c3ba0>, user=UserModel(id='54c8dc24-1d55-417b-9f2b-11ac44... │ └ <method 'run' of '_contextvars.Context' objects> └ <_contextvars.Context object at 0x7c3b24e21000> File "/app/backend/open_webui/routers/knowledge.py", line 398, in add_file_to_knowledge_by_id process_file( └ <function process_file at 0x7c3b749f6b60> > File "/app/backend/open_webui/routers/retrieval.py", line 1617, in process_file raise e └ ValueError(<ERROR_MESSAGES.DUPLICATE_CONTENT: 'Duplicate content detected. Please provide unique content to proceed.'>) File "/app/backend/open_webui/routers/retrieval.py", line 1581, in process_file result = save_docs_to_vector_db( └ <function save_docs_to_vector_db at 0x7c3b749f5d00> File "/app/backend/open_webui/routers/retrieval.py", line 1245, in save_docs_to_vector_db raise ValueError(ERROR_MESSAGES.DUPLICATE_CONTENT) │ └ <ERROR_MESSAGES.DUPLICATE_CONTENT: 'Duplicate content detected. Please provide unique content to proceed.'> └ <enum 'ERROR_MESSAGES'> ValueError: Duplicate content detected. Please provide unique content to proceed. ### Additional Information Document flow: Select files to upload -> extract with Apache Tika -> Calculate embeddings with gemma-embedding-300m running on vllm -> store vectors to milvus -> owui error
GiteaMirror added the bug label 2026-05-25 09:48:05 -05:00
Author
Owner

@tjbck commented on GitHub (Oct 7, 2025):

Unable to reproduce.

<!-- gh-comment-id:3377560189 --> @tjbck commented on GitHub (Oct 7, 2025): Unable to reproduce.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#138224