Is your feature request related to a problem? Please describe.
At present, the internal DB schema for open-webui specifies that collection_name is a primary key on the Document table.
From webui/models/documents.py:
class Document(Base):
__tablename__ = "document"
collection_name = Column(String, primary_key=True) <-- why is this the primary key?
name = Column(String, unique=True)
title = Column(Text)
filename = Column(Text)
content = Column(Text, nullable=True)
user_id = Column(String)
timestamp = Column(BigInteger)
This means that for every document uploaded to the vector store, open-webui has to create a new collection and will error if multiple documents are created with the same collection name. This means that collections within the vector store can't be used properly.
Describe the solution you'd like
A new primary key for the Document table could be created, such as a UUID for the document. The RAG and document storage code could then be updated to allow documents to be uploaded to the same collection.
Describe alternatives you've considered
An alternative would be to upload documents and embeddings to the vector store out of band, but I'm not clear that open-webui could then properly use the documents since they won't exist in the internal DB.
Additional context
A simple modification to apps/rag/main.py allows the RAGing and upload of documents with the same collection name to the vector store:
Originally created by @nta18705 on GitHub (Sep 19, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/5514
**Is your feature request related to a problem? Please describe.**
At present, the internal DB schema for open-webui specifies that `collection_name` is a primary key on the Document table.
From `webui/models/documents.py`:
```
class Document(Base):
__tablename__ = "document"
collection_name = Column(String, primary_key=True) <-- why is this the primary key?
name = Column(String, unique=True)
title = Column(Text)
filename = Column(Text)
content = Column(Text, nullable=True)
user_id = Column(String)
timestamp = Column(BigInteger)
```
This means that for every document uploaded to the vector store, open-webui has to create a new collection and will error if multiple documents are created with the same collection name. This means that collections within the vector store can't be used properly.
**Describe the solution you'd like**
A new primary key for the Document table could be created, such as a UUID for the document. The RAG and document storage code could then be updated to allow documents to be uploaded to the same collection.
**Describe alternatives you've considered**
An alternative would be to upload documents and embeddings to the vector store out of band, but I'm not clear that open-webui could then properly use the documents since they won't exist in the internal DB.
**Additional context**
A simple modification to `apps/rag/main.py` allows the RAGing and upload of documents with the same collection name to the vector store:
```
def store_docs_in_vector_db(
docs, collection_name, metadata: Optional[dict] = None, overwrite: bool = False
) -> bool:
log.info(f"store_docs_in_vector_db {docs} {collection_name}")
...
try:
# if overwrite:
# for collection in CHROMA_CLIENT.list_collections():
# if collection_name == collection.name:
# log.info(f"deleting existing collection {collection_name}")
# CHROMA_CLIENT.delete_collection(name=collection_name)
# collection = CHROMA_CLIENT.create_collection(name=collection_name)
collection = CHROMA_CLIENT.get_or_create_collection(collection_name)
...
```
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @nta18705 on GitHub (Sep 19, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/5514
Is your feature request related to a problem? Please describe.
At present, the internal DB schema for open-webui specifies that
collection_nameis a primary key on the Document table.From
webui/models/documents.py:This means that for every document uploaded to the vector store, open-webui has to create a new collection and will error if multiple documents are created with the same collection name. This means that collections within the vector store can't be used properly.
Describe the solution you'd like
A new primary key for the Document table could be created, such as a UUID for the document. The RAG and document storage code could then be updated to allow documents to be uploaded to the same collection.
Describe alternatives you've considered
An alternative would be to upload documents and embeddings to the vector store out of band, but I'm not clear that open-webui could then properly use the documents since they won't exist in the internal DB.
Additional context
A simple modification to
apps/rag/main.pyallows the RAGing and upload of documents with the same collection name to the vector store:@tjbck commented on GitHub (Sep 19, 2024):
https://github.com/open-webui/open-webui/issues/3527