Before submitting, make sure you've checked the following:
Target branch: Please verify that the pull request targets the dev branch.
Description: Provide a concise description of the changes made in this pull request.
Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources?
Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
Testing: Have you written and run sufficient tests for validating the changes?
Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
Prefix: To cleary categorize this pull request, prefix the pull request title, using one of the following:
Changelog Entry
Description
This MR is focused on implementing a new interface based on LangChain VectorStore Class to handle integration with different VectorStores. It is adapted to the use of the VectorStores by OpenWebUI solution, that means no breaking changes with the existing solution. This MR proposes the implementations for the existing VectorStore types (ChromaDB - Persisting mode and Server mode) and also two new VectorStore types (PGVector and Milvus).
With the use of LangChain VectorStore interface there is a need to use the LangChain Embeddings interface. So multiple modifications have been done to adapt to this. Further work can be done to create better Embedding interface adapted to OpenWebUI solution.
Some refactoring has been done to sort imports on file where changes have been made.
New VectorStore interface for adding easily new VectorStore integrations.
Two new VectorStore integrations have been added: PGVector & Milvus.
Changed
Migrate Chroma Persistent&Server integration to new interface.
Use of LangChain Embedding class as Embedding functions for VectorStore initialization.
Refactoring has been done to sort imports on file where changes have been made.
Additional Information
For the two new integrations (PGVector & Milvus), it is not possible to use the Reset VectorDB feature for now.
Testing
The impacts of this MR are on the following features: ✔️ Upload Document (via Chat or Workspace). ✔️ Chat Completions with Query to documents (Simple Query + Query with Reranker). ✔️ Scan of documents (via AdminPanel) ✔️ Reset VectorDB (via AdminPanel) ✔️ Edit of EmbeddingModel settings ✔️ Memory feature
The tests can be done with the following integrations:
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/open-webui/open-webui/pull/4839
**Author:** [@pascallim](https://github.com/pascallim)
**Created:** 8/23/2024
**Status:** ❌ Closed
**Base:** `dev` ← **Head:** `implement-multi-vectorstores`
---
### 📝 Commits (1)
- [`235793c`](https://github.com/open-webui/open-webui/commit/235793cfedbde285cb7b66b9076ba8ea7980d542) feat: implement interface for multiple vectorstores with usage of Langchain classes
### 📊 Changes
**16 files changed** (+948 additions, -418 deletions)
<details>
<summary>View changed files</summary>
📝 `backend/open_webui/apps/ollama/main.py` (+24 -52)
➕ `backend/open_webui/apps/rag/embedding_model.py` (+66 -0)
📝 `backend/open_webui/apps/rag/main.py` (+148 -130)
📝 `backend/open_webui/apps/rag/utils.py` (+39 -158)
➕ `backend/open_webui/apps/rag/vector_store/__init__.py` (+3 -0)
➕ `backend/open_webui/apps/rag/vector_store/implementations/__init__.py` (+5 -0)
➕ `backend/open_webui/apps/rag/vector_store/implementations/chroma.py` (+279 -0)
➕ `backend/open_webui/apps/rag/vector_store/implementations/milvus.py` (+89 -0)
➕ `backend/open_webui/apps/rag/vector_store/implementations/pgvector.py` (+94 -0)
➕ `backend/open_webui/apps/rag/vector_store/vector_store_connector.py` (+54 -0)
➕ `backend/open_webui/apps/rag/vector_store/vector_store_extension.py` (+34 -0)
📝 `backend/open_webui/apps/webui/routers/memories.py` (+67 -39)
📝 `backend/open_webui/config.py` (+4 -39)
📝 `backend/open_webui/env.py` (+30 -0)
📝 `backend/requirements.txt` (+6 -0)
📝 `pyproject.toml` (+6 -0)
</details>
### 📄 Description
# Pull Request Checklist
**Before submitting, make sure you've checked the following:**
- [x] **Target branch:** Please verify that the pull request targets the `dev` branch.
- [x] **Description:** Provide a concise description of the changes made in this pull request.
- [x] **Changelog:** Ensure a changelog entry following the format of [Keep a Changelog](https://keepachangelog.com/) is added at the bottom of the PR description.
- [x] **Documentation:** Have you updated relevant documentation [Open WebUI Docs](https://github.com/open-webui/docs), or other documentation sources?
- [x] **Dependencies:** Are there any new dependencies? Have you updated the dependency versions in the documentation?
- [x] **Testing:** Have you written and run sufficient tests for validating the changes?
- [x] **Code review:** Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
- [x] **Prefix:** To cleary categorize this pull request, prefix the pull request title, using one of the following:
# Changelog Entry
### Description
- This MR is focused on implementing a new interface based on LangChain VectorStore Class to handle integration with different VectorStores. It is adapted to the use of the VectorStores by OpenWebUI solution, that means no breaking changes with the existing solution. This MR proposes the implementations for the existing VectorStore types (ChromaDB - Persisting mode and Server mode) and also two new VectorStore types (PGVector and Milvus).
- With the use of LangChain VectorStore interface there is a need to use the LangChain Embeddings interface. So multiple modifications have been done to adapt to this. Further work can be done to create better Embedding interface adapted to OpenWebUI solution.
- Some refactoring has been done to sort imports on file where changes have been made.
- Here is the link to the MR link the modification of the documentation: https://github.com/open-webui/docs/pull/191
### Added
- New VectorStore interface for adding easily new VectorStore integrations.
- Two new VectorStore integrations have been added: PGVector & Milvus.
### Changed
- Migrate Chroma Persistent&Server integration to new interface.
- Use of LangChain Embedding class as Embedding functions for VectorStore initialization.
- Refactoring has been done to sort imports on file where changes have been made.
### Additional Information
- For the two new integrations (PGVector & Milvus), it is not possible to use the Reset VectorDB feature for now.
# Testing
The impacts of this MR are on the following features:
✔️ Upload Document (via Chat or Workspace).
✔️ Chat Completions with Query to documents (Simple Query + Query with Reranker).
✔️ Scan of documents (via AdminPanel)
✔️ Reset VectorDB (via AdminPanel)
✔️ Edit of EmbeddingModel settings
✔️ Memory feature
The tests can be done with the following integrations:
**Embeddings**:
✔️ HuggingFace Embeddings (ex: sentence-transformers/all-MiniLM-L6-v2)
✔️ OpenAI Embeddings
✔️ Ollama Embeddings
**VectorStores**:
✔️ Persistent Chroma
✔️ Chroma Server
✔️ PGVector
✔️ Milvus
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/open-webui/open-webui/pull/4839
Author: @pascallim
Created: 8/23/2024
Status: ❌ Closed
Base:
dev← Head:implement-multi-vectorstores📝 Commits (1)
235793cfeat: implement interface for multiple vectorstores with usage of Langchain classes📊 Changes
16 files changed (+948 additions, -418 deletions)
View changed files
📝
backend/open_webui/apps/ollama/main.py(+24 -52)➕
backend/open_webui/apps/rag/embedding_model.py(+66 -0)📝
backend/open_webui/apps/rag/main.py(+148 -130)📝
backend/open_webui/apps/rag/utils.py(+39 -158)➕
backend/open_webui/apps/rag/vector_store/__init__.py(+3 -0)➕
backend/open_webui/apps/rag/vector_store/implementations/__init__.py(+5 -0)➕
backend/open_webui/apps/rag/vector_store/implementations/chroma.py(+279 -0)➕
backend/open_webui/apps/rag/vector_store/implementations/milvus.py(+89 -0)➕
backend/open_webui/apps/rag/vector_store/implementations/pgvector.py(+94 -0)➕
backend/open_webui/apps/rag/vector_store/vector_store_connector.py(+54 -0)➕
backend/open_webui/apps/rag/vector_store/vector_store_extension.py(+34 -0)📝
backend/open_webui/apps/webui/routers/memories.py(+67 -39)📝
backend/open_webui/config.py(+4 -39)📝
backend/open_webui/env.py(+30 -0)📝
backend/requirements.txt(+6 -0)📝
pyproject.toml(+6 -0)📄 Description
Pull Request Checklist
Before submitting, make sure you've checked the following:
devbranch.Changelog Entry
Description
This MR is focused on implementing a new interface based on LangChain VectorStore Class to handle integration with different VectorStores. It is adapted to the use of the VectorStores by OpenWebUI solution, that means no breaking changes with the existing solution. This MR proposes the implementations for the existing VectorStore types (ChromaDB - Persisting mode and Server mode) and also two new VectorStore types (PGVector and Milvus).
With the use of LangChain VectorStore interface there is a need to use the LangChain Embeddings interface. So multiple modifications have been done to adapt to this. Further work can be done to create better Embedding interface adapted to OpenWebUI solution.
Some refactoring has been done to sort imports on file where changes have been made.
Here is the link to the MR link the modification of the documentation: https://github.com/open-webui/docs/pull/191
Added
Changed
Additional Information
Testing
The impacts of this MR are on the following features:
✔️ Upload Document (via Chat or Workspace).
✔️ Chat Completions with Query to documents (Simple Query + Query with Reranker).
✔️ Scan of documents (via AdminPanel)
✔️ Reset VectorDB (via AdminPanel)
✔️ Edit of EmbeddingModel settings
✔️ Memory feature
The tests can be done with the following integrations:
Embeddings:
✔️ HuggingFace Embeddings (ex: sentence-transformers/all-MiniLM-L6-v2)
✔️ OpenAI Embeddings
✔️ Ollama Embeddings
VectorStores:
✔️ Persistent Chroma
✔️ Chroma Server
✔️ PGVector
✔️ Milvus
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.