The intention is to reduce the size of the ChromaDB and PostSQL databases cleaning up orphaned records.
Note: --delete-vectors is extremely resource intensive and should only be used as part of regular maintenance and not a make good.
CHANGELOG ENTRY
Description
This Python script is designed to manage and clean up Postgres database entries, files, and vector store collections. It connects to a PostgreSQL database, examines stored chat and knowledge data, and optionally deletes unused files, database entries, and vector store collections to free up resources.
Path to the Chroma vector database directory. If not provided, defaults to script directory
None
/path/to/vector_db
-b, --batch-chats
Number of chat entries to process per batch (adjust for performance/memory usage)
10
50
-l, --list-files
List files marked for deletion without executing deletions
False
N/A
--delete-files
Delete identified unused files from storage
False
N/A
--delete-db-entries
Delete database entries (in file table) that are unused
False
N/A
--delete-vectors
Delete vector store collections not associated with current data
False
N/A
--no-confirm
Skip confirmation prompts before deletion actions
False
N/A
--log-memory
Log memory usage at different steps in the script
False
N/A
Script Functionality Breakdown
Connects to PostgreSQL
Uses provided database URL to connect.
Reads file table for IDs.
Reads knowledge table to extract knowledge IDs.
Streams chat entries.
Extracts IDs
File IDs: All IDs from file table.
Knowledge IDs: Extracted from knowledge table JSON data.
Chat File IDs: Extracted from chat content by recursive search for file IDs.
Checks for conflicts
Ensures no overlapping IDs between knowledge and chat files (raises error if found).
Determines files to delete
Finds file IDs that are not associated with current knowledge or chat data.
Reads files from the uploads directory, identifies files by prefix before underscore _.
Lists which files are safe to delete.
Works with Chroma vector store
Lists existing vector collections.
Identifies collections that are not associated with current data.
Optionally deletes these collections.
Deletes files and database entries
Files: Deletes files from storage if --delete-files.
Vector store collections: Deletes collections if --delete-vectors.
Database Entries: Deletes records from file table if --delete-db-entries.
Provides confirmation prompts (unless --no-confirm)
Before deletion actions, prompts the user unless overridden.
Logs memory usage
Optionally logs used memory during key steps if --log-memory.
By submitting this pull request, I confirm that I have read and fully agree to the CONTRIBUTOR_LICENSE_AGREEMENT, and I am providing my contributions under its terms.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/open-webui/open-webui/pull/13344
**Author:** [@spammenotinoz](https://github.com/spammenotinoz)
**Created:** 4/30/2025
**Status:** ❌ Closed
**Base:** `main` ← **Head:** `dev`
---
### 📝 Commits (10+)
- [`0a2747b`](https://github.com/open-webui/open-webui/commit/0a2747bc6a97f7dd14e2e08c5b5be4c11ab2f66a) Merge branch 'dev' of github.com:taylorwilsdon/open-webui into dev
- [`66bf6d8`](https://github.com/open-webui/open-webui/commit/66bf6d88bc298e1547f22f8baaf951a7c96b14bc) Merge branch 'dev' of github.com:taylorwilsdon/open-webui into dev
- [`b446598`](https://github.com/open-webui/open-webui/commit/b4465987f95fe062ef1df3c510ccd9631f58db0e) Environment variable to redirect on logout
- [`708fcdc`](https://github.com/open-webui/open-webui/commit/708fcdc6b786e8790d2ec30d72b7d3dc854f26dd) Merge branch 'dev' of github.com:taylorwilsdon/open-webui into dev
- [`a8bbaa6`](https://github.com/open-webui/open-webui/commit/a8bbaa61bd048a5d30273ad15840138a7ca479d2) Merge branch 'dev' into main
- [`839ba22`](https://github.com/open-webui/open-webui/commit/839ba22c90991dc4d72810aa6f6a527664db80c2) feat: Backend for Self-Hosted/External Web Search/Loader Engines
- [`22f0365`](https://github.com/open-webui/open-webui/commit/22f0365cef0d366632be48e20b25fbf09330f9ec) format
- [`008fec8`](https://github.com/open-webui/open-webui/commit/008fec80c10b911068205a1782ea28507e3dea6b) fix: Update external search/loader method to POST
- [`85f8e91`](https://github.com/open-webui/open-webui/commit/85f8e91288c201d68cece8f3ec3d08968a6d5fb1) feat: Allow admin editing external search/loader settings
- [`dcb3f18`](https://github.com/open-webui/open-webui/commit/dcb3f18e1e4e342c736dc1c86f337467883bc1bb) refac: styling
### 📊 Changes
**136 files changed** (+5113 additions, -2177 deletions)
<details>
<summary>View changed files</summary>
📝 `.github/pull_request_template.md` (+1 -1)
📝 `Dockerfile` (+1 -1)
📝 `LICENSE` (+7 -1)
📝 `README.md` (+1 -2)
📝 `backend/open_webui/__init__.py` (+1 -1)
📝 `backend/open_webui/config.py` (+95 -6)
📝 `backend/open_webui/env.py` (+61 -0)
📝 `backend/open_webui/main.py` (+31 -2)
➕ `backend/open_webui/retrieval/loaders/external.py` (+53 -0)
📝 `backend/open_webui/retrieval/utils.py` (+40 -16)
📝 `backend/open_webui/retrieval/vector/connector.py` (+4 -0)
📝 `backend/open_webui/retrieval/vector/dbs/chroma.py` (+7 -2)
📝 `backend/open_webui/retrieval/vector/dbs/elasticsearch.py` (+7 -2)
📝 `backend/open_webui/retrieval/vector/dbs/milvus.py` (+7 -2)
📝 `backend/open_webui/retrieval/vector/dbs/opensearch.py` (+7 -2)
📝 `backend/open_webui/retrieval/vector/dbs/pgvector.py` (+7 -2)
➕ `backend/open_webui/retrieval/vector/dbs/pinecone.py` (+412 -0)
📝 `backend/open_webui/retrieval/vector/dbs/qdrant.py` (+41 -9)
📝 `backend/open_webui/retrieval/vector/main.py` (+68 -1)
➕ `backend/open_webui/retrieval/web/external.py` (+47 -0)
_...and 80 more files_
</details>
### 📄 Description
The intention is to reduce the size of the ChromaDB and PostSQL databases cleaning up orphaned records.
Note: --delete-vectors is extremely resource intensive and should only be used as part of regular maintenance and not a make good.
### CHANGELOG ENTRY
### Description
This Python script is designed to manage and clean up Postgres database entries, files, and vector store collections. It connects to a PostgreSQL database, examines stored chat and knowledge data, and optionally deletes unused files, database entries, and vector store collections to free up resources.
By default it does not delete\change anything.
#### Requirements \ Dependencies
Python 3.x
psycopg2
psycopg2.extras
chromadb
psutil
json
argparse
os
sys
concurrent.futures
### USAGE
python postgres_chroma.cleanup.py [options]
<!--StartFragment--><h2 dir="auto" style="box-sizing: border-box; border: 0px solid oklch(0.21 0.034 264.665); margin: 2em 0px 1em; padding: 0px; font-size: 1.5em; font-weight: 600; color: oklch(0.21 0.034 264.665); line-height: 1.33333; margin-block: 4px; --tw-font-weight: 600; font-family: -apple-system, BlinkMacSystemFont, Inter, ui-sans-serif, system-ui, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif, "Helvetica Neue", Arial, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre-line; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Optional Arguments</h2><div class="relative w-full group" style="box-sizing: border-box; border: 0px solid rgb(78, 78, 78); margin: 0px; padding: 0px; position: relative; width: 936.009px; color: rgb(78, 78, 78); font-family: -apple-system, BlinkMacSystemFont, Inter, ui-sans-serif, system-ui, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif, "Helvetica Neue", Arial, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: pre-line; background-color: rgb(255, 255, 255); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><div class="scrollbar-hidden relative overflow-x-auto max-w-full rounded-lg" style="box-sizing: border-box; border: 0px solid rgb(78, 78, 78); margin: 0px; padding: 0px; position: relative; max-width: 100%; overflow-x: auto; border-radius: 8px;">
Option | Description | Default | Example
-- | -- | -- | --
--chroma-path | Path to the Chroma vector database directory. If not provided, defaults to script directory | None | /path/to/vector_db
-b, --batch-chats | Number of chat entries to process per batch (adjust for performance/memory usage) | 10 | 50
-l, --list-files | List files marked for deletion without executing deletions | False | N/A
--delete-files | Delete identified unused files from storage | False | N/A
--delete-db-entries | Delete database entries (in file table) that are unused | False | N/A
--delete-vectors | Delete vector store collections not associated with current data | False | N/A
--no-confirm | Skip confirmation prompts before deletion actions | False | N/A
--log-memory | Log memory usage at different steps in the script | False | N/A
</div></div><!--EndFragment-->
Script Functionality Breakdown
1. Connects to PostgreSQL
Uses provided database URL to connect.
Reads file table for IDs.
Reads knowledge table to extract knowledge IDs.
Streams chat entries.
2. Extracts IDs
File IDs: All IDs from file table.
Knowledge IDs: Extracted from knowledge table JSON data.
Chat File IDs: Extracted from chat content by recursive search for file IDs.
3. Checks for conflicts
Ensures no overlapping IDs between knowledge and chat files (raises error if found).
4. Determines files to delete
Finds file IDs that are not associated with current knowledge or chat data.
Reads files from the uploads directory, identifies files by prefix before underscore _.
Lists which files are safe to delete.
5. Works with Chroma vector store
Lists existing vector collections.
Identifies collections that are not associated with current data.
Optionally deletes these collections.
6. Deletes files and database entries
Files: Deletes files from storage if --delete-files.
Vector store collections: Deletes collections if --delete-vectors.
Database Entries: Deletes records from file table if --delete-db-entries.
7. Provides confirmation prompts (unless --no-confirm)
Before deletion actions, prompts the user unless overridden.
8. Logs memory usage
Optionally logs used memory during key steps if --log-memory.
By submitting this pull request, I confirm that I have read and fully agree to the [CONTRIBUTOR_LICENSE_AGREEMENT](CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms.
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/open-webui/open-webui/pull/13344
Author: @spammenotinoz
Created: 4/30/2025
Status: ❌ Closed
Base:
main← Head:dev📝 Commits (10+)
0a2747bMerge branch 'dev' of github.com:taylorwilsdon/open-webui into dev66bf6d8Merge branch 'dev' of github.com:taylorwilsdon/open-webui into devb446598Environment variable to redirect on logout708fcdcMerge branch 'dev' of github.com:taylorwilsdon/open-webui into deva8bbaa6Merge branch 'dev' into main839ba22feat: Backend for Self-Hosted/External Web Search/Loader Engines22f0365format008fec8fix: Update external search/loader method to POST85f8e91feat: Allow admin editing external search/loader settingsdcb3f18refac: styling📊 Changes
136 files changed (+5113 additions, -2177 deletions)
View changed files
📝
.github/pull_request_template.md(+1 -1)📝
Dockerfile(+1 -1)📝
LICENSE(+7 -1)📝
README.md(+1 -2)📝
backend/open_webui/__init__.py(+1 -1)📝
backend/open_webui/config.py(+95 -6)📝
backend/open_webui/env.py(+61 -0)📝
backend/open_webui/main.py(+31 -2)➕
backend/open_webui/retrieval/loaders/external.py(+53 -0)📝
backend/open_webui/retrieval/utils.py(+40 -16)📝
backend/open_webui/retrieval/vector/connector.py(+4 -0)📝
backend/open_webui/retrieval/vector/dbs/chroma.py(+7 -2)📝
backend/open_webui/retrieval/vector/dbs/elasticsearch.py(+7 -2)📝
backend/open_webui/retrieval/vector/dbs/milvus.py(+7 -2)📝
backend/open_webui/retrieval/vector/dbs/opensearch.py(+7 -2)📝
backend/open_webui/retrieval/vector/dbs/pgvector.py(+7 -2)➕
backend/open_webui/retrieval/vector/dbs/pinecone.py(+412 -0)📝
backend/open_webui/retrieval/vector/dbs/qdrant.py(+41 -9)📝
backend/open_webui/retrieval/vector/main.py(+68 -1)➕
backend/open_webui/retrieval/web/external.py(+47 -0)...and 80 more files
📄 Description
The intention is to reduce the size of the ChromaDB and PostSQL databases cleaning up orphaned records.
Note: --delete-vectors is extremely resource intensive and should only be used as part of regular maintenance and not a make good.
CHANGELOG ENTRY
Description
This Python script is designed to manage and clean up Postgres database entries, files, and vector store collections. It connects to a PostgreSQL database, examines stored chat and knowledge data, and optionally deletes unused files, database entries, and vector store collections to free up resources.
By default it does not delete\change anything.
Requirements \ Dependencies
Python 3.x
psycopg2
psycopg2.extras
chromadb
psutil
json
argparse
os
sys
concurrent.futures
USAGE
python postgres_chroma.cleanup.py [options]
Optional Arguments
Script Functionality Breakdown
Uses provided database URL to connect.
Reads file table for IDs.
Reads knowledge table to extract knowledge IDs.
Streams chat entries.
File IDs: All IDs from file table.
Knowledge IDs: Extracted from knowledge table JSON data.
Chat File IDs: Extracted from chat content by recursive search for file IDs.
Ensures no overlapping IDs between knowledge and chat files (raises error if found).
Finds file IDs that are not associated with current knowledge or chat data.
Reads files from the uploads directory, identifies files by prefix before underscore _.
Lists which files are safe to delete.
Lists existing vector collections.
Identifies collections that are not associated with current data.
Optionally deletes these collections.
Files: Deletes files from storage if --delete-files.
Vector store collections: Deletes collections if --delete-vectors.
Database Entries: Deletes records from file table if --delete-db-entries.
Before deletion actions, prompts the user unless overridden.
Optionally logs used memory during key steps if --log-memory.
By submitting this pull request, I confirm that I have read and fully agree to the CONTRIBUTOR_LICENSE_AGREEMENT, and I am providing my contributions under its terms.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.