github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-05-06 19:08:59 -05:00
Code Issues 1.3k Packages Projects Releases 126 Wiki Activity

[PR #16520] [CLOSED] 🚀 feat: COMPREHENSIVE DATA PRUNING SYSTEM - The Ultimate Storage Management Solution for Open WebUI #63016

New Issue
Closed
opened 2026-05-06 07:31:50 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-06 07:31:50 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/16520
Author: @Classic298
Created: 8/12/2025
Status: ❌ Closed

Base: dev ← Head: universal_file_deletion

📝 Commits (10+)

📊 Changes

6 files changed (+3015 additions, -0 deletions)

View changed files

📝 backend/open_webui/main.py (+2 -0)
📝 backend/open_webui/models/folders.py (+6 -0)
➕ backend/open_webui/routers/prune.py (+1793 -0)
➕ src/lib/apis/prune.ts (+66 -0)
📝 src/lib/components/admin/Settings/Database.svelte (+249 -0)
➕ src/lib/components/common/PruneDataDialog.svelte (+899 -0)

📄 Description

🚀 feat: COMPREHENSIVE DATA PRUNING SYSTEM - The Ultimate Storage Management Solution for Open WebUI

Before submitting, make sure you've checked the following:

  • Target branch: Please verify that the pull request targets the dev branch.
  • Description: Provide a concise description of the changes made in this pull request.
  • Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
  • Documentation: Have you updated relevant documentation Open WebUI Docs, or other documentation sources?
  • Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
  • Testing: Have you written and run sufficient tests to validate the changes?
  • Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
  • Prefix: To clearly categorize this pull request, prefix the pull request title using one of the following:
    • feat: Introduces a new feature or enhancement to the codebase

🎯 COMPREHENSIVE DATA MANAGEMENT SOLUTION

After MONTHS of development and addressing way over 20+ community issues, this PR introduces a complete data pruning system for Open WebUI. This implementation has been carefully designed over multiple months to address the most requested feature in the Open WebUI community - comprehensive storage management and cleanup capabilities.

🎉 ADDRESSES 25+ COMMUNITY ISSUES & DISCUSSIONS

This implementation closes/addresses:

Primary Issues / PRs / Discussions Resolved:

  • Addresses #13001 - feat: Add automatic chat record cleanup feature
  • Addresses #11582 - issue: Deleting a chat does not delete files uploaded in it
  • Addresses #13396 - WIP: add periodic data cleanup task
  • Addresses #10603 - "Delete All Chats" actually deletes ALL chats
  • Addresses #13551 - Issue: Audio data submitted via notes is not being deleted
  • Addresses #8902 - Many orphaned files in system after removed from knowledge base
  • Addresses #8862 - Introduce a Housekeeping Background Job
  • Addresses #5199 - Feature request: set retention for uploaded files
  • Addresses #6705 - Request for Transparency and Management Options for Audio File Retention
  • Addresses #7465 - Feature Request: Data Cleanup Policy
  • Addresses #12091 - Best practices for cleanup or how to avoid infinitely growing database
  • Addresses #12280 - feat: OPTIONAL deletion of files and db entries on chat deletion
  • Addresses #5468 - When the uploaded files will be deleted?
  • Addresses #13756 - feat: Avoid duplicate files on storage backend
  • Addresses #13869 - Button in the admin to delete all unused files
  • Addresses #8875 - Missing functionality for cleanup
  • Addresses #4580 - Automated deletion of old chats in history
  • Addresses #14129 - Enhancement: Add configurable retention and storage location for meeting audio notes
  • Addresses #13718 - issue: When deleting a knowledge base, it does not delete the collections corresponding to individual documents
  • (Kind-of) Addresses #10679 - Batch add file to knowledge doesn't check for existence
  • Addresses #12249 - issue: The Size of the webui.db File Continuously Increasing
  • Addresses #6935 - Chromadb warns that my database needs to be vacuumed
  • Addresses #7181 - File deletion doesn't properly clean up database entries, causing issues with re-uploads
  • Addresses #4035 - [bug/perf] large db, slow query to /chats/
  • Addresses #17065 - Files not getting deleted from Chroma DB
  • Addresses [GH-ISSUE #14601] feat: Admin creation from password hash (#17313)
  • https://github.com/open-webui/open-webui/discussions/3729

And definitely many more - in fact, i lost track of some of the discussions in my notifications.
Also there were PLENTY of feature requests, bug reports and discussions around this topic on the official Discord Server. If I had to guess, there were at least 40 real requests and discussions around this.

🛡️ PRESERVES EXISTING BEHAVIOR & FOLLOWS MAINTAINER VISION

🎯 EXACTLY THE APPROACH REQUESTED

This implementation follows the API endpoint + manual trigger approach outlined by @tjbck (maintainer) in previous discussions:

  • ✅ OPTIONAL feature - Completely optional and fully configurable to a minimum
  • ✅ API endpoint for automation - Manual execution only
  • ✅ NO changes to existing behavior - Files retained by default
  • ✅ Admin-only access - Users cannot trigger pruning
  • ✅ Explicit execution required - Nothing deleted without admin action

🔒 WHY THIS APPROACH WAS CHOSEN

Previous PRs proposing automated background deletion were correctly rejected because:

  • ❌ Changes existing behavior - Users expect files to persist
  • ❌ Violates regulatory requirements - Some jurisdictions or compliance and audit topics REQUIRE data retention (SEC financial records for 7 years, HIPAA medical documentation for 6+ years, legal discovery holds indefinitely, tax documentation for 3-7 years, employment records for 1-4 years, environmental compliance documentation for decades, tax records, employment law, environmental regulations - often 3-30+ years)
  • ❌ Removes admin control - Full automation by default can cause unexpected or unintended data loss (e.g. PRs that attempted to tie the file and vector deletion to the manual chat deletion by the user)
  • ❌ One-size-fits-all problems - Different organizations have different needs for what should get deleted, if at all, and when (automation perhaps not feasible, manual configuration is urgently needed).

This PR respects these constraints by providing optional, manual, admin-only, API-driven and FULLY CONFIGURABLE data pruning.

🌟 KEY DESIGN PRINCIPLES

🔒 SAFETY FIRST

  • Optional activation - You do not have to use the pruning feature at all, and if you do, you can enable or disable optional aspects
  • Admin-only access - Regular users cannot trigger pruning at all
  • Manual execution - No automated deletion without explicit admin action via API call or admin interface interaction
  • Comprehensive warnings - Clear documentation of destructive operations directly in the UI - docs to follow.
  • Granular control - Individual toggles for every cleanup type - fully configurable what to delete, including retention policies for automated deletion capabilities via external API call.

🌍 REGULATORY COMPLIANCE

  • Data retention flexibility - Supports organizations that MUST retain data (these companies proceed to not using this feature)
  • GDPR compliance tools - For organizations that need data minimization
  • Audit trail logging - Complete operation documentation
  • Configurable policies - Adaptable to any regulatory requirement

🤖 AUTOMATION READY

  • API endpoint (/api/v1/prune) for external scripts and automated calling
  • Copy-paste automation - Real-time API call generator - Open the menu in the admin panel, configure your policy and copy the finished API call to your desire!
  • Cron-job friendly - Perfect for scheduled maintenance
  • Enterprise integration - Fits existing infrastructure

🚀 COMPREHENSIVE FEATURE SET

👥 ADVANCED USER MANAGEMENT

  • ✅ Time-based inactive user deletion - Configurable retention periods for inactive accounts
  • ✅ Smart exemptions - Protect admin users and pending approvals
  • ✅ Activity tracking - Based on last_active_at timestamps for accurate detection
  • ✅ Cascade deletion - Automatically removes all user-associated data (chats, files, knowledge bases, etc.)
  • ✅ Safety guards - Strong defaults (90+ day minimum) with exemptions for critical accounts
  • ✅ Preview capability - Dry-run mode shows exactly which users would be affected

💬 ADVANCED CHAT MANAGEMENT

  • ✅ Age-based chat deletion using updated_at timestamps
  • ✅ Smart exemptions for archived chats
  • ✅ Folder protection - keeps organized chats safe
  • ✅ Pin preservation - respects user-pinned conversations
  • ✅ Orphaned chat cleanup from deleted users

📁 COMPREHENSIVE FILE SYSTEM INTEGRATION

  • ✅ Generated images - Already handled via file system! 🎨
  • ✅ Uploaded documents - Complete reference tracking
  • ✅ Orphaned file detection - Scans uploads directory
  • ✅ Vector collection cleanup - Removes unused embeddings
  • ✅ Knowledge base synchronization - Maintains data integrity - will delete files that failed the uploading procedure into a knowledge base (due to failed content extraction, failed vectorization or due to "duplicate content").

🎵 AUDIO CACHE MANAGEMENT

  • ✅ TTS file cleanup - Manages text-to-speech generated audio
  • ✅ STT transcription cleanup - Handles speech-to-text files
  • ✅ Configurable retention - Age-based audio file management
  • ✅ Storage reclamation - Reports cleaned space in MB

🗄️ DATABASE OPTIMIZATION

  • ✅ SQLite VACUUM - Optimizes main database - will free up Megabytes or even Gigabytes of storage - noticable speed and performance improvements as a result
  • ✅ PostgreSQL support - Works with PostgreSQL as well
  • ✅ ChromaDB cleanup - ENHANCED Vector database optimization with deep cleanup - Vector DB will shrink dramatically
  • ✅ Metadata synchronization - Keeps everything consistent

👥 USER & RESOURCE MANAGEMENT

  • ✅ Orphaned user content cleanup across 8 resource types:
    • 🗂️ Folders, 📝 Notes, 🛠️ Tools, ⚙️ Functions
    • 💬 Prompts, 🤖 Models, 📚 Knowledge Bases, 📁 Files
    • Fully configurable

🎛️ ADVANCED CONFIGURATION

  • ✅ Tabbed interface - Users, Chats, Workspace, Audio Cache
  • ✅ Toggle controls - Enable/disable any feature
  • ✅ Real-time API preview - Copy-paste automation scripts
  • ✅ Production-ready defaults - Safe out-of-the-box settings
  • ✅ Dry-run preview - See exactly what will be deleted before execution

🔧 TECHNICAL IMPLEMENTATION

🏗️ MULTI-STAGE PROCESSING ARCHITECTURE

  1. Inactive User Management - Time-based account cleanup with exemptions
  2. Smart Chat Deletion - Age-based with exemptions
  3. Preservation Set Building - Identifies all actively referenced data
  4. Orphaned Record Cleanup - Safely removes database entries
  5. Physical File Synchronization - Cleans actual storage
  6. Audio Cache Management - Handles TTS/STT files
  7. Database Optimization - VACUUM operations for peak performance

🏭 MODULAR VECTOR DATABASE FRAMEWORK

# Extensible architecture for community contributions:
class VectorDatabaseCleaner(ABC):
    # Abstract interface for all vector databases
    
class ChromaDatabaseCleaner(VectorDatabaseCleaner):
    # Full ChromaDB implementation with deep cleanup
    
class PGVectorDatabaseCleaner(VectorDatabaseCleaner):  
    # Complete PGVector implementation (community-ready)
    
def get_vector_database_cleaner() -> VectorDatabaseCleaner:
    # Factory pattern for automatic detection

🔧 CODE QUALITY IMPROVEMENTS

class JSONFileIDExtractor:
    # Extracted duplicate regex patterns into reusable utility
    # Compiles patterns once for better performance
    # Centralized validation logic

🔍 INTELLIGENT FILE SCANNING

# Scans ALL data sources for file references:
# ✅ Knowledge base data structures
# ✅ Chat message JSON content  
# ✅ Folder items and metadata
# ✅ Standalone message table
# ✅ URL pattern matching
# ✅ Database validation

🧠 ENHANCED VECTOR CLEANUP

# ChromaDB Deep Cleanup (fixes @mahenning's 2.2GB → 156KB issue):
# ✅ Orphaned embeddings cascade deletion
# ✅ Orphaned metadata cleanup
# ✅ Full-text search (FTS) selective rebuild
# ✅ Collection/segment metadata synchronization
# ✅ Proper VACUUM execution for space reclamation

# PGVector Integration:
# ✅ Uses existing client methods for reliability
# ✅ PostgreSQL-optimized VACUUM ANALYZE
# ✅ Collection discovery and cleanup

🎵 AUDIO CACHE INTELLIGENCE

# Comprehensive audio management:
# ✅ TTS file age-based cleanup
# ✅ STT transcription management
# ✅ Metadata file synchronization
# ✅ Storage space reporting

🔍 DRY-RUN PREVIEW SYSTEM

# Complete preview capabilities:
# ✅ Count inactive users before deletion
# ✅ Count old chats with exemption rules
# ✅ Count orphaned records across all resource types
# ✅ Count orphaned uploads and vector collections
# ✅ Count audio cache files for cleanup
# ✅ Real-time preview modal with detailed breakdown

⚡ OPTIMIZATION BENEFITS

  • Database size reduction - VACUUM operations reclaim unused space (Gigabytes in some cases)
  • Storage reclamation - Removes orphaned files and collections, saving space
  • Performance improvement - Cleaner databases run faster, less file handles, less unused references; less time for operations means faster performance
  • Cost savings - Reduced storage requirements and noticable speedup
  • Vector database efficiency - ChromaDB files reduce from 2.2GB+ to ~156KB (system tables only)

🎨 USER EXPERIENCE

🖼️ BEAUTIFUL INTERFACE

  • Modern, tabbed design - Now with Users tab
  • Expandable help sections with comprehensive documentation
  • Progress feedback and success notifications, incl. logging to CLI
  • Built-in help system explaining every feature and what the pruning feature does
  • Visual API configurator and preview for easily creating API calls tailored to your needs to creating automated scripts
  • Real-time dry-run preview - See exact counts before execution

🚧 FUTURE DEVELOPMENT OPPORTUNITIES

🗄️ VECTOR DATABASE SUPPORT

Currently Implemented:

  • ✅ ChromaDB - Full cleanup and optimization support with deep cleanup breakthrough
  • ✅ PGVector - Complete implementation using existing client methods

Community Extension Framework Ready:

  • 🔧 Milvus - Modular architecture ready for implementation
  • 🔧 Pinecone - API structure established
  • 🔧 Qdrant - Factory pattern supports easy addition
  • 🔧 Elasticsearch - Framework ready for community contribution
  • 🔧 OpenSearch - Modular design supports integration
  • 🔧 Oracle23AI - Extension pattern available

Adding New Vector Databases:

# Community contributors can easily add support by:
# 1. Extending VectorDatabaseCleaner abstract class
# 2. Implementing 3 required methods (count, cleanup, delete)
# 3. Adding detection logic to factory function
# Framework handles all integration automatically!

Changelog Entry

Description

COMPREHENSIVE DATA PRUNING SYSTEM - A production-ready, enterprise-grade pruning system developed over multiple months to address 20+ community issues. This optional, admin-controlled feature includes intelligent chat deletion, time-based user account management, comprehensive file cleanup, audio cache management, enhanced vector database optimization with modular framework, dry-run preview capabilities, and full GDPR compliance capabilities while preserving all existing behavior.

Added

  • 🎛️ Complete Admin Pruning Interface - Beautiful tabbed UI with docs, explanations and granular configuration controls
  • 👥 Time-Based Inactive User Management - Configurable deletion of inactive accounts with smart exemptions and cascade cleanup
  • 🔍 Dry-Run Preview System - Complete preview modal showing exact counts of what will be deleted before execution
  • 🏭 Modular Vector Database Framework - Extensible architecture supporting ChromaDB, PGVector, and community extensions
  • 🗄️ Enhanced ChromaDB Cleanup - Deep cleanup solving the 2.2GB+ file size issue (reduces to ~156KB)
  • 🗄️ Complete PGVector Integration - Full support using existing client methods for reliability
  • 🔧 Code Quality Improvements - Extracted duplicate patterns, optimized regex compilation, centralized validation
  • 📅 Smart Chat Age Management - Configurable deletion with optional and configurable archive/folder/pin exemptions
  • 📁 Comprehensive File System Integration - Complete orphaned file detection and cleanup (even for unindexed files)
  • 🎵 Audio Cache Management System - TTS/STT file cleanup with configurable retention
  • 👥 Orphaned User Content Cleanup - 8 resource types with individual toggles and granular control
  • 🤖 API Automation Endpoint - /api/v1/prune for external automated script-based integrations
  • 📋 Enhanced API Preview Generator - API call configurator with extensive comments, cron examples, and automation best practices
  • 🛡️ Multi-stage Safety Processing - Ground truth preservation with state synchronization
  • 🌍 GDPR Compliance Tools - Optional data minimization and retention policy enforcement
  • 📊 Comprehensive Logging - Detailed operation reporting
  • ⚡ Database Performance Optimization - VACUUM operations for SQLite, Chroma and PostgreSQL leading to major database performance gains

Deprecated / Changed

  • None - All changes are additive and backward compatible

Removed

  • None - All changes are additive and backward compatible

Fixed

  • 🐛 ChromaDB File Size Issue - Fixed @mahenning's reported issue where ChromaDB files remained 2.2GB+ after cleanup
  • 🐛 Vector Database Cleanup - Comprehensive orphaned record cleanup that ChromaDB's delete_collection() method missed
  • 🔧 Code Duplication - Extracted and centralized duplicate regex patterns for better maintainability

Security

  • 🔒 Admin-only API Access - Pruning endpoint restricted to administrators only
  • 🛡️ Multi-level Validation - Comprehensive safety checks before any deletion
  • 🌍 GDPR Compliance - Optional data minimization and retention policy enforcement
  • 🔐 Inactive User Security - Safe removal of long-inactive accounts with admin/pending exemptions

Additional Information

🎯 ADDRESSES COMMUNITY PAIN POINTS

This PR addresses years of community feedback about:

  • Runaway database growth affecting very large user instances
  • GDPR compliance concerns for EU deployments
  • Orphaned files consuming terabytes of storage
  • Audio retention violating confidentiality requirements
  • Manual SQL surgery being the only cleanup option
  • Vector databases growing to 200GB+ with zero active chats
  • Inactive user accounts accumulating over time
  • ChromaDB databases not properly shrinking after cleanup

Screenshots or Videos

Admin Panel - Database Section

image image

Admin Panel - Prune Modal

image image

Admin Panel - Prune Modal Docs

image image image image image

Admin Panel - Prune Modal Config

image image image image

Admin Panel - Inactive User Management Tab

image

Dry-Run Preview Modal

image image

Admin Panel - Prune Modal API helper

image

Shows the API call, fully configured according to the selections and settings you set in the configurator above.

Useful for external pruning automation.

API Helper with Advanced Comments

Example:

# Open WebUI Data Pruning API Call
# Use this template for automated maintenance scripts (cron jobs, etc.)

# AUTHENTICATION: Use API Key (not JWT token) for automation
# Get your API key from: Settings → Account → API Key → Generate new key
# Format: sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

curl -X POST "http://localhost:5173/api/v1/prune/" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-api-key>" \
  -d '{
    // SAFETY: Always test with dry_run=true first to preview results
    "dry_run": false,
    
    // AGE-BASED CHAT DELETION (null = disabled)
    "days": 0,
    "exempt_archived_chats": false,  // Keep archived chats even if old
    "exempt_chats_in_folders": false,  // Keep organized/pinned chats
    
    // INACTIVE USER DELETION (null = disabled, VERY DESTRUCTIVE)
    "delete_inactive_users_days": 90,
    "exempt_admin_users": true,  // Strongly recommended: true
    "exempt_pending_users": true,  // Recommended for user approval workflows
    
    // ORPHANED DATA CLEANUP (from deleted users)
    "delete_orphaned_chats": true,
    "delete_orphaned_tools": true,
    "delete_orphaned_functions": true,  // Actions, Pipes, Filters
    "delete_orphaned_prompts": true,
    "delete_orphaned_knowledge_bases": true,
    "delete_orphaned_models": true,
    "delete_orphaned_notes": true,
    "delete_orphaned_folders": true,
    
    // AUDIO CACHE CLEANUP (null = disabled)
    "audio_cache_max_age_days": 30  // TTS/STT files
  }'

# API KEY vs JWT TOKEN:
# - API Key: Persistent, use for automation (sk-xxxxxxxx...)
# - JWT Token: Session-bound, temporary, use for web UI only
# - ALWAYS use API Key for scripts/cron jobs

# AUTOMATION TIPS:
# 1. Run with dry_run=true first to preview what will be deleted
# 2. Schedule during low-usage hours to minimize performance impact  
# 3. Monitor logs: tail -f /path/to/open-webui/logs
# 4. Consider database backup before large cleanup operations
# 5. Test on staging environment with similar data size first

# EXAMPLE CRON JOB (runs weekly on Sunday at 2 AM):
# 0 2 * * 0 /path/to/your/prune-script.sh >> /var/log/openwebui-prune.log 2>&1

# RESPONSE HANDLING:
# - dry_run=true: Returns counts object with preview numbers
# - dry_run=false: Returns true on success, throws error on failure
# - Always check HTTP status code and response for errors
image

Confirmation of prune success

Info Level Logging

image

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

User Feedback Tracking

Thanks for your feedback and for testing the PR. This section of the PR description will be continuously updated to keep track of the last remaining points

Feature Wishes / To Do

  • ✅ Implement feature to optionally delete long inactive accounts (configurable)
  • ✅ Investigate modular architecture for other vector DB integrations
  • ✅ Attempt to integrate pgvector
  • ✅ Test pgvector integration
  • ✅ Improve chromaDB integration
    • ✅ Proper database vacuum, current implementation doesn't fully vacuum chromaDB
    • attempt to simplify implementation with .delete command (needs investigation if UUID matching still works, since chroma DB and the files itself and the file handles in Open WebUI's database have different UUID's each, requiring complex cross matching to even make it work in the first place)
      The amount of tinkering that is necessary to fully cleanup chroma db does not allow for this to be easy lol.
  • ✅ Extract Duplicate Regex Patterns and remove duplicates, simplifying the code a little bit
  • ✅ If possible, add a dry-run function (to preview what would get deleted, before deleting it)
  • ✅ Possibly expand the copy-paste API call section with a few more placeholders and comments for easy maintenance script creation

Tested by

  • Classic298 (sqlite / chromaDB)
  • robmurrer (sqlite / chromaDB)
  • spammenotinoz (PostgreSQL / ?)
  • mahenning (? / chromaDB)

Vector Database Integration Status:

  • ✅ ChromaDB - Complete with deep cleanup breakthrough
  • ✅ PGVector - Complete implementation ready for community testing
  • 🔧 Milvus, Pinecone, Qdrant, etc. - Framework ready for community contributions

Major Breakthroughs Achieved:

  • 🎯 ChromaDB File Size Issue - Solved @mahenning's 2.2GB → 156KB reduction
  • 🎯 Modular Vector Framework - Community-extensible architecture complete
  • 🎯 PGVector Integration - Full support using @recrudesce's "super easy" approach
  • 🎯 Dry-Run Preview System - Complete modal with detailed breakdown
  • 🎯 Time-Based User Management - Inactive account cleanup with smart exemptions

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/16520 **Author:** [@Classic298](https://github.com/Classic298) **Created:** 8/12/2025 **Status:** ❌ Closed **Base:** `dev` ← **Head:** `universal_file_deletion` --- ### 📝 Commits (10+) - [`d454e6a`](https://github.com/open-webui/open-webui/commit/d454e6a03359155a10fd6e8305f1a640945206ea) Feat/prune orphaned data (#16) - [`aadb296`](https://github.com/open-webui/open-webui/commit/aadb296577156668f065066a4544fe40ba6d4e8d) Merge branch 'open-webui:main' into universal_file_deletion - [`028a2e5`](https://github.com/open-webui/open-webui/commit/028a2e598497f4f28d0b583a309911af0f17dc8f) Update prune.py - [`0bd42e5`](https://github.com/open-webui/open-webui/commit/0bd42e5c6d93d2bea2930041636124148a8b47d0) Update Database.svelte - [`5ce002d`](https://github.com/open-webui/open-webui/commit/5ce002d5b3745f3eeb46cd614897d4f9a0efc6f8) Update PruneDataDialog.svelte - [`8d7273a`](https://github.com/open-webui/open-webui/commit/8d7273afaeb64e144b3cf91a26d2553df4db405a) Update prune.ts - [`e4a0bd8`](https://github.com/open-webui/open-webui/commit/e4a0bd86405d9eb7ba613e3401c221d9733ab35b) Update Database.svelte - [`60edac6`](https://github.com/open-webui/open-webui/commit/60edac6c3f47e453414dd09feaa968097163a7f1) Update Database.svelte - [`709c852`](https://github.com/open-webui/open-webui/commit/709c852917ca3e03c9af7434460943eee3508f69) Update prune.py - [`34c9a88`](https://github.com/open-webui/open-webui/commit/34c9a8825cf3802318c73829a569eb57780ab352) Update prune.py ### 📊 Changes **6 files changed** (+3015 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/main.py` (+2 -0) 📝 `backend/open_webui/models/folders.py` (+6 -0) ➕ `backend/open_webui/routers/prune.py` (+1793 -0) ➕ `src/lib/apis/prune.ts` (+66 -0) 📝 `src/lib/components/admin/Settings/Database.svelte` (+249 -0) ➕ `src/lib/components/common/PruneDataDialog.svelte` (+899 -0) </details> ### 📄 Description # 🚀 feat: COMPREHENSIVE DATA PRUNING SYSTEM - The Ultimate Storage Management Solution for Open WebUI **Before submitting, make sure you've checked the following:** - [x] **Target branch:** Please verify that the pull request targets the `dev` branch. - [x] **Description:** Provide a concise description of the changes made in this pull request. - [x] **Changelog:** Ensure a changelog entry following the format of [Keep a Changelog](https://keepachangelog.com/) is added at the bottom of the PR description. - [x] **Documentation:** Have you updated relevant documentation [Open WebUI Docs](https://github.com/open-webui/docs), or other documentation sources? - [x] **Dependencies:** Are there any new dependencies? Have you updated the dependency versions in the documentation? - [x] **Testing:** Have you written and run sufficient tests to validate the changes? - [x] **Code review:** Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards? - [x] **Prefix:** To clearly categorize this pull request, prefix the pull request title using one of the following: - **feat**: Introduces a new feature or enhancement to the codebase --- ## 🎯 **COMPREHENSIVE DATA MANAGEMENT SOLUTION** After **MONTHS of development** and addressing **way over 20+ community issues**, this PR introduces a complete data pruning system for Open WebUI. This implementation has been carefully designed over **multiple months** to address the most requested feature in the Open WebUI community - comprehensive storage management and cleanup capabilities. --- ## 🎉 **ADDRESSES 25+ COMMUNITY ISSUES & DISCUSSIONS** This implementation closes/addresses: **Primary Issues / PRs / Discussions Resolved:** - Addresses #13001 - feat: Add automatic chat record cleanup feature - Addresses #11582 - issue: Deleting a chat does not delete files uploaded in it - Addresses #13396 - WIP: add periodic data cleanup task - Addresses #10603 - "Delete All Chats" actually deletes ALL chats - Addresses #13551 - Issue: Audio data submitted via notes is not being deleted - Addresses #8902 - Many orphaned files in system after removed from knowledge base - Addresses #8862 - Introduce a Housekeeping Background Job - Addresses #5199 - Feature request: set retention for uploaded files - Addresses #6705 - Request for Transparency and Management Options for Audio File Retention - Addresses #7465 - Feature Request: Data Cleanup Policy - Addresses #12091 - Best practices for cleanup or how to avoid infinitely growing database - Addresses #12280 - feat: OPTIONAL deletion of files and db entries on chat deletion - Addresses #5468 - When the uploaded files will be deleted? - Addresses #13756 - feat: Avoid duplicate files on storage backend - Addresses #13869 - Button in the admin to delete all unused files - Addresses #8875 - Missing functionality for cleanup - Addresses #4580 - Automated deletion of old chats in history - Addresses #14129 - Enhancement: Add configurable retention and storage location for meeting audio notes - Addresses #13718 - issue: When deleting a knowledge base, it does not delete the collections corresponding to individual documents - (Kind-of) Addresses #10679 - Batch add file to knowledge doesn't check for existence - Addresses #12249 - issue: The Size of the webui.db File Continuously Increasing - Addresses #6935 - Chromadb warns that my database needs to be vacuumed - Addresses #7181 - File deletion doesn't properly clean up database entries, causing issues with re-uploads - Addresses #4035 - [bug/perf] large db, slow query to /chats/ - Addresses #17065 - Files not getting deleted from Chroma DB - Addresses #17313 - https://github.com/open-webui/open-webui/discussions/3729 **And definitely many more - in fact, i lost track of some of the discussions in my notifications. Also there were PLENTY of feature requests, bug reports and discussions around this topic on the official Discord Server. If I had to guess, there were at least 40 real requests and discussions around this.** --- ## 🛡️ **PRESERVES EXISTING BEHAVIOR & FOLLOWS MAINTAINER VISION** ### **🎯 EXACTLY THE APPROACH REQUESTED** This implementation follows the **API endpoint + manual trigger approach** outlined by **@tjbck** (maintainer) in previous discussions: - ✅ **OPTIONAL feature** - Completely optional and fully configurable to a minimum - ✅ **API endpoint for automation** - Manual execution only - ✅ **NO changes to existing behavior** - Files retained by default - ✅ **Admin-only access** - Users cannot trigger pruning - ✅ **Explicit execution required** - Nothing deleted without admin action ### **🔒 WHY THIS APPROACH WAS CHOSEN** Previous PRs proposing **automated background deletion** were correctly rejected because: - ❌ **Changes existing behavior** - Users expect files to persist - ❌ **Violates regulatory requirements** - Some jurisdictions or compliance and audit topics REQUIRE data retention (SEC financial records for 7 years, HIPAA medical documentation for 6+ years, legal discovery holds indefinitely, tax documentation for 3-7 years, employment records for 1-4 years, environmental compliance documentation for decades, tax records, employment law, environmental regulations - often 3-30+ years) - ❌ **Removes admin control** - Full automation by default can cause unexpected or unintended data loss (e.g. PRs that attempted to tie the file and vector deletion to the manual chat deletion by the user) - ❌ **One-size-fits-all problems** - Different organizations have different needs for what should get deleted, if at all, and when (automation perhaps not feasible, manual configuration is urgently needed). **This PR respects these constraints** by providing **optional, manual, admin-only, API-driven and FULLY CONFIGURABLE** data pruning. --- ## 🌟 **KEY DESIGN PRINCIPLES** ### **🔒 SAFETY FIRST** - **Optional activation** - You do not have to use the pruning feature at all, and if you do, you can enable or disable optional aspects - **Admin-only access** - Regular users cannot trigger pruning at all - **Manual execution** - No automated deletion without explicit admin action via API call or admin interface interaction - **Comprehensive warnings** - Clear documentation of destructive operations directly in the UI - docs to follow. - **Granular control** - Individual toggles for every cleanup type - fully configurable what to delete, including retention policies for automated deletion capabilities via external API call. ### **🌍 REGULATORY COMPLIANCE** - **Data retention flexibility** - Supports organizations that MUST retain data (these companies proceed to not using this feature) - **GDPR compliance tools** - For organizations that need data minimization - **Audit trail logging** - Complete operation documentation - **Configurable policies** - Adaptable to any regulatory requirement ### **🤖 AUTOMATION READY** - **API endpoint** (`/api/v1/prune`) for external scripts and automated calling - **Copy-paste automation** - Real-time API call generator - Open the menu in the admin panel, configure your policy and copy the finished API call to your desire! - **Cron-job friendly** - Perfect for scheduled maintenance - **Enterprise integration** - Fits existing infrastructure --- ## 🚀 **COMPREHENSIVE FEATURE SET** ### **👥 ADVANCED USER MANAGEMENT** - ✅ **Time-based inactive user deletion** - Configurable retention periods for inactive accounts - ✅ **Smart exemptions** - Protect admin users and pending approvals - ✅ **Activity tracking** - Based on `last_active_at` timestamps for accurate detection - ✅ **Cascade deletion** - Automatically removes all user-associated data (chats, files, knowledge bases, etc.) - ✅ **Safety guards** - Strong defaults (90+ day minimum) with exemptions for critical accounts - ✅ **Preview capability** - Dry-run mode shows exactly which users would be affected ### **💬 ADVANCED CHAT MANAGEMENT** - ✅ **Age-based chat deletion** using `updated_at` timestamps - ✅ **Smart exemptions** for archived chats - ✅ **Folder protection** - keeps organized chats safe - ✅ **Pin preservation** - respects user-pinned conversations - ✅ **Orphaned chat cleanup** from deleted users ### **📁 COMPREHENSIVE FILE SYSTEM INTEGRATION** - ✅ **Generated images** - Already handled via file system! 🎨 - ✅ **Uploaded documents** - Complete reference tracking - ✅ **Orphaned file detection** - Scans uploads directory - ✅ **Vector collection cleanup** - Removes unused embeddings - ✅ **Knowledge base synchronization** - Maintains data integrity - will delete files that failed the uploading procedure into a knowledge base (due to failed content extraction, failed vectorization or due to "duplicate content"). ### **🎵 AUDIO CACHE MANAGEMENT** - ✅ **TTS file cleanup** - Manages text-to-speech generated audio - ✅ **STT transcription cleanup** - Handles speech-to-text files - ✅ **Configurable retention** - Age-based audio file management - ✅ **Storage reclamation** - Reports cleaned space in MB ### **🗄️ DATABASE OPTIMIZATION** - ✅ **SQLite VACUUM** - Optimizes main database - will free up Megabytes or even Gigabytes of storage - noticable speed and performance improvements as a result - ✅ **PostgreSQL support** - Works with PostgreSQL as well - ✅ **ChromaDB cleanup** - **ENHANCED** Vector database optimization with **deep cleanup** - Vector DB will shrink dramatically - ✅ **Metadata synchronization** - Keeps everything consistent ### **👥 USER & RESOURCE MANAGEMENT** - ✅ **Orphaned user content cleanup** across 8 resource types: - 🗂️ Folders, 📝 Notes, 🛠️ Tools, ⚙️ Functions - 💬 Prompts, 🤖 Models, 📚 Knowledge Bases, 📁 Files - Fully configurable ### **🎛️ ADVANCED CONFIGURATION** - ✅ **Tabbed interface** - Users, Chats, Workspace, Audio Cache - ✅ **Toggle controls** - Enable/disable any feature - ✅ **Real-time API preview** - Copy-paste automation scripts - ✅ **Production-ready defaults** - Safe out-of-the-box settings - ✅ **Dry-run preview** - See exactly what will be deleted before execution --- ## 🔧 **TECHNICAL IMPLEMENTATION** ### **🏗️ MULTI-STAGE PROCESSING ARCHITECTURE** 1. **Inactive User Management** - Time-based account cleanup with exemptions 2. **Smart Chat Deletion** - Age-based with exemptions 3. **Preservation Set Building** - Identifies all actively referenced data 4. **Orphaned Record Cleanup** - Safely removes database entries 5. **Physical File Synchronization** - Cleans actual storage 6. **Audio Cache Management** - Handles TTS/STT files 7. **Database Optimization** - VACUUM operations for peak performance ### **🏭 MODULAR VECTOR DATABASE FRAMEWORK** ```python # Extensible architecture for community contributions: class VectorDatabaseCleaner(ABC): # Abstract interface for all vector databases class ChromaDatabaseCleaner(VectorDatabaseCleaner): # Full ChromaDB implementation with deep cleanup class PGVectorDatabaseCleaner(VectorDatabaseCleaner): # Complete PGVector implementation (community-ready) def get_vector_database_cleaner() -> VectorDatabaseCleaner: # Factory pattern for automatic detection ``` ### **🔧 CODE QUALITY IMPROVEMENTS** ```python class JSONFileIDExtractor: # Extracted duplicate regex patterns into reusable utility # Compiles patterns once for better performance # Centralized validation logic ``` ### **🔍 INTELLIGENT FILE SCANNING** ```python # Scans ALL data sources for file references: # ✅ Knowledge base data structures # ✅ Chat message JSON content # ✅ Folder items and metadata # ✅ Standalone message table # ✅ URL pattern matching # ✅ Database validation ``` ### **🧠 ENHANCED VECTOR CLEANUP** ```python # ChromaDB Deep Cleanup (fixes @mahenning's 2.2GB → 156KB issue): # ✅ Orphaned embeddings cascade deletion # ✅ Orphaned metadata cleanup # ✅ Full-text search (FTS) selective rebuild # ✅ Collection/segment metadata synchronization # ✅ Proper VACUUM execution for space reclamation # PGVector Integration: # ✅ Uses existing client methods for reliability # ✅ PostgreSQL-optimized VACUUM ANALYZE # ✅ Collection discovery and cleanup ``` ### **🎵 AUDIO CACHE INTELLIGENCE** ```python # Comprehensive audio management: # ✅ TTS file age-based cleanup # ✅ STT transcription management # ✅ Metadata file synchronization # ✅ Storage space reporting ``` ### **🔍 DRY-RUN PREVIEW SYSTEM** ```python # Complete preview capabilities: # ✅ Count inactive users before deletion # ✅ Count old chats with exemption rules # ✅ Count orphaned records across all resource types # ✅ Count orphaned uploads and vector collections # ✅ Count audio cache files for cleanup # ✅ Real-time preview modal with detailed breakdown ``` --- ## **⚡ OPTIMIZATION BENEFITS** - **Database size reduction** - VACUUM operations reclaim unused space (Gigabytes in some cases) - **Storage reclamation** - Removes orphaned files and collections, saving space - **Performance improvement** - Cleaner databases run faster, less file handles, less unused references; less time for operations means faster performance - **Cost savings** - Reduced storage requirements and noticable speedup - **Vector database efficiency** - ChromaDB files reduce from 2.2GB+ to ~156KB (system tables only) --- ## 🎨 **USER EXPERIENCE** ### **🖼️ BEAUTIFUL INTERFACE** - **Modern, tabbed design** - Now with Users tab - **Expandable help sections** with comprehensive documentation - **Progress feedback** and success notifications, incl. logging to CLI - **Built-in help system** explaining every feature and what the pruning feature does - **Visual API configurator and preview** for easily creating API calls tailored to your needs to creating automated scripts - **Real-time dry-run preview** - See exact counts before execution --- ## 🚧 **FUTURE DEVELOPMENT OPPORTUNITIES** ### **🗄️ VECTOR DATABASE SUPPORT** **Currently Implemented:** - ✅ **ChromaDB** - Full cleanup and optimization support with **deep cleanup breakthrough** - ✅ **PGVector** - Complete implementation using existing client methods **Community Extension Framework Ready:** - 🔧 **Milvus** - Modular architecture ready for implementation - 🔧 **Pinecone** - API structure established - 🔧 **Qdrant** - Factory pattern supports easy addition - 🔧 **Elasticsearch** - Framework ready for community contribution - 🔧 **OpenSearch** - Modular design supports integration - 🔧 **Oracle23AI** - Extension pattern available **Adding New Vector Databases:** ```python # Community contributors can easily add support by: # 1. Extending VectorDatabaseCleaner abstract class # 2. Implementing 3 required methods (count, cleanup, delete) # 3. Adding detection logic to factory function # Framework handles all integration automatically! ``` --- # Changelog Entry ### Description **COMPREHENSIVE DATA PRUNING SYSTEM** - A production-ready, enterprise-grade pruning system developed over multiple months to address 20+ community issues. This optional, admin-controlled feature includes intelligent chat deletion, **time-based user account management**, comprehensive file cleanup, audio cache management, **enhanced vector database optimization with modular framework**, **dry-run preview capabilities**, and full GDPR compliance capabilities while preserving all existing behavior. ### Added - **🎛️ Complete Admin Pruning Interface** - Beautiful tabbed UI with docs, explanations and granular configuration controls - **👥 Time-Based Inactive User Management** - Configurable deletion of inactive accounts with smart exemptions and cascade cleanup - **🔍 Dry-Run Preview System** - Complete preview modal showing exact counts of what will be deleted before execution - **🏭 Modular Vector Database Framework** - Extensible architecture supporting ChromaDB, PGVector, and community extensions - **🗄️ Enhanced ChromaDB Cleanup** - Deep cleanup solving the 2.2GB+ file size issue (reduces to ~156KB) - **🗄️ Complete PGVector Integration** - Full support using existing client methods for reliability - **🔧 Code Quality Improvements** - Extracted duplicate patterns, optimized regex compilation, centralized validation - **📅 Smart Chat Age Management** - Configurable deletion with optional and configurable archive/folder/pin exemptions - **📁 Comprehensive File System Integration** - Complete orphaned file detection and cleanup (even for unindexed files) - **🎵 Audio Cache Management System** - TTS/STT file cleanup with configurable retention - **👥 Orphaned User Content Cleanup** - 8 resource types with individual toggles and granular control - **🤖 API Automation Endpoint** - `/api/v1/prune` for external automated script-based integrations - **📋 Enhanced API Preview Generator** - API call configurator with extensive comments, cron examples, and automation best practices - **🛡️ Multi-stage Safety Processing** - Ground truth preservation with state synchronization - **🌍 GDPR Compliance Tools** - Optional data minimization and retention policy enforcement - **📊 Comprehensive Logging** - Detailed operation reporting - **⚡ Database Performance Optimization** - VACUUM operations for SQLite, Chroma and PostgreSQL leading to major database performance gains ### Deprecated / Changed - None - All changes are additive and backward compatible ### Removed - None - All changes are additive and backward compatible ### Fixed - **🐛 ChromaDB File Size Issue** - Fixed @mahenning's reported issue where ChromaDB files remained 2.2GB+ after cleanup - **🐛 Vector Database Cleanup** - Comprehensive orphaned record cleanup that ChromaDB's delete_collection() method missed - **🔧 Code Duplication** - Extracted and centralized duplicate regex patterns for better maintainability ### Security - **🔒 Admin-only API Access** - Pruning endpoint restricted to administrators only - **🛡️ Multi-level Validation** - Comprehensive safety checks before any deletion - **🌍 GDPR Compliance** - Optional data minimization and retention policy enforcement - **🔐 Inactive User Security** - Safe removal of long-inactive accounts with admin/pending exemptions --- ### Additional Information #### **🎯 ADDRESSES COMMUNITY PAIN POINTS** This PR addresses years of community feedback about: - Runaway database growth affecting very large user instances - GDPR compliance concerns for EU deployments - Orphaned files consuming terabytes of storage - Audio retention violating confidentiality requirements - Manual SQL surgery being the only cleanup option - Vector databases growing to 200GB+ with zero active chats - **Inactive user accounts accumulating over time** - **ChromaDB databases not properly shrinking after cleanup** ### Screenshots or Videos #### Admin Panel - Database Section <img width="2281" height="905" alt="image" src="https://github.com/user-attachments/assets/ee11a42b-db70-460b-bcfb-f007b0ae6ed6" /> <img width="1085" height="791" alt="image" src="https://github.com/user-attachments/assets/10f57313-c01b-4701-92af-fbba0f5762ee" /> #### Admin Panel - Prune Modal <img width="1320" height="1092" alt="image" src="https://github.com/user-attachments/assets/1e98b416-7f7d-4c62-b9e7-673922189fd0" /> <img width="948" height="959" alt="image" src="https://github.com/user-attachments/assets/d37e1a8a-188a-40dc-b0fc-57028667bab1" /> #### Admin Panel - Prune Modal Docs <img width="907" height="496" alt="image" src="https://github.com/user-attachments/assets/f466904c-26a0-48c1-973f-0a22e97fd6ec" /> <img width="894" height="457" alt="image" src="https://github.com/user-attachments/assets/7ff7b891-e560-4633-8363-ec5d2c9dfe3e" /> <img width="879" height="382" alt="image" src="https://github.com/user-attachments/assets/5dfa62a5-a351-4f29-9654-a5e76705bdae" /> <img width="897" height="372" alt="image" src="https://github.com/user-attachments/assets/051a824b-278f-40d8-ad91-04c52ce0707a" /> <img width="871" height="392" alt="image" src="https://github.com/user-attachments/assets/f547d2b4-7524-411a-bb5f-c41407bc303f" /> #### Admin Panel - Prune Modal Config <img width="896" height="608" alt="image" src="https://github.com/user-attachments/assets/ade5e9ba-8f40-45ee-bfe7-f4bc3fb07b1e" /> <img width="889" height="343" alt="image" src="https://github.com/user-attachments/assets/b73b1d6c-c6a6-4a5d-b529-6e08f5d9d61d" /> <img width="470" height="367" alt="image" src="https://github.com/user-attachments/assets/61c306d9-e87d-4544-9e8b-5c3296f5b1a3" /> <img width="883" height="453" alt="image" src="https://github.com/user-attachments/assets/7314187c-9d51-4deb-8ed5-905b5512bfc6" /> #### Admin Panel - Inactive User Management Tab <img width="2143" height="1254" alt="image" src="https://github.com/user-attachments/assets/6aae4680-9079-4ff5-8477-3c34927f0a0f" /> #### Dry-Run Preview Modal <img width="740" height="294" alt="image" src="https://github.com/user-attachments/assets/c47c50cb-eb2f-432e-a841-72b38db981bd" /> <img width="731" height="355" alt="image" src="https://github.com/user-attachments/assets/d842c0e3-a69a-481b-9400-a70dd5be4560" /> #### Admin Panel - Prune Modal API helper <img width="897" height="323" alt="image" src="https://github.com/user-attachments/assets/3e3dc2df-9042-44a9-bbe8-5f19a78ca429" /> Shows the API call, fully configured according to the selections and settings you set in the configurator above. Useful for external pruning automation. #### API Helper with Advanced Comments Example: ``` # Open WebUI Data Pruning API Call # Use this template for automated maintenance scripts (cron jobs, etc.) # AUTHENTICATION: Use API Key (not JWT token) for automation # Get your API key from: Settings → Account → API Key → Generate new key # Format: sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx curl -X POST "http://localhost:5173/api/v1/prune/" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer <your-api-key>" \ -d '{ // SAFETY: Always test with dry_run=true first to preview results "dry_run": false, // AGE-BASED CHAT DELETION (null = disabled) "days": 0, "exempt_archived_chats": false, // Keep archived chats even if old "exempt_chats_in_folders": false, // Keep organized/pinned chats // INACTIVE USER DELETION (null = disabled, VERY DESTRUCTIVE) "delete_inactive_users_days": 90, "exempt_admin_users": true, // Strongly recommended: true "exempt_pending_users": true, // Recommended for user approval workflows // ORPHANED DATA CLEANUP (from deleted users) "delete_orphaned_chats": true, "delete_orphaned_tools": true, "delete_orphaned_functions": true, // Actions, Pipes, Filters "delete_orphaned_prompts": true, "delete_orphaned_knowledge_bases": true, "delete_orphaned_models": true, "delete_orphaned_notes": true, "delete_orphaned_folders": true, // AUDIO CACHE CLEANUP (null = disabled) "audio_cache_max_age_days": 30 // TTS/STT files }' # API KEY vs JWT TOKEN: # - API Key: Persistent, use for automation (sk-xxxxxxxx...) # - JWT Token: Session-bound, temporary, use for web UI only # - ALWAYS use API Key for scripts/cron jobs # AUTOMATION TIPS: # 1. Run with dry_run=true first to preview what will be deleted # 2. Schedule during low-usage hours to minimize performance impact # 3. Monitor logs: tail -f /path/to/open-webui/logs # 4. Consider database backup before large cleanup operations # 5. Test on staging environment with similar data size first # EXAMPLE CRON JOB (runs weekly on Sunday at 2 AM): # 0 2 * * 0 /path/to/your/prune-script.sh >> /var/log/openwebui-prune.log 2>&1 # RESPONSE HANDLING: # - dry_run=true: Returns counts object with preview numbers # - dry_run=false: Returns true on success, throws error on failure # - Always check HTTP status code and response for errors ``` <img width="420" height="106" alt="image" src="https://github.com/user-attachments/assets/d6502b71-b0e7-414c-8f04-fcbbcbc8c8c6" /> Confirmation of prune success #### Info Level Logging <img width="988" height="258" alt="image" src="https://github.com/user-attachments/assets/8924f9a1-2944-43ae-b447-2413f023e4a7" /> --- ### Contributor License Agreement By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. --- # User Feedback Tracking Thanks for your feedback and for testing the PR. This section of the PR description will be continuously updated to keep track of the last remaining points **Feature Wishes / To Do** - [x] ✅ **Implement feature to optionally delete long inactive accounts (configurable)** - [x] ✅ **Investigate modular architecture for other vector DB integrations** - [x] ✅ **Attempt to integrate pgvector** - [x] ✅ **Test pgvector integration** - [x] ✅ **Improve chromaDB integration** - [x] ✅ **Proper database vacuum, current implementation doesn't fully vacuum chromaDB** - ~~attempt to simplify implementation with .delete command (needs investigation if UUID matching still works, since chroma DB and the files itself and the file handles in Open WebUI's database have different UUID's each, requiring complex cross matching to even make it work in the first place)~~ The amount of tinkering that is necessary to fully cleanup chroma db does not allow for this to be easy lol. - [x] ✅ **Extract Duplicate Regex Patterns and remove duplicates, simplifying the code a little bit** - [x] ✅ **If possible, add a dry-run function (to preview what would get deleted, before deleting it)** - [x] ✅ **Possibly expand the copy-paste API call section with a few more placeholders and comments for easy maintenance script creation** **Tested by** - Classic298 (sqlite / chromaDB) - robmurrer (sqlite / chromaDB) - spammenotinoz (PostgreSQL / ?) - mahenning (? / chromaDB) **Vector Database Integration Status:** - ✅ **ChromaDB** - Complete with deep cleanup breakthrough - ✅ **PGVector** - Complete implementation ready for community testing - 🔧 **Milvus, Pinecone, Qdrant, etc.** - Framework ready for community contributions **Major Breakthroughs Achieved:** - 🎯 **ChromaDB File Size Issue** - Solved @mahenning's 2.2GB → 156KB reduction - 🎯 **Modular Vector Framework** - Community-extensible architecture complete - 🎯 **PGVector Integration** - Full support using @recrudesce's "super easy" approach - 🎯 **Dry-Run Preview System** - Complete modal with detailed breakdown - 🎯 **Time-Based User Management** - Inactive account cleanup with smart exemptions --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-06 07:31:50 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#63016
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?