github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-05-06 10:58:17 -05:00
Code Issues 1.2k Packages Projects Releases 126 Wiki Activity

[PR #20966] feat: Smart sync for knowledge base directory uploads #48902

New Issue
Open
opened 2026-04-30 01:06:43 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-04-30 01:06:43 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/20966
Author: @Classic298
Created: 1/27/2026
Status: 🔄 Open

Base: dev ← Head: sync

📝 Commits (10+)

  • 6b2c927 feat: Smart sync for knowledge base directory uploads (#146)
  • 4c58134 fixes
  • 1cfd0fb fix form for upload file handler
  • 689f56b fix brittle form check for code QL
  • 0c61f0b fix: use size-based fallback for legacy files to prevent sync timeout
  • a22aaec fix: add separate original_path field for directory sync
  • d68d08b fix: sync compare syntax, replace scoping, and API error surfacing
  • a7cdc4d fix: await process_file, tighten replace validation, handle dup paths and network errors
  • 86e6e23 fix: close storage leaks, surface per-file sync failures, drop unused imports
  • 74163ce fix: purge orphan KB vectors, route sync deletes through file delete, reject duplicate sync paths

📊 Changes

7 files changed (+1087 additions, -191 deletions)

View changed files

📝 backend/open_webui/routers/files.py (+18 -3)
📝 backend/open_webui/routers/knowledge.py (+477 -14)
📝 backend/open_webui/routers/retrieval.py (+0 -1)
📝 backend/open_webui/utils/misc.py (+5 -0)
📝 src/lib/apis/knowledge/index.ts (+103 -0)
📝 src/lib/components/workspace/Knowledge/KnowledgeBase.svelte (+483 -172)
📝 src/lib/components/workspace/Knowledge/KnowledgeBase/AddContentMenu.svelte (+1 -1)

📄 Description

Co-authored-by: Stoyan Zlatev 47269030+Stoyan-Zlatev@users.noreply.github.com

Summary

Refactors the "Sync Directory" feature for knowledge bases to intelligently compare files instead of deleting everything and re-uploading. Previously, syncing a directory would reset the entire knowledge base and re-upload all files - expensive and slow for large knowledge bases where only a few files changed.

What Changed

Smart Comparison

When syncing a directory, the browser now calculates SHA-256 hashes for all local files and sends them to a new comparison endpoint. The server compares these against existing files in the knowledge base and returns exactly what needs to happen:

  • New files: Files that don't exist in the knowledge base yet
  • Changed files: Files where content has changed (hash mismatch)
  • Removed files: Files in the knowledge base that no longer exist in the directory
  • Unchanged files: Files that are identical (skipped entirely)

Atomic File Replacement

For changed files, a new atomic endpoint handles the entire replacement in a single server-side transaction:

  1. Upload new file
  2. Process and extract content
  3. Generate embeddings and store in vector database
  4. Add to knowledge base
  5. Delete old file (vectors, database record, storage)

The old file is only deleted after the new file is fully processed and embedded. If any step fails, the old file remains intact.

Backwards Compatibility

Files uploaded before this feature don't have stored hashes. For these legacy files:

  • Modern files (with stored hash): Uses accurate SHA-256 hash comparison
  • Legacy files (without stored hash): Falls back to file size comparison

This avoids expensive I/O operations (downloading from storage + hashing) during sync comparison, preventing timeouts for large knowledge bases. When legacy files are re-uploaded (either as new or changed), they receive stored hashes for accurate future comparisons.

Why This Approach

Interrupted Sync Safety

Each file operation is atomic. If the browser disconnects mid-sync:

  • Completed files are fully synced (old version deleted)
  • Incomplete files remain unchanged (old version preserved)
  • No partial state, no duplicates, no data loss

Performance

  • Unchanged files are skipped entirely (no re-upload, no re-embedding)
  • Only processes what actually changed
  • Hash comparison is fast (calculated in browser using Web Crypto API)
  • Legacy file comparison uses stored metadata (no I/O required)

User Experience

  • Clear progress: "3 new, 2 updated, 1 removed, 45 unchanged"
  • No confirmation needed for "delete all" since it's non-destructive
  • Tooltip explains the smart sync behavior

Endpoints Added

  • POST /knowledge/{id}/sync/compare - Compare file hashes, return sync plan
  • POST /knowledge/{id}/file/upload_and_replace - Atomic file replacement

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/20966 **Author:** [@Classic298](https://github.com/Classic298) **Created:** 1/27/2026 **Status:** 🔄 Open **Base:** `dev` ← **Head:** `sync` --- ### 📝 Commits (10+) - [`6b2c927`](https://github.com/open-webui/open-webui/commit/6b2c927828bc5986610677c9c5b7ffa79b1b5846) feat: Smart sync for knowledge base directory uploads (#146) - [`4c58134`](https://github.com/open-webui/open-webui/commit/4c581349bd395457e4d06731a13b191bab507ae5) fixes - [`1cfd0fb`](https://github.com/open-webui/open-webui/commit/1cfd0fb1ae75469d25485a7b2c2c043a3376eebf) fix form for upload file handler - [`689f56b`](https://github.com/open-webui/open-webui/commit/689f56b837562f49716aa848cc06a0c6c6084b4f) fix brittle form check for code QL - [`0c61f0b`](https://github.com/open-webui/open-webui/commit/0c61f0b9daa6171a23735bc969834cdea1473379) fix: use size-based fallback for legacy files to prevent sync timeout - [`a22aaec`](https://github.com/open-webui/open-webui/commit/a22aaecd4590470fb12d2d65f20e3e97753e8538) fix: add separate original_path field for directory sync - [`d68d08b`](https://github.com/open-webui/open-webui/commit/d68d08b67857da0e90804f3a65cab269ea4e19f8) fix: sync compare syntax, replace scoping, and API error surfacing - [`a7cdc4d`](https://github.com/open-webui/open-webui/commit/a7cdc4d1181e216c98529fcddf0b2d08bf836ecb) fix: await process_file, tighten replace validation, handle dup paths and network errors - [`86e6e23`](https://github.com/open-webui/open-webui/commit/86e6e239a46b9e99fe489b86d8af83ed55a4815d) fix: close storage leaks, surface per-file sync failures, drop unused imports - [`74163ce`](https://github.com/open-webui/open-webui/commit/74163ce9635f850244cf0675f8db562ce1e5263a) fix: purge orphan KB vectors, route sync deletes through file delete, reject duplicate sync paths ### 📊 Changes **7 files changed** (+1087 additions, -191 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/routers/files.py` (+18 -3) 📝 `backend/open_webui/routers/knowledge.py` (+477 -14) 📝 `backend/open_webui/routers/retrieval.py` (+0 -1) 📝 `backend/open_webui/utils/misc.py` (+5 -0) 📝 `src/lib/apis/knowledge/index.ts` (+103 -0) 📝 `src/lib/components/workspace/Knowledge/KnowledgeBase.svelte` (+483 -172) 📝 `src/lib/components/workspace/Knowledge/KnowledgeBase/AddContentMenu.svelte` (+1 -1) </details> ### 📄 Description Co-authored-by: Stoyan Zlatev <47269030+Stoyan-Zlatev@users.noreply.github.com> ## Summary Refactors the "Sync Directory" feature for knowledge bases to intelligently compare files instead of deleting everything and re-uploading. Previously, syncing a directory would reset the entire knowledge base and re-upload all files - expensive and slow for large knowledge bases where only a few files changed. ## What Changed ### Smart Comparison When syncing a directory, the browser now calculates SHA-256 hashes for all local files and sends them to a new comparison endpoint. The server compares these against existing files in the knowledge base and returns exactly what needs to happen: - **New files**: Files that don't exist in the knowledge base yet - **Changed files**: Files where content has changed (hash mismatch) - **Removed files**: Files in the knowledge base that no longer exist in the directory - **Unchanged files**: Files that are identical (skipped entirely) ### Atomic File Replacement For changed files, a new atomic endpoint handles the entire replacement in a single server-side transaction: 1. Upload new file 2. Process and extract content 3. Generate embeddings and store in vector database 4. Add to knowledge base 5. Delete old file (vectors, database record, storage) The old file is **only deleted after the new file is fully processed and embedded**. If any step fails, the old file remains intact. ### Backwards Compatibility Files uploaded before this feature don't have stored hashes. For these legacy files: - **Modern files** (with stored hash): Uses accurate SHA-256 hash comparison - **Legacy files** (without stored hash): Falls back to file size comparison This avoids expensive I/O operations (downloading from storage + hashing) during sync comparison, preventing timeouts for large knowledge bases. When legacy files are re-uploaded (either as new or changed), they receive stored hashes for accurate future comparisons. ## Why This Approach ### Interrupted Sync Safety Each file operation is atomic. If the browser disconnects mid-sync: - Completed files are fully synced (old version deleted) - Incomplete files remain unchanged (old version preserved) - No partial state, no duplicates, no data loss ### Performance - Unchanged files are skipped entirely (no re-upload, no re-embedding) - Only processes what actually changed - Hash comparison is fast (calculated in browser using Web Crypto API) - Legacy file comparison uses stored metadata (no I/O required) ### User Experience - Clear progress: "3 new, 2 updated, 1 removed, 45 unchanged" - No confirmation needed for "delete all" since it's non-destructive - Tooltip explains the smart sync behavior ## Endpoints Added - `POST /knowledge/{id}/sync/compare` - Compare file hashes, return sync plan - `POST /knowledge/{id}/file/upload_and_replace` - Atomic file replacement ### Contributor License Agreement By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-04-30 01:06:43 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#48902
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?