Refactors the "Sync Directory" feature for knowledge bases to intelligently compare files instead of deleting everything and re-uploading. Previously, syncing a directory would reset the entire knowledge base and re-upload all files - expensive and slow for large knowledge bases where only a few files changed.
What Changed
Smart Comparison
When syncing a directory, the browser now calculates SHA-256 hashes for all local files and sends them to a new comparison endpoint. The server compares these against existing files in the knowledge base and returns exactly what needs to happen:
New files: Files that don't exist in the knowledge base yet
Changed files: Files where content has changed (hash mismatch)
Removed files: Files in the knowledge base that no longer exist in the directory
Unchanged files: Files that are identical (skipped entirely)
Atomic File Replacement
For changed files, a new atomic endpoint handles the entire replacement in a single server-side transaction:
Upload new file
Process and extract content
Generate embeddings and store in vector database
Add to knowledge base
Delete old file (vectors, database record, storage)
The old file is only deleted after the new file is fully processed and embedded. If any step fails, the old file remains intact.
Backwards Compatibility
Files uploaded before this feature don't have stored hashes. For these legacy files:
Modern files (with stored hash): Uses accurate SHA-256 hash comparison
Legacy files (without stored hash): Falls back to file size comparison
This avoids expensive I/O operations (downloading from storage + hashing) during sync comparison, preventing timeouts for large knowledge bases. When legacy files are re-uploaded (either as new or changed), they receive stored hashes for accurate future comparisons.
Why This Approach
Interrupted Sync Safety
Each file operation is atomic. If the browser disconnects mid-sync:
Completed files are fully synced (old version deleted)
Incomplete files remain unchanged (old version preserved)
No partial state, no duplicates, no data loss
Performance
Unchanged files are skipped entirely (no re-upload, no re-embedding)
Only processes what actually changed
Hash comparison is fast (calculated in browser using Web Crypto API)
Legacy file comparison uses stored metadata (no I/O required)
No confirmation needed for "delete all" since it's non-destructive
Tooltip explains the smart sync behavior
Endpoints Added
POST /knowledge/{id}/sync/compare - Compare file hashes, return sync plan
POST /knowledge/{id}/file/upload_and_replace - Atomic file replacement
Contributor License Agreement
By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.
Note
Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/open-webui/open-webui/pull/20966
**Author:** [@Classic298](https://github.com/Classic298)
**Created:** 1/27/2026
**Status:** 🔄 Open
**Base:** `dev` ← **Head:** `sync`
---
### 📝 Commits (10+)
- [`6b2c927`](https://github.com/open-webui/open-webui/commit/6b2c927828bc5986610677c9c5b7ffa79b1b5846) feat: Smart sync for knowledge base directory uploads (#146)
- [`4c58134`](https://github.com/open-webui/open-webui/commit/4c581349bd395457e4d06731a13b191bab507ae5) fixes
- [`1cfd0fb`](https://github.com/open-webui/open-webui/commit/1cfd0fb1ae75469d25485a7b2c2c043a3376eebf) fix form for upload file handler
- [`689f56b`](https://github.com/open-webui/open-webui/commit/689f56b837562f49716aa848cc06a0c6c6084b4f) fix brittle form check for code QL
- [`0c61f0b`](https://github.com/open-webui/open-webui/commit/0c61f0b9daa6171a23735bc969834cdea1473379) fix: use size-based fallback for legacy files to prevent sync timeout
- [`a22aaec`](https://github.com/open-webui/open-webui/commit/a22aaecd4590470fb12d2d65f20e3e97753e8538) fix: add separate original_path field for directory sync
- [`d68d08b`](https://github.com/open-webui/open-webui/commit/d68d08b67857da0e90804f3a65cab269ea4e19f8) fix: sync compare syntax, replace scoping, and API error surfacing
- [`a7cdc4d`](https://github.com/open-webui/open-webui/commit/a7cdc4d1181e216c98529fcddf0b2d08bf836ecb) fix: await process_file, tighten replace validation, handle dup paths and network errors
- [`86e6e23`](https://github.com/open-webui/open-webui/commit/86e6e239a46b9e99fe489b86d8af83ed55a4815d) fix: close storage leaks, surface per-file sync failures, drop unused imports
- [`74163ce`](https://github.com/open-webui/open-webui/commit/74163ce9635f850244cf0675f8db562ce1e5263a) fix: purge orphan KB vectors, route sync deletes through file delete, reject duplicate sync paths
### 📊 Changes
**7 files changed** (+1087 additions, -191 deletions)
<details>
<summary>View changed files</summary>
📝 `backend/open_webui/routers/files.py` (+18 -3)
📝 `backend/open_webui/routers/knowledge.py` (+477 -14)
📝 `backend/open_webui/routers/retrieval.py` (+0 -1)
📝 `backend/open_webui/utils/misc.py` (+5 -0)
📝 `src/lib/apis/knowledge/index.ts` (+103 -0)
📝 `src/lib/components/workspace/Knowledge/KnowledgeBase.svelte` (+483 -172)
📝 `src/lib/components/workspace/Knowledge/KnowledgeBase/AddContentMenu.svelte` (+1 -1)
</details>
### 📄 Description
Co-authored-by: Stoyan Zlatev <47269030+Stoyan-Zlatev@users.noreply.github.com>
## Summary
Refactors the "Sync Directory" feature for knowledge bases to intelligently compare files instead of deleting everything and re-uploading. Previously, syncing a directory would reset the entire knowledge base and re-upload all files - expensive and slow for large knowledge bases where only a few files changed.
## What Changed
### Smart Comparison
When syncing a directory, the browser now calculates SHA-256 hashes for all local files and sends them to a new comparison endpoint. The server compares these against existing files in the knowledge base and returns exactly what needs to happen:
- **New files**: Files that don't exist in the knowledge base yet
- **Changed files**: Files where content has changed (hash mismatch)
- **Removed files**: Files in the knowledge base that no longer exist in the directory
- **Unchanged files**: Files that are identical (skipped entirely)
### Atomic File Replacement
For changed files, a new atomic endpoint handles the entire replacement in a single server-side transaction:
1. Upload new file
2. Process and extract content
3. Generate embeddings and store in vector database
4. Add to knowledge base
5. Delete old file (vectors, database record, storage)
The old file is **only deleted after the new file is fully processed and embedded**. If any step fails, the old file remains intact.
### Backwards Compatibility
Files uploaded before this feature don't have stored hashes. For these legacy files:
- **Modern files** (with stored hash): Uses accurate SHA-256 hash comparison
- **Legacy files** (without stored hash): Falls back to file size comparison
This avoids expensive I/O operations (downloading from storage + hashing) during sync comparison, preventing timeouts for large knowledge bases. When legacy files are re-uploaded (either as new or changed), they receive stored hashes for accurate future comparisons.
## Why This Approach
### Interrupted Sync Safety
Each file operation is atomic. If the browser disconnects mid-sync:
- Completed files are fully synced (old version deleted)
- Incomplete files remain unchanged (old version preserved)
- No partial state, no duplicates, no data loss
### Performance
- Unchanged files are skipped entirely (no re-upload, no re-embedding)
- Only processes what actually changed
- Hash comparison is fast (calculated in browser using Web Crypto API)
- Legacy file comparison uses stored metadata (no I/O required)
### User Experience
- Clear progress: "3 new, 2 updated, 1 removed, 45 unchanged"
- No confirmation needed for "delete all" since it's non-destructive
- Tooltip explains the smart sync behavior
## Endpoints Added
- `POST /knowledge/{id}/sync/compare` - Compare file hashes, return sync plan
- `POST /knowledge/{id}/file/upload_and_replace` - Atomic file replacement
### Contributor License Agreement
By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms.
> [!NOTE]
> Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/open-webui/open-webui/pull/20966
Author: @Classic298
Created: 1/27/2026
Status: 🔄 Open
Base:
dev← Head:sync📝 Commits (10+)
6b2c927feat: Smart sync for knowledge base directory uploads (#146)4c58134fixes1cfd0fbfix form for upload file handler689f56bfix brittle form check for code QL0c61f0bfix: use size-based fallback for legacy files to prevent sync timeouta22aaecfix: add separate original_path field for directory syncd68d08bfix: sync compare syntax, replace scoping, and API error surfacinga7cdc4dfix: await process_file, tighten replace validation, handle dup paths and network errors86e6e23fix: close storage leaks, surface per-file sync failures, drop unused imports74163cefix: purge orphan KB vectors, route sync deletes through file delete, reject duplicate sync paths📊 Changes
7 files changed (+1087 additions, -191 deletions)
View changed files
📝
backend/open_webui/routers/files.py(+18 -3)📝
backend/open_webui/routers/knowledge.py(+477 -14)📝
backend/open_webui/routers/retrieval.py(+0 -1)📝
backend/open_webui/utils/misc.py(+5 -0)📝
src/lib/apis/knowledge/index.ts(+103 -0)📝
src/lib/components/workspace/Knowledge/KnowledgeBase.svelte(+483 -172)📝
src/lib/components/workspace/Knowledge/KnowledgeBase/AddContentMenu.svelte(+1 -1)📄 Description
Co-authored-by: Stoyan Zlatev 47269030+Stoyan-Zlatev@users.noreply.github.com
Summary
Refactors the "Sync Directory" feature for knowledge bases to intelligently compare files instead of deleting everything and re-uploading. Previously, syncing a directory would reset the entire knowledge base and re-upload all files - expensive and slow for large knowledge bases where only a few files changed.
What Changed
Smart Comparison
When syncing a directory, the browser now calculates SHA-256 hashes for all local files and sends them to a new comparison endpoint. The server compares these against existing files in the knowledge base and returns exactly what needs to happen:
Atomic File Replacement
For changed files, a new atomic endpoint handles the entire replacement in a single server-side transaction:
The old file is only deleted after the new file is fully processed and embedded. If any step fails, the old file remains intact.
Backwards Compatibility
Files uploaded before this feature don't have stored hashes. For these legacy files:
This avoids expensive I/O operations (downloading from storage + hashing) during sync comparison, preventing timeouts for large knowledge bases. When legacy files are re-uploaded (either as new or changed), they receive stored hashes for accurate future comparisons.
Why This Approach
Interrupted Sync Safety
Each file operation is atomic. If the browser disconnects mid-sync:
Performance
User Experience
Endpoints Added
POST /knowledge/{id}/sync/compare- Compare file hashes, return sync planPOST /knowledge/{id}/file/upload_and_replace- Atomic file replacementContributor License Agreement
By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.