feat: Implement Automatic Cleanup of Local Files When Using S3 Backend #5359

New Issue

2025-11-11T16:18:41-06:00

GiteaMirror commented

2025-11-11 16:18:41 -06:00

Originally created by @calebhansard on GitHub (May 27, 2025).

Check Existing Issues

I have searched the existing issues and discussions.

Problem Description

When using S3 as a backend for uploaded files in Open WebUI, a local copy is also stored for processing. Currently, there is no automatic cleanup process for these local files. This leads to unnecessary storage consumption on the host system, as the local storage continues to grow over time.

Desired Solution you'd like

Implement an automatic cleanup process for local files after they are no longer needed. This solution should:

Periodically check for files in the local storage that have not been accessed recently.
Verify that the file exists in S3 and is accessible.
Delete the local copy of the file if it meets the criteria for removal (e.g., not recently accessed, successfully stored in S3).
Ensure that local storage does not grow indefinitely while maintaining efficient access to frequently used files.

Alternatives Considered

No response

Additional Context

Environment

Open WebUI deployed in Kubernetes and backed by S3 for uploads.
Data persistence is disabled in the Helm chart.
RAG database and OpenWebUI db are external.

Related discussions and issues

Originally created by @calebhansard on GitHub (May 27, 2025). ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description When using S3 as a backend for uploaded files in Open WebUI, a local copy is also stored for processing. Currently, there is no automatic cleanup process for these local files. This leads to unnecessary storage consumption on the host system, as the local storage continues to grow over time. ### Desired Solution you'd like Implement an automatic cleanup process for local files after they are no longer needed. This solution should: 1. Periodically check for files in the local storage that have not been accessed recently. 2. Verify that the file exists in S3 and is accessible. 3. Delete the local copy of the file if it meets the criteria for removal (e.g., not recently accessed, successfully stored in S3). 4. Ensure that local storage does not grow indefinitely while maintaining efficient access to frequently used files. ### Alternatives Considered _No response_ ### Additional Context ## Environment - Open WebUI deployed in Kubernetes and backed by S3 for uploads. - Data persistence is disabled in the Helm chart. - RAG database and OpenWebUI db are external. ## Related discussions and issues - [Files Saved Both Locally and on S3 Despite Using S3 Storage Provider](https://github.com/open-webui/open-webui/discussions/9487) - [Uploaded documents also saved on local when using external storage providers?](https://github.com/open-webui/open-webui/discussions/9035) - [enh: allow to use S3 for uploaded files](https://github.com/open-webui/open-webui/issues/5763) - [Document how to use S3](https://github.com/open-webui/helm-charts/issues/151)

GiteaMirror closed this issue

2025-11-11 16:18:41 -06:00

GiteaMirror referenced this issue

2026-04-19 20:28:52 -05:00

[GH-ISSUE #5359] pdf document upload failed #13954

GiteaMirror referenced this issue

2026-04-25 03:53:03 -05:00

[GH-ISSUE #5359] pdf document upload failed #29482

GiteaMirror referenced this issue

2026-05-05 13:43:26 -05:00

[GH-ISSUE #5359] pdf document upload failed #52620

GiteaMirror referenced this issue