github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-31 10:36:16 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

S3 provider uploads files to both S3 and local storage #3888

New Issue
Closed
opened 2025-11-11 15:41:52 -06:00 by GiteaMirror · 4 comments
GiteaMirror commented 2025-11-11 15:41:52 -06:00
Owner
Copy Link
Copy Source

Originally created by @JoeChen2me on GitHub (Feb 17, 2025).

Bug Report

Important Notes

Installation Method

Docker (latest image, version 0.5.12)

Environment

  • Open WebUI Version: v0.5.12
  • Ollama (if applicable): 不适用
  • Operating System: Ubuntu 22.04
  • Browser (if applicable): Edge

Confirmation:

  • I have read and followed all the instructions provided in the README.md.
  • I am on the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.

Expected Behavior:

When using the S3 storage provider, files should only be uploaded to S3 and not stored locally, thus saving local storage space and reducing I/O operations.

Actual Behavior:

When using the S3 storage provider, files are uploaded to S3, but they are also stored locally, which consumes local storage and increases I/O operations.

Description

Bug Summary:

The S3 storage provider uploads files to S3 as expected, but it unexpectedly also saves a copy of the files locally. This behavior defeats the purpose of using cloud storage to save local disk space. The issue stems from the upload_file method in the code, which calls LocalStorageProvider.upload_file before uploading to S3, resulting in duplicate storage. Similarly, delete_file and delete_all_files also call the corresponding local storage methods.

The relevant code snippet (from the upload_file method) is:

    def upload_file(self, file: BinaryIO, filename: str) -> Tuple[bytes, str]:
        """Handles uploading of the file to S3 storage."""
        _, file_path = LocalStorageProvider.upload_file(file, filename)
        try:
            s3_key = os.path.join(self.key_prefix, filename)
            self.s3_client.upload_file(file_path, self.bucket_name, s3_key)
            return (
                open(file_path, "rb").read(),
                "s3://" + self.bucket_name + "/" + s3_key,
            )

Reproduction Details

Steps to Reproduce:

  1. Configure Open WebUI to use the S3 storage provider.
  2. Upload a file through the Open WebUI interface.
  3. Observe that the file is successfully uploaded to the configured S3 bucket.
  4. Check the local storage (where Open WebUI is running) and observe that the same file is also stored locally.

Logs and Screenshots

Browser Console Logs:

Not provided.

Docker Container Logs:

Not provided.

Screenshots/Screen Recordings (if applicable):

Not provided.

Additional Information

  • Server OS: Ubuntu 22.04
  • Browser: Edge
  • Open WebUI Installation Method: Docker (latest, v0.5.12)

The current implementation increases I/O operations unnecessarily and consumes local storage space, negating the benefits of using cloud storage. It is expected that when using S3, files should only be stored in the S3 bucket.

Note

If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!

Originally created by @JoeChen2me on GitHub (Feb 17, 2025). # Bug Report ## Important Notes --- ## Installation Method Docker (latest image, version 0.5.12) ## Environment - **Open WebUI Version:** v0.5.12 - **Ollama (if applicable):** 不适用 - **Operating System:** Ubuntu 22.04 - **Browser (if applicable):** Edge **Confirmation:** - [x] I have read and followed all the instructions provided in the README.md. - [x] I am on the latest version of both Open WebUI and Ollama. - [ ] I have included the browser console logs. - [ ] I have included the Docker container logs. - [ ] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below. ## Expected Behavior: When using the S3 storage provider, files should only be uploaded to S3 and not stored locally, thus saving local storage space and reducing I/O operations. ## Actual Behavior: When using the S3 storage provider, files are uploaded to S3, but they are *also* stored locally, which consumes local storage and increases I/O operations. ## Description **Bug Summary:** The S3 storage provider uploads files to S3 as expected, but it unexpectedly also saves a copy of the files locally. This behavior defeats the purpose of using cloud storage to save local disk space. The issue stems from the `upload_file` method in the code, which calls `LocalStorageProvider.upload_file` before uploading to S3, resulting in duplicate storage. Similarly, `delete_file` and `delete_all_files` also call the corresponding local storage methods. The relevant code snippet (from the `upload_file` method) is: ```python def upload_file(self, file: BinaryIO, filename: str) -> Tuple[bytes, str]: """Handles uploading of the file to S3 storage.""" _, file_path = LocalStorageProvider.upload_file(file, filename) try: s3_key = os.path.join(self.key_prefix, filename) self.s3_client.upload_file(file_path, self.bucket_name, s3_key) return ( open(file_path, "rb").read(), "s3://" + self.bucket_name + "/" + s3_key, ) ``` ## Reproduction Details **Steps to Reproduce:** 1. Configure Open WebUI to use the S3 storage provider. 2. Upload a file through the Open WebUI interface. 3. Observe that the file is successfully uploaded to the configured S3 bucket. 4. Check the local storage (where Open WebUI is running) and observe that the same file is also stored locally. ## Logs and Screenshots **Browser Console Logs:** Not provided. **Docker Container Logs:** Not provided. **Screenshots/Screen Recordings (if applicable):** Not provided. ## Additional Information - Server OS: Ubuntu 22.04 - Browser: Edge - Open WebUI Installation Method: Docker (latest, v0.5.12) The current implementation increases I/O operations unnecessarily and consumes local storage space, negating the benefits of using cloud storage. It is expected that when using S3, files should only be stored in the S3 bucket. ## Note If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!
GiteaMirror commented 2025-11-11 15:41:53 -06:00
Author
Owner
Copy Link
Copy Source

@tjbck commented on GitHub (Feb 17, 2025):

Working as intended here, Open WebUI still requires local copy to serve the files to the users.

@tjbck commented on GitHub (Feb 17, 2025): Working as intended here, Open WebUI still requires local copy to serve the files to the users.
GiteaMirror commented 2025-11-11 15:41:54 -06:00
Author
Owner
Copy Link
Copy Source

@JoeChen2me commented on GitHub (Feb 17, 2025):

Working as intended here, Open WebUI still requires local copy to serve the files to the users.

Why can't it just use cloud storage?
If you still save a copy of the file locally after configuring the cloud storage, it seems that the meaning of setting up the cloud storage is not obvious, will still occupy the local space.

@JoeChen2me commented on GitHub (Feb 17, 2025): > Working as intended here, Open WebUI still requires local copy to serve the files to the users. Why can't it just use cloud storage? If you still save a copy of the file locally after configuring the cloud storage, it seems that the meaning of setting up the cloud storage is not obvious, will still occupy the local space.
GiteaMirror commented 2025-11-11 15:41:54 -06:00
Author
Owner
Copy Link
Copy Source

@hongbo-miao commented on GitHub (Mar 5, 2025):

Hi @tjbck could you clarify how does today Open WebUI uses local storage, is it pod local storage or Persistent Volume?
If it is Persistent Volume, is it just serving as cache, can we lose these data in Persistent Volume? As the data is already in S3. I am thinking the scenario that when we have multiple Open WebUI replicas, what happens to these "local" data. Thanks!

Hi @JoeChen2me as you have read source code and tried, if you know and could answer, that would be great too, thanks! ☺️

@hongbo-miao commented on GitHub (Mar 5, 2025): Hi @tjbck could you clarify how does today Open WebUI uses local storage, is it pod local storage or Persistent Volume? If it is Persistent Volume, is it just serving as cache, can we lose these data in Persistent Volume? As the data is already in S3. I am thinking the scenario that when we have multiple Open WebUI replicas, what happens to these "local" data. Thanks! Hi @JoeChen2me as you have read source code and tried, if you know and could answer, that would be great too, thanks! ☺️
GiteaMirror commented 2025-11-11 15:41:54 -06:00
Author
Owner
Copy Link
Copy Source

@biishan commented on GitHub (Apr 2, 2025):

Working as intended here, Open WebUI still requires local copy to serve the files to the users.

Why can't it just use cloud storage? If you still save a copy of the file locally after configuring the cloud storage, it seems that the meaning of setting up the cloud storage is not obvious, will still occupy the local space.

I am running into same issue. It keeps storing files locally as well in S3. When we refer to "local" its not keeping in memory. Rather its is actually storing in the Persistent Volume.

@tjbck Is there way to stop this behaviour ? The PV is full and now forcing to upgrade size of PV but imagine gigabyte or terabyte of files.

@biishan commented on GitHub (Apr 2, 2025): > > Working as intended here, Open WebUI still requires local copy to serve the files to the users. > > Why can't it just use cloud storage? If you still save a copy of the file locally after configuring the cloud storage, it seems that the meaning of setting up the cloud storage is not obvious, will still occupy the local space. I am running into same issue. It keeps storing files locally as well in S3. When we refer to "local" its not keeping in memory. Rather its is actually storing in the Persistent Volume. @tjbck Is there way to stop this behaviour ? The PV is full and now forcing to upgrade size of PV but imagine gigabyte or terabyte of files.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#3888