github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-05-06 10:58:17 -05:00
Code Issues 1.3k Packages Projects Releases 126 Wiki Activity

feat: Filter S3 objects using Prefix parameter instead of filtering locally in S3StorageProvider.delete_all_files #4852

New Issue
Closed
opened 2025-11-11 16:04:56 -06:00 by GiteaMirror · 2 comments
GiteaMirror commented 2025-11-11 16:04:56 -06:00
Owner
Copy Link

Originally created by @jcsaaddupuy on GitHub (Apr 15, 2025).

Check Existing Issues

  • I have searched the existing issues and discussions.

Problem Description

In S3StorageProvider.delete_all_files, all the files from the bucket are listed, then filtered in python on hte prefix.

boto3 S3 client accept a Prefix parameter that allow filtering S3 keys at the S3 api level, instead of fetching all items then filtering locally in python.

instead of

response = self.s3_client.list_objects_v2(Bucket=self.bucket_name)

we can do

response = self.s3_client.list_objects_v2(Bucket=self.bucket_name, Prefix=self.key_prefix)

Let me know whaat you think, i'll gladly provide a PR for this change

Desired Solution you'd like

Filtering S3 objects using the list_objects_v2 Prefix parameter

response = self.s3_client.list_objects_v2(Bucket=self.bucket_name, Prefix=self.key_prefix)

Alternatives Considered

No response

Additional Context

No response

Originally created by @jcsaaddupuy on GitHub (Apr 15, 2025). ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description In [S3StorageProvider.delete_all_files](https://github.com/open-webui/open-webui/blob/main/backend/open_webui/storage/provider.py#L168-L182), all the files from the bucket are listed, then filtered in python on hte prefix. `boto3` S3 client accept a `Prefix` parameter that allow filtering S3 keys at the S3 api level, instead of fetching all items then filtering locally in python. instead of ```python response = self.s3_client.list_objects_v2(Bucket=self.bucket_name) ``` we can do ```python response = self.s3_client.list_objects_v2(Bucket=self.bucket_name, Prefix=self.key_prefix) ``` Let me know whaat you think, i'll gladly provide a PR for this change ### Desired Solution you'd like Filtering S3 objects using the `list_objects_v2` `Prefix` parameter ```python response = self.s3_client.list_objects_v2(Bucket=self.bucket_name, Prefix=self.key_prefix) ``` ### Alternatives Considered _No response_ ### Additional Context _No response_
GiteaMirror commented 2025-11-11 16:04:57 -06:00
Author
Owner
Copy Link

@tjbck commented on GitHub (Apr 15, 2025):

PR welcome!

@tjbck commented on GitHub (Apr 15, 2025): PR welcome!
GiteaMirror commented 2025-11-11 16:04:57 -06:00
Author
Owner
Copy Link

@jcsaaddupuy commented on GitHub (Apr 15, 2025):

Hi !
I've started a fix here https://github.com/jcsaaddupuy/open-webui/tree/feat/12885-use-s3-prefix-parameter

I also started a discussion here https://github.com/open-webui/open-webui/discussions/12891 for follow up

@jcsaaddupuy commented on GitHub (Apr 15, 2025): Hi ! I've started a fix here https://github.com/jcsaaddupuy/open-webui/tree/feat/12885-use-s3-prefix-parameter I also started a discussion here https://github.com/open-webui/open-webui/discussions/12891 for follow up
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#4852
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?