mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-16 06:03:26 -05:00
[GH-ISSUE #10798] [FEAT][web loader proxy] retrieval web loader separeted proxy setting #86776
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#86776
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @KinglyWayne on GitHub (Feb 26, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/10798
Is your feature request related to a problem? Please describe.
Yes, during the retrieval process, network latency or disconnections often occur. Meanwhile, global proxy configuration may cause issues with internal component access. Therefore, having a dedicated network proxy configuration for the retrieval web loader is valuable. PS: This requirement might also apply to the network settings of web search engines.
Describe the solution you'd like
Introduce a separate network proxy configuration in open_webui/retrieval/web/utils.py, can which be configured through system parameters, similar to the proxy settings for YouTube.
Describe alternatives you've considered
Pipeline is also an optional solution. However, since it is a logically global setting, configuring individual pipelines and stacking them with others can make the entire system's configuration extremely complex and chaotic. Personally, I prefer to have a global setting.
Additional context
None
@FyZhu97 commented on GitHub (Feb 28, 2025):
same issue. And I found that global proxy configuration will not be applied to further rerieval process such as accessing urls listed in search results.
@tjbck commented on GitHub (May 23, 2025):
You can use external web loader server for this.