[PR #23808] [CLOSED] fix: honour HTTP proxy env vars for DuckDuckGo search and URL fetching #98421

New Issue

GiteaMirror · 2026-05-16T01:13:23-05:00

GiteaMirror commented

2026-05-16 01:13:23 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/23808
Author: @imoes
Created: 4/16/2026
Status: ❌ Closed

Base: main ← Head: fix/corporate-proxy-web-search

📝 Commits (1)

75eea5c fix: honour HTTP proxy env vars for DuckDuckGo search and URL fetching

📊 Changes

2 files changed (+15 additions, -7 deletions)

View changed files

📝 backend/open_webui/retrieval/web/duckduckgo.py (+3 -1)
📝 backend/open_webui/retrieval/web/utils.py (+12 -6)

📄 Description

Pull Request Checklist

Target branch: dev
Description: Provided below.
Changelog: Added below.
Documentation: No new env vars; existing WEB_SEARCH_TRUST_ENV behaviour unchanged.
Dependencies: No new dependencies.
Testing: Manually verified in Docker behind a corporate HTTP proxy.
Agentic AI Code: AI-assisted, with additional human review and manual testing.
Code review: Self-reviewed.
Git Hygiene: Single logical change, rebased on dev.

Problem

Web search failed in environments behind a corporate HTTP proxy (http_proxy/https_proxy env vars set in Docker).

1. DuckDuckGo search (`duckduckgo.py`)

DDGS() was instantiated without a proxy argument, so all requests ignored the proxy env vars and were blocked by the firewall.

2. URL fetching after search (`utils.py`)

SafeWebBaseLoader._fetch created an aiohttp.ClientSession with trust_env effectively hardcoded to False. Because WEB_SEARCH_TRUST_ENV is managed via PersistentConfig (database-backed), a stored False in the DB silently overrides the env var on every container restart — even when WEB_SEARCH_TRUST_ENV=true is set in the environment.

Fix

duckduckgo.py

Read https_proxy (fallback: http_proxy) from the environment and pass it to DDGS(proxy=proxy).

utils.py

Change all trust_env: bool = False defaults to True across all loader classes and get_web_loader().
In SafeWebBaseLoader._fetch, compute effective_trust_env = self.trust_env or bool(os.environ.get("https_proxy") or os.environ.get("http_proxy")) so the aiohttp session always uses the proxy when proxy env vars are present, regardless of the DB-cached config value.

Reproduction Steps

Deploy Open WebUI in Docker with:

http_proxy=http://proxy.example.com:80
https_proxy=http://proxy.example.com:80

Before fix: DuckDuckGo search → ConnectError; URL fetching → Connection timeout
After fix: Both work correctly (verified in production behind proxy.ippen.media:80)

Changelog Entry

Fixed

duckduckgo.py: Pass https_proxy/http_proxy env var to DDGS(proxy=...) so DuckDuckGo search works through a corporate HTTP proxy.
utils.py: SafeWebBaseLoader._fetch now auto-enables trust_env when proxy env vars are present, bypassing the PersistentConfig DB-cached False value that would otherwise silently ignore the proxy.
utils.py: Changed trust_env default from False to True in all loader classes and get_web_loader().

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/23808 **Author:** [@imoes](https://github.com/imoes) **Created:** 4/16/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix/corporate-proxy-web-search` --- ### 📝 Commits (1) - [`75eea5c`](https://github.com/open-webui/open-webui/commit/75eea5cd8af9eaa821458febd79a85e3e78b05ed) fix: honour HTTP proxy env vars for DuckDuckGo search and URL fetching ### 📊 Changes **2 files changed** (+15 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/retrieval/web/duckduckgo.py` (+3 -1) 📝 `backend/open_webui/retrieval/web/utils.py` (+12 -6) </details> ### 📄 Description # Pull Request Checklist - [x] **Target branch:** `dev` - [x] **Description:** Provided below. - [x] **Changelog:** Added below. - [ ] **Documentation:** No new env vars; existing `WEB_SEARCH_TRUST_ENV` behaviour unchanged. - [x] **Dependencies:** No new dependencies. - [x] **Testing:** Manually verified in Docker behind a corporate HTTP proxy. - [x] **Agentic AI Code:** AI-assisted, with additional human review and manual testing. - [x] **Code review:** Self-reviewed. - [x] **Git Hygiene:** Single logical change, rebased on `dev`. --- ## Problem Web search failed in environments behind a corporate HTTP proxy (`http_proxy`/`https_proxy` env vars set in Docker). ### 1. DuckDuckGo search (`duckduckgo.py`) `DDGS()` was instantiated without a `proxy` argument, so all requests ignored the proxy env vars and were blocked by the firewall. ### 2. URL fetching after search (`utils.py`) `SafeWebBaseLoader._fetch` created an `aiohttp.ClientSession` with `trust_env` effectively hardcoded to `False`. Because `WEB_SEARCH_TRUST_ENV` is managed via `PersistentConfig` (database-backed), a stored `False` in the DB silently overrides the env var on every container restart — even when `WEB_SEARCH_TRUST_ENV=true` is set in the environment. ## Fix **`duckduckgo.py`** - Read `https_proxy` (fallback: `http_proxy`) from the environment and pass it to `DDGS(proxy=proxy)`. **`utils.py`** - Change all `trust_env: bool = False` defaults to `True` across all loader classes and `get_web_loader()`. - In `SafeWebBaseLoader._fetch`, compute `effective_trust_env = self.trust_env or bool(os.environ.get("https_proxy") or os.environ.get("http_proxy"))` so the aiohttp session always uses the proxy when proxy env vars are present, regardless of the DB-cached config value. ## Reproduction Steps Deploy Open WebUI in Docker with: ``` http_proxy=http://proxy.example.com:80 https_proxy=http://proxy.example.com:80 ``` - **Before fix:** DuckDuckGo search → `ConnectError`; URL fetching → `Connection timeout` - **After fix:** Both work correctly (verified in production behind `proxy.ippen.media:80`) --- # Changelog Entry ### Fixed - `duckduckgo.py`: Pass `https_proxy`/`http_proxy` env var to `DDGS(proxy=...)` so DuckDuckGo search works through a corporate HTTP proxy. - `utils.py`: `SafeWebBaseLoader._fetch` now auto-enables `trust_env` when proxy env vars are present, bypassing the PersistentConfig DB-cached `False` value that would otherwise silently ignore the proxy. - `utils.py`: Changed `trust_env` default from `False` to `True` in all loader classes and `get_web_loader()`. --- ### Contributor License Agreement - [x] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-05-16 01:13:23 -05:00

GiteaMirror closed this issue

2026-05-16 01:13:26 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#98421

[PR #23808] [CLOSED] fix: honour HTTP proxy env vars for DuckDuckGo search and URL fetching #98421