github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-16 14:39:31 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #24630] bug: SafeWebBaseLoader._fetch() passes allow_redirects twice → TypeError on every async web search fetch #74967

New Issue
Closed
opened 2026-05-13 07:49:22 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-05-13 07:49:22 -05:00
Owner
Copy Link

Originally created by @da-astro on GitHub (May 12, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/24630

Check Existing Issues

  • I have searched for any existing and/or related issues.
  • I have searched for any existing and/or related discussions.
  • I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.9.5

Ollama Version (if applicable)

0.23.2

Operating System

Ubuntu Server 24.04 (Linux x86_64)

Browser (if applicable)

Brave 1.7x / N/A — bug reproduces via curl, no browser needed

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

When a user submits a chat query with features.web_search: true against a workspace model that has web search enabled, Open WebUI's web search pipeline should:

  1. Query the configured search engine (DDGS in this case) for relevant URLs.
  2. Use SafeWebBaseLoader to fetch each URL.
  3. Pass the fetched page content to the model as context.

Each sources[].source.docs[].content field in the response should contain the extracted page text (typically thousands of characters per URL).

Actual Behavior

Every URL returned by the search engine fails to fetch. The sources[] array is populated with URL metadata, but every content field is an empty string "". The model then responds from its training data only, with no web context — defeating the purpose of the web search feature.

The failure is silent because continue_on_failure=True is hardcoded in get_web_loader(). All 18 fetches "complete" in <10ms total (visible in logs as Fetching pages: 100%|##########| 18/18 [00:00<00:00, 1489.52it/s]), which shows that exceptions are firing before any I/O happens.

Running with continue_on_failure=False to surface the suppressed exception reveals:

TypeError: aiohttp.client.ClientSession.get() got multiple values for keyword argument 'allow_redirects'
  File "/app/backend/open_webui/retrieval/web/utils.py", line 529, in _fetch
    async with session.get(

Issue

In backend/open_webui/retrieval/web/utils.py, SafeWebBaseLoader.__init__() injects allow_redirects into self.requests_kwargs:

self.requests_kwargs = {
    **(self.requests_kwargs or {}),
    'allow_redirects': AIOHTTP_CLIENT_ALLOW_REDIRECTS,
}

Then _fetch() (line ~525) calls session.get() with allow_redirects passed both via **(self.requests_kwargs | kwargs) and as an explicit kwarg:

async with session.get(
    url,
    **(self.requests_kwargs | kwargs),
    allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS,
) as response:

Python raises TypeError: got multiple values for keyword argument 'allow_redirects'. langchain catches it via continue_on_failure=True and proceeds with empty content.

The synchronous load() path (using self.session.get() with requests) is unaffected because requests doesn't reject duplicate kwargs the same way. So manual SafeWebBaseLoader(...).load() succeeds and returns full content — but the production async path called by ascrape_all() → fetch_all() → _fetch() always fails.

Secondary issue (related, blocks fetches even after the TypeError is fixed)

get_web_loader() does not propagate the USER_AGENT environment variable to SafeWebBaseLoader. After fixing the TypeError, fetches still fail (or return Cloudflare/anti-bot pages) because the session uses the default python-requests/2.x UA. Setting self.session.headers['User-Agent'] = os.environ.get('USER_AGENT', '<default>') in SafeWebBaseLoader.__init__ after super().__init__() resolves this and gives a consistent UA across both sync and async paths.

Proposed fix

Either:

(a) Don't inject allow_redirects into self.requests_kwargs in __init__ (it's already passed explicitly in _fetch), or

(b) Pop it from the merged dict before the call:

_merged = self.requests_kwargs | kwargs
_merged.pop('allow_redirects', None)
async with session.get(
    url,
    **_merged,
    allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS,
) as response:

And separately, propagate USER_AGENT env into the session headers.

Steps to Reproduce

  1. Clean Ubuntu 24.04 host with Docker.
  2. Pull and run latest Open WebUI: 
    docker run -d --name open-webui \
  -p 3000:8080 \
  -v open-webui:/app/backend/data \
  -e ENABLE_API_KEY=true \
  ghcr.io/open-webui/open-webui:main
  3. Open http://localhost:3000, complete signup, log in.
  4. Admin Panel → Settings → Web Search:
    • Enable Web Search: ON
    • Web Search Engine: DDGS
    • DDGS Backend: Auto (Random)
    • Bypass Embedding and Retrieval: ON (to remove that as a variable)
    • Bypass Web Loader: OFF
    • Web Loader Engine: Default
    • Click Save.
  5. Workspace → Models → create new workspace model:
    • Base model: any Ollama model (e.g. llama3.2:latest)
    • Capabilities: enable web_search
    • Default features: web_search
    • Save (name it e.g. "Llama3.2 + web search" llama32--web-search).
  6. Submit a query via API: 
    TOKEN="<your-jwt>"
curl -s -X POST http://localhost:3000/api/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama32--web-search","messages":[{"role":"user","content":"latest SpaceX news"}],"stream":false,"features":{"web_search":true}}' \
  | jq '.sources[0].source.docs | map({url: .metadata.source, chars: (.content | length)})'
  7. Expected: most entries show thousands of chars in chars.
  8. Actual: every entry shows chars: 0.

To surface the underlying exception:

docker exec -e WEBUI_SECRET_KEY=x open-webui python3 -c "
import sys, asyncio; sys.path.insert(0, '/app/backend')
from open_webui.retrieval.web.utils import SafeWebBaseLoader
l = SafeWebBaseLoader(
    web_paths=['https://en.wikipedia.org/wiki/SpaceX'],
    requests_kwargs={'timeout': 15},
    trust_env=True,
    continue_on_failure=False,
)
docs = asyncio.run(l.aload())
"

This raises TypeError: aiohttp.client.ClientSession.get() got multiple values for keyword argument 'allow_redirects' deterministically.

Logs & Screenshots

Container log excerpt during a failing query (DEBUG level enabled via GLOBAL_LOG_LEVEL=DEBUG):

DEBUG | open_webui.retrieval.web.utils:get_web_loader:658 - Using WEB_LOADER_ENGINE SafeWebBaseLoader for 18 URLs
WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching <url>, skipping due to continue_on_failure=True
[... 17 more identical warnings, all within 6ms ...]
Fetching pages: 100%|##########| 18/18 [00:00<00:00, 1489.52it/s]

Full traceback (surfaced by setting continue_on_failure=False):

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/web_base.py", line 268, in _fetch_with_rate_limit
    return await self._fetch(url)
  File "/app/backend/open_webui/retrieval/web/utils.py", line 529, in _fetch
    async with session.get(
TypeError: aiohttp.client.ClientSession.get() got multiple values for keyword argument 'allow_redirects'

After applying the local patch (popping allow_redirects from the merged dict + setting USER_AGENT on self.session.headers in __init__), the same query returns real content — example sources from a "latest SpaceX news" query post-patch:

14,418 chars | https://www.space.com/space-exploration/launches-spacecraft/spacex-falcon-9-rocket-launch-nrol-172-spy-satellite-mission
23,251 chars | https://www.rocketlaunch.live/?filter=spacex
   141 chars | https://en.wikipedia.org/wiki/... (Wikipedia bot rate limit page — expected)
    34 chars | https://www.spacex.com/launches (Cloudflare block — expected)

Additional Information

Affects the default :main Docker image as of 2026-05-12.

Reproduces with both DDGS backend and external Search providers — the bug is in the loader (_fetch), not the search backend, so it impacts any web search provider whose pipeline uses SafeWebBaseLoader (which is the default and the recommended setting).

The synchronous path is unaffected, so any user testing WebBaseLoader manually in a Python shell will see it "work" — masking the bug in real usage where only the async path is invoked.

Originally created by @da-astro on GitHub (May 12, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/24630 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.9.5 ### Ollama Version (if applicable) 0.23.2 ### Operating System Ubuntu Server 24.04 (Linux x86_64) ### Browser (if applicable) Brave 1.7x / N/A — bug reproduces via curl, no browser needed ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior When a user submits a chat query with `features.web_search: true` against a workspace model that has web search enabled, Open WebUI's web search pipeline should: 1. Query the configured search engine (DDGS in this case) for relevant URLs. 2. Use `SafeWebBaseLoader` to fetch each URL. 3. Pass the fetched page content to the model as context. Each `sources[].source.docs[].content` field in the response should contain the extracted page text (typically thousands of characters per URL). ### Actual Behavior Every URL returned by the search engine fails to fetch. The `sources[]` array is populated with URL metadata, but every `content` field is an empty string `""`. The model then responds from its training data only, with no web context — defeating the purpose of the web search feature. The failure is silent because `continue_on_failure=True` is hardcoded in `get_web_loader()`. All 18 fetches "complete" in <10ms total (visible in logs as `Fetching pages: 100%|##########| 18/18 [00:00<00:00, 1489.52it/s]`), which shows that exceptions are firing before any I/O happens. Running with `continue_on_failure=False` to surface the suppressed exception reveals: ``` TypeError: aiohttp.client.ClientSession.get() got multiple values for keyword argument 'allow_redirects' File "/app/backend/open_webui/retrieval/web/utils.py", line 529, in _fetch async with session.get( ``` ### Issue In `backend/open_webui/retrieval/web/utils.py`, `SafeWebBaseLoader.__init__()` injects `allow_redirects` into `self.requests_kwargs`: ```python self.requests_kwargs = { **(self.requests_kwargs or {}), 'allow_redirects': AIOHTTP_CLIENT_ALLOW_REDIRECTS, } ``` Then `_fetch()` (line ~525) calls `session.get()` with `allow_redirects` passed **both** via `**(self.requests_kwargs | kwargs)` **and** as an explicit kwarg: ```python async with session.get( url, **(self.requests_kwargs | kwargs), allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS, ) as response: ``` Python raises `TypeError: got multiple values for keyword argument 'allow_redirects'`. langchain catches it via `continue_on_failure=True` and proceeds with empty content. The synchronous `load()` path (using `self.session.get()` with `requests`) is unaffected because `requests` doesn't reject duplicate kwargs the same way. So manual `SafeWebBaseLoader(...).load()` succeeds and returns full content — but the production async path called by `ascrape_all()` → `fetch_all()` → `_fetch()` always fails. ### Secondary issue (related, blocks fetches even after the TypeError is fixed) `get_web_loader()` does not propagate the `USER_AGENT` environment variable to `SafeWebBaseLoader`. After fixing the TypeError, fetches still fail (or return Cloudflare/anti-bot pages) because the session uses the default `python-requests/2.x` UA. Setting `self.session.headers['User-Agent'] = os.environ.get('USER_AGENT', '<default>')` in `SafeWebBaseLoader.__init__` after `super().__init__()` resolves this and gives a consistent UA across both sync and async paths. ### Proposed fix Either: **(a)** Don't inject `allow_redirects` into `self.requests_kwargs` in `__init__` (it's already passed explicitly in `_fetch`), or **(b)** Pop it from the merged dict before the call: ```python _merged = self.requests_kwargs | kwargs _merged.pop('allow_redirects', None) async with session.get( url, **_merged, allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS, ) as response: ``` And separately, propagate `USER_AGENT` env into the session headers. ### Steps to Reproduce 1. Clean Ubuntu 24.04 host with Docker. 2. Pull and run latest Open WebUI: ```bash docker run -d --name open-webui \ -p 3000:8080 \ -v open-webui:/app/backend/data \ -e ENABLE_API_KEY=true \ ghcr.io/open-webui/open-webui:main ``` 3. Open `http://localhost:3000`, complete signup, log in. 4. Admin Panel → Settings → Web Search: - Enable Web Search: ON - Web Search Engine: DDGS - DDGS Backend: Auto (Random) - Bypass Embedding and Retrieval: ON (to remove that as a variable) - Bypass Web Loader: OFF - Web Loader Engine: Default - Click Save. 5. Workspace → Models → create new workspace model: - Base model: any Ollama model (e.g. `llama3.2:latest`) - Capabilities: enable `web_search` - Default features: `web_search` - Save (name it e.g. "Llama3.2 + web search" `llama32--web-search`). 6. Submit a query via API: ```bash TOKEN="<your-jwt>" curl -s -X POST http://localhost:3000/api/chat/completions \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"model":"llama32--web-search","messages":[{"role":"user","content":"latest SpaceX news"}],"stream":false,"features":{"web_search":true}}' \ | jq '.sources[0].source.docs | map({url: .metadata.source, chars: (.content | length)})' ``` 7. **Expected:** most entries show thousands of chars in `chars`. 8. **Actual:** every entry shows `chars: 0`. To surface the underlying exception: ```bash docker exec -e WEBUI_SECRET_KEY=x open-webui python3 -c " import sys, asyncio; sys.path.insert(0, '/app/backend') from open_webui.retrieval.web.utils import SafeWebBaseLoader l = SafeWebBaseLoader( web_paths=['https://en.wikipedia.org/wiki/SpaceX'], requests_kwargs={'timeout': 15}, trust_env=True, continue_on_failure=False, ) docs = asyncio.run(l.aload()) " ``` This raises `TypeError: aiohttp.client.ClientSession.get() got multiple values for keyword argument 'allow_redirects'` deterministically. ### Logs & Screenshots Container log excerpt during a failing query (DEBUG level enabled via `GLOBAL_LOG_LEVEL=DEBUG`): ``` DEBUG | open_webui.retrieval.web.utils:get_web_loader:658 - Using WEB_LOADER_ENGINE SafeWebBaseLoader for 18 URLs WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching <url>, skipping due to continue_on_failure=True [... 17 more identical warnings, all within 6ms ...] Fetching pages: 100%|##########| 18/18 [00:00<00:00, 1489.52it/s] ``` Full traceback (surfaced by setting `continue_on_failure=False`): ``` Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/langchain_community/document_loaders/web_base.py", line 268, in _fetch_with_rate_limit return await self._fetch(url) File "/app/backend/open_webui/retrieval/web/utils.py", line 529, in _fetch async with session.get( TypeError: aiohttp.client.ClientSession.get() got multiple values for keyword argument 'allow_redirects' ``` After applying the local patch (popping `allow_redirects` from the merged dict + setting USER_AGENT on `self.session.headers` in `__init__`), the same query returns real content — example sources from a `"latest SpaceX news"` query post-patch: ``` 14,418 chars | https://www.space.com/space-exploration/launches-spacecraft/spacex-falcon-9-rocket-launch-nrol-172-spy-satellite-mission 23,251 chars | https://www.rocketlaunch.live/?filter=spacex 141 chars | https://en.wikipedia.org/wiki/... (Wikipedia bot rate limit page — expected) 34 chars | https://www.spacex.com/launches (Cloudflare block — expected) ``` ### Additional Information Affects the default `:main` Docker image as of 2026-05-12. Reproduces with both DDGS backend and external Search providers — the bug is in the loader (`_fetch`), not the search backend, so it impacts any web search provider whose pipeline uses `SafeWebBaseLoader` (which is the default and the recommended setting). The synchronous path is unaffected, so any user testing `WebBaseLoader` manually in a Python shell will see it "work" — masking the bug in real usage where only the async path is invoked.
GiteaMirror added the bug label 2026-05-13 07:49:22 -05:00
GiteaMirror commented 2026-05-13 07:49:24 -05:00
Author
Owner
Copy Link

@owui-terminator[bot] commented on GitHub (May 12, 2026):

⚠️ Missing Issue Title Prefix

@da-astro, your issue title is missing a categorising prefix (e.g. bug:, feat:, docs:).

Please update the title to lead with one of:

  • bug: bug report or error
  • feat: feature request or enhancement
  • docs: documentation issue
  • question: usage question
  • help: support request

Example: bug: Login fails when the password contains special characters

<!-- gh-comment-id:4433913679 --> @owui-terminator[bot] commented on GitHub (May 12, 2026): # ⚠️ Missing Issue Title Prefix @da-astro, your issue title is missing a categorising prefix (e.g. `bug:`, `feat:`, `docs:`). Please update the title to lead with one of: - **bug**: bug report or error - **feat**: feature request or enhancement - **docs**: documentation issue - **question**: usage question - **help**: support request Example: `bug: Login fails when the password contains special characters`
GiteaMirror commented 2026-05-13 07:49:24 -05:00
Author
Owner
Copy Link

@owui-terminator[bot] commented on GitHub (May 12, 2026):

🔍 Related Issues Found

I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:

  1. 🟢 #24560 issue: SafeWebBaseLoader._fetch crashes with TypeError on duplicate allow_redirects kwarg, breaking all web search in v0.9.5
    This is the same root cause: SafeWebBaseLoader._fetch() passes allow_redirects twice, causing the aiohttp TypeError and leaving web-search source contents empty in the async path.
    by blaknite

  2. 🟣 #17815 issue: Web search fails at content fetching stage, despite container having full internet access.
    This older web-search fetching failure is closely related because it also describes the pipeline succeeding at search but failing during URL content fetching, resulting in no web context being passed to the model.
    by dmitrysrtx · bug

  3. 🟣 #18290 Web Search returns incomplete or inconsistent results
    This is related at a higher level because it reports web search returning too little/incomplete fetched content, which overlaps with the symptom of missing or empty fetched page content in the web-search pipeline.
    by aimendenche-nw · bug

💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.

This comment was generated automatically. React with 👍 if helpful, 👎 if not.

<!-- gh-comment-id:4433915875 --> @owui-terminator[bot] commented on GitHub (May 12, 2026): <!-- terminator-bot:related-issues-reply --> 🔍 **Related Issues Found** I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions: 1. 🟢 [#24560](https://github.com/open-webui/open-webui/issues/24560) **issue: SafeWebBaseLoader._fetch crashes with TypeError on duplicate allow_redirects kwarg, breaking all web search in v0.9.5** *This is the same root cause: `SafeWebBaseLoader._fetch()` passes `allow_redirects` twice, causing the aiohttp TypeError and leaving web-search source contents empty in the async path.* *by blaknite* 2. 🟣 [#17815](https://github.com/open-webui/open-webui/issues/17815) **issue: Web search fails at content fetching stage, despite container having full internet access.** *This older web-search fetching failure is closely related because it also describes the pipeline succeeding at search but failing during URL content fetching, resulting in no web context being passed to the model.* *by dmitrysrtx · `bug`* 3. 🟣 [#18290](https://github.com/open-webui/open-webui/issues/18290) **Web Search returns incomplete or inconsistent results** *This is related at a higher level because it reports web search returning too little/incomplete fetched content, which overlaps with the symptom of missing or empty fetched page content in the web-search pipeline.* *by aimendenche-nw · `bug`* --- 💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead. *This comment was generated automatically.* React with 👍 if helpful, 👎 if not.
GiteaMirror commented 2026-05-13 07:49:24 -05:00
Author
Owner
Copy Link

@da-astro commented on GitHub (May 12, 2026):

Duplicate of #24560

<!-- gh-comment-id:4433960279 --> @da-astro commented on GitHub (May 12, 2026): Duplicate of #24560
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#74967
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?