mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-16 06:03:26 -05:00
[GH-ISSUE #24560] issue: SafeWebBaseLoader._fetch crashes with TypeError on duplicate allow_redirects kwarg, breaking all web search in v0.9.5 #123645
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#123645
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @blaknite on GitHub (May 11, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/24560
Check Existing Issues
Installation Method
Other (
uvx open-webui@latest serveunder systemd)Open WebUI Version
v0.9.5
Ollama Version (if applicable)
N/A
Operating System
Ubuntu 22.04 LTS
Browser (if applicable)
N/A — failure is server-side.
Confirmation
README.md.Expected Behavior
With web search enabled and
WEB_LOADER_ENGINEat its default (safe_web), the loader fetches the URLs returned by the search engine and feeds their content into the LLM context.Actual Behavior
Every URL fails to fetch. The failure is swallowed because
continue_on_failure=True, so all you see in the logs is:```
WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching , skipping due to continue_on_failure=True
```
After patching langchain to surface the swallowed exception, every fetch raises:
```
TypeError: aiohttp.client.ClientSession.get() got multiple values for keyword argument 'allow_redirects'
```
Root cause
In
backend/open_webui/retrieval/web/utils.py,SafeWebBaseLoader.__init__putsallow_redirectsintoself.requests_kwargs:```python
self.requests_kwargs = {
**(self.requests_kwargs or {}),
'allow_redirects': AIOHTTP_CLIENT_ALLOW_REDIRECTS,
}
```
Then
_fetchunpacksself.requests_kwargsintosession.get(...)and also passesallow_redirectsexplicitly:```python
async with session.get(
url,
**(self.requests_kwargs | kwargs),
allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS,
) as response:
```
aiohttprejects the duplicate kwarg on every call. The bug is onmainas well as the v0.9.5 tag, and was introduced by PR #24524.Fix
Drop the explicit
allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS,line from thesession.get(...)call. The value is already inself.requests_kwargs, so the SSRF protection is preserved.Steps to Reproduce
safe_weboption).Expected: search runs, URLs are fetched, content reaches the model.
Actual: search finds URLs, every fetch fails silently, the model gets no web context.
Logs & Screenshots
After patching
langchain_community/document_loaders/web_base.py:_fetch_with_rate_limitto log the swallowed exception:```
2026-05-11 02:27:28.466 | ERROR | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://openrouter.ai/qwen/qwen3.5-35b-a3b [TypeError: aiohttp.client.ClientSession.get() got multiple values for keyword argument 'allow_redirects'], skipping due to continue_on_failure=True
Traceback (most recent call last):
File ".../open_webui/retrieval/web/utils.py", line 521, in _fetch
async with session.get(
url,
**(self.requests_kwargs | kwargs),
allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS,
) as response:
TypeError: aiohttp.client.ClientSession.get() got multiple values for keyword argument 'allow_redirects'
```
Additional Information
Workarounds:
SafeWebBaseLoader._fetch.SafeWebBaseLoader._fetchlocally to remove the duplicateallow_redirectskwarg.Thanks for all your work on Open WebUI.
@owui-terminator[bot] commented on GitHub (May 11, 2026):
🔍 Related Issues Found
I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:
🟣 #24017 issue: Web search breaks persistently after setting Fetch URL Content Length Limit to 80000
This is another web-search regression in the same retrieval/web search pipeline, affecting fetch behavior after a configuration change. While the trigger differs, it is the closest existing report in the web search loader area.
by mero20j1 ·
bug🟣 #18686 issue: calling the /api/v1/retireval/process/web/search causes entire website to hang
This issue reports the web search endpoint hanging because the web loading path blocks the main thread. It is related to the same web retrieval flow, though the failure mode is different from the duplicate kwarg crash.
by OAburub ·
bug🟣 #15498 issue: "Bypass Web Loader" toggle gives error with Brave web search
This is a prior web-search/bypass-loader bug in the web retrieval subsystem. It involves the same web loader path and is useful as a related historical issue, even though the specific error is different.
by teese ·
bug💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.
This comment was generated automatically. React with 👍 if helpful, 👎 if not.
@farid-abu-issa commented on GitHub (May 11, 2026):
Confirming on Railway deployment, Tavily engine. v0.9.2 → v0.9.5
upgrade broke web search the moment Bypass Web Loader was OFF.
AIOHTTP_CLIENT_ALLOW_REDIRECTS=true did not help (consistent with
the duplicate kwarg diagnosis). Rolled back to v0.9.2, working again.
@bootcnode commented on GitHub (May 12, 2026):
Thank you for reporting this, I couldn't figure out why all my web-searches was broken. Rolling back to 0.9.2 has solved the issue for me.
@tezgno commented on GitHub (May 12, 2026):
I can also confirm this behavior and was diving into it when I saw this post. I rolled back to 0.9.2 and the issue is resolved as well.
@da-astro commented on GitHub (May 12, 2026):
Confirmed on
:devas of 2026-05-12I can confirm this bug still reproduces on
ghcr.io/open-webui/open-webui:dev(pulled fresh today). Same traceback:Applying the fix locally (popping
allow_redirectsfrom the merged dict before the call) resolves the TypeError and web search begins fetching content successfully.Secondary Issue: USER_AGENT not propagated
After fixing the
allow_redirectsbug, a second issue surfaces:get_web_loader()does not propagate theUSER_AGENTenvironment variable toSafeWebBaseLoader.The session defaults to
python-requests/2.xas the User-Agent, which many sites (Cloudflare-protected pages, anti-bot systems, rate-limiters like Wikipedia) block or throttle aggressively. This causes fetches to either fail outright (403/Cloudflare block pages) or return empty/error content even though the loader is working correctly.Fix
Add this in
SafeWebBaseLoader.__init__()aftersuper().__init__():This ensures the session sends a real browser UA on both the synchronous (
load()) and asynchronous (aload()→_fetch()) paths. The_fetch()method already propagatesself.session.headersto the aiohttp session (line ~513:headers=self.session.headers), so setting it once in__init__covers both.Validation
After applying both fixes (allow_redirects + USER_AGENT), web search fetches work end-to-end:
Before fixes: every entry shows
chars: 0After fixes: most entries show thousands of chars (some still fail due to paywalls/Cloudflare, which is expected)
Example output post-fix:
Recommendation: Address both issues together so web search works out-of-the-box without requiring users to set
USER_AGENTmanually or patch the loader.@farid-abu-issa commented on GitHub (May 12, 2026):
Adding a +1 on @da-astro's USER_AGENT finding. Worth flagging this
clearly so it isn't missed in the v0.9.6 review cycle.
The allow_redirects fix (PR #24602) addresses the immediate TypeError,
but if shipped alone, real-world web search will still degrade on
Cloudflare-protected sites, Wikipedia, and any host with bot
detection — exactly the sites users hit most often. The
python-requests UA gets blocked or rate-limited aggressively.
Would strongly recommend merging both fixes together in v0.9.6 so
users don't roll forward, find search "technically working" but
empty on half the sites, and roll back again.
@tjbck commented on GitHub (May 13, 2026):
Addressed in dev!