mirror of
https://github.com/open-webui/open-webui.git
synced 2026-11-17 15:53:59 -06:00
[GH-ISSUE #24793] issue: Web search discards all results — duplicate allow_redirects keyword in SafeWebBaseLoader._fetch() #107397
Reference in New Issue
Block a user
Originally created by @web2bruno on GitHub (May 15, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/24793
Check Existing Issues
Installation Method
Docker
Open WebUI Version
v0.9.5
Ollama Version (if applicable)
No response
Operating System
Unraid 7.2.5
Browser (if applicable)
No response
Confirmation
README.md.Expected Behavior
When performing a web search with SearXNG as the backend, search results should be fetched, their page content extracted via the web loader, and the content made available to the LLM for generating a response.
Actual Behavior
SearXNG returns results successfully, but the web loader fails to extract content from every URL. All fetches fail instantly (within the same millisecond), and Open WebUI discards all results and reports no search results found. The same search works correctly when "Bypass Web Loader" is enabled.
Steps to Reproduce
Logs & Screenshots
2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://docs.openwebui.com/troubleshooting/manual-database-migration/, skipping due to continue_on_failure=True
2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://discuss.huggingface.co/..., skipping due to continue_on_failure=True
2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://github.com/open-webui/open-webui/discussions/2764, skipping due to continue_on_failure=True
2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://github.com/open-webui/open-webui/discussions/11768, skipping due to continue_on_failure=True
Fetching pages: 100%|##########| 11/11 [00:00<00:00, 10151.23it/s]
All 11 fetches fail at the same millisecond, and the progress bar completes instantly with no actual I/O — indicating the error occurs before any HTTP request is made.
Subsequently:
chromadb.errors.NotFoundError: Collection [web-search-42b026dacf19d2d65cf82e58f945336a9546b1a7f60d1c40e9e] does not exist
Because no content was extracted, the vector collection is never created.
Also observed:
'coroutine' object has no attribute 'role'
Deployment: Docker on Unraid. Web Loader Engine: safe_web (default). langchain-community version: 0.4.1.
Additional Information
Root cause: Duplicate allow_redirects keyword argument in SafeWebBaseLoader._fetch() in /app/backend/open_webui/retrieval/web/utils.py.
The init method adds allow_redirects to self.requests_kwargs:
self.requests_kwargs = {
**(self.requests_kwargs or {}),
'allow_redirects': AIOHTTP_CLIENT_ALLOW_REDIRECTS,
}
Then _fetch() passes it both via dict unpacking AND as an explicit parameter:
async with session.get(
url,
**(self.requests_kwargs | kwargs), # ← allow_redirects here
allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS, # ← AND here (duplicate)
) as response:
This causes a TypeError: got multiple values for keyword argument 'allow_redirects', which fails instantly for every URL.
Fix: Remove the explicit allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS, line from _fetch(). The value is already passed through self.requests_kwargs.
Confirmed working after applying this one-line patch.
@owui-terminator[bot] commented on GitHub (May 15, 2026):
🔍 Related Issues Found
I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:
🟣 #24630 bug: SafeWebBaseLoader._fetch() passes
allow_redirectstwice → TypeError on every async web search fetchThis is the same bug: SafeWebBaseLoader._fetch passes
allow_redirectstwice, causing a TypeError that makes every web-search fetch fail instantly.by da-astro ·
bug🟣 #24560 issue: SafeWebBaseLoader._fetch crashes with TypeError on duplicate allow_redirects kwarg, breaking all web search in v0.9.5
This issue describes the identical duplicate
allow_redirectskeyword problem inSafeWebBaseLoader._fetch, with the same symptom of web search silently returning no content.by blaknite
🟣 #18030 issue: Web Search fails for local SearXNG with self-signed cert (requests missing verify=False)
This is a related web-search failure involving SearXNG, but it is a different root cause (certificate/verify handling), so it is only loosely related compared with the duplicate-kwarg bug.
by alexandrsuk ·
bug💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.
This comment was generated automatically. React with 👍 if helpful, 👎 if not.
@Classic298 commented on GitHub (May 15, 2026):
please test dev it should be fixed there
@Sean-Kenneth-Doherty commented on GitHub (May 17, 2026):
Tested current
devsource at359590ca9d06ee89f616b8a3caebdade93f6228bafter the maintainer note. The duplicateallow_redirectspath appears fixed there:SafeWebBaseLoader.__init__still storesallow_redirectsinself.requests_kwargsSafeWebBaseLoader._fetch()now callssession.get(url, **(self.requests_kwargs | kwargs))without a second explicitallow_redirects=...kwarggit greponbackend/open_webui/retrieval/web/utils.pyonly finds the env import and theself.requests_kwargsassignment forAIOHTTP_CLIENT_ALLOW_REDIRECTSSo this looks fixed on
devand likely just needs release/backport/duplicate handling rather than a new patch PR.