[GH-ISSUE #24793] issue: Web search discards all results — duplicate allow_redirects keyword in SafeWebBaseLoader._fetch() #123714

Closed
opened 2026-05-21 03:10:36 -05:00 by GiteaMirror · 6 comments
Owner

Originally created by @web2bruno on GitHub (May 15, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/24793

Check Existing Issues

  • I have searched for any existing and/or related issues.
  • I have searched for any existing and/or related discussions.
  • I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
  • I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.9.5

Ollama Version (if applicable)

No response

Operating System

Unraid 7.2.5

Browser (if applicable)

No response

Confirmation

  • I have read and followed all instructions in README.md.
  • I am using the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
  • I have provided every relevant configuration, setting, and environment variable used in my setup.
  • I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
  • I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
  • Start with the initial platform/version/OS and dependencies used,
  • Specify exact install/launch/configure commands,
  • List URLs visited, user input (incl. example values/emails/passwords if needed),
  • Describe all options and toggles enabled or changed,
  • Include any files or environmental changes,
  • Identify the expected and actual result at each stage,
  • Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

When performing a web search with SearXNG as the backend, search results should be fetched, their page content extracted via the web loader, and the content made available to the LLM for generating a response.

Actual Behavior

SearXNG returns results successfully, but the web loader fails to extract content from every URL. All fetches fail instantly (within the same millisecond), and Open WebUI discards all results and reports no search results found. The same search works correctly when "Bypass Web Loader" is enabled.

Steps to Reproduce

  • Configure Open WebUI with SearXNG as the web search backend
  • Set Content Extraction to "safe_web" (default Web Loader Engine)
  • Perform any web search
  • Observe results briefly appear then get discarded with "no results"

Logs & Screenshots

2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://docs.openwebui.com/troubleshooting/manual-database-migration/, skipping due to continue_on_failure=True
2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://discuss.huggingface.co/..., skipping due to continue_on_failure=True
2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://github.com/open-webui/open-webui/discussions/2764, skipping due to continue_on_failure=True
2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://github.com/open-webui/open-webui/discussions/11768, skipping due to continue_on_failure=True
Fetching pages: 100%|##########| 11/11 [00:00<00:00, 10151.23it/s]
All 11 fetches fail at the same millisecond, and the progress bar completes instantly with no actual I/O — indicating the error occurs before any HTTP request is made.

Subsequently:
chromadb.errors.NotFoundError: Collection [web-search-42b026dacf19d2d65cf82e58f945336a9546b1a7f60d1c40e9e] does not exist

Because no content was extracted, the vector collection is never created.

Also observed:
'coroutine' object has no attribute 'role'

Deployment: Docker on Unraid. Web Loader Engine: safe_web (default). langchain-community version: 0.4.1.

Additional Information

Root cause: Duplicate allow_redirects keyword argument in SafeWebBaseLoader._fetch() in /app/backend/open_webui/retrieval/web/utils.py.

The init method adds allow_redirects to self.requests_kwargs:
self.requests_kwargs = {
**(self.requests_kwargs or {}),
'allow_redirects': AIOHTTP_CLIENT_ALLOW_REDIRECTS,
}

Then _fetch() passes it both via dict unpacking AND as an explicit parameter:
async with session.get(
url,
**(self.requests_kwargs | kwargs), # ← allow_redirects here
allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS, # ← AND here (duplicate)
) as response:

This causes a TypeError: got multiple values for keyword argument 'allow_redirects', which fails instantly for every URL.

Fix: Remove the explicit allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS, line from _fetch(). The value is already passed through self.requests_kwargs.

Confirmed working after applying this one-line patch.

Originally created by @web2bruno on GitHub (May 15, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/24793 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.9.5 ### Ollama Version (if applicable) _No response_ ### Operating System Unraid 7.2.5 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior When performing a web search with SearXNG as the backend, search results should be fetched, their page content extracted via the web loader, and the content made available to the LLM for generating a response. ### Actual Behavior SearXNG returns results successfully, but the web loader fails to extract content from every URL. All fetches fail instantly (within the same millisecond), and Open WebUI discards all results and reports no search results found. The same search works correctly when "Bypass Web Loader" is enabled. ### Steps to Reproduce - Configure Open WebUI with SearXNG as the web search backend - Set Content Extraction to "safe_web" (default Web Loader Engine) - Perform any web search - Observe results briefly appear then get discarded with "no results" ### Logs & Screenshots 2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://docs.openwebui.com/troubleshooting/manual-database-migration/, skipping due to continue_on_failure=True 2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://discuss.huggingface.co/..., skipping due to continue_on_failure=True 2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://github.com/open-webui/open-webui/discussions/2764, skipping due to continue_on_failure=True 2026-05-15 12:40:58.865 | WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching https://github.com/open-webui/open-webui/discussions/11768, skipping due to continue_on_failure=True Fetching pages: 100%|##########| 11/11 [00:00<00:00, 10151.23it/s] All 11 fetches fail at the same millisecond, and the progress bar completes instantly with no actual I/O — indicating the error occurs before any HTTP request is made. Subsequently: chromadb.errors.NotFoundError: Collection [web-search-42b026dacf19d2d65cf82e58f945336a9546b1a7f60d1c40e9e] does not exist Because no content was extracted, the vector collection is never created. Also observed: 'coroutine' object has no attribute 'role' Deployment: Docker on Unraid. Web Loader Engine: safe_web (default). langchain-community version: 0.4.1. ### Additional Information Root cause: Duplicate allow_redirects keyword argument in SafeWebBaseLoader._fetch() in /app/backend/open_webui/retrieval/web/utils.py. The __init__ method adds allow_redirects to self.requests_kwargs: self.requests_kwargs = { **(self.requests_kwargs or {}), 'allow_redirects': AIOHTTP_CLIENT_ALLOW_REDIRECTS, } Then _fetch() passes it both via dict unpacking AND as an explicit parameter: async with session.get( url, **(self.requests_kwargs | kwargs), # ← allow_redirects here allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS, # ← AND here (duplicate) ) as response: This causes a TypeError: got multiple values for keyword argument 'allow_redirects', which fails instantly for every URL. Fix: Remove the explicit allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS, line from _fetch(). The value is already passed through self.requests_kwargs. Confirmed working after applying this one-line patch.
GiteaMirror added the bug label 2026-05-21 03:10:36 -05:00
Author
Owner

@owui-terminator[bot] commented on GitHub (May 15, 2026):

🔍 Related Issues Found

I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions:

  1. 🟣 #24630 bug: SafeWebBaseLoader._fetch() passes allow_redirects twice → TypeError on every async web search fetch
    This is the same bug: SafeWebBaseLoader._fetch passes allow_redirects twice, causing a TypeError that makes every web-search fetch fail instantly.
    by da-astro · bug

  2. 🟣 #24560 issue: SafeWebBaseLoader._fetch crashes with TypeError on duplicate allow_redirects kwarg, breaking all web search in v0.9.5
    This issue describes the identical duplicate allow_redirects keyword problem in SafeWebBaseLoader._fetch, with the same symptom of web search silently returning no content.
    by blaknite

  3. 🟣 #18030 issue: Web Search fails for local SearXNG with self-signed cert (requests missing verify=False)
    This is a related web-search failure involving SearXNG, but it is a different root cause (certificate/verify handling), so it is only loosely related compared with the duplicate-kwarg bug.
    by alexandrsuk · bug


💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead.

This comment was generated automatically. React with 👍 if helpful, 👎 if not.

<!-- gh-comment-id:4463401991 --> @owui-terminator[bot] commented on GitHub (May 15, 2026): <!-- terminator-bot:related-issues-reply --> 🔍 **Related Issues Found** I found some existing issues that might be related. Please check if any of these are duplicates or contain helpful solutions: 1. 🟣 [#24630](https://github.com/open-webui/open-webui/issues/24630) **bug: SafeWebBaseLoader._fetch() passes `allow_redirects` twice → TypeError on every async web search fetch** *This is the same bug: SafeWebBaseLoader._fetch passes `allow_redirects` twice, causing a TypeError that makes every web-search fetch fail instantly.* *by da-astro · `bug`* 2. 🟣 [#24560](https://github.com/open-webui/open-webui/issues/24560) **issue: SafeWebBaseLoader._fetch crashes with TypeError on duplicate allow_redirects kwarg, breaking all web search in v0.9.5** *This issue describes the identical duplicate `allow_redirects` keyword problem in `SafeWebBaseLoader._fetch`, with the same symptom of web search silently returning no content.* *by blaknite* 3. 🟣 [#18030](https://github.com/open-webui/open-webui/issues/18030) **issue: Web Search fails for local SearXNG with self-signed cert (requests missing verify=False)** *This is a related web-search failure involving SearXNG, but it is a different root cause (certificate/verify handling), so it is only loosely related compared with the duplicate-kwarg bug.* *by alexandrsuk · `bug`* --- 💡 If your issue is a duplicate, please close it and add any additional details to the existing issue instead. *This comment was generated automatically.* React with 👍 if helpful, 👎 if not.
Author
Owner

@Classic298 commented on GitHub (May 15, 2026):

please test dev it should be fixed there

<!-- gh-comment-id:4463465581 --> @Classic298 commented on GitHub (May 15, 2026): please test dev it should be fixed there
Author
Owner

@Sean-Kenneth-Doherty commented on GitHub (May 17, 2026):

Tested current dev source at 359590ca9d06ee89f616b8a3caebdade93f6228b after the maintainer note. The duplicate allow_redirects path appears fixed there:

  • SafeWebBaseLoader.__init__ still stores allow_redirects in self.requests_kwargs
  • SafeWebBaseLoader._fetch() now calls session.get(url, **(self.requests_kwargs | kwargs)) without a second explicit allow_redirects=... kwarg
  • git grep on backend/open_webui/retrieval/web/utils.py only finds the env import and the self.requests_kwargs assignment for AIOHTTP_CLIENT_ALLOW_REDIRECTS

So this looks fixed on dev and likely just needs release/backport/duplicate handling rather than a new patch PR.

<!-- gh-comment-id:4468621756 --> @Sean-Kenneth-Doherty commented on GitHub (May 17, 2026): Tested current `dev` source at `359590ca9d06ee89f616b8a3caebdade93f6228b` after the maintainer note. The duplicate `allow_redirects` path appears fixed there: - `SafeWebBaseLoader.__init__` still stores `allow_redirects` in `self.requests_kwargs` - `SafeWebBaseLoader._fetch()` now calls `session.get(url, **(self.requests_kwargs | kwargs))` without a second explicit `allow_redirects=...` kwarg - `git grep` on `backend/open_webui/retrieval/web/utils.py` only finds the env import and the `self.requests_kwargs` assignment for `AIOHTTP_CLIENT_ALLOW_REDIRECTS` So this looks fixed on `dev` and likely just needs release/backport/duplicate handling rather than a new patch PR.
Author
Owner

@jbgruenwald commented on GitHub (May 18, 2026):

i have the same issue in v0.9.5 Web search still worked in 0.9.4. I have openwebui installed using the docker container:

image: 'ghcr.io/open-webui/open-webui:0.9.4'

I guess there is no way to change to a different branch (I guess by dev you guys mean the branch?) there right? So i'll just for 0.9.6 or do I have other options?

<!-- gh-comment-id:4480778112 --> @jbgruenwald commented on GitHub (May 18, 2026): i have the same issue in v0.9.5 Web search still worked in 0.9.4. I have openwebui installed using the docker container: ``` image: 'ghcr.io/open-webui/open-webui:0.9.4' ``` I guess there is no way to change to a different branch (I guess by `dev` you guys mean the branch?) there right? So i'll just for 0.9.6 or do I have other options?
Author
Owner

@Classic298 commented on GitHub (May 18, 2026):

@jbgruenwald ghcr.io/open-webui/open-webui:dev is the dev branch (ahead of main)

more info here : https://docs.openwebui.com/getting-started/advanced-topics/development

<!-- gh-comment-id:4480855432 --> @Classic298 commented on GitHub (May 18, 2026): @jbgruenwald ghcr.io/open-webui/open-webui:dev is the dev branch (ahead of main) more info here : https://docs.openwebui.com/getting-started/advanced-topics/development
Author
Owner

@Classic298 commented on GitHub (May 18, 2026):

and thanks @Sean-Kenneth-Doherty for confirming. closing.

<!-- gh-comment-id:4480857590 --> @Classic298 commented on GitHub (May 18, 2026): and thanks @Sean-Kenneth-Doherty for confirming. closing.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#123714