issue: Web search fails at content fetching stage, despite container having full internet access. #6518

New Issue

2025-11-11T16:58:18-06:00

GiteaMirror commented

2025-11-11 16:58:18 -06:00

Originally created by @dmitrysrtx on GitHub (Sep 27, 2025).

Check Existing Issues

I have searched for any existing and/or related issues.
I have searched for any existing and/or related discussions.
I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.6.31

Ollama Version (if applicable)

0.12.0

Operating System

Ubuntu Server 24.04

Browser (if applicable)

Not applicable, issue is on the backend.

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

When web search is triggered, the system should fetch content from the search result URLs, analyze it, and use this information to provide an informed answer to the user's query.

Actual Behavior

The UI indicates that a web search is in progress. However, the model responds without any information from the web. The container logs show that the initial search API call is successful, but the subsequent step of fetching/scraping content from the resulting URLs fails with a "Error fetching" warning. This results in the model getting no web context and the user seeing a "0 sites checked" message.

Steps to Reproduce

Install Open WebUI using Docker Compose.
Configure a web search engine in the Admin Panel (tested with both DuckDuckGo and Google PSE with valid, correctly configured API keys).
Enable "Web Search" for any model in the chat settings.
Ask a simple question that requires a web search, e.g., "What is the weather in Haifa today?".

Logs & Screenshots

Here are the relevant log entries from the open-webui container that show the process failing after a successful search query:

Open WebUI Log Snippet:

INFO | open_webui.routers.retrieval:process_web_search:2049 - trying to web search with ('google_pse', ['weather Tel Aviv tomorrow', 'Tel Aviv weather forecast'])
WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching , skipping due to continue_on_failure=True
Fetching pages: 100%|##########| 1/1 [00:00<00:00, 2247.75it/s]
INFO | open_webui.routers.retrieval:save_docs_to_vector_db:1229 - save_docs_to_vector_db: document web-search-...

### Additional Information

To rule out network issues, I have performed diagnostic tests from inside the running open-webui container.

Diagnostic Command:
curl -v [https://www.wikipedia.org](https://www.wikipedia.org)

Result: The command completes successfully, proving the container has full internet access.
* Trying 185.15.59.224:443...
* Connected to www.wikipedia.org (185.15.59.224) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
...
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
* subject: CN=*.wikipedia.org
* start date: Aug 10 23:56:29 2025 GMT
* expire date: Nov 8 23:56:28 2025 GMT
* subjectAltName: host "www.wikipedia.org" matched cert's "*.wikipedia.org"
* issuer: C=US; O=Let's Encrypt; CN=E6
* SSL certificate verify ok.
...
< HTTP/2 200 
...
<!DOCTYPE html>
...
* Connection #0 to host www.wikipedia.org left intact

The key finding from the troubleshooting is that the container has full, confirmed internet access (DNS, TCP, and TLS/SSL are all working correctly). The failure of the web scraping component, despite these successful network tests with `curl` from within the same container, strongly suggests the issue lies within the application's scraping library (`langchain_community.document_loaders`) and not the user's network environment or configuration.

Originally created by @dmitrysrtx on GitHub (Sep 27, 2025). ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.6.31 ### Ollama Version (if applicable) 0.12.0 ### Operating System Ubuntu Server 24.04 ### Browser (if applicable) Not applicable, issue is on the backend. ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior When web search is triggered, the system should fetch content from the search result URLs, analyze it, and use this information to provide an informed answer to the user's query. ### Actual Behavior The UI indicates that a web search is in progress. However, the model responds without any information from the web. The container logs show that the initial search API call is successful, but the subsequent step of fetching/scraping content from the resulting URLs fails with a "Error fetching" warning. This results in the model getting no web context and the user seeing a "0 sites checked" message. ### Steps to Reproduce 1. Install Open WebUI using Docker Compose. 2. Configure a web search engine in the Admin Panel (tested with both DuckDuckGo and Google PSE with valid, correctly configured API keys). 3. Enable "Web Search" for any model in the chat settings. 4. Ask a simple question that requires a web search, e.g., "What is the weather in Haifa today?". ### Logs & Screenshots Here are the relevant log entries from the `open-webui` container that show the process failing after a successful search query: **Open WebUI Log Snippet:** ```log INFO | open_webui.routers.retrieval:process_web_search:2049 - trying to web search with ('google_pse', ['weather Tel Aviv tomorrow', 'Tel Aviv weather forecast']) WARNING | langchain_community.document_loaders.web_base:_fetch_with_rate_limit:271 - Error fetching , skipping due to continue_on_failure=True Fetching pages: 100%|##########| 1/1 [00:00<00:00, 2247.75it/s] INFO | open_webui.routers.retrieval:save_docs_to_vector_db:1229 - save_docs_to_vector_db: document web-search-... ### Additional Information To rule out network issues, I have performed diagnostic tests from inside the running open-webui container. Diagnostic Command: curl -v [https://www.wikipedia.org](https://www.wikipedia.org) Result: The command completes successfully, proving the container has full internet access. * Trying 185.15.59.224:443... * Connected to www.wikipedia.org (185.15.59.224) port 443 (#0) * ALPN: offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: /etc/ssl/certs * TLSv1.3 (IN), TLS handshake, Server hello (2): ... * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN: server accepted h2 * Server certificate: * subject: CN=*.wikipedia.org * start date: Aug 10 23:56:29 2025 GMT * expire date: Nov 8 23:56:28 2025 GMT * subjectAltName: host "www.wikipedia.org" matched cert's "*.wikipedia.org" * issuer: C=US; O=Let's Encrypt; CN=E6 * SSL certificate verify ok. ... < HTTP/2 200 ... <!DOCTYPE html> ... * Connection #0 to host www.wikipedia.org left intact The key finding from the troubleshooting is that the container has full, confirmed internet access (DNS, TCP, and TLS/SSL are all working correctly). The failure of the web scraping component, despite these successful network tests with `curl` from within the same container, strongly suggests the issue lies within the application's scraping library (`langchain_community.document_loaders`) and not the user's network environment or configuration.

GiteaMirror added the bug label 2025-11-11 16:58:18 -06:00

GiteaMirror closed this issue

2025-11-11 16:58:18 -06:00

GiteaMirror commented

2025-11-11 16:58:20 -06:00