[GH-ISSUE #18686] issue: calling the /api/v1/retireval/process/web/search causes entire website to hang #18671

New Issue

GiteaMirror · 2026-04-20T00:52:48-05:00

GiteaMirror commented

2026-04-20 00:52:48 -05:00

Originally created by @OAburub on GitHub (Oct 28, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/18686

Check Existing Issues

I have searched for any existing and/or related issues.
I have searched for any existing and/or related discussions.
I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

0.6.34

Ollama Version (if applicable)

No response

Operating System

ubuntu 24.04

Browser (if applicable)

No response

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

The api endpoint should process the web search and loading asynchronously without causing the entire website to hang

Actual Behavior

The web loading part executes in the main thread, causing other functions of the website to hang (this is clearer if the web loading enters a retry loop)

Steps to Reproduce

Start with a clean install of Ubuntu 24.04
Install docker 28.5.1
Use the default docker compose file from https://docs.openwebui.com/getting-started/quick-start/ to install and run open-webui
Install and configure Ollama (or any other AI providers, just for query generation)
Configure any web search method, as well as any web loader (from my inspection of the code, the web loader doesn't matter- I'll explain why in detail)
Invoke the /api/v1/retrieval/process/web/search endpoint, using either:
a. The generated bearer token from the accounts page in the settings
b. Environment Variable env=dev and the Swagger docs page (/docs)
c. a browser where you have logged in to open-webui
Use a lot of queries to give yourself more time to observe the problem happening
(Optional) Attempt to find a query that leads to a problematic URL that will trigger the retry loop to circumnavigate rate limiting
Quickly attempt to load a page, where you will observe that the website will not comply or load the page until the web search is complete

Logs & Screenshots

Technically speaking, the execution is working properly, it's just executing in the main thread rather than it's own task/another thread. No relevant logs or errors exist, and posting a screenshot of the loading circle is probably unhelpful.

Additional Information

Observing the process_web_search function in backend/open_webui/routers/retrieval.py, Here's what I think is happenning:
The search tasks are delegated to run_in_threadpool (meaning they don't execute in the main thread):

search_tasks = [
            run_in_threadpool(
                search_web,
                request,
                request.app.state.config.WEB_SEARCH_ENGINE,
                query,
            )
            for query in form_data.queries
        ]

        search_results = await asyncio.gather(*search_tasks)

However, after getting the search results, the web loader starts loading the results in the main thread:

            loader = get_web_loader(
                urls,
                verify_ssl=request.app.state.config.ENABLE_WEB_LOADER_SSL_VERIFICATION,
                requests_per_second=request.app.state.config.WEB_LOADER_CONCURRENT_REQUESTS,
                trust_env=request.app.state.config.WEB_SEARCH_TRUST_ENV,
            )
            docs = await loader.aload()

loader.aload will always attempt to connect with the pages and scrape their contents. The reason why this is more observable when encountering problematic URLS is that the retry loop also calls asyncio.sleep
in backend/open_webui/retrieval/web/utils.py,
case 1: default web loader, line 526:

                except aiohttp.ClientConnectionError as e:
                    if i == retries - 1:
                        raise
                    else:
                        log.warning(
                            f"Error fetching {url} with attempt "
                            f"{i + 1}/{retries}: {e}. Retrying..."
                        )
                        await asyncio.sleep(cooldown * backoff**i)

case 2: every other web loader, which all implement RateLimitMixin in line 124:

class RateLimitMixin:
    async def _wait_for_rate_limit(self):
        """Wait to respect the rate limit if specified."""
        if self.requests_per_second and self.last_request_time:
            min_interval = timedelta(seconds=1.0 / self.requests_per_second)
            time_since_last = datetime.now() - self.last_request_time
            if time_since_last < min_interval:
                await asyncio.sleep((min_interval - time_since_last).total_seconds())
        self.last_request_time = datetime.now()

I'm fairly certain the solution is to delegate the web loading to a seperate task.

Originally created by @OAburub on GitHub (Oct 28, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/18686 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version 0.6.34 ### Ollama Version (if applicable) _No response_ ### Operating System ubuntu 24.04 ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [ ] I have included the browser console logs. - [ ] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior The api endpoint should process the web search and loading asynchronously without causing the entire website to hang ### Actual Behavior The web loading part executes in the main thread, causing other functions of the website to hang (this is clearer if the web loading enters a retry loop) ### Steps to Reproduce 1. Start with a clean install of Ubuntu 24.04 2. Install docker 28.5.1 3. Use the default docker compose file from https://docs.openwebui.com/getting-started/quick-start/ to install and run open-webui 4. Install and configure Ollama (or any other AI providers, just for query generation) 5. Configure any web search method, as well as any web loader (from my inspection of the code, the web loader doesn't matter- I'll explain why in detail) 6. Invoke the /api/v1/retrieval/process/web/search endpoint, using either: a. The generated bearer token from the accounts page in the settings b. Environment Variable env=dev and the Swagger docs page (/docs) c. a browser where you have logged in to open-webui 7. Use a lot of queries to give yourself more time to observe the problem happening 8. (Optional) Attempt to find a query that leads to a problematic URL that will trigger the retry loop to circumnavigate rate limiting 9. Quickly attempt to load a page, where you will observe that the website will not comply or load the page until the web search is complete ### Logs & Screenshots Technically speaking, the execution is working properly, it's just executing in the main thread rather than it's own task/another thread. No relevant logs or errors exist, and posting a screenshot of the loading circle is probably unhelpful. ### Additional Information Observing the process_web_search function in backend/open_webui/routers/retrieval.py, Here's what I think is happenning: The search tasks are delegated to run_in_threadpool (meaning they don't execute in the main thread): ```py search_tasks = [ run_in_threadpool( search_web, request, request.app.state.config.WEB_SEARCH_ENGINE, query, ) for query in form_data.queries ] search_results = await asyncio.gather(*search_tasks) ``` However, after getting the search results, the web loader starts loading the results in the main thread: ```py loader = get_web_loader( urls, verify_ssl=request.app.state.config.ENABLE_WEB_LOADER_SSL_VERIFICATION, requests_per_second=request.app.state.config.WEB_LOADER_CONCURRENT_REQUESTS, trust_env=request.app.state.config.WEB_SEARCH_TRUST_ENV, ) docs = await loader.aload() ``` loader.aload will *always* attempt to connect with the pages and scrape their contents. The reason why this is more observable when encountering problematic URLS is that the retry loop also calls asyncio.sleep in backend/open_webui/retrieval/web/utils.py, case 1: default web loader, line 526: ```py except aiohttp.ClientConnectionError as e: if i == retries - 1: raise else: log.warning( f"Error fetching {url} with attempt " f"{i + 1}/{retries}: {e}. Retrying..." ) await asyncio.sleep(cooldown * backoff**i) ``` case 2: every other web loader, which all implement RateLimitMixin in line 124: ```py class RateLimitMixin: async def _wait_for_rate_limit(self): """Wait to respect the rate limit if specified.""" if self.requests_per_second and self.last_request_time: min_interval = timedelta(seconds=1.0 / self.requests_per_second) time_since_last = datetime.now() - self.last_request_time if time_since_last < min_interval: await asyncio.sleep((min_interval - time_since_last).total_seconds()) self.last_request_time = datetime.now() ``` I'm fairly certain the solution is to delegate the web loading to a seperate task.

GiteaMirror added the bug label 2026-04-20 00:52:48 -05:00

GiteaMirror closed this issue

2026-04-20 00:52:49 -05:00

GiteaMirror referenced this issue

2026-04-25 08:06:22 -05:00

[GH-ISSUE #18671] issue: Image Generation Display Issue with OpenRouter.ai API (Multimodal Output) #34195

GiteaMirror referenced this issue

2026-05-05 20:51:43 -05:00

[GH-ISSUE #18671] issue: Image Generation Display Issue with OpenRouter.ai API (Multimodal Output) #57332

Sign in to join this conversation.