[GH-ISSUE #23966] issue: v0.9.1 web search with Firecrawl fails: NameError: requests is not defined in retrieval/web/utils.py #58799

New Issue

GiteaMirror · 2026-05-05T23:59:14-05:00

GiteaMirror commented

2026-05-05 23:59:14 -05:00

Originally created by @bahree on GitHub (Apr 21, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/23966

Check Existing Issues

I have searched for any existing and/or related issues.
I have searched for any existing and/or related discussions.
I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!).
I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.9.1

Ollama Version (if applicable)

No response

Operating System

Ubuntu 24.04.4 LTS

Browser (if applicable)

No response

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

When web search is enabled in chat, Open WebUI should:

search via SearXNG
fetch/extract result pages via Firecrawl
inject grounded sources into the chat response

Actual Behavior

Web search appears to run, but the final response says no sources were found / falls back to a generic non-grounded answer.

In the backend logs, the real failure is:

File "/app/backend/open_webui/retrieval/web/utils.py", line 233, in lazy_load
    response = requests.post(
NameError: name 'requests' is not defined

This then cascades into missing temporary web-search vector collections, e.g.:

chromadb.errors.NotFoundError: Collection [web-search-...] does not exist

Steps to Reproduce

Configure:
- Web Search enabled
- Search engine = searxng
- Web loader engine = firecrawl
Start a fresh chat
Enable web search
Ask a current-events / live-status query
Observe:
- no grounded sources
- backend throws NameError: requests is not defined

Logs & Screenshots

ERROR | open_webui.retrieval.web.utils:lazy_load
Error extracting content from URLs: name 'requests' is not defined

Traceback (most recent call last):
  File "/app/backend/open_webui/retrieval/web/utils.py", line 233, in lazy_load
    response = requests.post(
NameError: name 'requests' is not defined

Then later:

ERROR | open_webui.retrieval.utils:_fetch_collection
Failed to fetch collection web-search-...: Collection [web-search-...] does not exist

Additional Information

Environment

Open WebUI behind reverse proxy / HTTPS
OpenAI-compatible local model backend
Web search enabled
Search provider: SearXNG
Web loader engine: Firecrawl

Suspected Root Cause

In backend/open_webui/retrieval/web/utils.py, SafeFireCrawlLoader.lazy_load() calls requests.post(...), but requests does not appear to be imported in that module.

Root Cause Confirmed

Confirmed locally on v0.9.1:

the shipped backend/open_webui/retrieval/web/utils.py calls requests.post(...)
the module does not import requests
adding import requests at module scope restores the Firecrawl web-search path

Local Workaround

As a temporary local hotfix, we mounted a patched copy of the v0.9.1 file back into the container with a one-line fix:

import requests

In our Docker deployment, the patched file is mounted over:

/app/backend/open_webui/retrieval/web/utils.py

After that:

web search started creating web-search-* collections again
embeddings and reranking resumed normally
fresh chats returned grounded sources instead of the generic fallback

Originally created by @bahree on GitHub (Apr 21, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/23966 ### Check Existing Issues - [x] I have searched for any existing and/or related issues. - [x] I have searched for any existing and/or related discussions. - [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.9.1 ### Ollama Version (if applicable) _No response_ ### Operating System Ubuntu 24.04.4 LTS ### Browser (if applicable) _No response_ ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior ## Expected Behavior When web search is enabled in chat, Open WebUI should: 1. search via SearXNG 2. fetch/extract result pages via Firecrawl 3. inject grounded sources into the chat response ### Actual Behavior Web search appears to run, but the final response says no sources were found / falls back to a generic non-grounded answer. In the backend logs, the real failure is: ```python File "/app/backend/open_webui/retrieval/web/utils.py", line 233, in lazy_load response = requests.post( NameError: name 'requests' is not defined ``` This then cascades into missing temporary web-search vector collections, e.g.: ```text chromadb.errors.NotFoundError: Collection [web-search-...] does not exist ``` ### Steps to Reproduce 1. Configure: - Web Search enabled - Search engine = `searxng` - Web loader engine = `firecrawl` 2. Start a fresh chat 3. Enable web search 4. Ask a current-events / live-status query 5. Observe: - no grounded sources - backend throws `NameError: requests is not defined` ### Logs & Screenshots ```text ERROR | open_webui.retrieval.web.utils:lazy_load Error extracting content from URLs: name 'requests' is not defined Traceback (most recent call last): File "/app/backend/open_webui/retrieval/web/utils.py", line 233, in lazy_load response = requests.post( NameError: name 'requests' is not defined ``` Then later: ```text ERROR | open_webui.retrieval.utils:_fetch_collection Failed to fetch collection web-search-...: Collection [web-search-...] does not exist ``` ### Additional Information ## Environment - Open WebUI behind reverse proxy / HTTPS - OpenAI-compatible local model backend - Web search enabled - Search provider: SearXNG - Web loader engine: Firecrawl ## Suspected Root Cause In `backend/open_webui/retrieval/web/utils.py`, `SafeFireCrawlLoader.lazy_load()` calls `requests.post(...)`, but `requests` does not appear to be imported in that module. ## Root Cause Confirmed Confirmed locally on `v0.9.1`: - the shipped `backend/open_webui/retrieval/web/utils.py` calls `requests.post(...)` - the module does not import `requests` - adding `import requests` at module scope restores the Firecrawl web-search path ## Local Workaround As a temporary local hotfix, we mounted a patched copy of the `v0.9.1` file back into the container with a one-line fix: ```python import requests ``` In our Docker deployment, the patched file is mounted over: ```text /app/backend/open_webui/retrieval/web/utils.py ``` After that: - web search started creating `web-search-*` collections again - embeddings and reranking resumed normally - fresh chats returned grounded sources instead of the generic fallback

GiteaMirror added the bug label 2026-05-05 23:59:14 -05:00

GiteaMirror closed this issue

2026-05-05 23:59:15 -05:00

GiteaMirror commented

2026-05-05 23:59:16 -05:00

@Balaxxe commented on GitHub (Apr 22, 2026):

Can confirm - this is the exact issue I am having. It's almost like every other update firecrawl takes a hit for some reason.

@Balaxxe commented on GitHub (Apr 22, 2026): Can confirm - this is the exact issue I am having. It's almost like every other update firecrawl takes a hit for some reason.

GiteaMirror commented

2026-05-05 23:59:19 -05:00

@gaby commented on GitHub (Apr 22, 2026):

@Balaxxe Lack of proper testing/CI processes is the reason.

@gaby commented on GitHub (Apr 22, 2026): @Balaxxe Lack of proper testing/CI processes is the reason.

GiteaMirror commented

2026-05-05 23:59:22 -05:00

@weekens commented on GitHub (Apr 22, 2026):

Same for me.

@weekens commented on GitHub (Apr 22, 2026): Same for me.

GiteaMirror commented

2026-05-05 23:59:26 -05:00

@frost19k commented on GitHub (Apr 22, 2026):

**Bug 1: Missing requests import

NameError: name 'requests' is not defined

Bug 2: Firecrawl response parsing fails

ERROR | open_webui.retrieval.web.firecrawl:search_firecrawl:51 - 
Error in FireCrawl search: 'list' object has no attribute 'get'

Code expects:

data = response.json().get('data', {})      # dict
for r in data.get('web', [])                # dict with 'web' key

Firecrawl actually returns:

{
  "success": true,
  "data": [
    {"url": "...", "title": "...", "description": "..."},
    ...
  ]
}

data is a list, not a dict. There is no web key — results are directly in data.

@frost19k commented on GitHub (Apr 22, 2026): **Bug 1: Missing `requests` import ``` NameError: name 'requests' is not defined ``` --- **Bug 2: Firecrawl response parsing fails** ``` ERROR | open_webui.retrieval.web.firecrawl:search_firecrawl:51 - Error in FireCrawl search: 'list' object has no attribute 'get' ``` **Code expects:** ```python data = response.json().get('data', {}) # dict for r in data.get('web', []) # dict with 'web' key ``` **Firecrawl actually returns:** ```json { "success": true, "data": [ {"url": "...", "title": "...", "description": "..."}, ... ] } ``` `data` is a **list**, not a dict. There is no `web` key — results are directly in `data`.

GiteaMirror commented

2026-05-05 23:59:27 -05:00

@gaby commented on GitHub (Apr 22, 2026):

**Bug 1: Missing requests import
NameError: name 'requests' is not defined
Bug 2: Firecrawl response parsing fails
ERROR | open_webui.retrieval.web.firecrawl:search_firecrawl:51 - 
Error in FireCrawl search: 'list' object has no attribute 'get'
Code expects:

data = response.json().get('data', {}) # dict
for r in data.get('web', []) # dict with 'web' key
Firecrawl actually returns:

{
"success": true,
"data": [
{"url": "...", "title": "...", "description": "..."},
...
]
}
data is a list, not a dict. There is no web key — results are directly in data.

Unrelated, requests import is missing.

@gaby commented on GitHub (Apr 22, 2026): > **Bug 1: Missing `requests` import > > ``` > NameError: name 'requests' is not defined > ``` > > **Bug 2: Firecrawl response parsing fails** > > ``` > ERROR | open_webui.retrieval.web.firecrawl:search_firecrawl:51 - > Error in FireCrawl search: 'list' object has no attribute 'get' > ``` > > **Code expects:** > > data = response.json().get('data', {}) # dict > for r in data.get('web', []) # dict with 'web' key > **Firecrawl actually returns:** > > { > "success": true, > "data": [ > {"url": "...", "title": "...", "description": "..."}, > ... > ] > } > `data` is a **list**, not a dict. There is no `web` key — results are directly in `data`. Unrelated, requests import is missing.

GiteaMirror commented

2026-05-05 23:59:29 -05:00

@frost19k commented on GitHub (Apr 22, 2026):

@gaby Eh, if you fix it you fix it. I've patched mine. I'm good.

@frost19k commented on GitHub (Apr 22, 2026): @gaby Eh, if you fix it you fix it. I've patched mine. I'm good.

GiteaMirror commented

2026-05-05 23:59:32 -05:00

@weekens commented on GitHub (Apr 22, 2026):

@bahree , @frost19k , thanks for your hints on fixing this! Made a PR.

@weekens commented on GitHub (Apr 22, 2026): @bahree , @frost19k , thanks for your hints on fixing this! Made a PR.

GiteaMirror commented

2026-05-05 23:59:35 -05:00