[PR #24183] fix: SafeFireCrawlLoader continue on per-URL errors and honor requests_per_second #131204

Open
opened 2026-05-21 16:21:55 -05:00 by GiteaMirror · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/24183
Author: @turicas
Created: 4/27/2026
Status: 🔄 Open

Base: dev ← Head: fix/firecrawl-multiple-requests


📝 Commits (4)

  • c3edd41 test: add SafeFireCrawlLoader contract tests
  • 03edefb fix: continue on per-URL errors in SafeFireCrawlLoader
  • 28a4df7 fix: honor requests_per_second in FireCrawl loader
  • 71e574e fix(ui): correct Web Loader config meaning

📊 Changes

3 files changed (+252 additions, -17 deletions)

View changed files

📝 backend/open_webui/retrieval/web/utils.py (+25 -14)
➕ backend/open_webui/test/test_retrieval_web_firecrawl_loader.py (+218 -0)
📝 src/lib/components/admin/Settings/WebSearch.svelte (+9 -3)

📄 Description

Before submitting, make sure you've checked the following:

  • Target branch: Verify that the pull request targets the dev branch. PRs targeting main will be immediately closed.
  • Description: Provide a concise description of the changes made in this pull request down below.
  • Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
  • Documentation: Add docs in Open WebUI Docs Repository. Document user-facing behavior, environment variables, public APIs/interfaces, or deployment steps.
  • Dependencies: Are there any new or upgraded dependencies? If so, explain why, update the changelog/docs, and include any compatibility notes. Actually run the code/function that uses updated library to ensure it doesn't crash.
  • Testing: Perform manual tests to verify the implemented fix/feature works as intended AND does not break any other functionality. Include reproducible steps to demonstrate the issue before the fix. Test edge cases (URL encoding, HTML entities, types). Take this as an opportunity to make screenshots of the feature/fix and include them in the PR description.
  • Agentic AI Code: Confirm this Pull Request is not written by any AI Agent or has at least gone through additional human review AND manual testing. If any AI Agent is the co-author of this PR, it may lead to immediate closure of the PR.
  • Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
  • Design & Architecture: Prefer smart defaults over adding new settings; use local state for ephemeral UI logic. Open a Discussion for major architectural or UX changes.
  • Git Hygiene: Keep PRs atomic (one logical change). Clean up commits and rebase on dev to ensure no unrelated commits (e.g. from main) are included. Push updates to the existing PR branch instead of closing and reopening.
  • Title Prefix: To clearly categorize this pull request, prefix the pull request title using one of the following:
    • fix: Bug fix or error correction

Description

SafeFireCrawlLoader.lazy_load and SafeFireCrawlLoader.alazy_load had try/except wrapping the entire for loop instead of each iteration. The first URL that returned an HTTP error from Firecrawl (e.g. 402, 403, 404, 400 -- none of which are in the helper's retry list) terminated the loop and silently dropped every URL that came after it. The continue_on_failure=True flag was effectively a no-op: it suppressed the exception but did not actually continue iterating.

This also has a direct cost impact: every successful scrape before the failing URL is billed by Firecrawl (the request completed on their end) but discarded before reaching the model, so the user pays for content that neither they nor the model ever see.

While investigating this, I found a second bug: SafeFireCrawlLoader inherits RateLimitMixin and stores requests_per_second from its constructor, but neither lazy_load nor alazy_load ever invoked the wait method. The configured value (admin setting and env var WEB_LOADER_CONCURRENT_REQUESTS) was silently ignored.

Both fixes mirror the existing pattern in SafePlaywrightURLLoader in the same file: the try/except moves inside the loop with continue in the handler and _sync_wait_for_rate_limit / _wait_for_rate_limit are invoked per iteration. alazy_load is also restructured to scrape one URL at a time in the threadpool and yield as it goes, instead of collecting the whole batch into a list first. The per-URL warning now identifies which URL failed.

The admin UI labeled the Web Loader's WEB_LOADER_CONCURRENT_REQUESTS setting as "Concurrent Requests", but the underlying value is passed as requests_per_second and used as a sequential rate limiter (the loop is sequential in every loader). The label was renamed to "Requests per second" and a tooltip was added describing the actual behavior. The config key and env var name were preserved for backward compatibility. The Web Search section's "Concurrent Requests" label is left unchanged because that one is genuinely concurrency control (asyncio.Semaphore over parallel queries).

Added

  • backend/open_webui/test/test_retrieval_web_firecrawl_loader.py: contract tests for SafeFireCrawlLoader.lazy_load and SafeFireCrawlLoader.alazy_load. Tests mock scrape_firecrawl_url and require no HTTP server, no Firecrawl SDK and no network access. They cover the happy path, per-URL failure recovery (the regression), the warning message identifying the failing URL, None-handling (empty markdown), continue_on_failure=False propagation and that the rate-limit wait is invoked before every scrape (sync and async).
  • Tooltip on the Web Loader admin setting describing what "Requests per second" actually does.

Fixed

  • SafeFireCrawlLoader.lazy_load and SafeFireCrawlLoader.alazy_load now actually continue past a per-URL failure when continue_on_failure=True (the default), instead of aborting the rest of the batch on the first error.
  • SafeFireCrawlLoader now honors the configured requests_per_second (admin setting, env var WEB_LOADER_CONCURRENT_REQUESTS). Previously the setting was silently ignored (the loader fired requests as fast as the loop ran).
  • The per-URL failure warning now includes the URL that failed, instead of just the Firecrawl endpoint, which made it impossible to tell which URL in a batch was responsible. This makes it easier to debug (e.g. searching for the URL in the FireCrawl dashboard).

Changed

  • SafeFireCrawlLoader.alazy_load is now a true streaming async generator: it scrapes URLs one at a time in the threadpool and yields each document as it becomes available. The previous implementation delegated to lazy_load via list(self.lazy_load()), blocking until every URL was scraped before yielding the first document.
  • Web Loader admin UI label renamed from "Concurrent Requests" to "Requests per second". The previous label misrepresented the setting (it is a sequential rate limit, not a concurrency cap). The config key and env var name weren't changed.

Security

None.

Breaking Changes

None.

Deprecated

None.

Removed

None.

Additional Information

Running the implemented tests against dev (i.e. before applying the fix commits in this PR) - 4 failures:

open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_yields_all_documents_when_all_urls_succeed PASSED                                                           [  9%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_skips_urls_that_return_none PASSED                                                                          [ 18%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_continues_after_per_url_error_when_continue_on_failure_true FAILED                                          [ 27%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_warning_message_includes_failing_url PASSED                                                                 [ 36%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_raises_when_continue_on_failure_is_false PASSED                                                             [ 45%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_waits_for_rate_limit_before_each_scrape FAILED                                                              [ 54%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_yields_all_documents_when_all_urls_succeed PASSED                                                          [ 63%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_continues_after_per_url_error_when_continue_on_failure_true FAILED                                         [ 72%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_warning_message_includes_failing_url PASSED                                                                [ 81%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_raises_when_continue_on_failure_is_false PASSED                                                            [ 90%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_waits_for_rate_limit_before_each_scrape FAILED                                                             [100%]

After - all 11 pass:

open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_yields_all_documents_when_all_urls_succeed PASSED                                                           [  9%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_skips_urls_that_return_none PASSED                                                                          [ 18%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_continues_after_per_url_error_when_continue_on_failure_true PASSED                                          [ 27%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_warning_message_includes_failing_url PASSED                                                                 [ 36%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_raises_when_continue_on_failure_is_false PASSED                                                             [ 45%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_waits_for_rate_limit_before_each_scrape PASSED                                                              [ 54%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_yields_all_documents_when_all_urls_succeed PASSED                                                          [ 63%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_continues_after_per_url_error_when_continue_on_failure_true PASSED                                         [ 72%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_warning_message_includes_failing_url PASSED                                                                [ 81%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_raises_when_continue_on_failure_is_false PASSED                                                            [ 90%]
open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_waits_for_rate_limit_before_each_scrape PASSED                                                             [100%]

Screenshots or Videos

The label change and the new tooltip:
2026-04-27_20-13

Contributor License Agreement

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/24183 **Author:** [@turicas](https://github.com/turicas) **Created:** 4/27/2026 **Status:** 🔄 Open **Base:** `dev` ← **Head:** `fix/firecrawl-multiple-requests` --- ### 📝 Commits (4) - [`c3edd41`](https://github.com/open-webui/open-webui/commit/c3edd41d1fcd749c4aa4529e5e1131bf9d393ae2) test: add SafeFireCrawlLoader contract tests - [`03edefb`](https://github.com/open-webui/open-webui/commit/03edefb8165917f2aa0993d3245d664a3ae11158) fix: continue on per-URL errors in SafeFireCrawlLoader - [`28a4df7`](https://github.com/open-webui/open-webui/commit/28a4df7921466d5dc839a13979956751b4e7a6bc) fix: honor requests_per_second in FireCrawl loader - [`71e574e`](https://github.com/open-webui/open-webui/commit/71e574ea606f0601546f4c9c67088f9401d10caa) fix(ui): correct Web Loader config meaning ### 📊 Changes **3 files changed** (+252 additions, -17 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/retrieval/web/utils.py` (+25 -14) ➕ `backend/open_webui/test/test_retrieval_web_firecrawl_loader.py` (+218 -0) 📝 `src/lib/components/admin/Settings/WebSearch.svelte` (+9 -3) </details> ### 📄 Description **Before submitting, make sure you've checked the following:** - [x] **Target branch:** Verify that the pull request targets the `dev` branch. **PRs targeting `main` will be immediately closed.** - [x] **Description:** Provide a concise description of the changes made in this pull request down below. - [x] **Changelog:** Ensure a changelog entry following the format of [Keep a Changelog](https://keepachangelog.com/) is added at the bottom of the PR description. - [x] **Documentation:** Add docs in [Open WebUI Docs Repository](https://github.com/open-webui/docs). Document user-facing behavior, environment variables, public APIs/interfaces, or deployment steps. - [x] **Dependencies:** Are there any new or upgraded dependencies? If so, explain why, update the changelog/docs, and include any compatibility notes. Actually run the code/function that uses updated library to ensure it doesn't crash. - [x] **Testing:** Perform manual tests to **verify the implemented fix/feature works as intended AND does not break any other functionality**. Include reproducible steps to demonstrate the issue before the fix. Test edge cases (URL encoding, HTML entities, types). Take this as an opportunity to **make screenshots of the feature/fix and include them in the PR description**. - [x] **Agentic AI Code:** Confirm this Pull Request is **not written by any AI Agent** or has at least **gone through additional human review AND manual testing**. If any AI Agent is the co-author of this PR, it may lead to immediate closure of the PR. - [x] **Code review:** Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards? - [x] **Design & Architecture:** Prefer smart defaults over adding new settings; use local state for ephemeral UI logic. Open a Discussion for major architectural or UX changes. - [x] **Git Hygiene:** Keep PRs atomic (one logical change). Clean up commits and rebase on `dev` to ensure no unrelated commits (e.g. from `main`) are included. Push updates to the existing PR branch instead of closing and reopening. - [x] **Title Prefix:** To clearly categorize this pull request, prefix the pull request title using one of the following: - **fix**: Bug fix or error correction ## Description `SafeFireCrawlLoader.lazy_load` and `SafeFireCrawlLoader.alazy_load` had `try/except` wrapping the entire `for` loop instead of each iteration. The first URL that returned an HTTP error from Firecrawl (e.g. 402, 403, 404, 400 -- none of which are in the helper's retry list) terminated the loop and silently dropped every URL that came after it. The `continue_on_failure=True` flag was effectively a no-op: it suppressed the exception but did not actually continue iterating. This also has a direct cost impact: every successful scrape before the failing URL is billed by Firecrawl (the request completed on their end) but discarded before reaching the model, so the user pays for content that neither they nor the model ever see. While investigating this, I found a second bug: `SafeFireCrawlLoader` inherits `RateLimitMixin` and stores `requests_per_second` from its constructor, but neither `lazy_load` nor `alazy_load` ever invoked the wait method. The configured value (admin setting and env var `WEB_LOADER_CONCURRENT_REQUESTS`) was silently ignored. Both fixes mirror the existing pattern in `SafePlaywrightURLLoader` in the same file: the `try/except` moves inside the loop with `continue` in the handler and `_sync_wait_for_rate_limit` / `_wait_for_rate_limit` are invoked per iteration. `alazy_load` is also restructured to scrape one URL at a time in the threadpool and yield as it goes, instead of collecting the whole batch into a list first. The per-URL warning now identifies which URL failed. The admin UI labeled the Web Loader's `WEB_LOADER_CONCURRENT_REQUESTS` setting as "Concurrent Requests", but the underlying value is passed as `requests_per_second` and used as a sequential rate limiter (the loop is sequential in every loader). The label was renamed to "Requests per second" and a tooltip was added describing the actual behavior. The config key and env var name were preserved for backward compatibility. The Web Search section's "Concurrent Requests" label is left unchanged because that one is genuinely concurrency control (`asyncio.Semaphore` over parallel queries). ### Added - `backend/open_webui/test/test_retrieval_web_firecrawl_loader.py`: contract tests for `SafeFireCrawlLoader.lazy_load` and `SafeFireCrawlLoader.alazy_load`. Tests mock `scrape_firecrawl_url` and require no HTTP server, no Firecrawl SDK and no network access. They cover the happy path, per-URL failure recovery (the regression), the warning message identifying the failing URL, `None`-handling (empty markdown), `continue_on_failure=False` propagation and that the rate-limit wait is invoked before every scrape (sync and async). - Tooltip on the Web Loader admin setting describing what "Requests per second" actually does. ### Fixed - `SafeFireCrawlLoader.lazy_load` and `SafeFireCrawlLoader.alazy_load` now actually continue past a per-URL failure when `continue_on_failure=True` (the default), instead of aborting the rest of the batch on the first error. - `SafeFireCrawlLoader` now honors the configured `requests_per_second` (admin setting, env var `WEB_LOADER_CONCURRENT_REQUESTS`). Previously the setting was silently ignored (the loader fired requests as fast as the loop ran). - The per-URL failure warning now includes the URL that failed, instead of just the Firecrawl endpoint, which made it impossible to tell which URL in a batch was responsible. This makes it easier to debug (e.g. searching for the URL in the FireCrawl dashboard). ### Changed - `SafeFireCrawlLoader.alazy_load` is now a true streaming async generator: it scrapes URLs one at a time in the threadpool and yields each document as it becomes available. The previous implementation delegated to `lazy_load` via `list(self.lazy_load())`, blocking until every URL was scraped before yielding the first document. - Web Loader admin UI label renamed from "Concurrent Requests" to "Requests per second". The previous label misrepresented the setting (it is a sequential rate limit, not a concurrency cap). The config key and env var name weren't changed. ## Security None. ## Breaking Changes None. ## Deprecated None. ## Removed None. ## Additional Information Running the implemented tests against `dev` (i.e. before applying the fix commits in this PR) - 4 failures: ``` open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_yields_all_documents_when_all_urls_succeed PASSED [ 9%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_skips_urls_that_return_none PASSED [ 18%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_continues_after_per_url_error_when_continue_on_failure_true FAILED [ 27%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_warning_message_includes_failing_url PASSED [ 36%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_raises_when_continue_on_failure_is_false PASSED [ 45%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_waits_for_rate_limit_before_each_scrape FAILED [ 54%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_yields_all_documents_when_all_urls_succeed PASSED [ 63%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_continues_after_per_url_error_when_continue_on_failure_true FAILED [ 72%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_warning_message_includes_failing_url PASSED [ 81%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_raises_when_continue_on_failure_is_false PASSED [ 90%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_waits_for_rate_limit_before_each_scrape FAILED [100%] ``` After - all 11 pass: ``` open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_yields_all_documents_when_all_urls_succeed PASSED [ 9%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_skips_urls_that_return_none PASSED [ 18%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_continues_after_per_url_error_when_continue_on_failure_true PASSED [ 27%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_warning_message_includes_failing_url PASSED [ 36%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_raises_when_continue_on_failure_is_false PASSED [ 45%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestLazyLoad::test_waits_for_rate_limit_before_each_scrape PASSED [ 54%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_yields_all_documents_when_all_urls_succeed PASSED [ 63%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_continues_after_per_url_error_when_continue_on_failure_true PASSED [ 72%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_warning_message_includes_failing_url PASSED [ 81%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_raises_when_continue_on_failure_is_false PASSED [ 90%] open_webui/test/test_retrieval_web_firecrawl_loader.py::TestAlazyLoad::test_waits_for_rate_limit_before_each_scrape PASSED [100%] ``` ## Screenshots or Videos The label change and the new tooltip: <img width="537" height="361" alt="2026-04-27_20-13" src="https://github.com/user-attachments/assets/dd871ee3-1423-4864-883c-1dcafc1bf6ec" /> ## Contributor License Agreement <!-- 🚨 DO NOT DELETE THE TEXT BELOW 🚨 Keep the "Contributor License Agreement" confirmation text intact. Deleting it will trigger the CLA-Bot to INVALIDATE your PR. Your PR will NOT be reviewed or merged until you check the box below confirming that you have read and agree to the terms of the CLA. --> - [x] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-21 16:21:55 -05:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#131204