mirror of
https://github.com/open-webui/open-webui.git
synced 2026-11-17 15:53:59 -06:00
[PR #24183] fix: SafeFireCrawlLoader continue on per-URL errors and honor requests_per_second #114848
Reference in New Issue
Block a user
📋 Pull Request Information
Original PR: https://github.com/open-webui/open-webui/pull/24183
Author: @turicas
Created: 4/27/2026
Status: 🔄 Open
Base:
dev← Head:fix/firecrawl-multiple-requests📝 Commits (4)
c3edd41test: add SafeFireCrawlLoader contract tests03edefbfix: continue on per-URL errors in SafeFireCrawlLoader28a4df7fix: honor requests_per_second in FireCrawl loader71e574efix(ui): correct Web Loader config meaning📊 Changes
3 files changed (+252 additions, -17 deletions)
View changed files
📝
backend/open_webui/retrieval/web/utils.py(+25 -14)➕
backend/open_webui/test/test_retrieval_web_firecrawl_loader.py(+218 -0)📝
src/lib/components/admin/Settings/WebSearch.svelte(+9 -3)📄 Description
Before submitting, make sure you've checked the following:
devbranch. PRs targetingmainwill be immediately closed.devto ensure no unrelated commits (e.g. frommain) are included. Push updates to the existing PR branch instead of closing and reopening.Description
SafeFireCrawlLoader.lazy_loadandSafeFireCrawlLoader.alazy_loadhadtry/exceptwrapping the entireforloop instead of each iteration. The first URL that returned an HTTP error from Firecrawl (e.g. 402, 403, 404, 400 -- none of which are in the helper's retry list) terminated the loop and silently dropped every URL that came after it. Thecontinue_on_failure=Trueflag was effectively a no-op: it suppressed the exception but did not actually continue iterating.This also has a direct cost impact: every successful scrape before the failing URL is billed by Firecrawl (the request completed on their end) but discarded before reaching the model, so the user pays for content that neither they nor the model ever see.
While investigating this, I found a second bug:
SafeFireCrawlLoaderinheritsRateLimitMixinand storesrequests_per_secondfrom its constructor, but neitherlazy_loadnoralazy_loadever invoked the wait method. The configured value (admin setting and env varWEB_LOADER_CONCURRENT_REQUESTS) was silently ignored.Both fixes mirror the existing pattern in
SafePlaywrightURLLoaderin the same file: thetry/exceptmoves inside the loop withcontinuein the handler and_sync_wait_for_rate_limit/_wait_for_rate_limitare invoked per iteration.alazy_loadis also restructured to scrape one URL at a time in the threadpool and yield as it goes, instead of collecting the whole batch into a list first. The per-URL warning now identifies which URL failed.The admin UI labeled the Web Loader's
WEB_LOADER_CONCURRENT_REQUESTSsetting as "Concurrent Requests", but the underlying value is passed asrequests_per_secondand used as a sequential rate limiter (the loop is sequential in every loader). The label was renamed to "Requests per second" and a tooltip was added describing the actual behavior. The config key and env var name were preserved for backward compatibility. The Web Search section's "Concurrent Requests" label is left unchanged because that one is genuinely concurrency control (asyncio.Semaphoreover parallel queries).Added
backend/open_webui/test/test_retrieval_web_firecrawl_loader.py: contract tests forSafeFireCrawlLoader.lazy_loadandSafeFireCrawlLoader.alazy_load. Tests mockscrape_firecrawl_urland require no HTTP server, no Firecrawl SDK and no network access. They cover the happy path, per-URL failure recovery (the regression), the warning message identifying the failing URL,None-handling (empty markdown),continue_on_failure=Falsepropagation and that the rate-limit wait is invoked before every scrape (sync and async).Fixed
SafeFireCrawlLoader.lazy_loadandSafeFireCrawlLoader.alazy_loadnow actually continue past a per-URL failure whencontinue_on_failure=True(the default), instead of aborting the rest of the batch on the first error.SafeFireCrawlLoadernow honors the configuredrequests_per_second(admin setting, env varWEB_LOADER_CONCURRENT_REQUESTS). Previously the setting was silently ignored (the loader fired requests as fast as the loop ran).Changed
SafeFireCrawlLoader.alazy_loadis now a true streaming async generator: it scrapes URLs one at a time in the threadpool and yields each document as it becomes available. The previous implementation delegated tolazy_loadvialist(self.lazy_load()), blocking until every URL was scraped before yielding the first document.Security
None.
Breaking Changes
None.
Deprecated
None.
Removed
None.
Additional Information
Running the implemented tests against
dev(i.e. before applying the fix commits in this PR) - 4 failures:After - all 11 pass:
Screenshots or Videos
The label change and the new tooltip:

Contributor License Agreement
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.