Target branch: This pull request targets the dev branch.
Description: Refactors Firecrawl web retrieval to use direct Firecrawl v2 HTTP API calls instead of the Python SDK.
Changelog: Changelog entry added below.
Documentation: No docs update included; this does not add new user-facing settings or UI.
Dependencies: Removes the unused firecrawl-py backend dependency. Firecrawl calls now use existing HTTP dependencies.
Testing: Performed targeted local checks listed below.
Agentic AI Code: This PR was AI-assisted and manually reviewed/tested before submission.
Code review: Self-reviewed for scope, dependency impact, and compatibility with the existing Firecrawl loader flow.
Design & Architecture: Keeps the existing single-URL direct scrape behavior introduced upstream and avoids adding new settings.
Git Hygiene: Atomic PR with one logical backend/dependency change.
Title Prefix:refactor
Changelog Entry
Description
Refactor Firecrawl web retrieval to call the Firecrawl v2 API directly, removing the need for the Python SDK while preserving the existing Open WebUI Firecrawl search and web loader interfaces.
Added
Added shared Firecrawl HTTP helpers for v2 URL construction, headers, timeout handling, and transient retry handling.
Added direct /v2/scrape usage for Firecrawl web loading.
Added direct /v2/search usage for Firecrawl web search.
Changed
Firecrawl search now uses the Firecrawl v2 HTTP API directly.
Firecrawl web loading now uses the Firecrawl v2 scrape endpoint directly.
Firecrawl request timeout handling keeps the local HTTP timeout slightly above the Firecrawl scrape timeout to reduce premature local timeout failures.
Firecrawl transient failures on 429 and 5xx responses are retried with a short backoff.
Deprecated
None.
Removed
Removed the unused firecrawl-py backend dependency from backend/requirements.txt, pyproject.toml, and uv.lock.
Fixed
Reduces dependency overhead now that Firecrawl runtime paths no longer require the SDK.
Improves resilience for transient Firecrawl 429 and 5xx responses.
Preserves local Firecrawl SSL behavior by keeping request certificate verification tied to the existing loader verify_ssl option.
Security
No new security-sensitive configuration added.
Existing URL validation and loader behavior are preserved.
Breaking Changes
BREAKING CHANGE: None expected for Firecrawl v2-compatible deployments.
Note: Firecrawl API base URLs must point to a Firecrawl deployment that supports the v2 API.
Additional Information
This follows the recent upstream direction of using direct Firecrawl scrape requests instead of the SDK, while moving those calls to Firecrawl v2 and removing the now-unused dependency.
Scope is intentionally limited: no UI changes, no new admin settings, no batch scrape mode, no crawl/map support.
Verified no remaining references to FirecrawlApp, from firecrawl, firecrawl-py, /v1/search, or /v1/scrape.
Ran a local stubbed smoke test for v2 URL construction, timeout clamping, and retry behavior.
Screenshots or Videos
Not applicable; backend-only refactor.
Contributor License Agreement
By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.
Note
Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/open-webui/open-webui/pull/23934
**Author:** [@RomualdYT](https://github.com/RomualdYT)
**Created:** 4/21/2026
**Status:** ✅ Merged
**Merged:** 4/24/2026
**Merged by:** [@tjbck](https://github.com/tjbck)
**Base:** `dev` ← **Head:** `firecrawl/v2-direct-api-pr`
---
### 📝 Commits (2)
- [`ec59a2a`](https://github.com/open-webui/open-webui/commit/ec59a2a9795d50a8eaeb0b65a526275fba151d1f) refactor(firecrawl): use v2 API directly
- [`281ab1e`](https://github.com/open-webui/open-webui/commit/281ab1e609342a8d1f42c9871b7805b9a25394cd) Merge branch 'dev' into firecrawl/v2-direct-api-pr
### 📊 Changes
**5 files changed** (+215 additions, -78 deletions)
<details>
<summary>View changed files</summary>
📝 `backend/open_webui/retrieval/web/firecrawl.py` (+202 -25)
📝 `backend/open_webui/retrieval/web/utils.py` (+12 -30)
📝 `backend/requirements.txt` (+0 -3)
📝 `pyproject.toml` (+0 -1)
📝 `uv.lock` (+1 -19)
</details>
### 📄 Description
<!--
⚠️ CRITICAL CHECKS FOR CONTRIBUTORS (READ, DON'T DELETE) ⚠️
1. Target the `dev` branch. PRs targeting `main` will be automatically closed.
2. Do NOT delete the CLA section at the bottom. It is required for the bot to accept your PR.
-->
# Pull Request Checklist
- [x] **Target branch:** This pull request targets the `dev` branch.
- [x] **Description:** Refactors Firecrawl web retrieval to use direct Firecrawl v2 HTTP API calls instead of the Python SDK.
- [x] **Changelog:** Changelog entry added below.
- [ ] **Documentation:** No docs update included; this does not add new user-facing settings or UI.
- [x] **Dependencies:** Removes the unused `firecrawl-py` backend dependency. Firecrawl calls now use existing HTTP dependencies.
- [x] **Testing:** Performed targeted local checks listed below.
- [x] **Agentic AI Code:** This PR was AI-assisted and manually reviewed/tested before submission.
- [x] **Code review:** Self-reviewed for scope, dependency impact, and compatibility with the existing Firecrawl loader flow.
- [x] **Design & Architecture:** Keeps the existing single-URL direct scrape behavior introduced upstream and avoids adding new settings.
- [x] **Git Hygiene:** Atomic PR with one logical backend/dependency change.
- [x] **Title Prefix:** `refactor`
# Changelog Entry
### Description
- Refactor Firecrawl web retrieval to call the Firecrawl v2 API directly, removing the need for the Python SDK while preserving the existing Open WebUI Firecrawl search and web loader interfaces.
### Added
- Added shared Firecrawl HTTP helpers for v2 URL construction, headers, timeout handling, and transient retry handling.
- Added direct `/v2/scrape` usage for Firecrawl web loading.
- Added direct `/v2/search` usage for Firecrawl web search.
### Changed
- Firecrawl search now uses the Firecrawl v2 HTTP API directly.
- Firecrawl web loading now uses the Firecrawl v2 scrape endpoint directly.
- Firecrawl request timeout handling keeps the local HTTP timeout slightly above the Firecrawl scrape timeout to reduce premature local timeout failures.
- Firecrawl transient failures on `429` and `5xx` responses are retried with a short backoff.
### Deprecated
- None.
### Removed
- Removed the unused `firecrawl-py` backend dependency from `backend/requirements.txt`, `pyproject.toml`, and `uv.lock`.
### Fixed
- Reduces dependency overhead now that Firecrawl runtime paths no longer require the SDK.
- Improves resilience for transient Firecrawl `429` and `5xx` responses.
- Preserves local Firecrawl SSL behavior by keeping request certificate verification tied to the existing loader `verify_ssl` option.
### Security
- No new security-sensitive configuration added.
- Existing URL validation and loader behavior are preserved.
### Breaking Changes
- **BREAKING CHANGE**: None expected for Firecrawl v2-compatible deployments.
- Note: Firecrawl API base URLs must point to a Firecrawl deployment that supports the v2 API.
---
### Additional Information
- This follows the recent upstream direction of using direct Firecrawl scrape requests instead of the SDK, while moving those calls to Firecrawl v2 and removing the now-unused dependency.
- Scope is intentionally limited: no UI changes, no new admin settings, no batch scrape mode, no crawl/map support.
### Testing
- `python -m ruff check backend/open_webui/retrieval/web/firecrawl.py`
- `python -m py_compile backend/open_webui/retrieval/web/firecrawl.py backend/open_webui/retrieval/web/utils.py`
- `git diff --check upstream/dev...HEAD`
- Verified no remaining references to `FirecrawlApp`, `from firecrawl`, `firecrawl-py`, `/v1/search`, or `/v1/scrape`.
- Ran a local stubbed smoke test for v2 URL construction, timeout clamping, and retry behavior.
### Screenshots or Videos
- Not applicable; backend-only refactor.
### Contributor License Agreement
<!--
🚨 DO NOT DELETE THE TEXT BELOW 🚨
Keep the "Contributor License Agreement" confirmation text intact.
Deleting it will trigger the CLA-Bot to INVALIDATE your PR.
Your PR will NOT be reviewed or merged until you check the box below confirming that you have read and agree to the terms of the CLA.
-->
- [X] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms.
> [!NOTE]
> Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/open-webui/open-webui/pull/23934
Author: @RomualdYT
Created: 4/21/2026
Status: ✅ Merged
Merged: 4/24/2026
Merged by: @tjbck
Base:
dev← Head:firecrawl/v2-direct-api-pr📝 Commits (2)
ec59a2arefactor(firecrawl): use v2 API directly281ab1eMerge branch 'dev' into firecrawl/v2-direct-api-pr📊 Changes
5 files changed (+215 additions, -78 deletions)
View changed files
📝
backend/open_webui/retrieval/web/firecrawl.py(+202 -25)📝
backend/open_webui/retrieval/web/utils.py(+12 -30)📝
backend/requirements.txt(+0 -3)📝
pyproject.toml(+0 -1)📝
uv.lock(+1 -19)📄 Description
Pull Request Checklist
devbranch.firecrawl-pybackend dependency. Firecrawl calls now use existing HTTP dependencies.refactorChangelog Entry
Description
Added
/v2/scrapeusage for Firecrawl web loading./v2/searchusage for Firecrawl web search.Changed
429and5xxresponses are retried with a short backoff.Deprecated
Removed
firecrawl-pybackend dependency frombackend/requirements.txt,pyproject.toml, anduv.lock.Fixed
429and5xxresponses.verify_ssloption.Security
Breaking Changes
Additional Information
Testing
python -m ruff check backend/open_webui/retrieval/web/firecrawl.pypython -m py_compile backend/open_webui/retrieval/web/firecrawl.py backend/open_webui/retrieval/web/utils.pygit diff --check upstream/dev...HEADFirecrawlApp,from firecrawl,firecrawl-py,/v1/search, or/v1/scrape.Screenshots or Videos
Contributor License Agreement
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.