github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-16 06:03:26 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[PR #23934] [MERGED] refactor(firecrawl): use v2 API #98484

New Issue
Closed
opened 2026-05-16 01:17:39 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-16 01:17:39 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/23934
Author: @RomualdYT
Created: 4/21/2026
Status: ✅ Merged
Merged: 4/24/2026
Merged by: @tjbck

Base: dev ← Head: firecrawl/v2-direct-api-pr

📝 Commits (2)

  • ec59a2a refactor(firecrawl): use v2 API directly
  • 281ab1e Merge branch 'dev' into firecrawl/v2-direct-api-pr

📊 Changes

5 files changed (+215 additions, -78 deletions)

View changed files

📝 backend/open_webui/retrieval/web/firecrawl.py (+202 -25)
📝 backend/open_webui/retrieval/web/utils.py (+12 -30)
📝 backend/requirements.txt (+0 -3)
📝 pyproject.toml (+0 -1)
📝 uv.lock (+1 -19)

📄 Description

Pull Request Checklist

  • Target branch: This pull request targets the dev branch.
  • Description: Refactors Firecrawl web retrieval to use direct Firecrawl v2 HTTP API calls instead of the Python SDK.
  • Changelog: Changelog entry added below.
  • Documentation: No docs update included; this does not add new user-facing settings or UI.
  • Dependencies: Removes the unused firecrawl-py backend dependency. Firecrawl calls now use existing HTTP dependencies.
  • Testing: Performed targeted local checks listed below.
  • Agentic AI Code: This PR was AI-assisted and manually reviewed/tested before submission.
  • Code review: Self-reviewed for scope, dependency impact, and compatibility with the existing Firecrawl loader flow.
  • Design & Architecture: Keeps the existing single-URL direct scrape behavior introduced upstream and avoids adding new settings.
  • Git Hygiene: Atomic PR with one logical backend/dependency change.
  • Title Prefix: refactor

Changelog Entry

Description

  • Refactor Firecrawl web retrieval to call the Firecrawl v2 API directly, removing the need for the Python SDK while preserving the existing Open WebUI Firecrawl search and web loader interfaces.

Added

  • Added shared Firecrawl HTTP helpers for v2 URL construction, headers, timeout handling, and transient retry handling.
  • Added direct /v2/scrape usage for Firecrawl web loading.
  • Added direct /v2/search usage for Firecrawl web search.

Changed

  • Firecrawl search now uses the Firecrawl v2 HTTP API directly.
  • Firecrawl web loading now uses the Firecrawl v2 scrape endpoint directly.
  • Firecrawl request timeout handling keeps the local HTTP timeout slightly above the Firecrawl scrape timeout to reduce premature local timeout failures.
  • Firecrawl transient failures on 429 and 5xx responses are retried with a short backoff.

Deprecated

  • None.

Removed

  • Removed the unused firecrawl-py backend dependency from backend/requirements.txt, pyproject.toml, and uv.lock.

Fixed

  • Reduces dependency overhead now that Firecrawl runtime paths no longer require the SDK.
  • Improves resilience for transient Firecrawl 429 and 5xx responses.
  • Preserves local Firecrawl SSL behavior by keeping request certificate verification tied to the existing loader verify_ssl option.

Security

  • No new security-sensitive configuration added.
  • Existing URL validation and loader behavior are preserved.

Breaking Changes

  • BREAKING CHANGE: None expected for Firecrawl v2-compatible deployments.
  • Note: Firecrawl API base URLs must point to a Firecrawl deployment that supports the v2 API.

Additional Information

  • This follows the recent upstream direction of using direct Firecrawl scrape requests instead of the SDK, while moving those calls to Firecrawl v2 and removing the now-unused dependency.
  • Scope is intentionally limited: no UI changes, no new admin settings, no batch scrape mode, no crawl/map support.

Testing

  • python -m ruff check backend/open_webui/retrieval/web/firecrawl.py
  • python -m py_compile backend/open_webui/retrieval/web/firecrawl.py backend/open_webui/retrieval/web/utils.py
  • git diff --check upstream/dev...HEAD
  • Verified no remaining references to FirecrawlApp, from firecrawl, firecrawl-py, /v1/search, or /v1/scrape.
  • Ran a local stubbed smoke test for v2 URL construction, timeout clamping, and retry behavior.

Screenshots or Videos

  • Not applicable; backend-only refactor.

Contributor License Agreement

Note

Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/23934 **Author:** [@RomualdYT](https://github.com/RomualdYT) **Created:** 4/21/2026 **Status:** ✅ Merged **Merged:** 4/24/2026 **Merged by:** [@tjbck](https://github.com/tjbck) **Base:** `dev` ← **Head:** `firecrawl/v2-direct-api-pr` --- ### 📝 Commits (2) - [`ec59a2a`](https://github.com/open-webui/open-webui/commit/ec59a2a9795d50a8eaeb0b65a526275fba151d1f) refactor(firecrawl): use v2 API directly - [`281ab1e`](https://github.com/open-webui/open-webui/commit/281ab1e609342a8d1f42c9871b7805b9a25394cd) Merge branch 'dev' into firecrawl/v2-direct-api-pr ### 📊 Changes **5 files changed** (+215 additions, -78 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/retrieval/web/firecrawl.py` (+202 -25) 📝 `backend/open_webui/retrieval/web/utils.py` (+12 -30) 📝 `backend/requirements.txt` (+0 -3) 📝 `pyproject.toml` (+0 -1) 📝 `uv.lock` (+1 -19) </details> ### 📄 Description <!-- ⚠️ CRITICAL CHECKS FOR CONTRIBUTORS (READ, DON'T DELETE) ⚠️ 1. Target the `dev` branch. PRs targeting `main` will be automatically closed. 2. Do NOT delete the CLA section at the bottom. It is required for the bot to accept your PR. --> # Pull Request Checklist - [x] **Target branch:** This pull request targets the `dev` branch. - [x] **Description:** Refactors Firecrawl web retrieval to use direct Firecrawl v2 HTTP API calls instead of the Python SDK. - [x] **Changelog:** Changelog entry added below. - [ ] **Documentation:** No docs update included; this does not add new user-facing settings or UI. - [x] **Dependencies:** Removes the unused `firecrawl-py` backend dependency. Firecrawl calls now use existing HTTP dependencies. - [x] **Testing:** Performed targeted local checks listed below. - [x] **Agentic AI Code:** This PR was AI-assisted and manually reviewed/tested before submission. - [x] **Code review:** Self-reviewed for scope, dependency impact, and compatibility with the existing Firecrawl loader flow. - [x] **Design & Architecture:** Keeps the existing single-URL direct scrape behavior introduced upstream and avoids adding new settings. - [x] **Git Hygiene:** Atomic PR with one logical backend/dependency change. - [x] **Title Prefix:** `refactor` # Changelog Entry ### Description - Refactor Firecrawl web retrieval to call the Firecrawl v2 API directly, removing the need for the Python SDK while preserving the existing Open WebUI Firecrawl search and web loader interfaces. ### Added - Added shared Firecrawl HTTP helpers for v2 URL construction, headers, timeout handling, and transient retry handling. - Added direct `/v2/scrape` usage for Firecrawl web loading. - Added direct `/v2/search` usage for Firecrawl web search. ### Changed - Firecrawl search now uses the Firecrawl v2 HTTP API directly. - Firecrawl web loading now uses the Firecrawl v2 scrape endpoint directly. - Firecrawl request timeout handling keeps the local HTTP timeout slightly above the Firecrawl scrape timeout to reduce premature local timeout failures. - Firecrawl transient failures on `429` and `5xx` responses are retried with a short backoff. ### Deprecated - None. ### Removed - Removed the unused `firecrawl-py` backend dependency from `backend/requirements.txt`, `pyproject.toml`, and `uv.lock`. ### Fixed - Reduces dependency overhead now that Firecrawl runtime paths no longer require the SDK. - Improves resilience for transient Firecrawl `429` and `5xx` responses. - Preserves local Firecrawl SSL behavior by keeping request certificate verification tied to the existing loader `verify_ssl` option. ### Security - No new security-sensitive configuration added. - Existing URL validation and loader behavior are preserved. ### Breaking Changes - **BREAKING CHANGE**: None expected for Firecrawl v2-compatible deployments. - Note: Firecrawl API base URLs must point to a Firecrawl deployment that supports the v2 API. --- ### Additional Information - This follows the recent upstream direction of using direct Firecrawl scrape requests instead of the SDK, while moving those calls to Firecrawl v2 and removing the now-unused dependency. - Scope is intentionally limited: no UI changes, no new admin settings, no batch scrape mode, no crawl/map support. ### Testing - `python -m ruff check backend/open_webui/retrieval/web/firecrawl.py` - `python -m py_compile backend/open_webui/retrieval/web/firecrawl.py backend/open_webui/retrieval/web/utils.py` - `git diff --check upstream/dev...HEAD` - Verified no remaining references to `FirecrawlApp`, `from firecrawl`, `firecrawl-py`, `/v1/search`, or `/v1/scrape`. - Ran a local stubbed smoke test for v2 URL construction, timeout clamping, and retry behavior. ### Screenshots or Videos - Not applicable; backend-only refactor. ### Contributor License Agreement <!-- 🚨 DO NOT DELETE THE TEXT BELOW 🚨 Keep the "Contributor License Agreement" confirmation text intact. Deleting it will trigger the CLA-Bot to INVALIDATE your PR. Your PR will NOT be reviewed or merged until you check the box below confirming that you have read and agree to the terms of the CLA. --> - [X] By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. > [!NOTE] > Deleting the CLA section will lead to immediate closure of your PR and it will not be merged in. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-16 01:17:39 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#98484
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?