[PR #18635] [MERGED] refactor: replace requests with Firecrawl SDK in search and requests Firecrawl SDK in scrape rather than langchain_community FireCrawlLoader #95896

New Issue

2026-05-15T22:06:11-05:00

GiteaMirror commented

2026-05-15 22:06:11 -05:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/18635
Author: @wei840222
Created: 10/26/2025
Status: ✅ Merged
Merged: 10/26/2025
Merged by: @tjbck

Base: dev ← Head: dev

📝 Commits (1)

7a3f4d8 refactor: replace requests with Firecrawl SDK in search and requests Firecrawl SDK in scrape rather than langchain_community FireCrawlLoader

📊 Changes

4 files changed (+87 additions, -63 deletions)

View changed files

📝 backend/open_webui/retrieval/web/firecrawl.py (+9 -18)
📝 backend/open_webui/retrieval/web/utils.py (+76 -43)
📝 backend/requirements.txt (+1 -1)
📝 pyproject.toml (+1 -1)

📄 Description

Pull Request Checklist

Note to first-time contributors: Please open a discussion post in Discussions and describe your changes before submitting a pull request.

https://github.com/open-webui/open-webui/discussions/18631

Before submitting, make sure you've checked the following:

Target branch: Verify that the pull request targets the dev branch. Not targeting the dev branch may lead to immediate closure of the PR.
Description: Provide a concise description of the changes made in this pull request.
Changelog: Ensure a changelog entry following the format of Keep a Changelog is added at the bottom of the PR description.
Documentation: If necessary, update relevant documentation Open WebUI Docs like environment variables, the tutorials, or other documentation sources.
Dependencies: Are there any new dependencies? Have you updated the dependency versions in the documentation?
Testing: Perform manual tests to verify the implemented fix/feature works as intended AND does not break any other functionality. Take this as an opportunity to make screenshots of the feature/fix and include it in the PR description.
Agentic AI Code:: Confirm this Pull Request is not written by any AI Agent or has at least gone through additional human review and manual testing. If any AI Agent is the co-author of this PR, it may lead to immediate closure of the PR.
Code review: Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards?
Title Prefix: To clearly categorize this pull request, prefix the pull request title using one of the following:
- refactor: Code restructuring for better maintainability, readability, or scalability

Changelog Entry

Description

Refactored Firecrawl integration to use the official Firecrawl SDK instead of direct HTTP requests and langchain_community FireCrawlLoader for improved reliability, performance, and maintainability

Added

Direct integration with Firecrawl SDK for search functionality
Batch scraping support using Firecrawl SDK's batch_scrape method
Enhanced error handling and timeout configuration
Support for markdown format extraction with base64 image removal

Changed

BREAKING CHANGE: Updated firecrawl-py dependency from version 1.12.0 to 4.5.0
Replaced manual HTTP requests with Firecrawl SDK in search functionality
Replaced langchain_community FireCrawlLoader with native Firecrawl SDK implementation
Updated SafeFireCrawlLoader to use batch_scrape for improved performance
Changed default mode from 'crawl' to 'scrape' in SafeFireCrawlLoader
Enhanced timeout handling with dynamic wait times based on URL count
Improved SSL verification and URL validation

Deprecated

Direct HTTP requests to Firecrawl API endpoints
Usage of langchain_community FireCrawlLoader for Firecrawl operations

Removed

Manual HTTP request implementation for Firecrawl search
Dependency on langchain_community.document_loaders.firecrawl.FireCrawlLoader
Unused imports (requests, urljoin, defaultdict, AIOHTTP_CLIENT_SESSION_SSL)

Fixed

Improved error handling in batch operations
Better URL processing and validation
Enhanced metadata handling for scraped documents

Security

Enhanced SSL/TLS verification options
Improved URL validation and filtering

Breaking Changes

BREAKING CHANGE: Major version update of firecrawl-py from 1.12.0 to 4.5.0 may require configuration updates
BREAKING CHANGE: API interface changes may affect custom Firecrawl implementations

Additional Information

This refactoring improves the Firecrawl integration by:

Leveraging the official Firecrawl SDK for better stability and feature support
Implementing batch processing for better performance when handling multiple URLs
Providing enhanced error handling and timeout management
Reducing maintenance overhead by removing custom HTTP implementation
Improving code readability and maintainability

The changes maintain backward compatibility for most use cases while providing performance improvements through batch operations.

Screenshots or Videos

[Manual testing screenshots would be added here demonstrating the Firecrawl functionality working correctly]

https://github.com/user-attachments/assets/c49f3cae-e193-4831-ac71-19f70768bd74

Contributor License Agreement

By submitting this pull request, I confirm that I have read and fully agree to the Contributor License Agreement (CLA), and I am providing my contributions under its terms.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/18635 **Author:** [@wei840222](https://github.com/wei840222) **Created:** 10/26/2025 **Status:** ✅ Merged **Merged:** 10/26/2025 **Merged by:** [@tjbck](https://github.com/tjbck) **Base:** `dev` ← **Head:** `dev` --- ### 📝 Commits (1) - [`7a3f4d8`](https://github.com/open-webui/open-webui/commit/7a3f4d85f6a1a9567ae6f45dd8e4bedc5b9f0c13) refactor: replace requests with Firecrawl SDK in search and requests Firecrawl SDK in scrape rather than langchain_community FireCrawlLoader ### 📊 Changes **4 files changed** (+87 additions, -63 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/retrieval/web/firecrawl.py` (+9 -18) 📝 `backend/open_webui/retrieval/web/utils.py` (+76 -43) 📝 `backend/requirements.txt` (+1 -1) 📝 `pyproject.toml` (+1 -1) </details> ### 📄 Description # Pull Request Checklist ### Note to first-time contributors: Please open a discussion post in [Discussions](https://github.com/open-webui/open-webui/discussions) and describe your changes before submitting a pull request. https://github.com/open-webui/open-webui/discussions/18631 **Before submitting, make sure you've checked the following:** - [x] **Target branch:** Verify that the pull request targets the `dev` branch. Not targeting the `dev` branch may lead to immediate closure of the PR. - [x] **Description:** Provide a concise description of the changes made in this pull request. - [x] **Changelog:** Ensure a changelog entry following the format of [Keep a Changelog](https://keepachangelog.com/) is added at the bottom of the PR description. - [x] **Documentation:** If necessary, update relevant documentation [Open WebUI Docs](https://github.com/open-webui/docs) like environment variables, the tutorials, or other documentation sources. - [x] **Dependencies:** Are there any new dependencies? Have you updated the dependency versions in the documentation? - [x] **Testing:** Perform manual tests to verify the implemented fix/feature works as intended AND does not break any other functionality. Take this as an opportunity to make screenshots of the feature/fix and include it in the PR description. - [x] **Agentic AI Code:**: Confirm this Pull Request is **not written by any AI Agent** or has at least gone through additional human review **and** manual testing. If any AI Agent is the co-author of this PR, it may lead to immediate closure of the PR. - [x] **Code review:** Have you performed a self-review of your code, addressing any coding standard issues and ensuring adherence to the project's coding standards? - [x] **Title Prefix:** To clearly categorize this pull request, prefix the pull request title using one of the following: - **refactor**: Code restructuring for better maintainability, readability, or scalability # Changelog Entry ### Description - Refactored Firecrawl integration to use the official Firecrawl SDK instead of direct HTTP requests and langchain_community FireCrawlLoader for improved reliability, performance, and maintainability ### Added - Direct integration with Firecrawl SDK for search functionality - Batch scraping support using Firecrawl SDK's batch_scrape method - Enhanced error handling and timeout configuration - Support for markdown format extraction with base64 image removal ### Changed - **BREAKING CHANGE**: Updated firecrawl-py dependency from version 1.12.0 to 4.5.0 - Replaced manual HTTP requests with Firecrawl SDK in search functionality - Replaced langchain_community FireCrawlLoader with native Firecrawl SDK implementation - Updated SafeFireCrawlLoader to use batch_scrape for improved performance - Changed default mode from 'crawl' to 'scrape' in SafeFireCrawlLoader - Enhanced timeout handling with dynamic wait times based on URL count - Improved SSL verification and URL validation ### Deprecated - Direct HTTP requests to Firecrawl API endpoints - Usage of langchain_community FireCrawlLoader for Firecrawl operations ### Removed - Manual HTTP request implementation for Firecrawl search - Dependency on langchain_community.document_loaders.firecrawl.FireCrawlLoader - Unused imports (requests, urljoin, defaultdict, AIOHTTP_CLIENT_SESSION_SSL) ### Fixed - Improved error handling in batch operations - Better URL processing and validation - Enhanced metadata handling for scraped documents ### Security - Enhanced SSL/TLS verification options - Improved URL validation and filtering ### Breaking Changes - **BREAKING CHANGE**: Major version update of firecrawl-py from 1.12.0 to 4.5.0 may require configuration updates - **BREAKING CHANGE**: API interface changes may affect custom Firecrawl implementations --- ### Additional Information This refactoring improves the Firecrawl integration by: - Leveraging the official Firecrawl SDK for better stability and feature support - Implementing batch processing for better performance when handling multiple URLs - Providing enhanced error handling and timeout management - Reducing maintenance overhead by removing custom HTTP implementation - Improving code readability and maintainability The changes maintain backward compatibility for most use cases while providing performance improvements through batch operations. ### Screenshots or Videos - [Manual testing screenshots would be added here demonstrating the Firecrawl functionality working correctly] https://github.com/user-attachments/assets/c49f3cae-e193-4831-ac71-19f70768bd74 ### Contributor License Agreement By submitting this pull request, I confirm that I have read and fully agree to the [Contributor License Agreement (CLA)](https://github.com/open-webui/open-webui/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT), and I am providing my contributions under its terms. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2026-05-15 22:06:11 -05:00

GiteaMirror closed this issue

2026-05-15 22:06:13 -05:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#95896