[PR #12822] [MERGED] feat: Support for Self-Hosted/External Web Search/Loader Engines #9824

New Issue

GiteaMirror · 2025-11-11T18:32:40-06:00

GiteaMirror commented

2025-11-11 18:32:40 -06:00

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/12822
Author: @tth37
Created: 4/13/2025
Status: ✅ Merged
Merged: 4/19/2025
Merged by: @tjbck

Base: dev ← Head: feat_external_search_loader

📝 Commits (5)

839ba22 feat: Backend for Self-Hosted/External Web Search/Loader Engines
22f0365 format
008fec8 fix: Update external search/loader method to POST
85f8e91 feat: Allow admin editing external search/loader settings
dcb3f18 refac: styling

📊 Changes

7 files changed (+237 additions, -2 deletions)

View changed files

📝 backend/open_webui/config.py (+23 -0)
📝 backend/open_webui/main.py (+8 -0)
➕ backend/open_webui/retrieval/loaders/external.py (+53 -0)
➕ backend/open_webui/retrieval/web/external.py (+47 -0)
📝 backend/open_webui/retrieval/web/utils.py (+8 -0)
📝 backend/open_webui/routers/retrieval.py (+33 -0)
📝 src/lib/components/admin/Settings/WebSearch.svelte (+65 -2)

📄 Description

Related discussion: #12783

Issues reported: #8843 #9646 #12711

This PR aims at introducing the capability to configure and utilize self-hosted or external web search and web loader engines within Open WebUI.

Roadmap

Add config entries for external web search/loader
Add web search/loader handlers to perform the functionality
Modify webconfig endpoint and WebSearch.svelte correspondingly to allow admin editing of these settings via UI
(Future PR) Update corresponding documentation
(Future PR) Allowing users to set their own external endpoints in user-scope settings
(Future PR) Support external document loader

Proposed Solution

This PR adds new configuration options and logic to enable routing web search and web loading requests to user-defined external HTTP endpoints. When the external engine is selected, Open WebUI will send POST request to the specified service url for operations.

External Search Engine
- Endpoint: POST <EXTERNAL_WEB_SEARCH_URL>
- Auth: Bearer <EXTERNAL_WEB_SEARCH_API_KEY>
- Body: {"query": str, "count": int}
- Response: list[{"link": str, "title": str, "snippet": str}]
External Web Loader
- Endpoint: POST <EXTERNAL_WEB_LOADER_URL>
- Auth: Bearer <EXTERNAL_WEB_LOADER_API_KEY>
- Body: {"urls": list[str]}
- Response: list[{"page_content": str, "metadata": object}]

Changes

config.py: Added the new configuration variables
main.py: Incorporateed new configs into app.state.config object
retrieval/web/external.py: Implemented search_external function to call the external search endpoint
routers/retrieval.py: Modified search_web handler to delegate to search_external when the engine is external
retrieval/loaders/external.py: Added ExternalLoader class to handle calls to the external loader endpoint
retrieval/web/utils.py: Updated get_web_loader to return ExternalLoader when the loader engine is external

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/12822 **Author:** [@tth37](https://github.com/tth37) **Created:** 4/13/2025 **Status:** ✅ Merged **Merged:** 4/19/2025 **Merged by:** [@tjbck](https://github.com/tjbck) **Base:** `dev` ← **Head:** `feat_external_search_loader` --- ### 📝 Commits (5) - [`839ba22`](https://github.com/open-webui/open-webui/commit/839ba22c90991dc4d72810aa6f6a527664db80c2) feat: Backend for Self-Hosted/External Web Search/Loader Engines - [`22f0365`](https://github.com/open-webui/open-webui/commit/22f0365cef0d366632be48e20b25fbf09330f9ec) format - [`008fec8`](https://github.com/open-webui/open-webui/commit/008fec80c10b911068205a1782ea28507e3dea6b) fix: Update external search/loader method to POST - [`85f8e91`](https://github.com/open-webui/open-webui/commit/85f8e91288c201d68cece8f3ec3d08968a6d5fb1) feat: Allow admin editing external search/loader settings - [`dcb3f18`](https://github.com/open-webui/open-webui/commit/dcb3f18e1e4e342c736dc1c86f337467883bc1bb) refac: styling ### 📊 Changes **7 files changed** (+237 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/config.py` (+23 -0) 📝 `backend/open_webui/main.py` (+8 -0) ➕ `backend/open_webui/retrieval/loaders/external.py` (+53 -0) ➕ `backend/open_webui/retrieval/web/external.py` (+47 -0) 📝 `backend/open_webui/retrieval/web/utils.py` (+8 -0) 📝 `backend/open_webui/routers/retrieval.py` (+33 -0) 📝 `src/lib/components/admin/Settings/WebSearch.svelte` (+65 -2) </details> ### 📄 Description Related discussion: #12783 Issues reported: #8843 #9646 #12711 This PR aims at introducing the capability to configure and utilize self-hosted or external web search and web loader engines within Open WebUI. ### Roadmap - [x] Add config entries for external web search/loader - [x] Add web search/loader handlers to perform the functionality - [x] Modify webconfig endpoint and `WebSearch.svelte` correspondingly to allow admin editing of these settings via UI - [ ] (Future PR) Update corresponding documentation - [ ] (Future PR) Allowing users to set their own external endpoints in user-scope settings - [ ] (Future PR) Support external document loader ### Proposed Solution This PR adds new configuration options and logic to enable routing web search and web loading requests to user-defined external HTTP endpoints. When the `external` engine is selected, Open WebUI will send POST request to the specified service url for operations. 1. External Search Engine - Endpoint: `POST <EXTERNAL_WEB_SEARCH_URL>` - Auth: `Bearer <EXTERNAL_WEB_SEARCH_API_KEY>` - Body: `{"query": str, "count": int}` - Response: `list[{"link": str, "title": str, "snippet": str}]` 2. External Web Loader - Endpoint: `POST <EXTERNAL_WEB_LOADER_URL>` - Auth: `Bearer <EXTERNAL_WEB_LOADER_API_KEY>` - Body: `{"urls": list[str]}` - Response: `list[{"page_content": str, "metadata": object}]` ### Changes - `config.py`: Added the new configuration variables - `main.py`: Incorporateed new configs into `app.state.config` object - `retrieval/web/external.py`: Implemented `search_external` function to call the external search endpoint - `routers/retrieval.py`: Modified `search_web` handler to delegate to `search_external` when the engine is `external` - `retrieval/loaders/external.py`: Added `ExternalLoader` class to handle calls to the external loader endpoint - `retrieval/web/utils.py`: Updated `get_web_loader` to return `ExternalLoader` when the loader engine is `external` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

GiteaMirror added the pull-request label 2025-11-11 18:32:40 -06:00

GiteaMirror closed this issue

2025-11-11 18:32:40 -06:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#9824