After web pages are loaded, the backend proceeds to save_doc_to_vector_db. However, if one of the documents has an empty page_content, the save_docs_to_vector_db operation fails, subsequently causing the entire web search to fail. Commit 70718dda90 partially addresses this issue by adding a check for empty doc.page_content.
However, there are cases where doc.page_content contains only whitespace, which also causes the save_docs_to_vector_db process to fail. This happens because save_docs_to_vector_db performs preprocessing on the page_content, including cleaning and splitting, which cannot handle whitespace-only content.
To ensure the web search process remains robust, we can wrap the save_docs_to_vector_db process in a try-catch block. This will handle potential errors gracefully, preventing them from failing the entire web search.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/open-webui/open-webui/pull/13079
**Author:** [@tth37](https://github.com/tth37)
**Created:** 4/20/2025
**Status:** ❌ Closed
**Base:** `dev` ← **Head:** `fix_web_rag_empty_content`
---
### 📝 Commits (1)
- [`ebb633f`](https://github.com/open-webui/open-webui/commit/ebb633fcf1a3cee00107f5735014bff6577ffe14) fix: Web RAG empty content
### 📊 Changes
**1 file changed** (+18 additions, -13 deletions)
<details>
<summary>View changed files</summary>
📝 `backend/open_webui/routers/retrieval.py` (+18 -13)
</details>
### 📄 Description
### Related Issue and Commit
#12832 70718dda90af07370414dea28f9a93058623a33b
### Problem Description
After web pages are loaded, the backend proceeds to `save_doc_to_vector_db`. However, if one of the documents has an empty `page_content`, the `save_docs_to_vector_db` operation fails, subsequently causing the entire web search to fail. Commit 70718dda90af07370414dea28f9a93058623a33b partially addresses this issue by adding a check for empty `doc.page_content`.
However, there are cases where `doc.page_content` contains **only whitespace**, which also causes the `save_docs_to_vector_db` process to fail. This happens because `save_docs_to_vector_db` performs preprocessing on the `page_content`, including cleaning and splitting, which cannot handle whitespace-only content.
https://github.com/open-webui/open-webui/blob/70718dda90af07370414dea28f9a93058623a33b/backend/open_webui/routers/retrieval.py#L862-L863
### Proposed Solution
To ensure the web search process remains robust, we can wrap the `save_docs_to_vector_db` process in a try-catch block. This will handle potential errors gracefully, preventing them from failing the entire web search.
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/open-webui/open-webui/pull/13079
Author: @tth37
Created: 4/20/2025
Status: ❌ Closed
Base:
dev← Head:fix_web_rag_empty_content📝 Commits (1)
ebb633ffix: Web RAG empty content📊 Changes
1 file changed (+18 additions, -13 deletions)
View changed files
📝
backend/open_webui/routers/retrieval.py(+18 -13)📄 Description
Related Issue and Commit
#12832
70718dda90Problem Description
After web pages are loaded, the backend proceeds to
save_doc_to_vector_db. However, if one of the documents has an emptypage_content, thesave_docs_to_vector_dboperation fails, subsequently causing the entire web search to fail. Commit70718dda90partially addresses this issue by adding a check for emptydoc.page_content.However, there are cases where
doc.page_contentcontains only whitespace, which also causes thesave_docs_to_vector_dbprocess to fail. This happens becausesave_docs_to_vector_dbperforms preprocessing on thepage_content, including cleaning and splitting, which cannot handle whitespace-only content.https://github.com/open-webui/open-webui/blob/70718dda90af07370414dea28f9a93058623a33b/backend/open_webui/routers/retrieval.py#L862-L863
Proposed Solution
To ensure the web search process remains robust, we can wrap the
save_docs_to_vector_dbprocess in a try-catch block. This will handle potential errors gracefully, preventing them from failing the entire web search.🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.