enh: playwright/selenium web search support for RAG #1051

New Issue

2025-11-11T14:36:18-06:00

GiteaMirror commented

2025-11-11 14:36:18 -06:00

Originally created by @tjbck on GitHub (May 28, 2024).

GiteaMirror closed this issue

2025-11-11 14:36:19 -06:00

GiteaMirror commented

2025-11-11 14:36:19 -06:00

@noperator commented on GitHub (Jan 5, 2025):

Is the idea here that, instead of just using langchain_community.document_loaders.WebBaseLoader._scrape() (below), we also optionally use something like AsyncChromiumLoader?

https://github.com/open-webui/open-webui/blob/4bc9904b3cd0726d3f9c3cbaeade972cf167b6c4/backend/open_webui/retrieval/web/utils.py#L57-L64

In addition to loading results dynamically, I'm also interested in adding support for:

loading results from archive sites: https://web.archive.org/, https://archive.ph/
extracting the "main text of a site" (e.g., with Trafilatura)
using multiple queries

@noperator commented on GitHub (Jan 5, 2025): Is the idea here that, instead of just using `langchain_community.document_loaders.WebBaseLoader._scrape()` (below), we also optionally use something like [`AsyncChromiumLoader`](https://python.langchain.com/docs/integrations/document_loaders/async_chromium/)? https://github.com/open-webui/open-webui/blob/4bc9904b3cd0726d3f9c3cbaeade972cf167b6c4/backend/open_webui/retrieval/web/utils.py#L57-L64 In addition to loading results dynamically, I'm also interested in adding support for: - loading results from archive sites: https://web.archive.org/, https://archive.ph/ - extracting the "main text of a site" (e.g., with [Trafilatura](https://github.com/adbar/trafilatura#:~:text=Robust%20and%20configurable%20extraction%20of%20key%20elements%3A)) - using [multiple queries](https://github.com/open-webui/open-webui/issues/7876)

GiteaMirror commented

2025-11-11 14:36:19 -06:00

@roryeckel commented on GitHub (Jan 28, 2025):

I am testing some code changes here. First roadblock with AsyncChromiumLoader is its internal usage of asyncio.run is causing "ERROR: asyncio.run() cannot be called from a running event loop"

I think PlaywrightURLLoader may be a better alternative because it has both sync and async implementations specific to Playwright.
However when I tried it, I also got "playwright._impl._errors.Error: It looks like you are using Playwright Sync API inside the asyncio loop. Please use the Async API instead."

Perhaps retrieval.py needs process_web_search to be refactored to async. However when I tried this, the event emitter events didn't go in order properly. Oof, can't catch a break, but I'll keep trying

Edit: managed to get a draft working & still cleaning up

@roryeckel commented on GitHub (Jan 28, 2025): I am testing some code changes here. First roadblock with AsyncChromiumLoader is its internal usage of asyncio.run is causing "ERROR: asyncio.run() cannot be called from a running event loop" I think [PlaywrightURLLoader](https://api.python.langchain.com/en/latest/community/document_loaders/langchain_community.document_loaders.url_playwright.PlaywrightURLLoader.html) may be a better alternative because it has both sync and async implementations specific to Playwright. However when I tried it, I also got "playwright._impl._errors.Error: It looks like you are using Playwright Sync API inside the asyncio loop. Please use the Async API instead." Perhaps retrieval.py needs process_web_search to be refactored to async. However when I tried this, the event emitter events didn't go in order properly. Oof, can't catch a break, but I'll keep trying Edit: managed to get a draft working & still cleaning up

GiteaMirror commented

2025-11-11 14:36:19 -06:00

@roryeckel commented on GitHub (Jan 29, 2025):

Please see my progress, I'm open to feedback. Will create PR soon

It is functioning great for me here. With my added URL validation, it seems to be more stable even in the old mode as well. https://github.com/roryeckel/open-webui/commit/4e8b3906821a9a10f4fd0038373291dff41b65cf

TODO for myself: update documentation for environment variable "RAG_WEB_LOADER", fix internationalization I forgot about, and test docker

@roryeckel commented on GitHub (Jan 29, 2025): Please see my progress, I'm open to feedback. Will create PR soon It is functioning great for me here. With my added URL validation, it seems to be more stable even in the old mode as well. https://github.com/roryeckel/open-webui/commit/4e8b3906821a9a10f4fd0038373291dff41b65cf TODO for myself: update documentation for environment variable "RAG_WEB_LOADER", fix internationalization I forgot about, and test docker

GiteaMirror commented

2025-11-11 14:36:20 -06:00

@roryeckel commented on GitHub (Jan 31, 2025):

Fixed the internationalization and cleaned up a bit.
However I did test docker and it does not work, 403s getting thrown by Playwright. I will investigate that

https://github.com/roryeckel/open-webui

Update: it seems like the 403 error is actually coming from download_nltk_packages. However, the startup script has already been adjusted to install the nltk package if the environment variable is set.
Error is caused by this access denial: https://utic-public-cf.s3.amazonaws.com/nltk_data_3.8.2.tar.gz
Our unstructured package is far too out of date, they revamped the downloading system to use a different method and URL

Also, I'd like to move the chromium dep installation into the data cache folder

@roryeckel commented on GitHub (Jan 31, 2025): Fixed the internationalization and cleaned up a bit. However I did test docker and it does not work, 403s getting thrown by Playwright. I will investigate that https://github.com/roryeckel/open-webui Update: it seems like the 403 error is actually coming from download_nltk_packages. However, the startup script has already been adjusted to install the nltk package if the environment variable is set. Error is caused by this access denial: https://utic-public-cf.s3.amazonaws.com/nltk_data_3.8.2.tar.gz Our unstructured package is far too out of date, they revamped the downloading system to use a different method and URL Also, I'd like to move the chromium dep installation into the data cache folder

GiteaMirror commented

2025-11-11 14:36:20 -06:00

@roryeckel commented on GitHub (Feb 1, 2025):

I've written documentation for my feature: https://github.com/roryeckel/open-webui-docs/commit/452c447edc1fd25d90b9f06e6d40108527b8c053

@roryeckel commented on GitHub (Feb 1, 2025): I've written documentation for my feature: https://github.com/roryeckel/open-webui-docs/commit/452c447edc1fd25d90b9f06e6d40108527b8c053

GiteaMirror commented

2025-11-11 14:36:20 -06:00

@roryeckel commented on GitHub (Feb 2, 2025):

After updating the unstructured package, Playwright + unstructured web loader is functioning in docker.
I tested by using the official docker compose with environment "RAG_WEB_LOADER=playwright" set on my branch:

https://github.com/roryeckel/open-webui
https://github.com/roryeckel/open-webui-docs

May someone please share the next steps for me to take to get this reviewed? Should I create a PR? Thanks in advance!

@roryeckel commented on GitHub (Feb 2, 2025): After updating the unstructured package, Playwright + unstructured web loader is functioning in docker. I tested by using the official docker compose with environment "RAG_WEB_LOADER=playwright" set on my branch: https://github.com/roryeckel/open-webui https://github.com/roryeckel/open-webui-docs May someone please share the next steps for me to take to get this reviewed? Should I create a PR? Thanks in advance!

GiteaMirror commented

2025-11-11 14:36:20 -06:00

@roryeckel commented on GitHub (Feb 2, 2025):

After discussion in discord, we want to allow a secondary docker container to serve the playwright API for increased security and bandwidth efficiency. I will make these changes to enable both modes ASAP

docker run --rm -it -p 3000:3000 mcr.microsoft.com/playwright/python:v1.27.1 playwright run-server --port=3000
Listening on ws://127.0.0.1:3000

@roryeckel commented on GitHub (Feb 2, 2025): After discussion in discord, we want to allow a secondary docker container to serve the playwright API for increased security and bandwidth efficiency. I will make these changes to enable both modes ASAP docker run --rm -it -p 3000:3000 mcr.microsoft.com/playwright/python:v1.27.1 playwright run-server --port=3000 Listening on ws://127.0.0.1:3000

GiteaMirror commented

2025-11-11 14:36:20 -06:00

@roryeckel commented on GitHub (Feb 3, 2025):

I was successful in implementing a separate playwright container, as well as being able to switch to a different mode to install the dependencies directly inside the Open WebUI container. Added docker-compose.playwright.yaml and updated run-compose.sh. Also updated docs

@roryeckel commented on GitHub (Feb 3, 2025): I was successful in implementing a separate playwright container, as well as being able to switch to a different mode to install the dependencies directly inside the Open WebUI container. Added docker-compose.playwright.yaml and updated run-compose.sh. Also updated docs

GiteaMirror commented

2025-11-11 14:36:20 -06:00