[GH-ISSUE #23491] feat: Recursively get all documentation from a site (made in Sphinx, etc.) #74603

Closed
opened 2026-05-13 07:24:23 -05:00 by GiteaMirror · 1 comment
Owner

Originally created by @ibobak on GitHub (Apr 7, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/23491

Check Existing Issues

  • I have searched for all existing open AND closed issues and discussions for similar requests. I have found none that is comparable to my request.

Verify Feature Scope

  • I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.

Problem Description

Imagine there is a site - documentation of some library, e.g. https://lifetimes.readthedocs.io/en/latest/

Without this feature I have three options now:

  1. If the product is open source: I have to Clone the repo (best for Sphinx/ReadTheDocs projects) - the source files are already there, no crawling needed. I will just add all markdown files from docs folder.
  2. Make a crawl script which will scan all the tree and outputs individual Markdown files, then I need to upload them to a Knowledge Base
  3. Same crawling, but sawing pages to pdf, for example, and then to manually add 100 pdf files.

A lot of work, really.

Desired Solution you'd like

I would like to provide a single link of the root page to Open WebUI, e.g. to provide it https://lifetimes.readthedocs.io/en/latest/, and I expect from it to crawl the whole tree of this documentation - all 100 pages (doesn't matter), and to add them as documents, so that I won't spend on programmatic crawling.

Alternatives Considered

No response

Additional Context

No response

Originally created by @ibobak on GitHub (Apr 7, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/23491 ### Check Existing Issues - [x] I have searched for all existing **open AND closed** issues and discussions for similar requests. I have found none that is comparable to my request. ### Verify Feature Scope - [x] I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions. ### Problem Description Imagine there is a site - documentation of some library, e.g. https://lifetimes.readthedocs.io/en/latest/ Without this feature I have three options now: 1. If the product is open source: I have to Clone the repo (best for Sphinx/ReadTheDocs projects) - the source files are already there, no crawling needed. I will just add all markdown files from docs folder. 2. Make a crawl script which will scan all the tree and outputs individual Markdown files, then I need to upload them to a Knowledge Base 3. Same crawling, but sawing pages to pdf, for example, and then to manually add 100 pdf files. A lot of work, really. ### Desired Solution you'd like I would like to provide a single link of the root page to Open WebUI, e.g. to provide it https://lifetimes.readthedocs.io/en/latest/, and I expect from it to crawl the whole tree of this documentation - all 100 pages (doesn't matter), and to add them as documents, so that I won't spend on programmatic crawling. ### Alternatives Considered _No response_ ### Additional Context _No response_
Author
Owner

@ibobak commented on GitHub (Apr 8, 2026):

This is what I am talking about

Image

And by the way, there is an equivalent which I've found yesterday, providing here just for reference purposes:
https://chromewebstore.google.com/detail/notebooklm-websync-full-s/hjoonjdnhagnpfgifhjolheimamcafok

<!-- gh-comment-id:4204273718 --> @ibobak commented on GitHub (Apr 8, 2026): This is what I am talking about <img width="540" height="221" alt="Image" src="https://github.com/user-attachments/assets/f1a78733-016f-4354-bfa9-2a0a1f6eefea" /> And by the way, there is an equivalent which I've found yesterday, providing here just for reference purposes: https://chromewebstore.google.com/detail/notebooklm-websync-full-s/hjoonjdnhagnpfgifhjolheimamcafok
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#74603