github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-10 22:22:56 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #21336] feat: Add option to make /api/v1/retrieval/process/web endpoint additive #106441

New Issue
Closed
opened 2026-05-18 04:47:33 -05:00 by GiteaMirror · 4 comments
GiteaMirror commented 2026-05-18 04:47:33 -05:00
Owner
Copy Link

Originally created by @jfahrenkrug on GitHub (Feb 12, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/21336

Check Existing Issues

  • I have searched for all existing open AND closed issues and discussions for similar requests. I have found none that is comparable to my request.

Verify Feature Scope

  • I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.

Problem Description

I've written a script that uses the sitemap of my website and then calls /api/v1/retrieval/process/web for every URL to ingest it into a knowledge base. It's working, but I was surprised how little data ended up in my Qdrant DB after successfully processing 602 webpages.
It turns out that every call to /api/v1/retrieval/process/web actually completely overrides the knowledge base. You can see that save_docs_to_vector_db is called with overwrite=True here: 2b26355002/backend/open_webui/routers/retrieval.py (L1884)

Desired Solution you'd like

I'd like to be able to add hundreds of URLs (processed website) to the same knowledge base, maybe by passing overwrite=False to the request.

Alternatives Considered

Using the files API or the text API endpoint.

Additional Context

No response

Originally created by @jfahrenkrug on GitHub (Feb 12, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/21336 ### Check Existing Issues - [x] I have searched for all existing **open AND closed** issues and discussions for similar requests. I have found none that is comparable to my request. ### Verify Feature Scope - [x] I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions. ### Problem Description I've written a script that uses the sitemap of my website and then calls /api/v1/retrieval/process/web for every URL to ingest it into a knowledge base. It's working, but I was surprised how little data ended up in my Qdrant DB after successfully processing 602 webpages. It turns out that every call to /api/v1/retrieval/process/web actually completely overrides the knowledge base. You can see that `save_docs_to_vector_db` is called with `overwrite=True` here: https://github.com/open-webui/open-webui/blob/2b26355002064228e9b671339f8f3fb9d1fafa73/backend/open_webui/routers/retrieval.py#L1884 ### Desired Solution you'd like I'd like to be able to add hundreds of URLs (processed website) to the same knowledge base, maybe by passing `overwrite=False` to the request. ### Alternatives Considered Using the files API or the text API endpoint. ### Additional Context _No response_
GiteaMirror commented 2026-05-18 04:47:34 -05:00
Author
Owner
Copy Link

@tjbck commented on GitHub (Feb 19, 2026):

Addressed in dev.

<!-- gh-comment-id:3930388424 --> @tjbck commented on GitHub (Feb 19, 2026): Addressed in dev.
GiteaMirror commented 2026-05-18 04:47:35 -05:00
Author
Owner
Copy Link

@jfahrenkrug commented on GitHub (Feb 19, 2026):

@tjbck Thank you for the update. Could you point me to the specific PR or commit? Thanks!

<!-- gh-comment-id:3930453091 --> @jfahrenkrug commented on GitHub (Feb 19, 2026): @tjbck Thank you for the update. Could you point me to the specific PR or commit? Thanks!
GiteaMirror commented 2026-05-18 04:47:35 -05:00
Author
Owner
Copy Link

@Classic298 commented on GitHub (Feb 19, 2026):

4bef69cc63

<!-- gh-comment-id:3930631851 --> @Classic298 commented on GitHub (Feb 19, 2026): https://github.com/open-webui/open-webui/commit/4bef69cc6344ff809090441aa6bced573a2aa838
GiteaMirror commented 2026-05-18 04:47:36 -05:00
Author
Owner
Copy Link

@jfahrenkrug commented on GitHub (Feb 23, 2026):

Related PR: https://github.com/open-webui/open-webui/pull/21786

<!-- gh-comment-id:3945047917 --> @jfahrenkrug commented on GitHub (Feb 23, 2026): Related PR: https://github.com/open-webui/open-webui/pull/21786
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#106441
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?