[GH-ISSUE #21336] feat: Add option to make /api/v1/retrieval/process/web endpoint additive #90186

New Issue

GiteaMirror · 2026-05-15T15:20:10-05:00

GiteaMirror commented

2026-05-15 15:20:10 -05:00

Originally created by @jfahrenkrug on GitHub (Feb 12, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/21336

Check Existing Issues

I have searched for all existing open AND closed issues and discussions for similar requests. I have found none that is comparable to my request.

Verify Feature Scope

I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions.

Problem Description

I've written a script that uses the sitemap of my website and then calls /api/v1/retrieval/process/web for every URL to ingest it into a knowledge base. It's working, but I was surprised how little data ended up in my Qdrant DB after successfully processing 602 webpages.
It turns out that every call to /api/v1/retrieval/process/web actually completely overrides the knowledge base. You can see that save_docs_to_vector_db is called with overwrite=True here: 2b26355002/backend/open_webui/routers/retrieval.py (L1884)

Desired Solution you'd like

I'd like to be able to add hundreds of URLs (processed website) to the same knowledge base, maybe by passing overwrite=False to the request.

Alternatives Considered

Using the files API or the text API endpoint.

Additional Context

No response

Originally created by @jfahrenkrug on GitHub (Feb 12, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/21336 ### Check Existing Issues - [x] I have searched for all existing **open AND closed** issues and discussions for similar requests. I have found none that is comparable to my request. ### Verify Feature Scope - [x] I have read through and understood the scope definition for feature requests in the Issues section. I believe my feature request meets the definition and belongs in the Issues section instead of the Discussions. ### Problem Description I've written a script that uses the sitemap of my website and then calls /api/v1/retrieval/process/web for every URL to ingest it into a knowledge base. It's working, but I was surprised how little data ended up in my Qdrant DB after successfully processing 602 webpages. It turns out that every call to /api/v1/retrieval/process/web actually completely overrides the knowledge base. You can see that `save_docs_to_vector_db` is called with `overwrite=True` here: https://github.com/open-webui/open-webui/blob/2b26355002064228e9b671339f8f3fb9d1fafa73/backend/open_webui/routers/retrieval.py#L1884 ### Desired Solution you'd like I'd like to be able to add hundreds of URLs (processed website) to the same knowledge base, maybe by passing `overwrite=False` to the request. ### Alternatives Considered Using the files API or the text API endpoint. ### Additional Context _No response_

GiteaMirror closed this issue

2026-05-15 15:20:11 -05:00

GiteaMirror commented

2026-05-15 15:20:12 -05:00

@tjbck commented on GitHub (Feb 19, 2026):

Addressed in dev.

@tjbck commented on GitHub (Feb 19, 2026): Addressed in dev.

GiteaMirror commented

2026-05-15 15:20:16 -05:00

@jfahrenkrug commented on GitHub (Feb 19, 2026):

@tjbck Thank you for the update. Could you point me to the specific PR or commit? Thanks!

@jfahrenkrug commented on GitHub (Feb 19, 2026): @tjbck Thank you for the update. Could you point me to the specific PR or commit? Thanks!

GiteaMirror commented

2026-05-15 15:20:20 -05:00

@Classic298 commented on GitHub (Feb 19, 2026):

4bef69cc63

@Classic298 commented on GitHub (Feb 19, 2026): https://github.com/open-webui/open-webui/commit/4bef69cc6344ff809090441aa6bced573a2aa838

GiteaMirror commented

2026-05-15 15:20:21 -05:00

@jfahrenkrug commented on GitHub (Feb 23, 2026):

Related PR: https://github.com/open-webui/open-webui/pull/21786

@jfahrenkrug commented on GitHub (Feb 23, 2026): Related PR: https://github.com/open-webui/open-webui/pull/21786

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#90186