github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-05-07 03:18:23 -05:00
Code Issues 1.3k Packages Projects Releases 126 Wiki Activity

[GH-ISSUE #10187] RAG: result duplicates #31330

New Issue
Closed
opened 2026-04-25 05:19:03 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-25 05:19:03 -05:00
Owner
Copy Link

Originally created by @mkhludnev on GitHub (Feb 17, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/10187

Discussed in https://github.com/open-webui/open-webui/discussions/10168

Originally posted by mkhludnev February 17, 2025
it's loosely related #8379.
When a several queries generated by the task. We get a result list per each of them. It's obvious that there are documents occurring in a few of the lists, then merge_and_sort_query_results doesn't does not deduplicate results, and we've got repeating docs in RAG context, beside of wasting LLM contexts with useless repetition, it might trigger repetition in the output.
Why don't drop duplicates in merge_and_sort_query_results?
May I come up with PR for it?

Originally created by @mkhludnev on GitHub (Feb 17, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/10187 ### Discussed in https://github.com/open-webui/open-webui/discussions/10168 <div type='discussions-op-text'> <sup>Originally posted by **mkhludnev** February 17, 2025</sup> it's loosely related #8379. When a several queries generated by the task. We get a result list per each of them. It's obvious that there are documents occurring in a few of the lists, then `merge_and_sort_query_results` doesn't does not deduplicate results, and we've got repeating docs in RAG context, beside of wasting LLM contexts with useless repetition, it might trigger repetition in the output. Why don't drop duplicates in `merge_and_sort_query_results`? May I come up with PR for it? </div>
GiteaMirror commented 2026-04-25 05:19:05 -05:00
Author
Owner
Copy Link

@tjbck commented on GitHub (Feb 17, 2025):

PR Welcome!

<!-- gh-comment-id:2663594090 --> @tjbck commented on GitHub (Feb 17, 2025): PR Welcome!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#31330
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?