github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-05-06 10:58:17 -05:00
Code Issues 1.3k Packages Projects Releases 126 Wiki Activity

[PR #10512] [CLOSED] fix: Rag dedupe (again) #61406

New Issue
Closed
opened 2026-05-06 04:55:22 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-05-06 04:55:22 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/open-webui/open-webui/pull/10512
Author: @mkhludnev
Created: 2/21/2025
Status: ❌ Closed

Base: dev ← Head: rag-dedupe-again

📝 Commits (3)

📊 Changes

1 file changed (+30 additions, -13 deletions)

View changed files

📝 backend/open_webui/retrieval/utils.py (+30 -13)

📄 Description

RAG: result duplicates

Discussed: https://github.com/open-webui/open-webui/discussions/10188

Previous attempt #10272 and the failure #10428
This time I use a document hash and metadata hash (but optional) for deduplication.

Changelog Entry

Description

  • fixes wasting RAG context by result duplicates

Fixed

  • removes result duplicates

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/open-webui/open-webui/pull/10512 **Author:** [@mkhludnev](https://github.com/mkhludnev) **Created:** 2/21/2025 **Status:** ❌ Closed **Base:** `dev` ← **Head:** `rag-dedupe-again` --- ### 📝 Commits (3) - [`a026686`](https://github.com/open-webui/open-webui/commit/a02668687a6aa885b62c430a44f2b3772a7ee118) dedupe results from multiple queries - [`66f4e02`](https://github.com/open-webui/open-webui/commit/66f4e0234a57419a50d21189bc51cde95203095a) fix dedupe - [`a66ffa0`](https://github.com/open-webui/open-webui/commit/a66ffa070170b3bb17a4742f3251a18ffccb97ca) format ### 📊 Changes **1 file changed** (+30 additions, -13 deletions) <details> <summary>View changed files</summary> 📝 `backend/open_webui/retrieval/utils.py` (+30 -13) </details> ### 📄 Description # RAG: result duplicates Discussed: https://github.com/open-webui/open-webui/discussions/10188 Previous attempt #10272 and the failure #10428 This time I use a document hash and metadata hash (but optional) for deduplication. # Changelog Entry ### Description - fixes wasting RAG context by result duplicates ### Fixed - removes result duplicates --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2026-05-06 04:55:22 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#61406
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?