RAG relevance percentage is misleading (inverse score) - High percentage scores for irrelevant chunks and vice-versa #2632

New Issue

GiteaMirror · 2025-11-11T15:11:02-06:00

GiteaMirror commented

2025-11-11 15:11:02 -06:00

Originally created by @clang88 on GitHub (Nov 13, 2024).

Bug Report

Installation Method

Docker via ghcr.io/open-webui/open-webui:main

Environment

Open WebUI Version: 0.3.35
Operating System: Ubuntu 22.04
Browser (if applicable): Chrome 130

Confirmation:

I have read and followed all the instructions provided in the README.md.
I am on the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.

Expected Behavior:

Percentage score for relevance should be high, if semantic distance is low (score is close to 1). Percentage score for relevance is low, if semantic distance is high (score is close to 0).

Actual Behavior:

Highly relevant chunks are marked as "low relevance" with a low percentage score (also colored in red), wheras irelvant chunks are marked with a high percentage score (green).

Description

Bug Summary:
Relevance percentage score is misleading, because irrelevant chunks have high percentage and highly relevant scores have low percentage.

Reproduction Details

Steps to Reproduce:

Upload a document or select a knowledge collection
Ask model a question that can be answered with knowledge from document
Check citations to see that percentage is the wrong way around.

Logs and Screenshots

Screenshots/Screen Recordings (if applicable):
Highly relevant chunk (answers exactly the question "How to create an entry in Kalcium Quickterm":

Irrelevant chunk (just a table of contents):

Additional Information

The issue is likely in this piece of code:
src/lib/components/chat/Messages/CitationsModal.svelte

function calculatePercentage(distance: number) {
		if (distance < 0) return 100;
		if (distance > 1) return 0;
		return Math.round((1 - distance) * 10000) / 100;
	}

It should be like this:

function calculatePercentage(distance: number) {
		if (distance < 0) return 0;
		if (distance > 1) return 100;
		return Math.round(distance * 10000) / 100;
	}

Originally created by @clang88 on GitHub (Nov 13, 2024). # Bug Report ## Installation Method Docker via ghcr.io/open-webui/open-webui:main ## Environment - **Open WebUI Version:** 0.3.35 - **Operating System:** Ubuntu 22.04 - **Browser (if applicable):** Chrome 130 **Confirmation:** - [x] I have read and followed all the instructions provided in the README.md. - [x] I am on the latest version of both Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below. ## Expected Behavior: Percentage score for relevance should be high, if semantic distance is low (score is close to 1). Percentage score for relevance is low, if semantic distance is high (score is close to 0). ## Actual Behavior: Highly relevant chunks are marked as "low relevance" with a low percentage score (also colored in red), wheras irelvant chunks are marked with a high percentage score (green). ## Description **Bug Summary:** Relevance percentage score is misleading, because irrelevant chunks have high percentage and highly relevant scores have low percentage. ## Reproduction Details **Steps to Reproduce:** 1) Upload a document or select a knowledge collection 2) Ask model a question that can be answered with knowledge from document 3) Check citations to see that percentage is the wrong way around. ## Logs and Screenshots **Screenshots/Screen Recordings (if applicable):** Highly relevant chunk (answers exactly the question "How to create an entry in Kalcium Quickterm": ![image](https://github.com/user-attachments/assets/670048f6-7b32-42d1-bfc6-8a38d25467d0) Irrelevant chunk (just a table of contents): ![image](https://github.com/user-attachments/assets/608edf08-3093-4aaa-8415-d0d64e277552) ## Additional Information The issue is likely in this piece of code: src/lib/components/chat/Messages/CitationsModal.svelte ``` function calculatePercentage(distance: number) { if (distance < 0) return 100; if (distance > 1) return 0; return Math.round((1 - distance) * 10000) / 100; } ``` It should be like this: ``` function calculatePercentage(distance: number) { if (distance < 0) return 0; if (distance > 1) return 100; return Math.round(distance * 10000) / 100; } ```

GiteaMirror closed this issue

2025-11-11 15:11:03 -06:00

GiteaMirror referenced this issue

2026-04-19 19:46:21 -05:00

[GH-ISSUE #2632] feat: team workspace #12955

GiteaMirror referenced this issue

2026-04-25 03:05:50 -05:00

[GH-ISSUE #2632] feat: team workspace #28483

GiteaMirror referenced this issue

2026-05-05 12:42:03 -05:00

[GH-ISSUE #2632] feat: team workspace #51621

Sign in to join this conversation.