[GH-ISSUE #9106] Error at RAG function chromadb sorting and RAG optimization #30915

New Issue

GiteaMirror · 2026-04-25T05:04:00-05:00

GiteaMirror commented

2026-04-25 05:04:00 -05:00

Originally created by @Schwenn2002 on GitHub (Jan 30, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/9106

Discussed in https://github.com/open-webui/open-webui/discussions/9001

^{Originally posted by Schwenn2002 January 27, 2025}
There is a faulty code for sorting the document list in ChromaDB because the values of the cosine vector are misinterpreted (see below).

Furthermore, the RAG function can significantly speed up the query speed due to much smaller context lengths if the sorted return list from the RAG is limited to the best X hits:

A suggestion for optimizing the RAG function.

Currently, you can improve the quality of the query using Top k and ReRanking.

It would be optimal to filter out the most important documents using Top k=50 or 70 and ReRanking (0,6 - 0.8). Then another parameter would be optimal if, after the reranking, maximum the best (Top Best=10 or 20) hits were passed on to the LLM as context.

This way, the context length can be kept shorter and the response time improved. This also ensures that the context cannot become larger than configured in the LLM.

The current behavior of the LLM is that a context that is too large cannot be processed (i.e. it is interpreted as empty). It would therefore make sense to truncate the context from the RAG query after reranking to the context length of the LLM, so that only the best hits are passed on

backend/open_webui/retrieval/utils.py

Why is there a distinction made here for sorting for ChromaDB? Is this a bug?

    if VECTOR_DB == "chroma":
        # Chroma uses unconventional cosine similarity, so we don't need to reverse the results
        # https://docs.trychroma.com/docs/collections/configure#configuring-chroma-collections
        return merge_and_sort_query_results(results, k=k, reverse=False)
    else:
        return merge_and_sort_query_results(results, k=k, reverse=True)

The result of cosine similarity always lies within the range of -1 to 1.

Interpretation of Values:

1 (Maximum Similarity):
The two vectors are identical or perfectly aligned.
The angle between the vectors is 0°.
0 (No Similarity):
The vectors are orthogonal (perpendicular) to each other.
They have no common direction or meaning.
-1 (Maximum Oppositeness):
The two vectors are exactly opposite in direction.
The angle between the vectors is 180°.

Practical Significance:

Cosine Similarity in the Context of Embeddings:

A high value (close to 1) indicates that two embeddings share similar meanings or characteristics.
A low value (close to -1) indicates that the embeddings are opposite in meaning.
A value near 0 suggests that there is no significant similarity between the embeddings.

In Machine Learning or Information Retrieval:

Document Comparison: Two documents with similar content will have a high cosine similarity.
User Preference Similarity: Vectors representing users with similar preferences will have a high similarity.

Shouldn't the following command also be correct for ChromaDB?

return merge_and_sort_query_results(results, k=k, reverse=True)

As a test, I adapted the function backend/open_webui/retrieval/utils.py:

def merge_and_sort_query_results(
    query_results: list[dict], k: int, reverse: bool = False
) -> list[dict]:
    # Initialize lists to store combined data
    combined_distances = []
    combined_documents = []
    combined_metadatas = []

    for data in query_results:
        combined_distances.extend(data["distances"][0])
        combined_documents.extend(data["documents"][0])
        combined_metadatas.extend(data["metadatas"][0])

    # Create a list of tuples (distance, document, metadata)
    combined = list(zip(combined_distances, combined_documents, combined_metadatas))

    # Sort the list based on distances
    #combined.sort(key=lambda x: x[0], reverse=reverse)
    # combined.sort(key=lambda x: abs(x[0]), reverse=True) # for maximum oppositeness
    combined.sort(key=lambda x: x[0], reverse=True)

    # We don't have anything :-(
    if not combined:
        sorted_distances = []
        sorted_documents = []
        sorted_metadatas = []
    else:
        # Unzip the sorted list
        sorted_distances, sorted_documents, sorted_metadatas = zip(*combined)

        # Slicing the lists to include only k elements
        #sorted_distances = list(sorted_distances)[:k]
        #sorted_documents = list(sorted_documents)[:k]
        #sorted_metadatas = list(sorted_metadatas)[:k]
        # only 20
        sorted_distances = list(sorted_distances)[:20]
        sorted_documents = list(sorted_documents)[:20]
        sorted_metadatas = list(sorted_metadatas)[:20]

    # Create the output dictionary
    result = {
        "distances": [sorted_distances],
        "documents": [sorted_documents],
        "metadatas": [sorted_metadatas],
    }

    return result

I have also enabled descending sorting for ChromaDB:
combined.sort(key=lambda x: x[0], reverse=True)

If you want to take conflicting chunks into account, you could also sort the list by absolute value:
combined.sort(key=lambda x: abs(x[0]), reverse=True)

Additionally, I have hard-coded that the best 20 chunks should be kept in the sorted list(Top k was used to query 50 documents in the RAG). Due to the correct sorting, the best hits are returned here.

This could also be controlled via an additional parameter and called via query_collection or query_collection_with_hybrid_search accordingly. Perhaps this parameter can also be anchored in the model parameters.

        # only 20
        sorted_distances = list(sorted_distances)[:20]
        sorted_documents = list(sorted_documents)[:20]
        sorted_metadatas = list(sorted_metadatas)[:20]

The answers in the RAG will then be much better!

Due to the sorted list, the most probable chunks are returned from the RAG. A significantly larger number of documents are initially retrieved from the RAG. The reranking leads to better results. The query from the RAG still runs very efficiently even for 70 documents (top k=70). By shortening the list to the best 20 hits, the context window for the model can be reduced. Overall, the performance and quality are significantly improved.

Originally created by @Schwenn2002 on GitHub (Jan 30, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/9106 ### Discussed in https://github.com/open-webui/open-webui/discussions/9001 <div type='discussions-op-text'> <sup>Originally posted by **Schwenn2002** January 27, 2025</sup> There is a faulty code for sorting the document list in ChromaDB because the values of the cosine vector are misinterpreted (see below). Furthermore, the RAG function can significantly speed up the query speed due to much smaller context lengths if the sorted return list from the RAG is limited to the best X hits: A suggestion for optimizing the RAG function. Currently, you can improve the quality of the query using Top k and ReRanking. It would be optimal to filter out the most important documents using Top k=50 or 70 and ReRanking (0,6 - 0.8). Then another parameter would be optimal if, after the reranking, maximum the best (Top Best=10 or 20) hits were passed on to the LLM as context. This way, the context length can be kept shorter and the response time improved. This also ensures that the context cannot become larger than configured in the LLM. The current behavior of the LLM is that a context that is too large cannot be processed (i.e. it is interpreted as empty). It would therefore make sense to truncate the context from the RAG query after reranking to the context length of the LLM, so that only the best hits are passed on</div> --- ### backend/open_webui/retrieval/utils.py **Why is there a distinction made here for sorting for ChromaDB? Is this a bug?** ``` if VECTOR_DB == "chroma": # Chroma uses unconventional cosine similarity, so we don't need to reverse the results # https://docs.trychroma.com/docs/collections/configure#configuring-chroma-collections return merge_and_sort_query_results(results, k=k, reverse=False) else: return merge_and_sort_query_results(results, k=k, reverse=True) ``` The result of cosine similarity always lies within the range of -1 to 1. --- ### **Interpretation of Values:** - **1 (Maximum Similarity):** The two vectors are identical or perfectly aligned. The angle between the vectors is **0°**. - **0 (No Similarity):** The vectors are orthogonal (perpendicular) to each other. They have no common direction or meaning. - **-1 (Maximum Oppositeness):** The two vectors are exactly opposite in direction. The angle between the vectors is **180°**. ### **Practical Significance:** #### **Cosine Similarity in the Context of Embeddings:** - A high value (close to 1) indicates that two embeddings share similar meanings or characteristics. - A low value (close to -1) indicates that the embeddings are opposite in meaning. - A value near 0 suggests that there is no significant similarity between the embeddings. #### **In Machine Learning or Information Retrieval:** - **Document Comparison:** Two documents with similar content will have a high cosine similarity. - **User Preference Similarity:** Vectors representing users with similar preferences will have a high similarity. --- **Shouldn't the following command also be correct for ChromaDB?** `return merge_and_sort_query_results(results, k=k, reverse=True)` --- As a test, I adapted the function _backend/open_webui/retrieval/utils.py_: ``` def merge_and_sort_query_results( query_results: list[dict], k: int, reverse: bool = False ) -> list[dict]: # Initialize lists to store combined data combined_distances = [] combined_documents = [] combined_metadatas = [] for data in query_results: combined_distances.extend(data["distances"][0]) combined_documents.extend(data["documents"][0]) combined_metadatas.extend(data["metadatas"][0]) # Create a list of tuples (distance, document, metadata) combined = list(zip(combined_distances, combined_documents, combined_metadatas)) # Sort the list based on distances #combined.sort(key=lambda x: x[0], reverse=reverse) # combined.sort(key=lambda x: abs(x[0]), reverse=True) # for maximum oppositeness combined.sort(key=lambda x: x[0], reverse=True) # We don't have anything :-( if not combined: sorted_distances = [] sorted_documents = [] sorted_metadatas = [] else: # Unzip the sorted list sorted_distances, sorted_documents, sorted_metadatas = zip(*combined) # Slicing the lists to include only k elements #sorted_distances = list(sorted_distances)[:k] #sorted_documents = list(sorted_documents)[:k] #sorted_metadatas = list(sorted_metadatas)[:k] # only 20 sorted_distances = list(sorted_distances)[:20] sorted_documents = list(sorted_documents)[:20] sorted_metadatas = list(sorted_metadatas)[:20] # Create the output dictionary result = { "distances": [sorted_distances], "documents": [sorted_documents], "metadatas": [sorted_metadatas], } return result ``` I have also enabled descending sorting for ChromaDB: ` combined.sort(key=lambda x: x[0], reverse=True)` If you want to take conflicting chunks into account, you could also sort the list by absolute value: ` combined.sort(key=lambda x: abs(x[0]), reverse=True)` Additionally, I have hard-coded that the best 20 chunks should be kept in the sorted list(Top k was used to query 50 documents in the RAG). Due to the correct sorting, the best hits are returned here. This could also be controlled via an additional parameter and called via _query_collection_ or _query_collection_with_hybrid_search_ accordingly. Perhaps this parameter can also be anchored in the model parameters. ``` # only 20 sorted_distances = list(sorted_distances)[:20] sorted_documents = list(sorted_documents)[:20] sorted_metadatas = list(sorted_metadatas)[:20] ``` **The answers in the RAG will then be much better!** Due to the sorted list, the most probable chunks are returned from the RAG. A significantly larger number of documents are initially retrieved from the RAG. The reranking leads to better results. The query from the RAG still runs very efficiently even for 70 documents (top k=70). By shortening the list to the best 20 hits, the context window for the model can be reduced. **Overall, the performance and quality are significantly improved.**

GiteaMirror closed this issue

2026-04-25 05:04:00 -05:00

GiteaMirror commented

2026-04-25 05:04:00 -05:00

@tjbck commented on GitHub (Jan 30, 2025):

https://github.com/open-webui/open-webui/pull/8379#issuecomment-2579428766

@tjbck commented on GitHub (Jan 30, 2025): https://github.com/open-webui/open-webui/pull/8379#issuecomment-2579428766

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#30915