[GH-ISSUE #12150] issue: Bug Report: Controlling Vector Results Count in RAG System #136209

New Issue

Closed

opened 2026-05-25 03:53:33 -05:00 by GiteaMirror · 16 comments

GiteaMirror commented 2026-05-25 03:53:33 -05:00

Owner

Copy Link

Originally created by @chalitbkb on GitHub (Mar 28, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/12150

Problem Description: The number of vector data references related to user inputs remains high even after reducing the "RAG_TOP_K" results to consider for embedding when using RAG, and decreasing the "RAG_TOP_K_RERANKER" results as specified in the documentation. This causes errors in certain models that support small context window sizes, such as OpenAI's services. However, when I switched to using Gemini 2.0 Pro, which supports larger context window sizes, the model successfully generated responses to user questions because Gemini 2.0 Pro can handle a context window size of up to 2 million tokens, thus avoiding this issue. For other models, this problem persists because the "vector" dataset returns an excessive number of aggregated results. Therefore, I attempted to reduce the number of aggregated results from the "vector DB" but was unsuccessful in enabling models that support small context windows to function properly.

Additionally, regarding the "RAG_TOP_K_RERANKER" setting, I do not see it in the UI as the documentation states that "This environment variable is a PersistentConfig variable." I only see the "RAG_TOP_K" and "RAG_RELEVANCE_THRESHOLD" sections. Therefore, I am unsure where the "RAG_TOP_K_RERANKER" setting can be found in the UI as described in the documentation as a "PersistentConfig." I tried setting the number of search results for "vector" by configuring "RAG_TOP_K_RERANKER" directly in the environment variables, but it still did not work.

Summary: Currently, I am experiencing an issue with an excessive number of vector data search results, ranging from 50-100 sets, each containing large content sizes due to my own specifications for Chunk Size and Chunk Overlap. This occurs even though the "RAG_TOP_K" and "RAG_RELEVANCE_THRESHOLD" settings have been configured with lower values.

Check Existing Issues

• I have searched the existing issues and discussions.
• I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.5.20

Ollama Version (if applicable)

No response

Operating System

Windows 11

Browser (if applicable)

Chrome 134.0.6998.118

Confirmation

• I have read and followed all instructions in README.md.
• I am using the latest version of both Open WebUI and Ollama.
• I have included the browser console logs.
• I have included the Docker container logs.
• I have listed steps to reproduce the bug in detail.

Expected Behavior

• The system should limit the number of vector results according to the value set in RAG_TOP_K_RERANKER. When this value is set lower, the number of results sent to the LLM model should decrease accordingly.
• Adjusting the RAG_TOP_K_RERANKER setting should be possible through the UI or environment variables as specified in the documentation. This value should directly control the amount of data aggregated from the vector database.
• Models with small context windows (such as OpenAI's services) should be able to function properly when the number of vector results is reduced to fit within the model's limitations.

Actual Behavior

• Despite reducing the RAG_TOP_K value and setting RAG_TOP_K_RERANKER to a low value, the system continues to send an excessive number of vector results (50-100 sets) to the LLM model.
• The RAG_TOP_K_RERANKER option is not found in the UI as described in the documentation as a "PersistentConfig variable". Only RAG_TOP_K and RAG_RELEVANCE_THRESHOLD are visible.
• Setting RAG_TOP_K_RERANKER directly through environment variables has no effect on reducing the number of vector results.
• Models with small context windows (such as OpenAI's services) encounter errors as they cannot handle the excessive amount of data, while Gemini 2.0 Pro, which supports a large context window of up to 2 million tokens, can operate without issues.

Steps to Reproduce

1. Set RAG_TOP_K to a low value (e.g., 3-5) through the UI.
2. Set RAG_RELEVANCE_THRESHOLD to a high value (e.g., 0.7-0.8) to limit results.
3. Attempt to set RAG_TOP_K_RERANKER through environment variables to a low value (e.g., 3-5).
4. Test the system with queries related to data in the vector database.
5. Check the number of vector results sent to the model, which will still be excessively high (50-100 sets).
6. Test with models that have small context windows (such as OpenAI's services) and observe that errors occur.
7. Test with models that have large context windows (such as Gemini 2.0 Pro) and observe that they function properly.

Logs & Screenshots

Try using a model that supports large context windows. The Gemini 2.0 Pro model is capable of processing.
It’s noticeable that the up and down sliders are small, indicating that there are numerous vector data components with many items. I examined the list and found that there are 50 to 100 entries. Even though I attempted to adjust the relevant sections for re-ranking, it still produces a large number of related results.

When testing models with smaller context windows, like the o3-mini, the model couldn't process them because there were too many vector results.

Additional Information

The main issue is the inability to effectively control the number of aggregated vector results through the RAG_TOP_K_RERANKER setting as described in the documentation, which prevents proper use of models with context window limitations.

Originally created by @chalitbkb on GitHub (Mar 28, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/12150 **Problem Description:** The number of vector data references related to user inputs remains high even after reducing the "RAG_TOP_K" results to consider for embedding when using RAG, and decreasing the "RAG_TOP_K_RERANKER" results as specified in the documentation. This causes errors in certain models that support small context window sizes, such as OpenAI's services. However, when I switched to using Gemini 2.0 Pro, which supports larger context window sizes, the model successfully generated responses to user questions because Gemini 2.0 Pro can handle a context window size of up to 2 million tokens, thus avoiding this issue. For other models, this problem persists because the "vector" dataset returns an excessive number of aggregated results. Therefore, I attempted to reduce the number of aggregated results from the "vector DB" but was unsuccessful in enabling models that support small context windows to function properly. Additionally, regarding the "RAG_TOP_K_RERANKER" setting, I do not see it in the UI as the documentation states that "This environment variable is a PersistentConfig variable." I only see the "RAG_TOP_K" and "RAG_RELEVANCE_THRESHOLD" sections. Therefore, I am unsure where the "RAG_TOP_K_RERANKER" setting can be found in the UI as described in the documentation as a "PersistentConfig." I tried setting the number of search results for "vector" by configuring "RAG_TOP_K_RERANKER" directly in the environment variables, but it still did not work. **Summary:** Currently, I am experiencing an issue with an excessive number of vector data search results, ranging from 50-100 sets, each containing large content sizes due to my own specifications for Chunk Size and Chunk Overlap. This occurs even though the "RAG_TOP_K" and "RAG_RELEVANCE_THRESHOLD" settings have been configured with lower values. ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version v0.5.20 ### Ollama Version (if applicable) _No response_ ### Operating System Windows 11 ### Browser (if applicable) Chrome 134.0.6998.118 ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have listed steps to reproduce the bug in detail. ### Expected Behavior * The system should limit the number of vector results according to the value set in `RAG_TOP_K_RERANKER`. When this value is set lower, the number of results sent to the LLM model should decrease accordingly. * Adjusting the `RAG_TOP_K_RERANKER` setting should be possible through the UI or environment variables as specified in the documentation. This value should directly control the amount of data aggregated from the vector database. * Models with small context windows (such as OpenAI's services) should be able to function properly when the number of vector results is reduced to fit within the model's limitations. ### Actual Behavior * Despite reducing the `RAG_TOP_K` value and setting `RAG_TOP_K_RERANKER` to a low value, the system continues to send an excessive number of vector results (50-100 sets) to the LLM model. * The `RAG_TOP_K_RERANKER` option is not found in the UI as described in the documentation as a "PersistentConfig variable". Only `RAG_TOP_K` and `RAG_RELEVANCE_THRESHOLD` are visible. * Setting `RAG_TOP_K_RERANKER` directly through environment variables has no effect on reducing the number of vector results. * Models with small context windows (such as OpenAI's services) encounter errors as they cannot handle the excessive amount of data, while Gemini 2.0 Pro, which supports a large context window of up to 2 million tokens, can operate without issues. ### Steps to Reproduce 1. Set `RAG_TOP_K` to a low value (e.g., 3-5) through the UI. 2. Set `RAG_RELEVANCE_THRESHOLD` to a high value (e.g., 0.7-0.8) to limit results. 3. Attempt to set `RAG_TOP_K_RERANKER` through environment variables to a low value (e.g., 3-5). 4. Test the system with queries related to data in the vector database. 5. Check the number of vector results sent to the model, which will still be excessively high (50-100 sets). 6. Test with models that have small context windows (such as OpenAI's services) and observe that errors occur. 7. Test with models that have large context windows (such as Gemini 2.0 Pro) and observe that they function properly. ### Logs & Screenshots Try using a model that supports large context windows. The Gemini 2.0 Pro model is capable of processing. It’s noticeable that the up and down sliders are small, indicating that there are numerous vector data components with many items. I examined the list and found that there are 50 to 100 entries. Even though I attempted to adjust the relevant sections for re-ranking, it still produces a large number of related results.  When testing models with smaller context windows, like the o3-mini, the model couldn't process them because there were too many vector results.   ### Additional Information The main issue is the inability to effectively control the number of aggregated vector results through the `RAG_TOP_K_RERANKER` setting as described in the documentation, which prevents proper use of models with context window limitations.

GiteaMirror added the bug label 2026-05-25 03:53:33 -05:00

GiteaMirror closed this issue 2026-05-25 03:53:36 -05:00

GiteaMirror commented 2026-05-25 03:53:39 -05:00

Author

Owner

Copy Link

@tjbck commented on GitHub (Mar 28, 2025):

Are you 100% sure you configured everything right? @chalitbkb

cc @mahenning

<!-- gh-comment-id:2760710536 --> @tjbck commented on GitHub (Mar 28, 2025): Are you 100% sure you configured everything right? @chalitbkb cc @mahenning

GiteaMirror commented 2026-05-25 03:53:41 -05:00

Author

Owner

Copy Link

@chalitbkb commented on GitHub (Mar 28, 2025):

Are you 100% sure you configured everything right? @chalitbkb

cc @mahenning

I’ve thoroughly reviewed all the relevant documents and feel reasonably confident that I’ve configured everything correctly. However, I’m uncertain if I’ve overlooked something. Even though I’ve tried to reduce the vector aggregation settings, the aggregation results still remain high. Therefore, I’m bringing up this issue.

<!-- gh-comment-id:2760731244 --> @chalitbkb commented on GitHub (Mar 28, 2025): > Are you 100% sure you configured everything right? [@chalitbkb](https://github.com/chalitbkb) > > cc [@mahenning](https://github.com/mahenning) I’ve thoroughly reviewed all the relevant documents and feel reasonably confident that I’ve configured everything correctly. However, I’m uncertain if I’ve overlooked something. Even though I’ve tried to reduce the vector aggregation settings, the aggregation results still remain high. Therefore, I’m bringing up this issue.

GiteaMirror commented 2026-05-25 03:53:42 -05:00

Author

Owner

Copy Link

@mahenning commented on GitHub (Mar 28, 2025):

You wrote you're on Open-WebUI v0.5.20. Did you clone the repo and run on the dev branch or do you run Open-WebUI normally and are on v0.5.20? The RAG_TOP_K_RERANKER feature is not yet in v0.5.20, only on the dev branch at the moment. The feature is likely included in the next version.

When I run Open-WebUI on the dev branch, I get

<!-- gh-comment-id:2761699485 --> @mahenning commented on GitHub (Mar 28, 2025): You wrote you're on Open-WebUI `v0.5.20`. Did you clone the repo and run on the `dev` branch or do you run Open-WebUI normally and are on `v0.5.20`? The `RAG_TOP_K_RERANKER` feature is not yet in `v0.5.20`, only on the dev branch at the moment. The feature is likely included in the next version. When I run Open-WebUI on the dev branch, I get 

GiteaMirror commented 2026-05-25 03:53:45 -05:00

Author

Owner

Copy Link

@mahenning commented on GitHub (Mar 28, 2025):

Ah I see, the variable is already listed in the list of environment variables, because https://github.com/open-webui/docs (the repo for the documentation and stuff) has no dev branch, an my PR to include the documentation for RAG_TOP_K_RERANKER was merged into main. It seems the documentation is a bit ahead of the actual features of the main version of Open-WebUI ;)

<!-- gh-comment-id:2761711402 --> @mahenning commented on GitHub (Mar 28, 2025): Ah I see, the variable is already listed in [the list of environment variables](https://docs.openwebui.com/getting-started/env-configuration/), because https://github.com/open-webui/docs (the repo for the documentation and stuff) has no `dev` branch, an my PR to include the documentation for `RAG_TOP_K_RERANKER` was merged into main. It seems the documentation is a bit ahead of the actual features of the main version of Open-WebUI ;)

GiteaMirror commented 2026-05-25 03:53:47 -05:00

Author

Owner

Copy Link

@chalitbkb commented on GitHub (Mar 28, 2025):

You wrote you're on Open-WebUI v0.5.20. Did you clone the repo and run on the dev branch or do you run Open-WebUI normally and are on v0.5.20? The RAG_TOP_K_RERANKER feature is not yet in v0.5.20, only on the dev branch at the moment. The feature is likely included in the next version.

When I run Open-WebUI on the dev branch, I get

Thank you. I'm currently using the DEV version of Open-WebUI and am enjoying the RAG_TOP_K_RERANKER feature. However, I've tested configuring RAG_TOP_K_RERANKER, but I still encounter the aforementioned problem.

<!-- gh-comment-id:2761714644 --> @chalitbkb commented on GitHub (Mar 28, 2025): > You wrote you're on Open-WebUI `v0.5.20`. Did you clone the repo and run on the `dev` branch or do you run Open-WebUI normally and are on `v0.5.20`? The `RAG_TOP_K_RERANKER` feature is not yet in `v0.5.20`, only on the dev branch at the moment. The feature is likely included in the next version. > > When I run Open-WebUI on the dev branch, I get > Thank you. I'm currently using the DEV version of Open-WebUI and am enjoying the RAG_TOP_K_RERANKER feature. However, I've tested configuring RAG_TOP_K_RERANKER, but I still encounter the aforementioned problem.

GiteaMirror commented 2026-05-25 03:53:50 -05:00

Author

Owner

Copy Link

@mahenning commented on GitHub (Mar 28, 2025):

Thanks for the fast reply. Just to make sure I follow everything correctly:
Can you tell me how you start your dev version? Maybe our ways of starting/building differ and I can reconstruct it that way.
You said earlier you can neither set the variable in the UI nor as an environment variable, in the comment above you write you're "enjoying the RAG_TOP_K_RERANKER feature". Is it somehow/sometimes working for you now, or still not set-able in both ways?

If you have the local code at hand, can you look into the code if you find "k_reranker"? For instance, it should be a parameter for the function query_collection_with_hybrid_search in backend/open_webui/retrieval/utils.py:280.

Another question (please don't be mad at me for this), but did you make sure you're on the latest dev version? Did you pull changes/update your local version?

<!-- gh-comment-id:2761740469 --> @mahenning commented on GitHub (Mar 28, 2025): Thanks for the fast reply. Just to make sure I follow everything correctly: Can you tell me how you start your dev version? Maybe our ways of starting/building differ and I can reconstruct it that way. You said earlier you can neither set the variable in the UI nor as an environment variable, in the comment above you write you're "enjoying the RAG_TOP_K_RERANKER feature". Is it somehow/sometimes working for you now, or still not set-able in both ways? If you have the local code at hand, can you look into the code if you find "k_reranker"? For instance, it should be a parameter for the function `query_collection_with_hybrid_search` in backend/open_webui/retrieval/utils.py:280. Another question (please don't be mad at me for this), but did you make sure you're on the latest dev version? Did you pull changes/update your local version?

GiteaMirror commented 2026-05-25 03:53:52 -05:00

Author

Owner

Copy Link

@chalitbkb commented on GitHub (Mar 28, 2025):

Thanks for the fast reply. Just to make sure I follow everything correctly: Can you tell me how you start your dev version? Maybe our ways of starting/building differ and I can reconstruct it that way. You said earlier you can neither set the variable in the UI nor as an environment variable, in the comment above you write you're "enjoying the RAG_TOP_K_RERANKER feature". Is it somehow/sometimes working for you now, or still not set-able in both ways?

If you have the local code at hand, can you look into the code if you find "k_reranker"? For instance, it should be a parameter for the function query_collection_with_hybrid_search in backend/open_webui/retrieval/utils.py:280.

Another question (please don't be mad at me for this), but did you make sure you're on the latest dev version? Did you pull changes/update your local version?

Is it currently working intermittently or is it still unable to be configured in both ways?

Ans: I mean, I tried testing with the dev version and found the "RAG_TOP_K_RERANKER" option on the UI. I set it to 1, but it still didn't work. It still returns a large number of vector components. It seems that configuring RAG_TOP_K_RERANKER has no effect.

To your question, "How do I start the development version?"

Ans: I use GHCR to update the version with a Dockerfile like:

FROM ghcr.io/open-webui/open-webui:dev

RUN chmod 777 -R /app /root

ENV PORT=7860

<!-- gh-comment-id:2761768762 --> @chalitbkb commented on GitHub (Mar 28, 2025): > Thanks for the fast reply. Just to make sure I follow everything correctly: Can you tell me how you start your dev version? Maybe our ways of starting/building differ and I can reconstruct it that way. You said earlier you can neither set the variable in the UI nor as an environment variable, in the comment above you write you're "enjoying the RAG_TOP_K_RERANKER feature". Is it somehow/sometimes working for you now, or still not set-able in both ways? > > If you have the local code at hand, can you look into the code if you find "k_reranker"? For instance, it should be a parameter for the function `query_collection_with_hybrid_search` in backend/open_webui/retrieval/utils.py:280. > > Another question (please don't be mad at me for this), but did you make sure you're on the latest dev version? Did you pull changes/update your local version? Is it currently working intermittently or is it still unable to be configured in both ways? Ans: I mean, I tried testing with the dev version and found the "RAG_TOP_K_RERANKER" option on the UI. I set it to 1, but it still didn't work. It still returns a large number of vector components. It seems that configuring RAG_TOP_K_RERANKER has no effect. To your question, "How do I start the development version?" Ans: I use GHCR to update the version with a Dockerfile like: ``` FROM ghcr.io/open-webui/open-webui:dev RUN chmod 777 -R /app /root ENV PORT=7860 ```

GiteaMirror commented 2026-05-25 03:53:54 -05:00

Author

Owner

Copy Link

@mahenning commented on GitHub (Mar 28, 2025):

(For a (maybe) quick answer and solution, scroll down to the edit. It seems you asked 7 min before a new dev version was pushed)

I tried to reproduce your problem, but alas, "for me it works".. (That sentence again..)
Steps:
docker pull ghcr.io/open-webui/open-webui:dev
docker run -p 3003:8080 --add-host=host.docker.internal:host-gateway --name open-webui-dev ghcr.io/open-webui/open-webui :dev
I used the default settings for chunks, etc. I used sentence-transformers/all-MiniLM-L6-v2 and BAAI/bge-reranker-v2-m3. In the UI, Top K and Top K Reranker were set default to 3 and 3. I changed it to top k=10 and top k reranker=1, uploaded a document in the chat and asked a question. Then I changed the settings to top k=10, top k reranker=5.

In the logging of the container, I could see the following (truncated and shortened for readability)
top k reranker = 1:

Batches: 100%|██████████| 1/1 [00:00<00:00, 21.30it/s]
Batches: 100%|██████████| 1/1 [00:02<00:00, 2.06s/it]
2025-03-28 16:08:47.179 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result [...] 'score': 0.11954475939273834} 0.11954475939273834 - {}
Batches: 100%|██████████| 1/1 [00:00<00:00, 3.97it/s]
Batches: 100%|██████████| 1/1 [00:02<00:00, 2.14s/it]
2025-03-28 16:08:49.613 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result [...] 'score': 0.01118550170212984} 0.01118550170212984 - {}
Batches: 100%|██████████| 1/1 [00:00<00:00, 71.87it/s]
Batches: 100%|██████████| 1/1 [00:01<00:00, 1.90s/it]
2025-03-28 16:08:51.567 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result [....] 'score': 0.017430100589990616} 0.017430100589990616 - {}

top k reranker=5
Batches: 100%|██████████| 1/1 [00:00<00:00, 21.85it/s]
Batches: 100%|██████████| 1/1 [00:02<00:00, 2.07s/it]
2025-03-28 16:09:59.597 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result 'score': 0.002910345559939742} 0.01118550170212984, 0.009643329307436943, 0.006689450703561306, 0.0032938995864242315, 0.002910345559939742 - {}
Batches: 100%|██████████| 1/1 [00:00<00:00, 97.11it/s]
Batches: 100%|██████████| 1/1 [00:01<00:00, 1.57s/it]
2025-03-28 16:10:01.210 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result 'score': 0.0007439087494276464} 0.007581176236271858, 0.003316872753202915, 0.0012548574013635516, 0.0012110307579860091, 0.0007439087494276464 - {}
Batches: 100%|██████████| 1/1 [00:00<00:00, 74.65it/s]
Batches: 100%|██████████| 1/1 [00:01<00:00, 1.96s/it]
2025-03-28 16:10:03.216 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result 'score': 0.0025328141637146473} 0.01953815668821335, 0.016017155721783638, 0.011503961868584156, 0.008491785265505314, 0.0025328141637146473 - {}

I tried to highlight the important part bold. Each chat question started 3 queries for documents. You can see that for top k reranker=1, there is only one score vector at the end in bold, while for top k reranker=5 there are 5 scores each. This and the fact that I got many more references with top k reranker=5 shows me that it worked, at least for me.

The question now is, why is it not working for you? You could try to again pull the newest image for dev. Both your issue and the newest dev image are "6 hours ago" and for the image, I can not see the exact time it got pushed. It could be that your image version is not the latest anymore?

Edit: Okay, found the time (2025-03-28T02:37:39-7:00), which is apparently 9:37 (UTC, London time). You started the issue at 9:30 UTC if I'm not mistaken, so the newest image should be newer than your issue. Can you re-pull and try again? :)

<!-- gh-comment-id:2761850442 --> @mahenning commented on GitHub (Mar 28, 2025): (For a (maybe) quick answer and solution, scroll down to the edit. It seems you asked 7 min before a new dev version was pushed) I tried to reproduce your problem, but alas, "for me it works".. (_That_ sentence again..) Steps: `docker pull ghcr.io/open-webui/open-webui:dev` `docker run -p 3003:8080 --add-host=host.docker.internal:host-gateway --name open-webui-dev ghcr.io/open-webui/open-webui :dev` I used the default settings for chunks, etc. I used `sentence-transformers/all-MiniLM-L6-v2` and `BAAI/bge-reranker-v2-m3`. In the UI, Top K and Top K Reranker were set default to 3 and 3. I changed it to top k=10 and top k reranker=1, uploaded a document in the chat and asked a question. Then I changed the settings to top k=10, top k reranker=5. In the logging of the container, I could see the following (truncated and shortened for readability) top k reranker = 1: Batches: 100%|██████████| 1/1 [00:00<00:00, 21.30it/s] Batches: 100%|██████████| 1/1 [00:02<00:00, 2.06s/it] 2025-03-28 16:08:47.179 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result [[[...] 'score': 0.11954475939273834}]] **[[0.11954475939273834]]** - {} Batches: 100%|██████████| 1/1 [00:00<00:00, 3.97it/s] Batches: 100%|██████████| 1/1 [00:02<00:00, 2.14s/it] 2025-03-28 16:08:49.613 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result [[[...] 'score': 0.01118550170212984}]] **[[0.01118550170212984]]** - {} Batches: 100%|██████████| 1/1 [00:00<00:00, 71.87it/s] Batches: 100%|██████████| 1/1 [00:01<00:00, 1.90s/it] 2025-03-28 16:08:51.567 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result [[[....] 'score': 0.017430100589990616}]] **[[0.017430100589990616]]** - {} top k reranker=5 Batches: 100%|██████████| 1/1 [00:00<00:00, 21.85it/s] Batches: 100%|██████████| 1/1 [00:02<00:00, 2.07s/it] 2025-03-28 16:09:59.597 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result [[ 'score': 0.002910345559939742}]] **[[0.01118550170212984, 0.009643329307436943, 0.006689450703561306, 0.0032938995864242315, 0.002910345559939742]]** - {} Batches: 100%|██████████| 1/1 [00:00<00:00, 97.11it/s] Batches: 100%|██████████| 1/1 [00:01<00:00, 1.57s/it] 2025-03-28 16:10:01.210 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result [[ 'score': 0.0007439087494276464}]] **[[0.007581176236271858, 0.003316872753202915, 0.0012548574013635516, 0.0012110307579860091, 0.0007439087494276464]]** - {} Batches: 100%|██████████| 1/1 [00:00<00:00, 74.65it/s] Batches: 100%|██████████| 1/1 [00:01<00:00, 1.96s/it] 2025-03-28 16:10:03.216 | INFO | open_webui.retrieval.utils:query_doc_with_hybrid_search:160 - query_doc_with_hybrid_search:result [[ 'score': 0.0025328141637146473}]] **[[0.01953815668821335, 0.016017155721783638, 0.011503961868584156, 0.008491785265505314, 0.0025328141637146473]]** - {} I tried to highlight the important part bold. Each chat question started 3 queries for documents. You can see that for top k reranker=1, there is only one score vector at the end in bold, while for top k reranker=5 there are 5 scores each. This and the fact that I got many more references with top k reranker=5 shows me that it worked, at least for me. The question now is, why is it not working for you? You could try to again pull the newest image for dev. Both your issue and the newest dev image are "6 hours ago" and for the image, I can not see the exact time it got pushed. It could be that your image version is not the latest anymore? Edit: Okay, found the time (2025-03-28T02:37:39-7:00), which is apparently 9:37 (UTC, London time). You started the issue at 9:30 UTC if I'm not mistaken, so the newest image should be newer than your issue. Can you re-pull and try again? :)

GiteaMirror commented 2026-05-25 03:53:56 -05:00

Author

Owner

Copy Link

@chalitbkb commented on GitHub (Mar 28, 2025):

@mahenning

I'm confident that it's the latest version, but I'm still encountering this problem. The results from the vector data still have too many components, even after reducing the Top K and Top K Reranker.

<!-- gh-comment-id:2762091791 --> @chalitbkb commented on GitHub (Mar 28, 2025): @mahenning I'm confident that it's the latest version, but I'm still encountering this problem. The results from the vector data still have too many components, even after reducing the Top K and Top K Reranker.     

GiteaMirror commented 2026-05-25 03:53:59 -05:00

Author

Owner

Copy Link

@chalitbkb commented on GitHub (Mar 28, 2025):

@mahenning

Then I switched to using the Gemini model to test how many results obtained from the vector were discovered. I found that there were still many items even though the model was able to respond to me. This means that the "RAG_TOP_K_RERANKER" setting was not actually implemented or may be due to a defect elsewhere.

<!-- gh-comment-id:2762122412 --> @chalitbkb commented on GitHub (Mar 28, 2025): @mahenning Then I switched to using the Gemini model to test how many results obtained from the vector were discovered. I found that there were still many items even though the model was able to respond to me. This means that the "RAG_TOP_K_RERANKER" setting was not actually implemented or may be due to a defect elsewhere.  

GiteaMirror commented 2026-05-25 03:54:02 -05:00

Author

Owner

Copy Link

@mahenning commented on GitHub (Mar 28, 2025):

I see in your screenshot below that "full context mode" is enabled. If thats enabled, the full document(s) are loaded into the chat, disabling the "chunking -> scoring -> reranking" pipeline that retrieves only (relevant) chunks of the documents. That could also explain your many results, and why top k and top k reranker settings are ignored. There is no filtering for the best, the LLM gets it all.

@mahenning

<!-- gh-comment-id:2762343077 --> @mahenning commented on GitHub (Mar 28, 2025): I see in your screenshot below that "full context mode" is enabled. If thats enabled, the full document(s) are loaded into the chat, disabling the "chunking -> scoring -> reranking" pipeline that retrieves only (relevant) chunks of the documents. That could also explain your many results, and why top k and top k reranker settings are ignored. There is no filtering for the best, the LLM gets it all. > [@mahenning](https://github.com/mahenning) > > 

GiteaMirror commented 2026-05-25 03:54:04 -05:00

Author

Owner

Copy Link

@mahenning commented on GitHub (Mar 28, 2025):

@tjbck Maybe it would be a good ida to hide the hybrid / top k / etc stuff if full context mode is enabled to signal that these settings are ignored in that case.

<!-- gh-comment-id:2762354808 --> @mahenning commented on GitHub (Mar 28, 2025): @tjbck Maybe it would be a good ida to hide the hybrid / top k / etc stuff if full context mode is enabled to signal that these settings are ignored in that case.

GiteaMirror commented 2026-05-25 03:54:07 -05:00

Author

Owner

Copy Link

@tjbck commented on GitHub (Mar 28, 2025):

@mahenning Addressed with 0413c747a9ba33275dc4e4d3146581ed52979474!

<!-- gh-comment-id:2762361519 --> @tjbck commented on GitHub (Mar 28, 2025): @mahenning Addressed with 0413c747a9ba33275dc4e4d3146581ed52979474!

GiteaMirror commented 2026-05-25 03:54:10 -05:00

Author

Owner

Copy Link

@mahenning commented on GitHub (Mar 28, 2025):

Committed 2 mins ago? That was ninja-level fast.

<!-- gh-comment-id:2762363861 --> @mahenning commented on GitHub (Mar 28, 2025): Committed 2 mins ago? That was ninja-level fast.

GiteaMirror commented 2026-05-25 03:54:12 -05:00

Author

Owner

Copy Link

@chalitbkb commented on GitHub (Mar 28, 2025):

@tjbck Maybe it would be a good ida to hide the hybrid / top k / etc stuff if full context mode is enabled to signal that these settings are ignored in that case.

@mahenning
I was so focused on a tiny problem, it was like staring at a breadcrumb and forgetting I had a whole pizza!🍕😂

Thank you, it's working now. However, there seems to be a 20-30 second delay for the RERANKER.

<!-- gh-comment-id:2762372223 --> @chalitbkb commented on GitHub (Mar 28, 2025): > [@tjbck](https://github.com/tjbck) Maybe it would be a good ida to hide the hybrid / top k / etc stuff if full context mode is enabled to signal that these settings are ignored in that case. @mahenning I was so focused on a tiny problem, it was like staring at a breadcrumb and forgetting I had a whole pizza!🍕😂 Thank you, it's working now. However, there seems to be a 20-30 second delay for the RERANKER.

GiteaMirror commented 2026-05-25 03:54:12 -05:00

Author

Owner

Copy Link

@mahenning commented on GitHub (Mar 28, 2025):

If Open-WebUI does not run with GPU support, the embedding/reranking models runs on CPU. Depending on how many document chunks have to be reranked, this could take a while. Embedding only has do be done one time per document, reranking every time you ask something in the chat with documents available. If you habe a few GB VRAM to spare, you can run the :cuda (or :dev-cuda) image.

Would be actually a nice PR to enable e.g. only the reranker to GPU and letting the embedding model run on CPU..

<!-- gh-comment-id:2762388386 --> @mahenning commented on GitHub (Mar 28, 2025): If Open-WebUI does not run with GPU support, the embedding/reranking models runs on CPU. Depending on how many document chunks have to be reranked, this could take a while. Embedding only has do be done one time per document, reranking every time you ask something in the chat with documents available. If you habe a few GB VRAM to spare, you can run the `:cuda` (or :dev-cuda) image. Would be actually a nice PR to enable e.g. only the reranker to GPU and letting the embedding model run on CPU..

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dev

v0.10.2

v0.10.1

v0.10.0

v0.9.6

v0.9.5

v0.9.4

v0.9.3

v0.9.2

v0.9.1

v0.9.0

v0.8.12

v0.8.11

v0.8.10

v0.8.9

v0.8.8

v0.8.7

v0.8.6

v0.8.5

v0.8.4

v0.8.3

v0.8.2

v0.8.1

v0.8.0

v0.7.2

v0.7.1

v0.7.0

v0.6.43

v0.6.42

v0.6.41

v0.6.40

v0.6.39

v0.6.38

v0.6.37

v0.6.36

v0.6.35

v0.6.34

v0.6.33

v0.6.32

v0.6.31

v0.6.30

v0.6.29

v0.6.28

v0.6.27

v0.6.26

v0.6.25

v0.6.24

v0.6.23

v0.6.22

v0.6.21

v0.6.20

v0.6.19

v0.6.18

v0.6.17

v0.6.16

v0.6.15

v0.6.14

v0.6.13

v0.6.12

v0.6.11

v0.6.10

v0.6.9

v0.6.8

v0.6.7

v0.6.6

v0.6.5

v0.6.4

v0.6.3

v0.6.2

v0.6.1

v0.6.0

v0.5.20

v0.5.19

v0.5.18

v0.5.17

v0.5.16

v0.5.15

v0.5.14

v0.5.13

v0.5.12

v0.5.11

v0.5.10

v0.5.9

v0.5.8

v0.5.7

v0.5.6

v0.5.5

v0.5.4

v0.5.3

v0.5.2

v0.5.1

v0.5.0

v0.4.8

v0.4.7

v0.4.6

v0.4.5

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.4.0

v0.3.35

v0.3.34

v0.3.33

v0.3.32

v0.3.31

v0.3.30

v0.3.29

v0.3.28

v0.3.27

v0.3.26

v0.3.25

v0.3.24

v0.3.23

v0.3.22

v0.3.21

v0.3.20

v0.3.19

v0.3.18

v0.3.17

v0.3.16

v0.3.15

v0.3.14

v0.3.13

v0.3.12

v0.3.11

v0.3.10

v0.3.9

v0.3.8

v0.3.7

v0.3.6

v0.3.5

v0.3.4

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.2.5

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.125

v0.1.124

v0.1.123

v0.1.122

v0.1.121

v0.1.120

v0.1.119

v0.1.118

v0.1.117

v0.1.116

v0.1.115

v0.1.114

v0.1.113

v0.1.112

v0.1.111

v0.1.110

v0.1.109

v0.1.108

v0.1.107

v0.1.106

v0.1.105

v0.1.104

v0.1.103

v0.1.102

Labels

Clear labels

bug

confirmed

confirmed issue

core

documentation

enhancement

good first issue

help wanted

non-core

pull-request

Mirrored from GitHub Pull Request

python

question

testing wanted

No Label bug

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

GiteaMirror ninjasurge

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#136209