[GH-ISSUE #7143] Degraded RAG results in v0.4.2 (compared to v0.3.35) #30155

Closed
opened 2026-04-25 04:27:23 -05:00 by GiteaMirror · 3 comments
Owner

Originally created by @firoagni on GitHub (Nov 21, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/7143

Bug Report

Installation Method

Using Docker to deploy Open WebUI

Environment

  • Open WebUI Version: v0.4.2

  • Ollama: 0.4.2

  • Operating System: macOS 15.0.1

Confirmation:

  • [X ] I have read and followed all the instructions provided in the README.md.
  • [X ] I am on the latest version of both Open WebUI and Ollama.
  • [ ] I have included the browser console logs.
  • [ ] I have included the Docker container logs.
  • [X ] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below.

Expected Behavior:

I am running two instances of Open WebUI:
• Port 2000: Open WebUI v0.4.2
• Port 3000: Open WebUI v0.3.35

Both are fresh installations. I created the same knowledge object and added an identical markdown file to it on both instances. Subsequently, I created a custom model based on Llama:3.2 using the knowledge object in both instances, setting the TEMPERATURE value to 0 (accuracy over creativity).

Expectation: Both Open WebUI v0.4.2 and Open WebUI v0.3.35 should produce similar results since they are running the same custom RAG model.

Actual Behavior:

For the same questions, my custom model in Open WebUI v0.4.2 consistently provides worse responses compared to Open WebUI v0.3.35. I tested with different files, including various file formats, and multiple questions. The answers from v0.4.2 are consistently inferior to those from v0.3.35. Strangely, when I click on citations, they are mostly the same between the two versions. However, v0.4.2 tends not to follow the citations as closely.

Even tried changing the TEMPERATURE values. No luck.

What changes might account for this discrepancy?

Reproduction Details

  1. Setup Instances:
  • Deploy two instances of Open WebUI on different ports:
    • Port 2000: Install Open WebUI v0.4.2
    • Port 3000: Install Open WebUI v0.3.35
      Ensure both installations are fresh and have no prior configuration.
  1. Create Knowledge Object:
  • On both instances, create the same knowledge object.
  • Add an identical markdown file to the knowledge object in each instance.
  1. Configure Custom Model:
  • In both WebUI instances, create a custom model based on Llama:3.2 using the knowledge object.
  • Set the TEMPERATURE value to 0 for both models to prioritize accuracy over creativity.
  1. Run Tests:
  • Ask identical questions to both models in each WebUI instance.
  • Observe and document the responses from each version.
Originally created by @firoagni on GitHub (Nov 21, 2024). Original GitHub issue: https://github.com/open-webui/open-webui/issues/7143 # Bug Report ## Installation Method Using Docker to deploy Open WebUI ## Environment - Open WebUI Version: v0.4.2 - Ollama: 0.4.2 - Operating System: macOS 15.0.1 **Confirmation:** - [X ] I have read and followed all the instructions provided in the README.md. - [X ] I am on the latest version of both Open WebUI and Ollama. - [ ] I have included the browser console logs. - [ ] I have included the Docker container logs. - [X ] I have provided the exact steps to reproduce the bug in the "Steps to Reproduce" section below. ## Expected Behavior: I am running two instances of Open WebUI: • Port 2000: Open WebUI v0.4.2 • Port 3000: Open WebUI v0.3.35 Both are fresh installations. I created the same knowledge object and added an identical markdown file to it on both instances. Subsequently, I created a custom model based on `Llama:3.2` using the knowledge object in both instances, setting the TEMPERATURE value to 0 (accuracy over creativity). Expectation: Both Open WebUI v0.4.2 and Open WebUI v0.3.35 should produce similar results since they are running the same custom RAG model. ## Actual Behavior: For the same questions, my custom model in Open WebUI v0.4.2 consistently provides worse responses compared to Open WebUI v0.3.35. I tested with different files, including various file formats, and multiple questions. The answers from v0.4.2 are consistently inferior to those from v0.3.35. Strangely, when I click on citations, they are mostly the same between the two versions. However, v0.4.2 tends not to follow the citations as closely. Even tried changing the TEMPERATURE values. No luck. What changes might account for this discrepancy? ## Reproduction Details 1. Setup Instances: - Deploy two instances of Open WebUI on different ports: - Port 2000: Install Open WebUI v0.4.2 - Port 3000: Install Open WebUI v0.3.35 Ensure both installations are fresh and have no prior configuration. 2. Create Knowledge Object: - On both instances, create the same knowledge object. - Add an identical markdown file to the knowledge object in each instance. 3. Configure Custom Model: - In both WebUI instances, create a custom model based on Llama:3.2 using the knowledge object. - Set the TEMPERATURE value to 0 for both models to prioritize accuracy over creativity. 4. Run Tests: - Ask identical questions to both models in each WebUI instance. - Observe and document the responses from each version.
Author
Owner

@tjbck commented on GitHub (Nov 21, 2024):

You can disable RAG query generation. Please check the changelogs.

<!-- gh-comment-id:2491595795 --> @tjbck commented on GitHub (Nov 21, 2024): You can disable RAG query generation. Please check the changelogs.
Author
Owner

@firoagni commented on GitHub (Nov 21, 2024):

@tjbck

changelogs

Dumb question - From where I can disable RAG query generation? Couldn't able to find the checkbox

<!-- gh-comment-id:2491909919 --> @firoagni commented on GitHub (Nov 21, 2024): @tjbck > changelogs Dumb question - From where I can disable RAG query generation? Couldn't able to find the checkbox
Author
Owner

@morgan55555 commented on GitHub (Nov 22, 2024):

From where I can disable RAG query generation? Couldn't able to find the checkbox

In Interface tab.
Also, you can change Query Generation Prompt. This prompt is used both for internet search and RAG search. I found that better results will be with prompt like that:

`

Task:

Based on the user's message and interaction history, generate search query. Refrain from verbose responses or any additional commentary. Prefer suggesting a search if uncertain to provide comprehensive or updated information. Then output both search query and user message. Today's date is: {{CURRENT_DATE}}. Search query must be in a user's message language.

Guidelines:

  • Respond exclusively with a JSON object.
  • Be concise, focusing strictly on composing search queries with no additional commentary or text.

Output:

JSON format: {
"queries": ["Search query", "User message"]
}

User message:

{{prompt:end:4000}}

Chat History:

{{MESSAGES:END:6}}
`

<!-- gh-comment-id:2493855997 --> @morgan55555 commented on GitHub (Nov 22, 2024): > From where I can disable RAG query generation? Couldn't able to find the checkbox In Interface tab. Also, you can change Query Generation Prompt. This prompt is used both for internet search and RAG search. I found that better results will be with prompt like that: ` ### Task: Based on the user's message and interaction history, generate search query. Refrain from verbose responses or any additional commentary. Prefer suggesting a search if uncertain to provide comprehensive or updated information. Then output both search query and user message. Today's date is: {{CURRENT_DATE}}. Search query must be in a user's message language. ### Guidelines: - Respond exclusively with a JSON object. - Be concise, focusing strictly on composing search queries with no additional commentary or text. ### Output: JSON format: { "queries": ["Search query", "User message"] } ### User message: {{prompt:end:4000}} ### Chat History: {{MESSAGES:END:6}} `
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#30155