github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-07-16 23:21:44 -05:00
Code Issues 2.4k Packages Projects Releases 129 Wiki Activity

[GH-ISSUE #21888] bug: RAG query generation fails with thinking models (<think> tags break JSON extraction) #58269

New Issue
Closed
opened 2026-05-05 22:44:04 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-05-05 22:44:04 -05:00
Owner
Copy Link

Originally created by @rockinyp on GitHub (Feb 26, 2026).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/21888

GitHub Issue Draft for open-webui/open-webui

Title: RAG query generation fails with thinking models (<think> tags break JSON extraction)

Description

RAG query generation fails when using thinking models (Qwen3, DeepSeek R1, etc.) because these models wrap their responses in <think>...</think> tags. The current JSON extraction logic doesn't handle this, causing RAG to silently fail with "No sources found."

Steps to Reproduce

  1. Configure an OpenAI-compatible endpoint with a thinking model (e.g., Qwen3.5, DeepSeek R1)
  2. Enable RAG with a knowledge folder
  3. Ask a question that should trigger RAG retrieval
  4. Observe "No sources found" even though the knowledge base has relevant documents

Expected Behavior

RAG should work regardless of whether the model outputs thinking tags.

Actual Behavior

  • "No sources found" displayed
  • Error in logs: TypeError: 'NoneType' object is not subscriptable in generate_embeddings
  • The same knowledge base works fine with non-thinking models

Root Cause

In open_webui/utils/middleware.py (around line 1763), the query generation response is parsed like this:

queries_response = queries_response["choices"][0]["message"]["content"]

try:
    bracket_start = queries_response.find("{")
    bracket_end = queries_response.rfind("}") + 1
    # ... JSON extraction

When a thinking model responds, the content looks like:

<think>
The user is asking about X. I should generate queries for...
</think>
{"queries": ["query1", "query2"]}

The JSON extraction either fails or extracts garbage because:

  1. The { inside thinking content may be found first
  2. The parser doesn't expect text before the JSON
  3. Fallback logic puts the entire malformed response into queries, which then fails in embedding

Proposed Fix

Strip <think>...</think> tags before JSON extraction:

queries_response = queries_response["choices"][0]["message"]["content"]

# Strip <think>...</think> tags from thinking models
import re
queries_response = re.sub(r'<think>.*?</think>', '', queries_response, flags=re.DOTALL).strip()

try:
    bracket_start = queries_response.find("{")
    bracket_end = queries_response.rfind("}") + 1

This is a minimal, non-breaking change that:

  • Handles thinking models correctly
  • Has no effect on non-thinking models (no <think> tags to strip)
  • Uses re.DOTALL to handle multi-line thinking blocks

Environment

  • Open WebUI version: 0.8.5
  • Model: Qwen3.5-35B-A3B-8bit via mlx_vlm.server (OpenAI-compatible API)
  • Also affects: DeepSeek R1, any model with <think> output format

Additional Context

Labels to add

  • bug
  • rag
  • enhancement (if treating as feature request for thinking model support)
Originally created by @rockinyp on GitHub (Feb 26, 2026). Original GitHub issue: https://github.com/open-webui/open-webui/issues/21888 # GitHub Issue Draft for open-webui/open-webui **Title:** RAG query generation fails with thinking models (`<think>` tags break JSON extraction) --- ## Description RAG query generation fails when using thinking models (Qwen3, DeepSeek R1, etc.) because these models wrap their responses in `<think>...</think>` tags. The current JSON extraction logic doesn't handle this, causing RAG to silently fail with "No sources found." ## Steps to Reproduce 1. Configure an OpenAI-compatible endpoint with a thinking model (e.g., Qwen3.5, DeepSeek R1) 2. Enable RAG with a knowledge folder 3. Ask a question that should trigger RAG retrieval 4. Observe "No sources found" even though the knowledge base has relevant documents ## Expected Behavior RAG should work regardless of whether the model outputs thinking tags. ## Actual Behavior - "No sources found" displayed - Error in logs: `TypeError: 'NoneType' object is not subscriptable` in `generate_embeddings` - The same knowledge base works fine with non-thinking models ## Root Cause In `open_webui/utils/middleware.py` (around line 1763), the query generation response is parsed like this: ```python queries_response = queries_response["choices"][0]["message"]["content"] try: bracket_start = queries_response.find("{") bracket_end = queries_response.rfind("}") + 1 # ... JSON extraction ``` When a thinking model responds, the content looks like: ``` <think> The user is asking about X. I should generate queries for... </think> {"queries": ["query1", "query2"]} ``` The JSON extraction either fails or extracts garbage because: 1. The `{` inside thinking content may be found first 2. The parser doesn't expect text before the JSON 3. Fallback logic puts the entire malformed response into queries, which then fails in embedding ## Proposed Fix Strip `<think>...</think>` tags before JSON extraction: ```python queries_response = queries_response["choices"][0]["message"]["content"] # Strip <think>...</think> tags from thinking models import re queries_response = re.sub(r'<think>.*?</think>', '', queries_response, flags=re.DOTALL).strip() try: bracket_start = queries_response.find("{") bracket_end = queries_response.rfind("}") + 1 ``` This is a minimal, non-breaking change that: - Handles thinking models correctly - Has no effect on non-thinking models (no `<think>` tags to strip) - Uses `re.DOTALL` to handle multi-line thinking blocks ## Environment - Open WebUI version: 0.8.5 - Model: Qwen3.5-35B-A3B-8bit via mlx_vlm.server (OpenAI-compatible API) - Also affects: DeepSeek R1, any model with `<think>` output format ## Additional Context - Gist with patch: https://gist.github.com/rockinyp/a622dd40fb25f838cd2916bb521bd3e4 - The same issue may affect other task generation endpoints (title generation, autocomplete) if they also don't handle thinking tags --- ## Labels to add - `bug` - `rag` - `enhancement` (if treating as feature request for thinking model support)
GiteaMirror added the bug label 2026-05-05 22:44:04 -05:00
GiteaMirror commented 2026-05-05 22:44:06 -05:00
Author
Owner
Copy Link

@pr-validator-bot commented on GitHub (Feb 26, 2026):

⚠️ Missing Issue Title Prefix

@rockinyp, your issue title is missing a prefix (e.g., bug:, feat:, docs:).

Please update your issue title to include one of the following prefixes:

  • bug: Bug report or error you've encountered
  • feat: Feature request or enhancement suggestion
  • docs: Documentation issue or improvement request
  • question: Question about usage or functionality
  • help: Request for help or support

Example: bug: Login fails when using special characters in password

<!-- gh-comment-id:3963764223 --> @pr-validator-bot commented on GitHub (Feb 26, 2026): # ⚠️ Missing Issue Title Prefix @rockinyp, your issue title is missing a prefix (e.g., `bug:`, `feat:`, `docs:`). Please update your issue title to include one of the following prefixes: - **bug**: Bug report or error you've encountered - **feat**: Feature request or enhancement suggestion - **docs**: Documentation issue or improvement request - **question**: Question about usage or functionality - **help**: Request for help or support Example: `bug: Login fails when using special characters in password`
GiteaMirror commented 2026-05-05 22:44:06 -05:00
Author
Owner
Copy Link

@rockinyp commented on GitHub (Feb 26, 2026):

⚠️ Missing Issue Title Prefix

@rockinyp, your issue title is missing a prefix (e.g., bug:, feat:, docs:).

Please update your issue title to include one of the following prefixes:

  • bug: Bug report or error you've encountered
  • feat: Feature request or enhancement suggestion
  • docs: Documentation issue or improvement request
  • question: Question about usage or functionality
  • help: Request for help or support

Example: bug: Login fails when using special characters in password

Should be updated now.

<!-- gh-comment-id:3963786132 --> @rockinyp commented on GitHub (Feb 26, 2026): > # ⚠️ Missing Issue Title Prefix > [@rockinyp](https://github.com/rockinyp), your issue title is missing a prefix (e.g., `bug:`, `feat:`, `docs:`). > > Please update your issue title to include one of the following prefixes: > > * **bug**: Bug report or error you've encountered > * **feat**: Feature request or enhancement suggestion > * **docs**: Documentation issue or improvement request > * **question**: Question about usage or functionality > * **help**: Request for help or support > > Example: `bug: Login fails when using special characters in password` Should be updated now.
GiteaMirror commented 2026-05-05 22:44:07 -05:00
Author
Owner
Copy Link

@Classic298 commented on GitHub (Mar 7, 2026):

Generally it is not recommended to use reasoning models as task models but I'll leave this open for now

<!-- gh-comment-id:4014967597 --> @Classic298 commented on GitHub (Mar 7, 2026): Generally it is not recommended to use reasoning models as task models but I'll leave this open for now
GiteaMirror commented 2026-05-05 22:44:07 -05:00
Author
Owner
Copy Link

@tjbck commented on GitHub (Mar 24, 2026):

Addressed in dev.

<!-- gh-comment-id:4121834714 --> @tjbck commented on GitHub (Mar 24, 2026): Addressed in dev.
GiteaMirror commented 2026-05-05 22:44:08 -05:00
Author
Owner
Copy Link

@Yasand123 commented on GitHub (Apr 15, 2026):

Is this fix included in v0.8.12? since I still have the issue

<!-- gh-comment-id:4250107625 --> @Yasand123 commented on GitHub (Apr 15, 2026): Is this fix included in v0.8.12? since I still have the issue
GiteaMirror commented 2026-05-05 22:44:08 -05:00
Author
Owner
Copy Link

@rockinyp commented on GitHub (Apr 15, 2026):

Is this fix included in v0.8.12? since I still have the issue

My solution was to switch from MLX to Ollama. Ollama adds better support for the thinking tags than MLX.

<!-- gh-comment-id:4251116400 --> @rockinyp commented on GitHub (Apr 15, 2026): > Is this fix included in v0.8.12? since I still have the issue My solution was to switch from MLX to Ollama. Ollama adds better support for the thinking tags than MLX.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#58269
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?