[GH-ISSUE #15081] issue: Default follow-up generation consumes full context, causing unexpected high credit usage on long prompts #104456

New Issue

GiteaMirror · 2026-05-18T02:08:11-05:00

GiteaMirror commented

2026-05-18 02:08:11 -05:00

Originally created by @inno3759 on GitHub (Jun 17, 2025).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/15081

Check Existing Issues

I have searched the existing issues and discussions.
I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

0.6.15

Operating System

Debian 12

Confirmation

I have read and followed all instructions in README.md.
I am using the latest version of both Open WebUI and Ollama.
I have included the browser console logs.
I have included the Docker container logs.
I have provided every relevant configuration, setting, and environment variable used in my setup.
I have clearly listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc).
I have documented step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation. My steps:
Start with the initial platform/version/OS and dependencies used,
Specify exact install/launch/configure commands,
List URLs visited, user input (incl. example values/emails/passwords if needed),
Describe all options and toggles enabled or changed,
Include any files or environmental changes,
Identify the expected and actual result at each stage,
Ensure any reasonably skilled user can follow and hit the same issue.

Expected Behavior

 The follow-up generator should not resend extremely large messages by default.  
 Or it should at least warn/log when any pulled-in message exceeds a configurable token threshold.  
 Documentation should note that the default prompt, which uses {{MESSAGES:END:6}}, includes full messages and may re-bill large tokens.

Actual Behavior

The default follow-up prompt blindly includes the entire conversation history, causing each follow-up call to resend the full context and incur high credit usage in the UI auxiliary model.

The replace_messages_variable function implements {{MESSAGES:END:X}} by doing:

elif end_length is not None:
    return get_messages_content(messages[-int(end_length):])

In the default follow-up prompt template, {{MESSAGES:END:6}} therefore pulls in the last six messages verbatim. If one of those messages contains a very large document, every follow-up generation call resends that entire document to the auxiliary model and incurs a huge token cost.

Steps to Reproduce

 Run openweb-ui with the default DEFAULT_FOLLOW_UP_GENERATION_PROMPT_TEMPLATE.

 Send an initial user message containing a large document (~600 k tokens).

 Continue the conversation (e.g. ask a follow-up question).

 Observe in your usage logs that the follow-up model is billed for ~600 k tokens on each call.

Additional Information

The default follow-up prompt template pulls in the last six messages ({{MESSAGES:END:6}}) regardless of their size. If one of those messages is a very large document, every follow-up call resends that large prompt to the auxiliary model, driving up token usage and costs unexpectedly in that model.

My temporary prompt:

Ignore all previous instructions.
Suggest 3-5 relevant follow-up questions or prompts that the user might naturally ask next in this conversation as a user, based on the chat history, to help continue or deepen the discussion.

Guidelines:

Write all follow-up questions from the user’s point of view, directed to the assistant.
Make questions concise, clear, and directly related to the discussed topic(s).
Only suggest follow-ups that make sense given the chat content and do not repeat what was already covered.
If the conversation is very short or not specific, suggest more general (but relevant) follow-ups the user might ask.
Use the conversation's primary language; default to English if multilingual.
Response must be a JSON array of strings, no extra text or formatting.
Avoid formatting.

Output:

JSON format: { "follow_ups": ["Question 1?", "Question 2?", "Question 3?"] }

Chat History:

<chat_history>
{{prompt:middletruncate:7500}}
</chat_history>

Originally created by @inno3759 on GitHub (Jun 17, 2025). Original GitHub issue: https://github.com/open-webui/open-webui/issues/15081 ### Check Existing Issues - [x] I have searched the existing issues and discussions. - [x] I am using the latest version of Open WebUI. ### Installation Method Docker ### Open WebUI Version 0.6.15 ### Operating System Debian 12 ### Confirmation - [x] I have read and followed all instructions in `README.md`. - [x] I am using the latest version of **both** Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs. - [x] I have **provided every relevant configuration, setting, and environment variable used in my setup.** - [x] I have clearly **listed every relevant configuration, custom setting, environment variable, and command-line option that influences my setup** (such as Docker Compose overrides, .env values, browser settings, authentication configurations, etc). - [x] I have documented **step-by-step reproduction instructions that are precise, sequential, and leave nothing to interpretation**. My steps: - Start with the initial platform/version/OS and dependencies used, - Specify exact install/launch/configure commands, - List URLs visited, user input (incl. example values/emails/passwords if needed), - Describe all options and toggles enabled or changed, - Include any files or environmental changes, - Identify the expected and actual result at each stage, - Ensure any reasonably skilled user can follow and hit the same issue. ### Expected Behavior The follow-up generator should not resend extremely large messages by default. Or it should at least warn/log when any pulled-in message exceeds a configurable token threshold. Documentation should note that the default prompt, which uses {{MESSAGES:END:6}}, includes full messages and may re-bill large tokens. ### Actual Behavior The default follow-up prompt blindly includes the entire conversation history, causing each follow-up call to resend the full context and incur high credit usage in the UI auxiliary model. The replace_messages_variable function implements {{MESSAGES:END:X}} by doing: ``` elif end_length is not None: return get_messages_content(messages[-int(end_length):]) ``` In the default follow-up prompt template, {{MESSAGES:END:6}} therefore pulls in the last six messages verbatim. If one of those messages contains a very large document, every follow-up generation call resends that entire document to the auxiliary model and incurs a huge token cost. ### Steps to Reproduce 1. Run openweb-ui with the default DEFAULT_FOLLOW_UP_GENERATION_PROMPT_TEMPLATE. 2. Send an initial user message containing a large document (~600 k tokens). 3. Continue the conversation (e.g. ask a follow-up question). 4. Observe in your usage logs that the follow-up model is billed for ~600 k tokens on each call. ### Additional Information The default follow-up prompt template pulls in the last six messages ({{MESSAGES:END:6}}) regardless of their size. If one of those messages is a very large document, every follow-up call resends that large prompt to the auxiliary model, driving up token usage and costs unexpectedly in that model. My temporary prompt: Ignore all previous instructions. Suggest 3-5 relevant follow-up questions or prompts that the user might naturally ask next in this conversation as a **user**, based on the chat history, to help continue or deepen the discussion. ### Guidelines: - Write all follow-up questions from the user’s point of view, directed to the assistant. - Make questions concise, clear, and directly related to the discussed topic(s). - Only suggest follow-ups that make sense given the chat content and do not repeat what was already covered. - If the conversation is very short or not specific, suggest more general (but relevant) follow-ups the user might ask. - Use the conversation's primary language; default to English if multilingual. - Response must be a JSON array of strings, no extra text or formatting. - Avoid formatting. ### Output: JSON format: { "follow_ups": ["Question 1?", "Question 2?", "Question 3?"] } ### Chat History: <chat_history> {{prompt:middletruncate:7500}} </chat_history>

GiteaMirror added the bug label 2026-05-18 02:08:11 -05:00

GiteaMirror closed this issue

2026-05-18 02:08:12 -05:00

GiteaMirror commented

2026-05-18 02:08:12 -05:00

@Classic298 commented on GitHub (Jun 17, 2025):

Generally, it is advisable to use very cheap models as a task model something like gemini 2.0 flash lite or gpt 4.1 nano.

Even with large inputs you will be billed at most a cent or so typically.

But be that as it may, unlike you say, it's not the entire conversation history that gets sent to the task model, just a few messages. And the other tasks do that too (title generation, tag generation, etc.)

But your proposal to add a character based truncation point to truncate each message of those 6 fetched messages is not bad. Perhaps it can be added as a universal setting for all task model tasks.

I.e. every single task will fetch the messages as normal, then truncate them to ABCXYZ characters (whatever has been set in the document settings in the admin panel) and only then insert it into the task's prompt and send it to the task model.

@Classic298 commented on GitHub (Jun 17, 2025): Generally, it is advisable to use very cheap models as a task model something like gemini 2.0 flash lite or gpt 4.1 nano. Even with large inputs you will be billed at most a cent or so typically. But be that as it may, unlike you say, it's not the entire conversation history that gets sent to the task model, just a few messages. And the other tasks do that too (title generation, tag generation, etc.) But your proposal to add a character based truncation point to truncate each message of those 6 fetched messages is not bad. Perhaps it can be added as a universal setting for all task model tasks. I.e. every single task will fetch the messages as normal, then truncate them to ABCXYZ characters (whatever has been set in the document settings in the admin panel) and only then insert it into the task's prompt and send it to the task model.

GiteaMirror commented

2026-05-18 02:08:13 -05:00

@Classic298 commented on GitHub (Jun 17, 2025):

Proposed a PR https://github.com/open-webui/open-webui/pull/15087

@Classic298 commented on GitHub (Jun 17, 2025): Proposed a PR https://github.com/open-webui/open-webui/pull/15087

GiteaMirror commented

2026-05-18 02:08:14 -05:00

@tjbck commented on GitHub (Jun 18, 2025):

For this exactly reason you can write your own custom prompt template for follow up generation. This should be added to our docs, PR welcome!

@tjbck commented on GitHub (Jun 18, 2025): For this exactly reason you can write your own custom prompt template for follow up generation. This should be added to our docs, PR welcome!

GiteaMirror commented

2026-05-18 02:08:14 -05:00

@arrr444 commented on GitHub (Aug 3, 2025):

How can i enable Followup to certain custom models we created and disable it to others, to control which models the followup appear to and dont every model, i tried tweaking the prompt by saying if model name is """" show questions or if so dont, but it didnt work

@arrr444 commented on GitHub (Aug 3, 2025): How can i enable Followup to certain custom models we created and disable it to others, to control which models the followup appear to and dont every model, i tried tweaking the prompt by saying if model name is """" show questions or if so dont, but it didnt work

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/open-webui#104456