mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-16 06:03:26 -05:00
[GH-ISSUE #1997] Reduce title auto-generation costs by supporting a token limit #132445
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#132445
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @varenc on GitHub (May 6, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/1997
Is your feature request related to a problem? Please describe.
I like having title auto-generation, but I hate that it essentially doubles the cost for my first query. When using a hosted model like gpt-4-turbo and title auto-generation is on, starting a new chat makes one request to answer my prompt, and then another request with the entire prompt contents again asking for a concise summary title.
I noticed this recently because I made a new chat and sent a prompt to gpt-4-turbo that had 10k tokens, and I observed in LangFuse that I had 2x request with 10k+ tokens made: one to answer my question and one to generate the title. Ultimately this made my query cost $0.24 instead of $0.12. For this query the only information needed to generate the title would have been in my prompt's first ~200 tokens and not in the large block of text pasted in.
Describe the solution you'd like
Allow users to configure a title auto-generate token limit and default that limit to something reasonable like 2048.
Describe alternatives you've considered
I searched the GitHub issues and docs for something like this and couldn't find anything. Apologies if I missed something. At the moment my best workaround is when I want to make a large prompt query, I start a new chat with a brief summary of the question but without the bulk of the tokens, let the title-auto generate, and then I make my large query. Alternatively I now realize I can set the title auto-generation model to something cheaper, but it still feels wasteful to be sending my entire prompt that could be 10k+ tokens just for a 3-5 title summary.
Additional context
This doesn't matter for local models where queries are free, which many users are using, but it will matter for people making large queries to pay-per-token models like OpenAI's.
@varenc commented on GitHub (May 6, 2024):
relatedly, I noticed a bug with title auto-generation. If your chat is using multiple models, Open WebUI seems to make duplicate title auto-generation queries for every model. Since the query is the same the result is the same (usually) and even if it wasn't I assume Open WebUI would still only use one of them.
@cheahjs commented on GitHub (May 6, 2024):
This was why I added the ability to modify the the prompt template to take a subset of the start and end of the prompt, so it doesn't use a massive prompt: https://github.com/open-webui/open-webui/pull/1539
@tjbck commented on GitHub (May 7, 2024):
Thanks for the suggestion! Would our current implementation on our dev branch suffice for your use case? @varenc
@varenc commented on GitHub (May 7, 2024):
Thanks for the info! The ability to use
{prompt:start:NN}to truncate the prompt being sent with the title gen prompt would definitely meet my needs. Apologies I didn't notice that feature existed. I'd considered this feature request closed!That said, it's a little roundabout from what I was imagining since I just want title generation to cost less and care less about customizing the title generation prompt to suit my needs. I'd suggest making the default title generation prompt use
{prompt:start:4096}or something. I had some other thoughts on #1539 I'll comment on too.