mirror of
https://github.com/open-webui/open-webui.git
synced 2026-03-09 23:35:09 -05:00
feat: beam search #68
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#68
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @walking-octopus on GitHub (Dec 1, 2023).
Is your feature request related to a problem? Please describe.
Small LLMs make silly mistakes in reasoning. With less parameters to capture the real world with more certainty, nonsense starts to seem almost as immediately probably as the correct continuation, even if the model understands the end result of running with it is not as likely as the correct solution.
Describe the solution you'd like
Beam search is sampling mechanism to maximize the probability of the entire completion, not just the next token. This allows for higher quality generations in use-cases where one step logically follows from the next, such as writing a short story with a twist near the end or solving a physics word problem.
ChatGPT is speculated to use beam search, as it's cheaper to use a smaller indecisive model and let it make more guesses then a behemoth like the original GPT-3 Davinci.
Describe alternatives you've considered
Other methods such as context grammars, Classifier-Free Guidance, and dynamic temperature can increase output quality, but they are to be regarded as separate issues and haven't yet established themselves as well as beam search.
Additional context
Blocked by jmorganca/ollama#1344.