mirror of
https://github.com/open-webui/open-webui.git
synced 2026-03-14 11:05:45 -05:00
feat: Increase context length setting for new models #754
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#754
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @notasquid1938 on GitHub (Apr 28, 2024).
Now that models such as this one https://ollama.com/library/dolphin-llama3:256k have created contexts lengths up to 256k, can the context length slider be adjusted to not max out at 16k? Also would it be possible to make context length settable by model? Is there a downside to just using the highest context value even for models that don't support it?
@NiLon commented on GitHub (Apr 28, 2024):
Short answer would be that yes, it's bad to use bigger context value than the model is trained for. The model starts to generate nonsense. There are some techniques to reach longer context windows by playing with the index positions but these are generally things for the model to do rather than setting it to unsupported values. Like for example the model you are speaking of likely already has these tricks.
@notasquid1938 commented on GitHub (Apr 29, 2024):
Went ahead and made a pr with this change #1843 just waiting on approval
@notasquid1938 commented on GitHub (Apr 30, 2024):
Looks like another model just dropped:
https://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048k
The context limit needs to be increased to 1048576 (Or maybe add an option for the user to set it as high as they want to avoid constantly increasing it?)
@NiLon commented on GitHub (Apr 30, 2024):
This setting might be fundamentally flawed to start with. It is inherently model-specific. Setting it too high consumes more memory and makes the model output nonsense because it wasn't trained for it. Having such global value might not be at all correct thing to do.
https://github.com/ggerganov/llama.cpp/discussions/3111
@notasquid1938 commented on GitHub (Apr 30, 2024):
Maybe adding all the parameters as an option for modelfiles instead of a global setting could serve as a case by case solution? The only issue I see is not being able to adjust non-modelfile models.
@GrayXu commented on GitHub (Aug 7, 2024):
So when using an external API, these settings would not take effect?
@max5800 commented on GitHub (Aug 26, 2024):
do you got an answer for that?
Is there an option to set context windows per model which are connected via api?