mirror of
https://github.com/open-webui/open-webui.git
synced 2026-07-16 23:21:44 -05:00
[GH-ISSUE #3148] Feature: Allow setting num_batch and make basic parameters like this available in the chat UI #13149
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#13149
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @sammcj on GitHub (Jun 14, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/3148
Is your feature request related to a problem? Please describe.
num_batch can greatly impact inference performance at the cost of more VRAM usage.
Depending on the task, it can be beneficial to sacrifice say some context size if it allows you to increase num_batch or vice versa.
e.g:
Describe the solution you'd like
It would be great if you could set
num_batchin Open WebUI.It would also be really useful if basic parameters such as num_ctx, num_batch, temp/top, num_keep etc... were available in the chat interface without having to go into settings -> advanced and tweak them there each time.
Describe alternatives you've considered
Right now I'm having to create several copies of models with the num_ctx/num_batch in their name in order to quickly switch between settings. It works, but it's painful.
@tjbck commented on GitHub (Jun 14, 2024):
Great suggestion, PR welcome!
@sammcj commented on GitHub (Jun 14, 2024):
PR incoming :)
@tjbck commented on GitHub (Jun 14, 2024):
@sammcj
@sammcj commented on GitHub (Jun 14, 2024):
@tjbck