mirror of
https://github.com/open-webui/open-webui.git
synced 2026-05-30 05:18:47 -05:00
Add support to use custom chat template for usage with koboldcpp #1267
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/open-webui#1267
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Tureti on GitHub (Jun 15, 2024).
Is your feature request related to a problem? Please describe.
Koboldcpp uses Alpaca as the default chat template for all v1 api completion requests, causing models with other chat templates to not work correctly.
Describe the solution you'd like
Koboldcpps work around is described here: https://github.com/LostRuins/koboldcpp/pull/466 and
fb6b9a8c41.It would be great if we could specify a custom chat template for a model to use with koboldcpp.
Describe alternatives you've considered
I currently hardcoded the chat template for Llama-3 into backend/apps/openai/main.py: async def generate_chat_completion() adding this code:
payload["adapter"] = {"system_start":"<|start_header_id|>system<|end_header_id|>\n\n", "system_end":"<|eot_id|>", "user_start":"<|start_header_id|>user<|end_header_id|>\n\n", "user_end":"<|eot_id|>", "assistant_start":"<|start_header_id|>assistant<|end_header_id|>\n\n", "assistant_end":"<|eot_id|>"}
@LostRuins commented on GitHub (Jun 15, 2024):
Fyi, the adapter can be sent over the API, but it also accepts plain JSON files that can be loaded independently of any frontend such as open-webui.
Here's a sample file: https://github.com/LostRuins/koboldcpp/wiki#what-is---chatcompletionsadapter
When launching KoboldCpp, you can apply this adapter manually using this technique. It can also be selected in the GUI loader.
Or if using CLI, with the
--chatcompletionsadapterlaunch flag.Of course, this does not preclude sending an adapter over the API, and if an adapter is sent over the API it will take precedence over the one selected on load.
@tukangcode commented on GitHub (Jun 15, 2024):
Sorry if out of topic of issue but are openwebui support koboldcpp and horde api?
@LostRuins commented on GitHub (Jun 15, 2024):
Koboldcpp provides a OpenAI compatible endpoint for completions and chat completions, so it should.
@TheTerrasque commented on GitHub (Jun 15, 2024):
On a side note, I've noticed many gguf's have a jinja2 based template value in them, that at least llama-cpp-python uses. Does koboldcpp support or plan to support that? If so, that could be one workaround for this problem.