Mistral Nemo chat template is not applied correctly when using openai compatible endpoint (llama-server) #1588

Closed
opened 2025-11-11 14:47:54 -06:00 by GiteaMirror · 1 comment
Owner

Originally created by @Tureti on GitHub (Jul 23, 2024).

Bug Report

Description

Bug Summary:
System message is added at the start with only one newline instead of adding it before the last user message with two newlines.
A space before [/INST] is added that shouldn't be there.
Space after [/INST] and before assistant reply isn't added.

Steps to Reproduce:
Start llama-server with Mistral-Nemo-Instruct model and use it as an openai compatible endpoint in open-webui.

Expected Behavior:
Apply the correct chat template to the prompt as specified in https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407/blob/main/tokenizer_config.json

Actual Behavior:
"messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What is 1234+321?"},{"role":"assistant","content":"The sum of 1234 and 321 is 1555."},{"role":"user","content":"Good job!"},{"role":"assistant","content":"Thank you! Is there anything else I can help you with?"},{"role":"user","content":"No."}]

Open-webui prompt:
"prompt": "[INST] You are a helpful assistant.\nWhat is 1234+321? [/INST]The sum of 1234 and 321 is 1555.[INST] Good job! [/INST]Thank you! Is there anything else I can help you with?[INST] No. [/INST]"

Expected prompt:
"prompt":"[INST] What is 1234+321?[/INST] The sum of 1234 and 321 is 1555.[INST] Good job![/INST] Thank you! Is there anything else I can help you with?[INST] You are a helpful assistant.\n\nNo.[/INST]"

Environment

  • Open WebUI Version: v0.3.10

  • Operating System: Ubuntu 22.04.4 LTS

Confirmation:

  • I have read and followed all the instructions provided in the README.md.
  • I am on the latest version of both Open WebUI and Ollama.
  • I have included the browser console logs.
  • I have included the Docker container logs.
Originally created by @Tureti on GitHub (Jul 23, 2024). # Bug Report ## Description **Bug Summary:** System message is added at the start with only one newline instead of adding it before the last user message with two newlines. A space before [/INST] is added that shouldn't be there. Space after [/INST] and before assistant reply isn't added. **Steps to Reproduce:** Start llama-server with Mistral-Nemo-Instruct model and use it as an openai compatible endpoint in open-webui. **Expected Behavior:** Apply the correct chat template to the prompt as specified in [https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407/blob/main/tokenizer_config.json](url) **Actual Behavior:** "messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What is 1234+321?"},{"role":"assistant","content":"The sum of 1234 and 321 is 1555."},{"role":"user","content":"Good job!"},{"role":"assistant","content":"Thank you! Is there anything else I can help you with?"},{"role":"user","content":"No."}] Open-webui prompt: "prompt": "[INST] You are a helpful assistant.\nWhat is 1234+321? [/INST]The sum of 1234 and 321 is 1555.</s>[INST] Good job! [/INST]Thank you! Is there anything else I can help you with?</s>[INST] No. [/INST]" Expected prompt: "prompt":"[INST] What is 1234+321?[/INST] The sum of 1234 and 321 is 1555.</s>[INST] Good job![/INST] Thank you! Is there anything else I can help you with?</s>[INST] You are a helpful assistant.\n\nNo.[/INST]" ## Environment - **Open WebUI Version:** v0.3.10 - **Operating System:** Ubuntu 22.04.4 LTS **Confirmation:** - [x] I have read and followed all the instructions provided in the README.md. - [x] I am on the latest version of both Open WebUI and Ollama. - [x] I have included the browser console logs. - [x] I have included the Docker container logs.
Author
Owner

@justinh-rahb commented on GitHub (Jul 23, 2024):

WebUI does not handle the chat templates of any of the inference backends, you should take this issue up with llama-server.

@justinh-rahb commented on GitHub (Jul 23, 2024): WebUI does not handle the chat templates of any of the inference backends, you should take this issue up with llama-server.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#1588