Regarding non-conversational tasks such as tool usage, it would be interesting to force JSON usage. Since instruct models are fine-tuned towards conversation, it's hard to get proper responses related to feature extraction and arbitrary choices by LLMs. Libraries such as Ollama provide the option for models to answer in JSON with a provided schema. These features use GBNF (refer to llama.cpp docs) in order to restrict grammar, making JSON responses very efficient, or at least more efficient than prompting the LLM to do so. I believe this will be important, especially towards tool usage, but it can also be used in query and title generation.
<!-- gh-comment-id:2157852260 -->
@IIPedro commented on GitHub (Jun 10, 2024):
Regarding non-conversational tasks such as tool usage, it would be interesting to force JSON usage. Since instruct models are fine-tuned towards conversation, it's hard to get proper responses related to feature extraction and arbitrary choices by LLMs. Libraries such as Ollama provide the option for models to answer in JSON with a provided schema. These features use GBNF (refer to llama.cpp docs) in order to restrict grammar, making JSON responses very efficient, or at least more efficient than prompting the LLM to do so. I believe this will be important, especially towards tool usage, but it can also be used in query and title generation.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @tjbck on GitHub (Jun 4, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/2825
#798 #2175
@IIPedro commented on GitHub (Jun 10, 2024):
Regarding non-conversational tasks such as tool usage, it would be interesting to force JSON usage. Since instruct models are fine-tuned towards conversation, it's hard to get proper responses related to feature extraction and arbitrary choices by LLMs. Libraries such as Ollama provide the option for models to answer in JSON with a provided schema. These features use GBNF (refer to llama.cpp docs) in order to restrict grammar, making JSON responses very efficient, or at least more efficient than prompting the LLM to do so. I believe this will be important, especially towards tool usage, but it can also be used in query and title generation.