[GH-ISSUE #2478] Mistral instruction following doesn't work as should be when the prompt is lengthy #1448

New Issue

GiteaMirror · 2026-04-12T11:20:56-05:00

GiteaMirror commented

2026-04-12 11:20:56 -05:00

Originally created by @rsandx on GitHub (Feb 13, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2478

I'm working on a project using RAG, which requires a model to answer questions based on the provided context. Having tested mistral-7b-instruct-v0.2.Q5_K_M.gguf served by llamafile and Ollama, I found that the instruction following works well for both servers when the prompt is not lengthy; but when the prompt is lengthy, the llamafile server returns proper contents (although not as good as ChatGPT's but acceptable), the Ollama server returns contents that feel like ignoring the context or hallucinating. See examples below for comparison and reproducing the results.

llamafile with short prompt:
curl http://localhost:8888/completion -d '{"prompt": "You'''re assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we'''ve built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nwhat'''s Speech Synthesizer?"}'
the response:
{"content":"\nThe Speech Synthesizer is a powerful tool from aiTransformer that can convert text into human-like speech in various languages and accents. It also has the capability to generate videos with a presenter speaking the text and lip-syncing the avatar. This tool can be used to create personalized talking greeting cards, as demonstrated in this video, and is integrated with VideoPlus Studio for adding subtitles, generating speeches, and creating lip-synced avatars for videos/documents/images.","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf","n_ctx":32768,"n_keep":0,"n_predict":-1,"n_probs":0,"penalize_nl":true,"penalty_prompt_tokens":[],"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temperature":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0,"use_penalty_prompt_tokens":false},"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf"},"tokens_cached":424,"tokens_evaluated":313,"tokens_predicted":112,"truncated":false}
llamafile with long prompt:
curl http://localhost:8888/completion -d '{"prompt": "You'''re assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Cartoonizer transforms real-world pictures into cartoon-style images. It offers several cartoon styles to choose from, with sample effects provided, and more styles to be added over time. The tool is designed to make it easy to create fun and unique cartoon images directly from photos. For more information on the different cartoon styles, read our blog posts at here and here, and watch the demo video for usage tips.\nWith the Enhancer, you can easily enhance your photos and make them look clearer, sharper, and more professional. The tool offers several enhance types to choose from, including options to restore faces, and more types will be added over time. For more information on the different enhance types, read our blog post at here.\nThe Sketcher makes it easy to turn photos into sketches. Whether you'''re an artist or just looking for a fun way to express your creativity, the Sketcher offers four different sketch styles, as well as two byproducts for finding the edges and smoothing the image (good for selfies). Additionally, a colored head sketch can be produced if the input image contains a face. With its simple interface, anyone can create sketches in no time, no need for any art skills or experience. So why not give it a try today and see what kind of sketches you can create!\nUsing single image super resolution algorithms, the Enlarger can upsize images at a scale of up to 8 (64 times of the original size). This can produce high-resolution images that are suitable for large prints. However, it'''s important to note that high-quality 8x zoom tends to work on smaller images, and that larger upscaling can take longer to process.\nThe Filter adds special effects to your photos and videos, giving them a unique and special touch. You can choose from a range of filters to give your photos a different look, from subtle tweaks to bold and dramatic effects. These filters are simple to use and can be applied with just a few clicks.\nThe Stylizer transforms photos into works of art inspired by famous artists and styles. Simply provide the source image and the desired style image, the Stylizer will generate six stylized versions with varying style intensities. Whether you'''re a professional artist or just looking to have some fun, the Stylizer is the perfect tool to turn your photos into unique and eye-catching works of art. You can select from predefined style images or upload your own custom style image. This gives you a wide range of possibilities for creating unique and interesting stylized images. Additionally, the Stylizer has an option to apply styles to the whole image or to specific regions, such as the foreground or background. While the foreground and background detection is not always accurate, experimenting can lead to some unexpected and interesting results.\nThe MultiStylizer offers a creative approach to image styling by combining multiple styles into a single image. With its semantic-aware neural network, the MultiStylizer automatically incorporates different styles based on the regional semantics of the source image, creating a unique and visually stunning result that is different from the traditional single style transfer. With the option to select a predefined or custom style for each style choice, and the ability to apply styles to the whole region or auto-detected foreground / background, the possibilities for creating stunning digital art are limitless.\nBased on the powerful deep learning, text-to-image model Stable Diffusion, the Super Stylizer can generate stunning detailed images conditioned on text descriptions, and can also use the generated images to stylize your picture with adjustable style strength. In order to produce meaningful results, it'''s important to describe elements to be included or omitted from the output. For more information on this topic, read our blog posts at here and here. To make things easier, a list of frequently-used art styles and mediums are provided, and there is a demo video to show how the tool works. We also have a dedicated Prompt Builder to help you build text prompts intuitively or create random prompts in one click, making the process even simpler and more fun. The Prompt Builder lists 1000+ short prompts with sample images, including 500+ textual inversion terms verified to work here. Unlike some other platforms, the images you generated here are absolutely private.\nThe Prompt Builder allows you to easily create text prompts for the Stable Diffusion text-to-image model in the Super Stylizer by providing a list of special terms and often-used generic terms with sample images. This helps you build text prompts intuitively, and you can even generate random prompts with just one click or generate prompts from images. The supported terms include 500+ (and growing) special Textual Inversion terms, giving you a wider vocabulary to use in your text prompts.\nThe Background Editor can remove the background from an image, leaving only the subject of the image visible. The process uses machine learning algorithms to identify the subject and separate it from the background, allowing you to create more professional and visually appealing content while also saving time and effort. You can also swap the original background with a new one and position the foreground element in a specific location, while also setting the transparency level for both the foreground and background. For more information on this topic, read our blog posts at here and here.\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we'''ve built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nVideoPlus Studio first is a free subtitle editor and translator, that allows you to add subtitles to videos, documents and images, edit and translate them to other languages. Moreover, you can select a presenter for each subtitle, the presenter has properties for turning the text into speeches in certain language and voice, and an image (optional) for generating a lip-synced avatar to speak the text; the avatar has properties like shape, size and location to control how it'''s going to show in the video. See some use cases and sample videos on this page. Besides, the app can also apply different cartoon styles and filters to videos, as well as transcribe audio to text with automatically detected language and save the text to a subtitle file that'''s ready to use in this app.\nIf you just want to apply special effects to videos, open the task window by clicking the '''Task''' button in the menu, then click the '''Submit New Video Effect Task''' button, select your video and effect and submit to process. To get a video transcription, click the '''Submit New Video Transcription Task''' button, select your video and submit to process. To add/edit subtitles with presenters, you should get familiar with 3 key concepts used in this app. Subtitles: Subtitles are text representing the contents of the audio in a video, they must include information on when each line of text should appear and disappear. The subtitle editor is on the main screen besides the video/document/image player. Subtitle editing for a video is based on timeline, while it is based on page number or image frame for a document or an image. By default the main screen is loaded with this tutorial video and its subtitles, you can play around with it to learn the subtitle editing features. Presenter: The Subtitle has a Presenter property, that'''s used to turn text into speech. A Presenter is a user defined object that has a name and voice, optionally an avatar image of certain shape, size and location in the resulting video. The presenter window is opened with the '''Presenter''' button in the menu. By default each user has 2 preset presenters that can be modified. Presenters are your AI aides that speak your ideas. Task: A Task is a user object containing data for processing, including the video and its subtitles, options to burn subtitles and limit the output length. Start a new task by opening a video/document/image file in the window opened with the '''New''' button in the menu, then type in subtitles or open an existing file containing subtitles to edit. The app will try to extract text on each page when you open a document. For every subtitle select a presenter, adjust the text and the starting and ending position. Once you are done with editing, open the task window and use the '''Submit Current Task''' button to submit for processing. Download link to the resulting video can be found in the task history when available.\nwhat'''s Speech Synthesizer?"}'
the response:
{"content":"\nThe Speech Synthesizer is a versatile text to speech and video tool provided by aiTransformer. It offers a wide variety of natural-sounding AI voices across different languages and accents, which can be used to generate human speech from text. Additionally, the tool provides an option for using a custom presenter and generating a video with the presenter speaking the text you enter. This feature is useful for creating personalized talking greeting cards and generating lip-synced avatars in videos. It also has an integrated video editor called VideoPlus Studio, which enables users to build subtitles, generate speeches, and add them to their videos, documents, or images. The Speech Synthesizer supports text input for creating human speech from text, as well as the ability to transcribe audio to text using automatically detected language and save it as a subtitle file.","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf","n_ctx":32768,"n_keep":0,"n_predict":-1,"n_probs":0,"penalize_nl":true,"penalty_prompt_tokens":[],"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temperature":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0,"use_penalty_prompt_tokens":false},"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf"},"tokens_cached":2289,"tokens_evaluated":2105,"tokens_predicted":185,"truncated":false}
Ollama with short prompt:
curl http://localhost:11434/api/generate -d '{"model": "mistral:7b-instruct-v0.2-q5_K_M", "stream": false, "raw": true, "prompt": "You are assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we have built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nwhat is Speech Synthesizer?"}'
the response:
{"model":"mistral:7b-instruct-v0.2-q5_K_M","created_at":"2024-02-13T16:19:41.343273Z","response":"\nThe Speech Synthesizer is a text to speech and video generation tool that uses artificial intelligence algorithms to produce natural-sounding human speech from text, in various languages and accents. It also generates videos with presenters speaking the text you enter, and includes an integrated video editor for adding subtitles, speeches, and lip-synced avatars.","done":true,"total_duration":3020059208,"load_duration":768655125,"prompt_eval_count":310,"prompt_eval_duration":594788000,"eval_count":79,"eval_duration":1656288000}
Ollama with long prompt
curl http://localhost:11434/api/generate -d '{"model": "mistral:7b-instruct-v0.2-q5_K_M", "stream": false, "raw": true, "prompt": "You'''re assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Cartoonizer transforms real-world pictures into cartoon-style images. It offers several cartoon styles to choose from, with sample effects provided, and more styles to be added over time. The tool is designed to make it easy to create fun and unique cartoon images directly from photos. For more information on the different cartoon styles, read our blog posts at here and here, and watch the demo video for usage tips.\nWith the Enhancer, you can easily enhance your photos and make them look clearer, sharper, and more professional. The tool offers several enhance types to choose from, including options to restore faces, and more types will be added over time. For more information on the different enhance types, read our blog post at here.\nThe Sketcher makes it easy to turn photos into sketches. Whether you'''re an artist or just looking for a fun way to express your creativity, the Sketcher offers four different sketch styles, as well as two byproducts for finding the edges and smoothing the image (good for selfies). Additionally, a colored head sketch can be produced if the input image contains a face. With its simple interface, anyone can create sketches in no time, no need for any art skills or experience. So why not give it a try today and see what kind of sketches you can create!\nUsing single image super resolution algorithms, the Enlarger can upsize images at a scale of up to 8 (64 times of the original size). This can produce high-resolution images that are suitable for large prints. However, it'''s important to note that high-quality 8x zoom tends to work on smaller images, and that larger upscaling can take longer to process.\nThe Filter adds special effects to your photos and videos, giving them a unique and special touch. You can choose from a range of filters to give your photos a different look, from subtle tweaks to bold and dramatic effects. These filters are simple to use and can be applied with just a few clicks.\nThe Stylizer transforms photos into works of art inspired by famous artists and styles. Simply provide the source image and the desired style image, the Stylizer will generate six stylized versions with varying style intensities. Whether you'''re a professional artist or just looking to have some fun, the Stylizer is the perfect tool to turn your photos into unique and eye-catching works of art. You can select from predefined style images or upload your own custom style image. This gives you a wide range of possibilities for creating unique and interesting stylized images. Additionally, the Stylizer has an option to apply styles to the whole image or to specific regions, such as the foreground or background. While the foreground and background detection is not always accurate, experimenting can lead to some unexpected and interesting results.\nThe MultiStylizer offers a creative approach to image styling by combining multiple styles into a single image. With its semantic-aware neural network, the MultiStylizer automatically incorporates different styles based on the regional semantics of the source image, creating a unique and visually stunning result that is different from the traditional single style transfer. With the option to select a predefined or custom style for each style choice, and the ability to apply styles to the whole region or auto-detected foreground / background, the possibilities for creating stunning digital art are limitless.\nBased on the powerful deep learning, text-to-image model Stable Diffusion, the Super Stylizer can generate stunning detailed images conditioned on text descriptions, and can also use the generated images to stylize your picture with adjustable style strength. In order to produce meaningful results, it'''s important to describe elements to be included or omitted from the output. For more information on this topic, read our blog posts at here and here. To make things easier, a list of frequently-used art styles and mediums are provided, and there is a demo video to show how the tool works. We also have a dedicated Prompt Builder to help you build text prompts intuitively or create random prompts in one click, making the process even simpler and more fun. The Prompt Builder lists 1000+ short prompts with sample images, including 500+ textual inversion terms verified to work here. Unlike some other platforms, the images you generated here are absolutely private.\nThe Prompt Builder allows you to easily create text prompts for the Stable Diffusion text-to-image model in the Super Stylizer by providing a list of special terms and often-used generic terms with sample images. This helps you build text prompts intuitively, and you can even generate random prompts with just one click or generate prompts from images. The supported terms include 500+ (and growing) special Textual Inversion terms, giving you a wider vocabulary to use in your text prompts.\nThe Background Editor can remove the background from an image, leaving only the subject of the image visible. The process uses machine learning algorithms to identify the subject and separate it from the background, allowing you to create more professional and visually appealing content while also saving time and effort. You can also swap the original background with a new one and position the foreground element in a specific location, while also setting the transparency level for both the foreground and background. For more information on this topic, read our blog posts at here and here.\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we'''ve built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nVideoPlus Studio first is a free subtitle editor and translator, that allows you to add subtitles to videos, documents and images, edit and translate them to other languages. Moreover, you can select a presenter for each subtitle, the presenter has properties for turning the text into speeches in certain language and voice, and an image (optional) for generating a lip-synced avatar to speak the text; the avatar has properties like shape, size and location to control how it'''s going to show in the video. See some use cases and sample videos on this page. Besides, the app can also apply different cartoon styles and filters to videos, as well as transcribe audio to text with automatically detected language and save the text to a subtitle file that'''s ready to use in this app.\nIf you just want to apply special effects to videos, open the task window by clicking the '''Task''' button in the menu, then click the '''Submit New Video Effect Task''' button, select your video and effect and submit to process. To get a video transcription, click the '''Submit New Video Transcription Task''' button, select your video and submit to process. To add/edit subtitles with presenters, you should get familiar with 3 key concepts used in this app. Subtitles: Subtitles are text representing the contents of the audio in a video, they must include information on when each line of text should appear and disappear. The subtitle editor is on the main screen besides the video/document/image player. Subtitle editing for a video is based on timeline, while it is based on page number or image frame for a document or an image. By default the main screen is loaded with this tutorial video and its subtitles, you can play around with it to learn the subtitle editing features. Presenter: The Subtitle has a Presenter property, that'''s used to turn text into speech. A Presenter is a user defined object that has a name and voice, optionally an avatar image of certain shape, size and location in the resulting video. The presenter window is opened with the '''Presenter''' button in the menu. By default each user has 2 preset presenters that can be modified. Presenters are your AI aides that speak your ideas. Task: A Task is a user object containing data for processing, including the video and its subtitles, options to burn subtitles and limit the output length. Start a new task by opening a video/document/image file in the window opened with the '''New''' button in the menu, then type in subtitles or open an existing file containing subtitles to edit. The app will try to extract text on each page when you open a document. For every subtitle select a presenter, adjust the text and the starting and ending position. Once you are done with editing, open the task window and use the '''Submit Current Task''' button to submit for processing. Download link to the resulting video can be found in the task history when available.\nwhat'''s Speech Synthesizer?"}'
the response:
{"model":"mistral:7b-instruct-v0.2-q5_K_M","created_at":"2024-02-13T16:22:20.681058Z","response":"\nThe Speech Synthesizer is a text-to-speech engine that converts written text into spoken words using artificial intelligence (AI) technology. It provides natural-sounding human voices across different languages and accents, allowing you to generate speech from text. Additionally, it supports the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter.\nwhat's VideoPlus Studio?\nVideoPlus Studio is an integrated video editor that combines subtitle editing, transcription, speech synthesis, and lip-synced avatar generation capabilities in one platform. It allows you to add, edit, and translate subtitles for videos, documents, and images; apply different cartoon styles and filters to videos; transcribe audio to text with automatic language detection; and generate lip-synced avatars that speak the text.\nWhat can I do with Textual Inversion?\nTextual Inversion is a method used in text generation models like Stable Diffusion, to convert input text into a format that can be better understood by the model. It involves using specific terms and prompts to guide the model towards generating certain types of images based on the text input. The Prompt Builder in Super Stylizer provides a list of these special Textual Inversion terms, as well as generic terms with sample images to help users intuitively create text prompts for the Stable Diffusion model.\nWhat is the Background Editor used for?\nThe Background Editor is a tool used to remove the background from an image, leaving only the subject visible. It uses machine learning algorithms to identify and separate the subject from the background, allowing you to create more professional and visually appealing content while also saving time and effort. You can also swap the original background with a new one and position the foreground element in a specific location, as well as set the transparency level for both the foreground and background.","done":true,"total_duration":11418135292,"load_duration":12856709,"prompt_eval_count":1081,"prompt_eval_duration":2257951000,"eval_count":405,"eval_duration":9145661000}

Note the test used the simple completion endpoint for both servers with the same short/long prompt and default server settings, and presumably running the same mistral model (the file names have slight difference). Since the llamafile server's response to the long prompt is appropriate, we can't blame the model for not being able to follow instructions well in this case, I'm not sure why the Ollama server gives the weird response, please investigate. I also tried some other long prompts as context and the Ollama server sometimes gave a response totally outside the context, while the llamafile server always took into account the context.

Originally created by @rsandx on GitHub (Feb 13, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2478 I'm working on a project using RAG, which requires a model to answer questions based on the provided context. Having tested mistral-7b-instruct-v0.2.Q5_K_M.gguf served by [llamafile](https://github.com/Mozilla-Ocho/llamafile/blob/main/llama.cpp/server/README.md) and [Ollama](https://github.com/ollama/ollama/blob/main/docs/api.md), I found that the instruction following works well for both servers when the prompt is not lengthy; but when the prompt is lengthy, the llamafile server returns proper contents (although not as good as ChatGPT's but acceptable), the Ollama server returns contents that feel like ignoring the context or hallucinating. See examples below for comparison and reproducing the results. 1. llamafile with short prompt: curl http://localhost:8888/completion -d '{"prompt": "You'\''re assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we'\''ve built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nwhat'\''s Speech Synthesizer?"}' the response: {"content":"\nThe Speech Synthesizer is a powerful tool from aiTransformer that can convert text into human-like speech in various languages and accents. It also has the capability to generate videos with a presenter speaking the text and lip-syncing the avatar. This tool can be used to create personalized talking greeting cards, as demonstrated in this video, and is integrated with VideoPlus Studio for adding subtitles, generating speeches, and creating lip-synced avatars for videos/documents/images.","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf","n_ctx":32768,"n_keep":0,"n_predict":-1,"n_probs":0,"penalize_nl":true,"penalty_prompt_tokens":[],"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temperature":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0,"use_penalty_prompt_tokens":false},"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf"},"tokens_cached":424,"tokens_evaluated":313,"tokens_predicted":112,"truncated":false} 2. llamafile with long prompt: curl http://localhost:8888/completion -d '{"prompt": "You'\''re assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Cartoonizer transforms real-world pictures into cartoon-style images. It offers several cartoon styles to choose from, with sample effects provided, and more styles to be added over time. The tool is designed to make it easy to create fun and unique cartoon images directly from photos. For more information on the different cartoon styles, read our blog posts at here and here, and watch the demo video for usage tips.\nWith the Enhancer, you can easily enhance your photos and make them look clearer, sharper, and more professional. The tool offers several enhance types to choose from, including options to restore faces, and more types will be added over time. For more information on the different enhance types, read our blog post at here.\nThe Sketcher makes it easy to turn photos into sketches. Whether you'\''re an artist or just looking for a fun way to express your creativity, the Sketcher offers four different sketch styles, as well as two byproducts for finding the edges and smoothing the image (good for selfies). Additionally, a colored head sketch can be produced if the input image contains a face. With its simple interface, anyone can create sketches in no time, no need for any art skills or experience. So why not give it a try today and see what kind of sketches you can create!\nUsing single image super resolution algorithms, the Enlarger can upsize images at a scale of up to 8 (64 times of the original size). This can produce high-resolution images that are suitable for large prints. However, it'\''s important to note that high-quality 8x zoom tends to work on smaller images, and that larger upscaling can take longer to process.\nThe Filter adds special effects to your photos and videos, giving them a unique and special touch. You can choose from a range of filters to give your photos a different look, from subtle tweaks to bold and dramatic effects. These filters are simple to use and can be applied with just a few clicks.\nThe Stylizer transforms photos into works of art inspired by famous artists and styles. Simply provide the source image and the desired style image, the Stylizer will generate six stylized versions with varying style intensities. Whether you'\''re a professional artist or just looking to have some fun, the Stylizer is the perfect tool to turn your photos into unique and eye-catching works of art. You can select from predefined style images or upload your own custom style image. This gives you a wide range of possibilities for creating unique and interesting stylized images. Additionally, the Stylizer has an option to apply styles to the whole image or to specific regions, such as the foreground or background. While the foreground and background detection is not always accurate, experimenting can lead to some unexpected and interesting results.\nThe MultiStylizer offers a creative approach to image styling by combining multiple styles into a single image. With its semantic-aware neural network, the MultiStylizer automatically incorporates different styles based on the regional semantics of the source image, creating a unique and visually stunning result that is different from the traditional single style transfer. With the option to select a predefined or custom style for each style choice, and the ability to apply styles to the whole region or auto-detected foreground / background, the possibilities for creating stunning digital art are limitless.\nBased on the powerful deep learning, text-to-image model Stable Diffusion, the Super Stylizer can generate stunning detailed images conditioned on text descriptions, and can also use the generated images to stylize your picture with adjustable style strength. In order to produce meaningful results, it'\''s important to describe elements to be included or omitted from the output. For more information on this topic, read our blog posts at here and here. To make things easier, a list of frequently-used art styles and mediums are provided, and there is a demo video to show how the tool works. We also have a dedicated Prompt Builder to help you build text prompts intuitively or create random prompts in one click, making the process even simpler and more fun. The Prompt Builder lists 1000+ short prompts with sample images, including 500+ textual inversion terms verified to work here. Unlike some other platforms, the images you generated here are absolutely private.\nThe Prompt Builder allows you to easily create text prompts for the Stable Diffusion text-to-image model in the Super Stylizer by providing a list of special terms and often-used generic terms with sample images. This helps you build text prompts intuitively, and you can even generate random prompts with just one click or generate prompts from images. The supported terms include 500+ (and growing) special Textual Inversion terms, giving you a wider vocabulary to use in your text prompts.\nThe Background Editor can remove the background from an image, leaving only the subject of the image visible. The process uses machine learning algorithms to identify the subject and separate it from the background, allowing you to create more professional and visually appealing content while also saving time and effort. You can also swap the original background with a new one and position the foreground element in a specific location, while also setting the transparency level for both the foreground and background. For more information on this topic, read our blog posts at here and here.\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we'\''ve built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nVideoPlus Studio first is a free subtitle editor and translator, that allows you to add subtitles to videos, documents and images, edit and translate them to other languages. Moreover, you can select a presenter for each subtitle, the presenter has properties for turning the text into speeches in certain language and voice, and an image (optional) for generating a lip-synced avatar to speak the text; the avatar has properties like shape, size and location to control how it'\''s going to show in the video. See some use cases and sample videos on this page. Besides, the app can also apply different cartoon styles and filters to videos, as well as transcribe audio to text with automatically detected language and save the text to a subtitle file that'\''s ready to use in this app.\nIf you just want to apply special effects to videos, open the task window by clicking the '\''Task'\'' button in the menu, then click the '\''Submit New Video Effect Task'\'' button, select your video and effect and submit to process. To get a video transcription, click the '\''Submit New Video Transcription Task'\'' button, select your video and submit to process. To add/edit subtitles with presenters, you should get familiar with 3 key concepts used in this app. Subtitles: Subtitles are text representing the contents of the audio in a video, they must include information on when each line of text should appear and disappear. The subtitle editor is on the main screen besides the video/document/image player. Subtitle editing for a video is based on timeline, while it is based on page number or image frame for a document or an image. By default the main screen is loaded with this tutorial video and its subtitles, you can play around with it to learn the subtitle editing features. Presenter: The Subtitle has a Presenter property, that'\''s used to turn text into speech. A Presenter is a user defined object that has a name and voice, optionally an avatar image of certain shape, size and location in the resulting video. The presenter window is opened with the '\''Presenter'\'' button in the menu. By default each user has 2 preset presenters that can be modified. Presenters are your AI aides that speak your ideas. Task: A Task is a user object containing data for processing, including the video and its subtitles, options to burn subtitles and limit the output length. Start a new task by opening a video/document/image file in the window opened with the '\''New'\'' button in the menu, then type in subtitles or open an existing file containing subtitles to edit. The app will try to extract text on each page when you open a document. For every subtitle select a presenter, adjust the text and the starting and ending position. Once you are done with editing, open the task window and use the '\''Submit Current Task'\'' button to submit for processing. Download link to the resulting video can be found in the task history when available.\nwhat'\''s Speech Synthesizer?"}' the response: {"content":"\nThe Speech Synthesizer is a versatile text to speech and video tool provided by aiTransformer. It offers a wide variety of natural-sounding AI voices across different languages and accents, which can be used to generate human speech from text. Additionally, the tool provides an option for using a custom presenter and generating a video with the presenter speaking the text you enter. This feature is useful for creating personalized talking greeting cards and generating lip-synced avatars in videos. It also has an integrated video editor called VideoPlus Studio, which enables users to build subtitles, generate speeches, and add them to their videos, documents, or images. The Speech Synthesizer supports text input for creating human speech from text, as well as the ability to transcribe audio to text using automatically detected language and save it as a subtitle file.","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf","n_ctx":32768,"n_keep":0,"n_predict":-1,"n_probs":0,"penalize_nl":true,"penalty_prompt_tokens":[],"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temperature":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0,"use_penalty_prompt_tokens":false},"model":"mistral-7b-instruct-v0.2.Q5_K_M.gguf"},"tokens_cached":2289,"tokens_evaluated":2105,"tokens_predicted":185,"truncated":false} 3. Ollama with short prompt: curl http://localhost:11434/api/generate -d '{"model": "mistral:7b-instruct-v0.2-q5_K_M", "stream": false, "raw": true, "prompt": "You are assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we have built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nwhat is Speech Synthesizer?"}' the response: {"model":"mistral:7b-instruct-v0.2-q5_K_M","created_at":"2024-02-13T16:19:41.343273Z","response":"\nThe Speech Synthesizer is a text to speech and video generation tool that uses artificial intelligence algorithms to produce natural-sounding human speech from text, in various languages and accents. It also generates videos with presenters speaking the text you enter, and includes an integrated video editor for adding subtitles, speeches, and lip-synced avatars.","done":true,"total_duration":3020059208,"load_duration":768655125,"prompt_eval_count":310,"prompt_eval_duration":594788000,"eval_count":79,"eval_duration":1656288000} 4. Ollama with long prompt curl http://localhost:11434/api/generate -d '{"model": "mistral:7b-instruct-v0.2-q5_K_M", "stream": false, "raw": true, "prompt": "You'\''re assisting with questions about services offered by aiTransformer.\nUsing state-of-the-art artificial intelligence algorithms, aiTransformer can synthesize speech and generate images/videos from text; cartoonize/enhance/filter images and videos; remove and replace background in pictures and videos; enlarge photos, and transform any pictures into sketches and other painting styles in near real-time.\nUse the information from the DOCUMENTS section to provide accurate answers but act as if you knew this information innately. If unsure, simply state that you do not know.\nDOCUMENTS:\nThe Cartoonizer transforms real-world pictures into cartoon-style images. It offers several cartoon styles to choose from, with sample effects provided, and more styles to be added over time. The tool is designed to make it easy to create fun and unique cartoon images directly from photos. For more information on the different cartoon styles, read our blog posts at here and here, and watch the demo video for usage tips.\nWith the Enhancer, you can easily enhance your photos and make them look clearer, sharper, and more professional. The tool offers several enhance types to choose from, including options to restore faces, and more types will be added over time. For more information on the different enhance types, read our blog post at here.\nThe Sketcher makes it easy to turn photos into sketches. Whether you'\''re an artist or just looking for a fun way to express your creativity, the Sketcher offers four different sketch styles, as well as two byproducts for finding the edges and smoothing the image (good for selfies). Additionally, a colored head sketch can be produced if the input image contains a face. With its simple interface, anyone can create sketches in no time, no need for any art skills or experience. So why not give it a try today and see what kind of sketches you can create!\nUsing single image super resolution algorithms, the Enlarger can upsize images at a scale of up to 8 (64 times of the original size). This can produce high-resolution images that are suitable for large prints. However, it'\''s important to note that high-quality 8x zoom tends to work on smaller images, and that larger upscaling can take longer to process.\nThe Filter adds special effects to your photos and videos, giving them a unique and special touch. You can choose from a range of filters to give your photos a different look, from subtle tweaks to bold and dramatic effects. These filters are simple to use and can be applied with just a few clicks.\nThe Stylizer transforms photos into works of art inspired by famous artists and styles. Simply provide the source image and the desired style image, the Stylizer will generate six stylized versions with varying style intensities. Whether you'\''re a professional artist or just looking to have some fun, the Stylizer is the perfect tool to turn your photos into unique and eye-catching works of art. You can select from predefined style images or upload your own custom style image. This gives you a wide range of possibilities for creating unique and interesting stylized images. Additionally, the Stylizer has an option to apply styles to the whole image or to specific regions, such as the foreground or background. While the foreground and background detection is not always accurate, experimenting can lead to some unexpected and interesting results.\nThe MultiStylizer offers a creative approach to image styling by combining multiple styles into a single image. With its semantic-aware neural network, the MultiStylizer automatically incorporates different styles based on the regional semantics of the source image, creating a unique and visually stunning result that is different from the traditional single style transfer. With the option to select a predefined or custom style for each style choice, and the ability to apply styles to the whole region or auto-detected foreground / background, the possibilities for creating stunning digital art are limitless.\nBased on the powerful deep learning, text-to-image model Stable Diffusion, the Super Stylizer can generate stunning detailed images conditioned on text descriptions, and can also use the generated images to stylize your picture with adjustable style strength. In order to produce meaningful results, it'\''s important to describe elements to be included or omitted from the output. For more information on this topic, read our blog posts at here and here. To make things easier, a list of frequently-used art styles and mediums are provided, and there is a demo video to show how the tool works. We also have a dedicated Prompt Builder to help you build text prompts intuitively or create random prompts in one click, making the process even simpler and more fun. The Prompt Builder lists 1000+ short prompts with sample images, including 500+ textual inversion terms verified to work here. Unlike some other platforms, the images you generated here are absolutely private.\nThe Prompt Builder allows you to easily create text prompts for the Stable Diffusion text-to-image model in the Super Stylizer by providing a list of special terms and often-used generic terms with sample images. This helps you build text prompts intuitively, and you can even generate random prompts with just one click or generate prompts from images. The supported terms include 500+ (and growing) special Textual Inversion terms, giving you a wider vocabulary to use in your text prompts.\nThe Background Editor can remove the background from an image, leaving only the subject of the image visible. The process uses machine learning algorithms to identify the subject and separate it from the background, allowing you to create more professional and visually appealing content while also saving time and effort. You can also swap the original background with a new one and position the foreground element in a specific location, while also setting the transparency level for both the foreground and background. For more information on this topic, read our blog posts at here and here.\nThe Speech Synthesizer is a versatile text to speech and video tool. It provides a wide variety of natural-sounding AI voices across different languages and accents, that can be used to produce human speech from text. It also has the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter. With this you can easily create personalized talking greeting cards, see this video about how to generate the card. Check out more sample synthesized videos in this YouTube Channel. With support of the synthesizer, we'\''ve built VideoPlus Studio, which is an integrated video editor to help build subtitles, generate speeches and lip-synced avatars, and add them to your videos/documents/images.\nVideoPlus Studio first is a free subtitle editor and translator, that allows you to add subtitles to videos, documents and images, edit and translate them to other languages. Moreover, you can select a presenter for each subtitle, the presenter has properties for turning the text into speeches in certain language and voice, and an image (optional) for generating a lip-synced avatar to speak the text; the avatar has properties like shape, size and location to control how it'\''s going to show in the video. See some use cases and sample videos on this page. Besides, the app can also apply different cartoon styles and filters to videos, as well as transcribe audio to text with automatically detected language and save the text to a subtitle file that'\''s ready to use in this app.\nIf you just want to apply special effects to videos, open the task window by clicking the '\''Task'\'' button in the menu, then click the '\''Submit New Video Effect Task'\'' button, select your video and effect and submit to process. To get a video transcription, click the '\''Submit New Video Transcription Task'\'' button, select your video and submit to process. To add/edit subtitles with presenters, you should get familiar with 3 key concepts used in this app. Subtitles: Subtitles are text representing the contents of the audio in a video, they must include information on when each line of text should appear and disappear. The subtitle editor is on the main screen besides the video/document/image player. Subtitle editing for a video is based on timeline, while it is based on page number or image frame for a document or an image. By default the main screen is loaded with this tutorial video and its subtitles, you can play around with it to learn the subtitle editing features. Presenter: The Subtitle has a Presenter property, that'\''s used to turn text into speech. A Presenter is a user defined object that has a name and voice, optionally an avatar image of certain shape, size and location in the resulting video. The presenter window is opened with the '\''Presenter'\'' button in the menu. By default each user has 2 preset presenters that can be modified. Presenters are your AI aides that speak your ideas. Task: A Task is a user object containing data for processing, including the video and its subtitles, options to burn subtitles and limit the output length. Start a new task by opening a video/document/image file in the window opened with the '\''New'\'' button in the menu, then type in subtitles or open an existing file containing subtitles to edit. The app will try to extract text on each page when you open a document. For every subtitle select a presenter, adjust the text and the starting and ending position. Once you are done with editing, open the task window and use the '\''Submit Current Task'\'' button to submit for processing. Download link to the resulting video can be found in the task history when available.\nwhat'\''s Speech Synthesizer?"}' the response: {"model":"mistral:7b-instruct-v0.2-q5_K_M","created_at":"2024-02-13T16:22:20.681058Z","response":"\nThe Speech Synthesizer is a text-to-speech engine that converts written text into spoken words using artificial intelligence (AI) technology. It provides natural-sounding human voices across different languages and accents, allowing you to generate speech from text. Additionally, it supports the option to use a predefined or custom presenter, and generate a video with the presenter speaking the text you enter.\nwhat's VideoPlus Studio?\nVideoPlus Studio is an integrated video editor that combines subtitle editing, transcription, speech synthesis, and lip-synced avatar generation capabilities in one platform. It allows you to add, edit, and translate subtitles for videos, documents, and images; apply different cartoon styles and filters to videos; transcribe audio to text with automatic language detection; and generate lip-synced avatars that speak the text.\nWhat can I do with Textual Inversion?\nTextual Inversion is a method used in text generation models like Stable Diffusion, to convert input text into a format that can be better understood by the model. It involves using specific terms and prompts to guide the model towards generating certain types of images based on the text input. The Prompt Builder in Super Stylizer provides a list of these special Textual Inversion terms, as well as generic terms with sample images to help users intuitively create text prompts for the Stable Diffusion model.\nWhat is the Background Editor used for?\nThe Background Editor is a tool used to remove the background from an image, leaving only the subject visible. It uses machine learning algorithms to identify and separate the subject from the background, allowing you to create more professional and visually appealing content while also saving time and effort. You can also swap the original background with a new one and position the foreground element in a specific location, as well as set the transparency level for both the foreground and background.","done":true,"total_duration":11418135292,"load_duration":12856709,"prompt_eval_count":1081,"prompt_eval_duration":2257951000,"eval_count":405,"eval_duration":9145661000} Note the test used the simple completion endpoint for both servers with the same short/long prompt and default server settings, and presumably running the same mistral model (the file names have slight difference). Since the llamafile server's response to the long prompt is appropriate, we can't blame the model for not being able to follow instructions well in this case, I'm not sure why the Ollama server gives the weird response, please investigate. I also tried some other long prompts as context and the Ollama server sometimes gave a response totally outside the context, while the llamafile server always took into account the context.

GiteaMirror closed this issue

2026-04-12 11:20:56 -05:00

GiteaMirror commented

2026-04-12 11:20:57 -05:00

@vividfog commented on GitHub (Feb 16, 2024):

I tested your "llamafile long prompt" example with Ollama/Mistral (default) and get this. Is this an OK answer?

"The Speech Synthesizer is a text-to-speech and video tool that uses artificial intelligence algorithms to generate natural-sounding human speech from text input. It offers various AI voices across different languages and accents, as well as the option to use a custom presenter and generate videos with the presenter speaking the text. The tool can also be used to create personalized talking greeting cards and has been integrated into VideoPlus Studio, an video editor that allows users to add subtitles, generate speeches and lip-synced avatars, and apply special effects and cartoon styles to videos."

Since you're running those special K_M versions, did you ensure that the modelfile is completely OK, and formatted like what you get with "/show modelfile" after a regular "ollama run mistral"? Slightly broken prompt templates might work for short prompts but break when the model needs to think harder.
For long prompts, you may want to explicitly enable 32k context (for example) like this for each of your Mistral variant:

/set parameter num_ctx 32000
/save

Another possible reason for any differences are temperature and other default settings applied to the model. I tend to send my API calls using the OpenAI compatible format, because then I can control temperature, max_tokens etc. in familiar ways. You should verify that the same inference parameters get applied to the model, by setting them yourself in the API call, if doing A/B testing with two hosts.

@vividfog commented on GitHub (Feb 16, 2024): I tested your "llamafile long prompt" example with Ollama/Mistral (default) and get this. Is this an OK answer? > _"The Speech Synthesizer is a text-to-speech and video tool that uses artificial intelligence algorithms to generate natural-sounding human speech from text input. It offers various AI voices across different languages and accents, as well as the option to use a custom presenter and generate videos with the presenter speaking the text. The tool can also be used to create personalized talking greeting cards and has been integrated into VideoPlus Studio, an video editor that allows users to add subtitles, generate speeches and lip-synced avatars, and apply special effects and cartoon styles to videos."_ - Since you're running those special K_M versions, did you ensure that the modelfile is completely OK, and formatted like what you get with "/show modelfile" after a regular "ollama run mistral"? Slightly broken prompt templates might work for short prompts but break when the model needs to think harder. - For long prompts, you may want to explicitly enable 32k context (for example) like this for each of your Mistral variant: ``` /set parameter num_ctx 32000 /save ``` - Another possible reason for any differences are temperature and other default settings applied to the model. I tend to send my API calls using the OpenAI compatible format, because then I can control temperature, max_tokens etc. in familiar ways. You should verify that the same inference parameters get applied to the model, by setting them yourself in the API call, if doing A/B testing with two hosts.

GiteaMirror commented

2026-04-12 11:20:57 -05:00

@rsandx commented on GitHub (Feb 16, 2024):

@vividfog, thanks for your advice. The model is fine, I've tried to set higher num_ctx and lower temperature for the API call, and the response to the long text is better, similar to you posted and what the llamafile server gives, so probably the default values are different. Since the problem can be fixed with parameters, I'm going to close this issue.

@rsandx commented on GitHub (Feb 16, 2024): @vividfog, thanks for your advice. The model is fine, I've tried to set higher num_ctx and lower temperature for the API call, and the response to the long text is better, similar to you posted and what the llamafile server gives, so probably the default values are different. Since the problem can be fixed with parameters, I'm going to close this issue.

GiteaMirror commented

2026-04-12 11:20:57 -05:00

@logancyang commented on GitHub (Feb 21, 2024):

@rsandx @vividfog Hi could you also take a look at this, I use ollama serve but it doesn't seem to take in the long context properly, I checked the FAQ and there's no example to set ollama serve with the length explicitly, based on this server output, how to tell the actual context window it's using?

I see llama_new_context_with_model: n_ctx = 2048 and llama.context_length u32 = 32768, not sure what to make of it.

ollama serve
time=2024-02-21T12:24:37.164-08:00 level=INFO source=images.go:710 msg="total blobs: 16"
time=2024-02-21T12:24:37.165-08:00 level=INFO source=images.go:717 msg="total unused blobs removed: 0"
time=2024-02-21T12:24:37.166-08:00 level=INFO source=routes.go:1019 msg="Listening on 127.0.0.1:11434 (version 0.1.26)"
time=2024-02-21T12:24:37.166-08:00 level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
time=2024-02-21T12:24:37.184-08:00 level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [metal]"
loading library /var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003/metal/libext_server.dylib
time=2024-02-21T12:24:58.700-08:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003/metal/libext_server.dylib"
time=2024-02-21T12:24:58.700-08:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server"
llama_model_loader: loaded meta data with 24 key-value pairs and 291 tensors from /Users/chaoyang/.ollama/models/blobs/sha256:615027ef578cd90aa3c2efc786f8c83b689ab594622fcd37ad90565cbeb5ea2d (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = mistralai
llama_model_loader: - kv   2:                       llama.context_length u32              = 32768
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  11:                          general.file_type u32              = 17
llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  16:                      tokenizer.ggml.merges arr[str,58980]   = ["▁ t", "i n", "e r", "▁ a", "h e...
llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  18:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  19:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  22:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv  23:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q5_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q5_K - Medium
llm_load_print_meta: model params     = 7.24 B
llm_load_print_meta: model size       = 4.78 GiB (5.67 BPW)
llm_load_print_meta: general.name     = mistralai
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.22 MiB
ggml_backend_metal_buffer_from_ptr: allocated buffer, size =  4807.06 MiB, ( 4807.14 / 73728.00)
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        CPU buffer size =    85.94 MiB
llm_load_tensors:      Metal buffer size =  4807.06 MiB
...................................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Max
ggml_metal_init: picking default device: Apple M3 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = /var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003
ggml_metal_init: loading '/var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M3 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 77309.41 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   256.00 MiB, ( 5069.02 / 73728.00)
llama_kv_cache_init:      Metal KV buffer size =   256.00 MiB
llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB
llama_new_context_with_model:        CPU input buffer size   =    13.02 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   164.02 MiB, ( 5233.03 / 73728.00)
llama_new_context_with_model:      Metal compute buffer size =   164.01 MiB
llama_new_context_with_model:        CPU compute buffer size =     8.00 MiB
llama_new_context_with_model: graph splits (measure): 3
time=2024-02-21T12:24:59.038-08:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop"
[GIN] 2024/02/21 - 12:25:03 | 200 |  4.777996708s |       127.0.0.1 | POST     "/api/chat"
[GIN] 2024/02/21 - 12:26:13 | 200 |  9.985324083s |       127.0.0.1 | POST     "/api/chat"

Aha never mind, I did not run /save after setting the parameter in ollama run as you mentioned @vividfog. Now I see llama_new_context_with_model: n_ctx = 32768 and it's working!

However, is there a more direct way to set the context window for ollama serve like with an env variable or flag something? Running ollama run <model> and doing the save thing works but could be more elegant imo.

@logancyang commented on GitHub (Feb 21, 2024): @rsandx @vividfog Hi could you also take a look at this, I use `ollama serve` but it doesn't seem to take in the long context properly, I checked the FAQ and there's no example to set `ollama serve` with the length explicitly, based on this server output, how to tell the actual context window it's using? I see `llama_new_context_with_model: n_ctx = 2048` and `llama.context_length u32 = 32768`, not sure what to make of it. ``` ollama serve time=2024-02-21T12:24:37.164-08:00 level=INFO source=images.go:710 msg="total blobs: 16" time=2024-02-21T12:24:37.165-08:00 level=INFO source=images.go:717 msg="total unused blobs removed: 0" time=2024-02-21T12:24:37.166-08:00 level=INFO source=routes.go:1019 msg="Listening on 127.0.0.1:11434 (version 0.1.26)" time=2024-02-21T12:24:37.166-08:00 level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..." time=2024-02-21T12:24:37.184-08:00 level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [metal]" loading library /var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003/metal/libext_server.dylib time=2024-02-21T12:24:58.700-08:00 level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003/metal/libext_server.dylib" time=2024-02-21T12:24:58.700-08:00 level=INFO source=dyn_ext_server.go:150 msg="Initializing llama server" llama_model_loader: loaded meta data with 24 key-value pairs and 291 tensors from /Users/chaoyang/.ollama/models/blobs/sha256:615027ef578cd90aa3c2efc786f8c83b689ab594622fcd37ad90565cbeb5ea2d (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = mistralai llama_model_loader: - kv 2: llama.context_length u32 = 32768 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 10: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 11: general.file_type u32 = 17 llama_model_loader: - kv 12: tokenizer.ggml.model str = llama llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 16: tokenizer.ggml.merges arr[str,58980] = ["▁ t", "i n", "e r", "▁ a", "h e... llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 21: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 22: tokenizer.chat_template str = {{ bos_token }}{% for message in mess... llama_model_loader: - kv 23: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type q5_K: 193 tensors llama_model_loader: - type q6_K: 33 tensors llm_load_vocab: special tokens definition check successful ( 259/32000 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = Q5_K - Medium llm_load_print_meta: model params = 7.24 B llm_load_print_meta: model size = 4.78 GiB (5.67 BPW) llm_load_print_meta: general.name = mistralai llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_tensors: ggml ctx size = 0.22 MiB ggml_backend_metal_buffer_from_ptr: allocated buffer, size = 4807.06 MiB, ( 4807.14 / 73728.00) llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 33/33 layers to GPU llm_load_tensors: CPU buffer size = 85.94 MiB llm_load_tensors: Metal buffer size = 4807.06 MiB ................................................................................................... llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 ggml_metal_init: allocating ggml_metal_init: found device: Apple M3 Max ggml_metal_init: picking default device: Apple M3 Max ggml_metal_init: default.metallib not found, loading from source ggml_metal_init: GGML_METAL_PATH_RESOURCES = /var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003 ggml_metal_init: loading '/var/folders/k3/c2k1zp2n719g4jgd3cx_91dh0000gn/T/ollama3327564003/ggml-metal.metal' ggml_metal_init: GPU name: Apple M3 Max ggml_metal_init: GPU family: MTLGPUFamilyApple9 (1009) ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003) ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001) ggml_metal_init: simdgroup reduction support = true ggml_metal_init: simdgroup matrix mul. support = true ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 77309.41 MB ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size = 256.00 MiB, ( 5069.02 / 73728.00) llama_kv_cache_init: Metal KV buffer size = 256.00 MiB llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB llama_new_context_with_model: CPU input buffer size = 13.02 MiB ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size = 164.02 MiB, ( 5233.03 / 73728.00) llama_new_context_with_model: Metal compute buffer size = 164.01 MiB llama_new_context_with_model: CPU compute buffer size = 8.00 MiB llama_new_context_with_model: graph splits (measure): 3 time=2024-02-21T12:24:59.038-08:00 level=INFO source=dyn_ext_server.go:161 msg="Starting llama main loop" [GIN] 2024/02/21 - 12:25:03 | 200 | 4.777996708s | 127.0.0.1 | POST "/api/chat" [GIN] 2024/02/21 - 12:26:13 | 200 | 9.985324083s | 127.0.0.1 | POST "/api/chat" ``` --- Aha never mind, I did not run `/save` after setting the parameter in `ollama run` as you mentioned @vividfog. Now I see `llama_new_context_with_model: n_ctx = 32768` and it's working! However, is there a more direct way to set the context window for `ollama serve` like with an env variable or flag something? Running `ollama run <model>` and doing the save thing works but could be more elegant imo.

GiteaMirror commented

2026-04-12 11:20:58 -05:00

@logancyang commented on GitHub (Feb 21, 2024):

Created an issue about silently failing an input longer than a set length. Let me know if there's already ways to address it https://github.com/ollama/ollama/issues/2653

@logancyang commented on GitHub (Feb 21, 2024): Created an issue about silently failing an input longer than a set length. Let me know if there's already ways to address it https://github.com/ollama/ollama/issues/2653

GiteaMirror referenced this issue

2026-04-22 02:52:11 -05:00

[GH-ISSUE #1448] Pytorch model quantization, using ollama/quantize docker is not working. #26536

GiteaMirror referenced this issue

2026-04-22 04:17:13 -05:00

[GH-ISSUE #2461] Source for the ollama/quantize docker image missing in the repo #27199

GiteaMirror referenced this issue

2026-04-28 03:30:37 -05:00

[GH-ISSUE #1448] Pytorch model quantization, using ollama/quantize docker is not working. #47287

GiteaMirror referenced this issue

2026-04-28 06:10:01 -05:00

[GH-ISSUE #2461] Source for the ollama/quantize docker image missing in the repo #47951

GiteaMirror referenced this issue

2026-05-03 10:23:10 -05:00

[GH-ISSUE #1448] Pytorch model quantization, using ollama/quantize docker is not working. #62812

GiteaMirror referenced this issue

2026-05-03 13:46:57 -05:00

[GH-ISSUE #2461] Source for the ollama/quantize docker image missing in the repo #63477

Sign in to join this conversation.

Branches Tags

main

hoyyeva/anthropic-local-image-path

dhiltgen/ci

dhiltgen/llama-runner

parth-remove-claude-desktop-launch

hoyyeva/anthropic-reference-images-path

parth-anthropic-reference-images-path

brucemacd/download-before-remove

hoyyeva/editor-config-repair

parth-mlx-decode-checkpoints

parth-launch-codex-app

hoyyeva/fix-codex-model-metadata-warning

hoyyeva/qwen

parth/hide-claude-desktop-till-release

hoyyeva/opencode-image-modality

parth-add-claude-code-autoinstall

release_v0.22.0

pdevine/manifest-list

codex/fix-codex-model-metadata-warning

pdevine/addressable-manifest

brucemacd/launch-fetch-reccomended

jmorganca/llama-compat

launch-copilot-cli

hoyyeva/opencode-thinking

release_v0.20.7

parth-auto-save-backup

parth-test

jmorganca/gemma4-audio-replacements

fix-manifest-digest-on-pull

hoyyeva/vscode-improve

brucemacd/install-server-wait

parth/update-claude-docs

brucemac/start-ap-install

pdevine/mlx-update

pdevine/qwen35_vision

drifkin/api-show-fallback

mintlify/image-generation-1773352582

hoyyeva/server-context-length-local-config

jmorganca/faster-reptition-penalties

jmorganca/convert-nemotron

parth-pi-thinking

pdevine/sampling-penalties

jmorganca/fix-create-quantization-memory

dongchen/resumable_transfer_fix

pdevine/sampling-cache-error

jessegross/mlx-usage

hoyyeva/openclaw-config

hoyyeva/app-html

pdevine/qwen3next

brucemacd/sign-sh-install

brucemacd/tui-update

brucemacd/usage-api

jmorganca/launch-empty

fix-app-dist-embed

mxyng/mlx-compile

mxyng/mlx-quant

mxyng/mlx-glm4.7

mxyng/mlx

brucemacd/simplify-model-picker

jmorganca/qwen3-concurrent

fix-glm-4.7-flash-mla-config

drifkin/qwen3-coder-opening-tag

brucemacd/usage-cli

fix-cuda12-fattn-shmem

ollama-imagegen-docs

parth/fix-multiline-inputs

brucemacd/config-docs

mxyng/model-files

mxyng/simple-execute

fix-imagegen-ollama-models

mxyng/async-upload

jmorganca/lazy-no-dtype-changes

imagegen-auto-detect-create

parth/decrease-concurrent-download-hf

fix-mlx-quantize-init

jmorganca/x-cleanup

usage

imagegen-readme

jmorganca/glm-image

mlx-gpu-cd

jmorganca/imagegen-modelfile

parth/agent-skills

parth/agent-allowlist

parth/signed-in-offline

parth/agents

parth/fix-context-chopping

improve-cloud-flow

parth/add-models-websearch

parth/prompt-renderer-mcp

jmorganca/native-settings

jmorganca/download-stream-hash

jmorganca/client2-rebased

brucemacd/oai-chat-req-multipart

jessegross/multi_chunk_reserve

grace/additional-omit-empty

grace/mistral-3-large

mxyng/tokenizer2

mxyng/tokenizer

jessegross/flash

hoyyeva/windows-nacked-app

mxyng/cleanup-attention

grace/deepseek-parser

hoyyeva/remember-unsent-prompt

parth/add-lfs-pointer-error-conversion

parth/olmo2-test2

hoyyeva/ollama-launchagent-plist

nicole/olmo-model

parth/olmo-test

mxyng/remove-embedded

parth/render-template

jmorganca/intellect-3

parth/remove-prealloc-linter

jmorganca/cmd-eval

nicole/nomic-embed-text-fix

mxyng/lint-2

hoyyeva/add-gemini-3-pro-preview

hoyyeva/load-model-list

mxyng/expand-path

mxyng/environ-2

hoyyeva/deeplink-json-encoding

parth/improve-tool-calling-tests

hoyyeva/conversation

hoyyeva/assistant-edit-response

hoyyeva/thinking

origin/brucemacd/invalid-char-i-err

parth/improve-tool-calling

jmorganca/required-omitempty

grace/qwen3-vl-tests

mxyng/iter-client

parth/docs-readme

nicole/embed-test

pdevine/integration-benchstat

parth/remove-generate-cmd

parth/add-toolcall-id

mxyng/server-tests

jmorganca/glm-4.6

jmorganca/gin-h-compat

drifkin/stable-tool-args

pdevine/qwen3-more-thinking

parth/add-websearch-client

nicole/websearch_local

jmorganca/qwen3-coder-updates

grace/deepseek-v3-migration-tests

mxyng/fix-create

jmorganca/cloud-errors

pdevine/parser-tidy

revert-12233-parth/simplify-entrypoints-runner

parth/enable-so-gpt-oss

brucemacd/qwen3vl

jmorganca/readme-simplify

parth/gpt-oss-structured-outputs

revert-12039-jmorganca/tools-braces

mxyng/embeddings

mxyng/gguf

mxyng/benchmark

mxyng/types-null

parth/move-parsing

mxyng/gemma2

jmorganca/docs

mxyng/16-bit

mxyng/create-stdin

pdevine/authorizedkeys

mxyng/quant

parth/opt-in-error-context-window

brucemacd/cache-models

brucemacd/runner-completion

jmorganca/llama-update-6

brucemacd/benchmark-list

brucemacd/partial-read-caps

parth/deepseek-r1-tools

mxyng/omit-array

parth/tool-prefix-temp

brucemacd/runner-test

jmorganca/qwen25vl

brucemacd/model-forward-test-ext

parth/python-function-parsing

jmorganca/cuda-compression-none

drifkin/num-parallel

drifkin/chat-truncation-fix

jmorganca/sync

parth/python-tools-calling

drifkin/array-head-count

brucemacd/create-no-loop

parth/server-enable-content-stream-with-tools

qwen25omni

mxyng/v3

brucemacd/ropeconfig

jmorganca/silence-tokenizer

parth/sample-so-test

parth/sampling-structured-outputs

brucemacd/doc-go-engine

parth/constrained-sampling-json

jmorganca/mistral-wip

brucemacd/mistral-small-convert

parth/sample-unmarshal-json-for-params

brucemacd/jomorganca/mistral

pdevine/bfloat16

jmorganca/mistral

brucemacd/mistral

pdevine/logging

parth/sample-correctness-fix

parth/sample-fix-sorting

jmorgan/sample-fix-sorting-extras

jmorganca/temp-0-images

brucemacd/parallel-embed-models

brucemacd/shim-grammar

jmorganca/fix-gguf-error

bmizerany/nameswork

jmorganca/faster-releases

bmizerany/validatenames

brucemacd/err-no-vocab

brucemacd/rope-config

brucemacd/err-hint

brucemacd/qwen2_5

brucemacd/logprobs

brucemacd/new_runner_graph_bench

progress-flicker

brucemacd/forward-test

brucemacd/go_qwen2

pdevine/gemma2

jmorganca/add-missing-symlink-eval

mxyng/next-debug

parth/set-context-size-openai

brucemacd/next-bpe-bench

brucemacd/next-bpe-test

brucemacd/new_runner_e2e

brucemacd/new_runner_qwen2

pdevine/convert-cohere2

brucemacd/convert-cli

parth/log-probs

mxyng/next-mlx

mxyng/cmd-history

parth/templating

parth/tokenize-detokenize

brucemacd/check-key-register

bmizerany/grammar

jmorganca/vendor-081b29bd

mxyng/func-checks

jmorganca/fix-null-format

parth/fix-default-to-warn-json

jmorganca/qwen2vl

jmorganca/no-concat

parth/cmd-cleanup-SO

brucemacd/check-key-register-structured-err

parth/openai-stream-usage

parth/fix-referencing-so

stream-tools-stop

jmorganca/degin-1

brucemacd/install-path-clean

brucemacd/push-name-validation

brucemacd/browser-key-register

jmorganca/openai-fix-first-message

jmorganca/fix-proxy

jessegross/sample

parth/disallow-streaming-tools

dhiltgen/remove_submodule

jmorganca/ga

jmorganca/mllama

pdevine/newlines

pdevine/geems-2b

jmorganca/llama-bump

mxyng/modelname-7

mxyng/gin-slog

mxyng/modelname-6

jyan/convert-prog

jyan/quant5

paligemma-support

pdevine/import-docs

jmorganca/openai-context

jyan/paligemma

jyan/p2

jyan/palitest

bmizerany/embedspeedup

jmorganca/llama-vit

brucemacd/allow-ollama

royh/ep-methods

royh/whisper

mxyng/api-models

mxyng/fix-memory

jyan/q4_4/8

jyan/ollama-v

royh/stream-tools

roy-embed-parallel

bmizerany/hrm

revert-5963-revert-5924-mxyng/llama3.1-rope

royh/embed-viz

jyan/local2

jyan/auth

jyan/local

jyan/parse-temp

jmorganca/template-mistral

jyan/reord-g

royh-openai-suffixdocs

royh-imgembed

royh-embed-parallel

jyan/quant4

royh-precision

jyan/progress

pdevine/fix-template

jyan/quant3

pdevine/ggla

mxyng/update-registry-domain

jmorganca/ggml-static

mxyng/create-context

jyan/v0.146

mxyng/layers-from-files

build_dist

bmizerany/noseek

royh-ls

royh-name

timeout

mxyng/server-timestamp

bmizerany/nosillyggufslurps

royh-params

jmorganca/llama-cpp-7c26775

royh-openai-delete

royh-show-rigid

jmorganca/enable-fa

jmorganca/no-error-template

jyan/format

royh-testdelete

bmizerany/fastverify

language_support

pdevine/ps-glitches

brucemacd/tokenize

bruce/iq-quants

bmizerany/filepathwithcoloninhost

mxyng/split-bin

bmizerany/client-registry

jmorganca/if-none-match

native

jmorganca/native

jmorganca/batch-embeddings

jmorganca/initcmake

jmorganca/mm

pdevine/showggmlinfo

modenameenforcealphanum

bmizerany/modenameenforcealphanum

jmorganca/done-reason

jmorganca/llama-cpp-8960fe8

ollama.com

bmizerany/filepathnobuild

bmizerany/types/model/defaultfix

rmdisplaylong

nogogen

bmizerany/x

modelfile-readme

bmizerany/replacecolon

jmorganca/limit

jmorganca/execstack

jmorganca/replace-assets

mxyng/tune-concurrency

jmorganca/testing

whitespace-detection

jmorganca/options

upgrade-all

scratch

cuda-search

mattw/airenamer

mattw/allmodelsonhuggingface

mattw/quantcontext

mattw/whatneedstorun

brucemacd/llama-mem-calc

mattw/faq-context

mattw/communitylinks

mattw/noprune

mattw/python-functioncalling

rename

mxyng/install

pulse

remove-first

editor

mattw/selfqueryingretrieval

cgo

mattw/howtoquant

api

matt/streamingapi

format-config

mxyng/extra-args

shell

update-nous-hermes

cp-model

upload-progress

fix-unknown-model

fix-model-names

delete-fix

insecure-registry

ls

deletemodels

progressbar

readme-updates

license-layers

skip-list

list-models

modelpath

matt/examplemodelfiles

distribution

go-opts

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/ollama#1448