github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6007] Qwen2 tool calling support #50268

New Issue
Closed
opened 2026-04-28 14:54:14 -05:00 by GiteaMirror · 31 comments
GiteaMirror commented 2026-04-28 14:54:14 -05:00
Owner
Copy Link

Originally created by @jiandandema on GitHub (Jul 27, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6007

Originally assigned to: @jmorganca on GitHub.

Please support the API tool calling capability of Qwen2. By the way, if I want to implement the API tool calling capability of Qwen2 in Ollma myself, can I make it? What shall I do?
thank you!!!

Originally created by @jiandandema on GitHub (Jul 27, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6007 Originally assigned to: @jmorganca on GitHub. Please support the API tool calling capability of Qwen2. By the way, if I want to implement the API tool calling capability of Qwen2 in Ollma myself, can I make it? What shall I do? thank you!!!
GiteaMirror added the feature request label 2026-04-28 14:54:14 -05:00
GiteaMirror commented 2026-04-28 14:54:16 -05:00
Author
Owner
Copy Link

@mmmmm21 commented on GitHub (Jul 27, 2024):

I also want to know that .
I downloaded Qwen/Qwen2-7B-Instruct , which supports tools but cannot be used in Ollama

<!-- gh-comment-id:2253876722 --> @mmmmm21 commented on GitHub (Jul 27, 2024): I also want to know that . I downloaded Qwen/Qwen2-7B-Instruct , which supports tools but cannot be used in Ollama
GiteaMirror commented 2026-04-28 14:54:17 -05:00
Author
Owner
Copy Link

@jiandandema commented on GitHub (Jul 27, 2024):

I also want to know that . I downloaded Qwen/Qwen2-7B-Instruct , which supports tools but cannot be used in Ollama

glm4 also do not work. Maybe ollama only support llama3.1, mistral for now.

<!-- gh-comment-id:2254086667 --> @jiandandema commented on GitHub (Jul 27, 2024): > I also want to know that . I downloaded Qwen/Qwen2-7B-Instruct , which supports tools but cannot be used in Ollama glm4 also do not work. Maybe ollama only support llama3.1, mistral for now.
GiteaMirror commented 2026-04-28 14:54:18 -05:00
Author
Owner
Copy Link

@JianxinMa commented on GitHub (Jul 27, 2024):

FYI, this is the reference implementation from Qwen:

Example usages:

I'm looking into how to port Qwen's implementation to Ollama as well. Unfortunately, Ollama is written in Go, not Python. I'm not familiar with Go. So the progress is slow. If the upcoming Qwen 2.5 supports the tool-calling prompt used by LLaMA 3.1, things might be easier.

cc @JustinLin610

<!-- gh-comment-id:2254091074 --> @JianxinMa commented on GitHub (Jul 27, 2024): FYI, this is the reference implementation from Qwen: - https://github.com/QwenLM/Qwen-Agent/blob/main/qwen_agent/llm/function_calling.py#L381 Example usages: - https://github.com/QwenLM/Qwen-Agent/blob/main/examples/function_calling.py - https://github.com/QwenLM/Qwen-Agent/blob/main/examples/function_calling_in_parallel.py I'm looking into how to port Qwen's implementation to Ollama as well. Unfortunately, Ollama is written in Go, not Python. I'm not familiar with Go. So the progress is slow. If the upcoming Qwen 2.5 supports the tool-calling prompt used by LLaMA 3.1, things might be easier. cc @JustinLin610
GiteaMirror commented 2026-04-28 14:54:19 -05:00
Author
Owner
Copy Link

@jiandandema commented on GitHub (Jul 27, 2024):

FYI, this is the reference implementation from Qwen:

Example usages:

I'm looking into how to port Qwen's implementation to Ollama as well. Unfortunately, Ollama is written in Go, not Python. I'm not familiar with Go. So the progress is slow. If the upcoming Qwen 2.5 supports the tool-calling prompt used by LLaMA 3.1, things might be easier.

cc @JustinLin610

Thank you for your information! I have been searching for ways to use API tools calling for a long time, such as: fastchat, vllm. But they do not support glm4, Qwen2, etc. In my observation, many methods in Langchain require API tools calling, such as llm.bind_tools(), llm.with_structured_output(), etc. So if there is a middleware that can implement API tool calling for Chinese open source llm in the future, that would be excellent.

<!-- gh-comment-id:2254095407 --> @jiandandema commented on GitHub (Jul 27, 2024): > FYI, this is the reference implementation from Qwen: > > * https://github.com/QwenLM/Qwen-Agent/blob/main/qwen_agent/llm/function_calling.py#L381 > > Example usages: > > * https://github.com/QwenLM/Qwen-Agent/blob/main/examples/function_calling.py > * https://github.com/QwenLM/Qwen-Agent/blob/main/examples/function_calling_in_parallel.py > > I'm looking into how to port Qwen's implementation to Ollama as well. Unfortunately, Ollama is written in Go, not Python. I'm not familiar with Go. So the progress is slow. If the upcoming Qwen 2.5 supports the tool-calling prompt used by LLaMA 3.1, things might be easier. > > cc @JustinLin610 Thank you for your information! I have been searching for ways to use API tools calling for a long time, such as: fastchat, vllm. But they do not support glm4, Qwen2, etc. In my observation, many methods in Langchain require API tools calling, such as llm.bind_tools(), llm.with_structured_output(), etc. So if there is a middleware that can implement API tool calling for Chinese open source llm in the future, that would be excellent.
GiteaMirror commented 2026-04-28 14:54:21 -05:00
Author
Owner
Copy Link

@jiandandema commented on GitHub (Jul 29, 2024):

I have read the source code of Ollma, but since I am not very familiar with Go, I am not sure if my understanding is correct.

  1. Receiving a POST request from this line of code:
r.POST("/v1/chat/completions", openai.ChatMiddleware(), s.ChatHandler)
  1. s.GeneratedHandler will determine whether the model has tool capability in following code:
r, m, opts, err := s.scheduleRunner(c.Request.Context(), req.Model, caps, req.Options, req.KeepAlive)
  1. s.scheduleRunner function first reads the corresponding prompt template file under /usr/share/ollama/.ollama/models/Blobs/, which code is:
        mp := ParseModelPath(name)
	manifest, digest, err := GetManifest(mp)
	if err != nil {
		return nil, err
	}

	model := &Model{
		Name:      mp.GetFullTagname(),
		ShortName: mp.GetShortTagname(),
		Digest:    digest,
		Template:  template.DefaultTemplate,
	}

	filename, err := GetBlobsPath(manifest.Config.Digest)
  1. If there is {{. tool}} in this template file, model.CheckCapabilities(caps...) can pass(in s.scheduleRunner), otherwise it will report an error
  2. Assembling template files as prompt:
prompt, images, err := chatPrompt(c.Request.Context(), m, r.Tokenize, opts, req.Messages, req.Tools)
  1. llm responds based on prompt(Tools information only appear in prompt)
r.Completion(c.Request.Context(), llm.CompletionRequest{
			Prompt:  prompt,
			Images:  images,
			Format:  req.Format,
			Options: opts,
		}
<!-- gh-comment-id:2254873081 --> @jiandandema commented on GitHub (Jul 29, 2024): I have read the source code of Ollma, but since I am not very familiar with Go, I am not sure if my understanding is correct. 1. Receiving a POST request from this line of code: ```go r.POST("/v1/chat/completions", openai.ChatMiddleware(), s.ChatHandler) ``` 2. `s.GeneratedHandler` will determine whether the model has tool capability in following code: ```go r, m, opts, err := s.scheduleRunner(c.Request.Context(), req.Model, caps, req.Options, req.KeepAlive) ``` 3. `s.scheduleRunner` function first reads the corresponding prompt template file under `/usr/share/ollama/.ollama/models/Blobs/`, which code is: ```go mp := ParseModelPath(name) manifest, digest, err := GetManifest(mp) if err != nil { return nil, err } model := &Model{ Name: mp.GetFullTagname(), ShortName: mp.GetShortTagname(), Digest: digest, Template: template.DefaultTemplate, } filename, err := GetBlobsPath(manifest.Config.Digest) ``` 4. If there is {{. tool}} in this template file, `model.CheckCapabilities(caps...)` can pass(in `s.scheduleRunner`), otherwise it will report an error 5. Assembling template files as prompt: ```go prompt, images, err := chatPrompt(c.Request.Context(), m, r.Tokenize, opts, req.Messages, req.Tools) ``` 6. llm responds based on prompt(Tools information only appear in prompt) ```go r.Completion(c.Request.Context(), llm.CompletionRequest{ Prompt: prompt, Images: images, Format: req.Format, Options: opts, } ```
GiteaMirror commented 2026-04-28 14:54:24 -05:00
Author
Owner
Copy Link

@jiandandema commented on GitHub (Jul 29, 2024):

I have read the source code of Ollma, but since I am not very familiar with Go, I am not sure if my understanding is correct.

  1. Receiving a POST request from this line of code:
r.POST("/v1/chat/completions", openai.ChatMiddleware(), s.ChatHandler)
  1. s.GeneratedHandler will determine whether the model has tool capability in following code:
r, m, opts, err := s.scheduleRunner(c.Request.Context(), req.Model, caps, req.Options, req.KeepAlive)
  1. s.scheduleRunner function first reads the corresponding prompt template file under /usr/share/ollama/.ollama/models/Blobs/, which code is:
        mp := ParseModelPath(name)
	manifest, digest, err := GetManifest(mp)
	if err != nil {
		return nil, err
	}

	model := &Model{
		Name:      mp.GetFullTagname(),
		ShortName: mp.GetShortTagname(),
		Digest:    digest,
		Template:  template.DefaultTemplate,
	}

	filename, err := GetBlobsPath(manifest.Config.Digest)
  1. If there is {{. tool}} in this template file, model.CheckCapabilities(caps...) can pass(in s.scheduleRunner), otherwise it will report an error
  2. Assembling template files as prompt:
prompt, images, err := chatPrompt(c.Request.Context(), m, r.Tokenize, opts, req.Messages, req.Tools)
  1. llm responds based on prompt(Tools information only appear in prompt)
r.Completion(c.Request.Context(), llm.CompletionRequest{
			Prompt:  prompt,
			Images:  images,
			Format:  req.Format,
			Options: opts,
		}

So, Is it possible for Qwen2 to implement API tool calling as long as I adapt the prompt template in /usr/share/ollama/.ollama/models/Blobs/?

<!-- gh-comment-id:2254875497 --> @jiandandema commented on GitHub (Jul 29, 2024): > I have read the source code of Ollma, but since I am not very familiar with Go, I am not sure if my understanding is correct. > > 1. Receiving a POST request from this line of code: > > ```go > r.POST("/v1/chat/completions", openai.ChatMiddleware(), s.ChatHandler) > ``` > > 2. `s.GeneratedHandler` will determine whether the model has tool capability in following code: > > ```go > r, m, opts, err := s.scheduleRunner(c.Request.Context(), req.Model, caps, req.Options, req.KeepAlive) > ``` > > 3. `s.scheduleRunner` function first reads the corresponding prompt template file under `/usr/share/ollama/.ollama/models/Blobs/`, which code is: > > ```go > mp := ParseModelPath(name) > manifest, digest, err := GetManifest(mp) > if err != nil { > return nil, err > } > > model := &Model{ > Name: mp.GetFullTagname(), > ShortName: mp.GetShortTagname(), > Digest: digest, > Template: template.DefaultTemplate, > } > > filename, err := GetBlobsPath(manifest.Config.Digest) > ``` > > 4. If there is {{. tool}} in this template file, `model.CheckCapabilities(caps...)` can pass(in `s.scheduleRunner`), otherwise it will report an error > 5. Assembling template files as prompt: > > ```go > prompt, images, err := chatPrompt(c.Request.Context(), m, r.Tokenize, opts, req.Messages, req.Tools) > ``` > > 6. llm responds based on prompt(Tools information only appear in prompt) > > ```go > r.Completion(c.Request.Context(), llm.CompletionRequest{ > Prompt: prompt, > Images: images, > Format: req.Format, > Options: opts, > } > ``` So, Is it possible for Qwen2 to implement API tool calling as long as I adapt the prompt template in /usr/share/ollama/.ollama/models/Blobs/?
GiteaMirror commented 2026-04-28 14:54:27 -05:00
Author
Owner
Copy Link

@JianxinMa commented on GitHub (Jul 30, 2024):

I just made a tool call template for qwen2:7b: https://ollama.com/majx13/test

However, the latest install.sh installs Ollama version 0.3.0 for me (which is weird), not 0.3.1 that includes the tool call functionality. Therefore, I haven't tested if the template is correct. I'm just sharing this and hope it helps.

I'll test it later once I have 0.3.1 installed.

<!-- gh-comment-id:2257841869 --> @JianxinMa commented on GitHub (Jul 30, 2024): I just made a tool call template for `qwen2:7b`: https://ollama.com/majx13/test However, the latest install.sh installs Ollama version 0.3.0 for me (which is weird), not 0.3.1 that includes the tool call functionality. Therefore, I haven't tested if the template is correct. I'm just sharing this and hope it helps. I'll test it later once I have 0.3.1 installed.
GiteaMirror commented 2026-04-28 14:54:29 -05:00
Author
Owner
Copy Link

@jiandandema commented on GitHub (Jul 31, 2024):

I just made a tool call template for qwen2:7b: https://ollama.com/majx13/test

However, the latest install.sh installs Ollama version 0.3.0 for me (which is weird), not 0.3.1 that includes the tool call functionality. Therefore, I haven't tested if the template is correct. I'm just sharing this and hope it helps.

I'll test it later once I have 0.3.1 installed.

@JianxinMa Firstly, thank you for your attention and work. I downloaded the test model you provided, made a request using curl and received the following response. I feel like it works.
微信截图_20240731140832
However, further experiments and adaptation may be necessary.
This is the content field of response:

<tool_call>
{"name": "get_current_weather", "arguments": {"location": "Paris", "format": "celsius"}}
<tool_call>
<!-- gh-comment-id:2259751644 --> @jiandandema commented on GitHub (Jul 31, 2024): > I just made a tool call template for `qwen2:7b`: https://ollama.com/majx13/test > > However, the latest install.sh installs Ollama version 0.3.0 for me (which is weird), not 0.3.1 that includes the tool call functionality. Therefore, I haven't tested if the template is correct. I'm just sharing this and hope it helps. > > I'll test it later once I have 0.3.1 installed. @JianxinMa Firstly, thank you for your attention and work. I downloaded the test model you provided, made a request using curl and received the following response. I feel like it works. ![微信截图_20240731140832](https://github.com/user-attachments/assets/dd5d2bcf-a1e2-4168-a4df-e37f2ce52735) However, further experiments and adaptation may be necessary. This is the content field of response: ```json <tool_call> {"name": "get_current_weather", "arguments": {"location": "Paris", "format": "celsius"}} <tool_call> ```
GiteaMirror commented 2026-04-28 14:54:30 -05:00
Author
Owner
Copy Link

@JianxinMa commented on GitHub (Aug 1, 2024):

@jiandandema Hi, I have just updated the template at https://ollama.com/majx13/test Please make sure to ollama pull majx13/test. It should work now. Tested with ollama 0.3.2.

The script for testing:

from openai import OpenAI
import json

model = "majx13/test"
client = OpenAI(
    base_url="http://localhost:11434/v1/",
    # required but ignored
    api_key="ollama",
)


# Example dummy function hard coded to return the same weather
# In production, this could be your backend API or an external API
def get_current_weather(location, unit="fahrenheit"):
    """Get the current weather in a given location"""
    if "tokyo" in location.lower():
        return json.dumps({"location": "Tokyo", "temperature": "10", "unit": unit})
    elif "san francisco" in location.lower():
        return json.dumps(
            {"location": "San Francisco", "temperature": "72", "unit": unit}
        )
    elif "paris" in location.lower():
        return json.dumps({"location": "Paris", "temperature": "22", "unit": unit})
    else:
        return json.dumps({"location": location, "temperature": "unknown"})


def run_conversation():
    # Step 1: send the conversation and available functions to the model
    messages = [
        {
            "role": "user",
            "content": "What's the weather like in San Francisco, Tokyo, and Paris?",
        }
    ]
    tools = [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        },
                        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                    },
                    "required": ["location"],
                },
            },
        }
    ]
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        tools=tools,
    )
    print(response)
    response_message = response.choices[0].message
    tool_calls = response_message.tool_calls
    # Step 2: check if the model wanted to call a function
    if tool_calls:
        # Step 3: call the function
        # Note: the JSON response may not always be valid; be sure to handle errors
        available_functions = {
            "get_current_weather": get_current_weather,
        }  # only one function in this example, but you can have multiple
        messages.append(response_message)  # extend conversation with assistant's reply
        # Step 4: send the info for each function call and function response to the model
        for tool_call in tool_calls:
            function_name = tool_call.function.name
            function_to_call = available_functions[function_name]
            function_args = json.loads(tool_call.function.arguments)
            print(f'Calling Function {function_name} with Argument {function_args}...')
            function_response = function_to_call(
                location=function_args.get("location"),
                unit=function_args.get("unit"),
            )
            messages.append(
                {
                    "tool_call_id": tool_call.id,
                    "role": "tool",
                    "name": function_name,
                    "content": function_response,
                }
            )  # extend conversation with function response
        second_response = client.chat.completions.create(
            model=model,
            messages=messages,
            tools=tools,
        )  # get a new response from the model where it can see the function response
        print(second_response)


run_conversation()

which should output:

ChatCompletion(id='chatcmpl-697', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='', role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_kknhr1o0', function=Function(arguments='{"location":"San Francisco, CA"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_8cqnavtc', function=Function(arguments='{"location":"Tokyo","unit":"celsius"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_9jzm14a3', function=Function(arguments='{"location":"Paris, FR"}', name='get_current_weather'), type='function')]))], created=1722493532, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=89, prompt_tokens=200, total_tokens=289))
Calling Function get_current_weather with Argument {'location': 'San Francisco, CA'}...
Calling Function get_current_weather with Argument {'location': 'Tokyo', 'unit': 'celsius'}...
Calling Function get_current_weather with Argument {'location': 'Paris, FR'}...
ChatCompletion(id='chatcmpl-175', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="The current weather in San Francisco is 72 degrees. In Tokyo, the temperature is 10 degrees Celsius. And in Paris, it's around 22 degrees.", role='assistant', function_call=None, tool_calls=None))], created=1722493548, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=37, prompt_tokens=275, total_tokens=312))

I will try to contact the maintainer of Qwen2 to use this template if it works for you as well.

<!-- gh-comment-id:2262154658 --> @JianxinMa commented on GitHub (Aug 1, 2024): @jiandandema Hi, I have just updated the template at https://ollama.com/majx13/test Please make sure to `ollama pull majx13/test`. It should work now. Tested with ollama 0.3.2. The script for testing: ```py from openai import OpenAI import json model = "majx13/test" client = OpenAI( base_url="http://localhost:11434/v1/", # required but ignored api_key="ollama", ) # Example dummy function hard coded to return the same weather # In production, this could be your backend API or an external API def get_current_weather(location, unit="fahrenheit"): """Get the current weather in a given location""" if "tokyo" in location.lower(): return json.dumps({"location": "Tokyo", "temperature": "10", "unit": unit}) elif "san francisco" in location.lower(): return json.dumps( {"location": "San Francisco", "temperature": "72", "unit": unit} ) elif "paris" in location.lower(): return json.dumps({"location": "Paris", "temperature": "22", "unit": unit}) else: return json.dumps({"location": location, "temperature": "unknown"}) def run_conversation(): # Step 1: send the conversation and available functions to the model messages = [ { "role": "user", "content": "What's the weather like in San Francisco, Tokyo, and Paris?", } ] tools = [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model=model, messages=messages, tools=tools, ) print(response) response_message = response.choices[0].message tool_calls = response_message.tool_calls # Step 2: check if the model wanted to call a function if tool_calls: # Step 3: call the function # Note: the JSON response may not always be valid; be sure to handle errors available_functions = { "get_current_weather": get_current_weather, } # only one function in this example, but you can have multiple messages.append(response_message) # extend conversation with assistant's reply # Step 4: send the info for each function call and function response to the model for tool_call in tool_calls: function_name = tool_call.function.name function_to_call = available_functions[function_name] function_args = json.loads(tool_call.function.arguments) print(f'Calling Function {function_name} with Argument {function_args}...') function_response = function_to_call( location=function_args.get("location"), unit=function_args.get("unit"), ) messages.append( { "tool_call_id": tool_call.id, "role": "tool", "name": function_name, "content": function_response, } ) # extend conversation with function response second_response = client.chat.completions.create( model=model, messages=messages, tools=tools, ) # get a new response from the model where it can see the function response print(second_response) run_conversation() ``` which should output: ``` ChatCompletion(id='chatcmpl-697', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='', role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_kknhr1o0', function=Function(arguments='{"location":"San Francisco, CA"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_8cqnavtc', function=Function(arguments='{"location":"Tokyo","unit":"celsius"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_9jzm14a3', function=Function(arguments='{"location":"Paris, FR"}', name='get_current_weather'), type='function')]))], created=1722493532, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=89, prompt_tokens=200, total_tokens=289)) Calling Function get_current_weather with Argument {'location': 'San Francisco, CA'}... Calling Function get_current_weather with Argument {'location': 'Tokyo', 'unit': 'celsius'}... Calling Function get_current_weather with Argument {'location': 'Paris, FR'}... ChatCompletion(id='chatcmpl-175', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="The current weather in San Francisco is 72 degrees. In Tokyo, the temperature is 10 degrees Celsius. And in Paris, it's around 22 degrees.", role='assistant', function_call=None, tool_calls=None))], created=1722493548, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=37, prompt_tokens=275, total_tokens=312)) ``` I will try to contact the maintainer of Qwen2 to use this template if it works for you as well.
GiteaMirror commented 2026-04-28 14:54:32 -05:00
Author
Owner
Copy Link

@jiandandema commented on GitHub (Aug 3, 2024):

@jiandandema Hi, I have just updated the template at https://ollama.com/majx13/test Please make sure to ollama pull majx13/test. It should work now. Tested with ollama 0.3.2.

The script for testing:

from openai import OpenAI
import json

model = "majx13/test"
client = OpenAI(
    base_url="http://localhost:11434/v1/",
    # required but ignored
    api_key="ollama",
)


# Example dummy function hard coded to return the same weather
# In production, this could be your backend API or an external API
def get_current_weather(location, unit="fahrenheit"):
    """Get the current weather in a given location"""
    if "tokyo" in location.lower():
        return json.dumps({"location": "Tokyo", "temperature": "10", "unit": unit})
    elif "san francisco" in location.lower():
        return json.dumps(
            {"location": "San Francisco", "temperature": "72", "unit": unit}
        )
    elif "paris" in location.lower():
        return json.dumps({"location": "Paris", "temperature": "22", "unit": unit})
    else:
        return json.dumps({"location": location, "temperature": "unknown"})


def run_conversation():
    # Step 1: send the conversation and available functions to the model
    messages = [
        {
            "role": "user",
            "content": "What's the weather like in San Francisco, Tokyo, and Paris?",
        }
    ]
    tools = [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        },
                        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                    },
                    "required": ["location"],
                },
            },
        }
    ]
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        tools=tools,
    )
    print(response)
    response_message = response.choices[0].message
    tool_calls = response_message.tool_calls
    # Step 2: check if the model wanted to call a function
    if tool_calls:
        # Step 3: call the function
        # Note: the JSON response may not always be valid; be sure to handle errors
        available_functions = {
            "get_current_weather": get_current_weather,
        }  # only one function in this example, but you can have multiple
        messages.append(response_message)  # extend conversation with assistant's reply
        # Step 4: send the info for each function call and function response to the model
        for tool_call in tool_calls:
            function_name = tool_call.function.name
            function_to_call = available_functions[function_name]
            function_args = json.loads(tool_call.function.arguments)
            print(f'Calling Function {function_name} with Argument {function_args}...')
            function_response = function_to_call(
                location=function_args.get("location"),
                unit=function_args.get("unit"),
            )
            messages.append(
                {
                    "tool_call_id": tool_call.id,
                    "role": "tool",
                    "name": function_name,
                    "content": function_response,
                }
            )  # extend conversation with function response
        second_response = client.chat.completions.create(
            model=model,
            messages=messages,
            tools=tools,
        )  # get a new response from the model where it can see the function response
        print(second_response)


run_conversation()

which should output:

ChatCompletion(id='chatcmpl-697', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='', role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_kknhr1o0', function=Function(arguments='{"location":"San Francisco, CA"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_8cqnavtc', function=Function(arguments='{"location":"Tokyo","unit":"celsius"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_9jzm14a3', function=Function(arguments='{"location":"Paris, FR"}', name='get_current_weather'), type='function')]))], created=1722493532, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=89, prompt_tokens=200, total_tokens=289))
Calling Function get_current_weather with Argument {'location': 'San Francisco, CA'}...
Calling Function get_current_weather with Argument {'location': 'Tokyo', 'unit': 'celsius'}...
Calling Function get_current_weather with Argument {'location': 'Paris, FR'}...
ChatCompletion(id='chatcmpl-175', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="The current weather in San Francisco is 72 degrees. In Tokyo, the temperature is 10 degrees Celsius. And in Paris, it's around 22 degrees.", role='assistant', function_call=None, tool_calls=None))], created=1722493548, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=37, prompt_tokens=275, total_tokens=312))

I will try to contact the maintainer of Qwen2 to use this template if it works for you as well.

@JianxinMa Yes, I received the expected output with ollama 0.3.3! Here is the output:

ChatCompletion(id='chatcmpl-356', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='', role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_73l616ue', function=Function(arguments='{"location":"San Francisco","unit":"celsius"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_6hm997fw', function=Function(arguments='{"location":"Tokyo","unit":"celsius"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_8eofcwc0', function=Function(arguments='{"location":"Paris","unit":"celsius"}', name='get_current_weather'), type='function')]))], created=1722660482, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=99, prompt_tokens=200, total_tokens=299))
Calling Function get_current_weather with Argument {'location': 'San Francisco', 'unit': 'celsius'}...
Calling Function get_current_weather with Argument {'location': 'Tokyo', 'unit': 'celsius'}...
Calling Function get_current_weather with Argument {'location': 'Paris', 'unit': 'celsius'}...
ChatCompletion(id='chatcmpl-152', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="The weather in San Francisco is currently 72 degrees Celsius. In Tokyo, it's 10 degrees Celsius and in Paris, the temperature is 22 degrees Celsius.", role='assistant', function_call=None, tool_calls=None))], created=1722660482, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=37, prompt_tokens=279, total_tokens=316))

Finally, thank you again for your attention and work!

<!-- gh-comment-id:2266367401 --> @jiandandema commented on GitHub (Aug 3, 2024): > @jiandandema Hi, I have just updated the template at https://ollama.com/majx13/test Please make sure to `ollama pull majx13/test`. It should work now. Tested with ollama 0.3.2. > > The script for testing: > > ```python > from openai import OpenAI > import json > > model = "majx13/test" > client = OpenAI( > base_url="http://localhost:11434/v1/", > # required but ignored > api_key="ollama", > ) > > > # Example dummy function hard coded to return the same weather > # In production, this could be your backend API or an external API > def get_current_weather(location, unit="fahrenheit"): > """Get the current weather in a given location""" > if "tokyo" in location.lower(): > return json.dumps({"location": "Tokyo", "temperature": "10", "unit": unit}) > elif "san francisco" in location.lower(): > return json.dumps( > {"location": "San Francisco", "temperature": "72", "unit": unit} > ) > elif "paris" in location.lower(): > return json.dumps({"location": "Paris", "temperature": "22", "unit": unit}) > else: > return json.dumps({"location": location, "temperature": "unknown"}) > > > def run_conversation(): > # Step 1: send the conversation and available functions to the model > messages = [ > { > "role": "user", > "content": "What's the weather like in San Francisco, Tokyo, and Paris?", > } > ] > tools = [ > { > "type": "function", > "function": { > "name": "get_current_weather", > "description": "Get the current weather in a given location", > "parameters": { > "type": "object", > "properties": { > "location": { > "type": "string", > "description": "The city and state, e.g. San Francisco, CA", > }, > "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, > }, > "required": ["location"], > }, > }, > } > ] > response = client.chat.completions.create( > model=model, > messages=messages, > tools=tools, > ) > print(response) > response_message = response.choices[0].message > tool_calls = response_message.tool_calls > # Step 2: check if the model wanted to call a function > if tool_calls: > # Step 3: call the function > # Note: the JSON response may not always be valid; be sure to handle errors > available_functions = { > "get_current_weather": get_current_weather, > } # only one function in this example, but you can have multiple > messages.append(response_message) # extend conversation with assistant's reply > # Step 4: send the info for each function call and function response to the model > for tool_call in tool_calls: > function_name = tool_call.function.name > function_to_call = available_functions[function_name] > function_args = json.loads(tool_call.function.arguments) > print(f'Calling Function {function_name} with Argument {function_args}...') > function_response = function_to_call( > location=function_args.get("location"), > unit=function_args.get("unit"), > ) > messages.append( > { > "tool_call_id": tool_call.id, > "role": "tool", > "name": function_name, > "content": function_response, > } > ) # extend conversation with function response > second_response = client.chat.completions.create( > model=model, > messages=messages, > tools=tools, > ) # get a new response from the model where it can see the function response > print(second_response) > > > run_conversation() > ``` > > which should output: > > ``` > ChatCompletion(id='chatcmpl-697', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='', role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_kknhr1o0', function=Function(arguments='{"location":"San Francisco, CA"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_8cqnavtc', function=Function(arguments='{"location":"Tokyo","unit":"celsius"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_9jzm14a3', function=Function(arguments='{"location":"Paris, FR"}', name='get_current_weather'), type='function')]))], created=1722493532, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=89, prompt_tokens=200, total_tokens=289)) > Calling Function get_current_weather with Argument {'location': 'San Francisco, CA'}... > Calling Function get_current_weather with Argument {'location': 'Tokyo', 'unit': 'celsius'}... > Calling Function get_current_weather with Argument {'location': 'Paris, FR'}... > ChatCompletion(id='chatcmpl-175', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="The current weather in San Francisco is 72 degrees. In Tokyo, the temperature is 10 degrees Celsius. And in Paris, it's around 22 degrees.", role='assistant', function_call=None, tool_calls=None))], created=1722493548, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=37, prompt_tokens=275, total_tokens=312)) > ``` > > I will try to contact the maintainer of Qwen2 to use this template if it works for you as well. @JianxinMa Yes, I received the expected output with ollama 0.3.3! Here is the output: ``` ChatCompletion(id='chatcmpl-356', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='', role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_73l616ue', function=Function(arguments='{"location":"San Francisco","unit":"celsius"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_6hm997fw', function=Function(arguments='{"location":"Tokyo","unit":"celsius"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_8eofcwc0', function=Function(arguments='{"location":"Paris","unit":"celsius"}', name='get_current_weather'), type='function')]))], created=1722660482, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=99, prompt_tokens=200, total_tokens=299)) Calling Function get_current_weather with Argument {'location': 'San Francisco', 'unit': 'celsius'}... Calling Function get_current_weather with Argument {'location': 'Tokyo', 'unit': 'celsius'}... Calling Function get_current_weather with Argument {'location': 'Paris', 'unit': 'celsius'}... ChatCompletion(id='chatcmpl-152', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="The weather in San Francisco is currently 72 degrees Celsius. In Tokyo, it's 10 degrees Celsius and in Paris, the temperature is 22 degrees Celsius.", role='assistant', function_call=None, tool_calls=None))], created=1722660482, model='majx13/test', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=37, prompt_tokens=279, total_tokens=316)) ``` Finally, thank you again for your attention and work!
GiteaMirror commented 2026-04-28 14:54:34 -05:00
Author
Owner
Copy Link

@jiandandema commented on GitHub (Aug 3, 2024):

I also tested the llm.with_structured_output() method in Langchain and was able to obtain the expected results. This is fantastic!
760fb4f4197d02d951877faf2737757
I've been aiming for this result for the past month or two, and I was able to achieve it using OpenAI's API earlier on. I will continue to test other methods in Langchain going forward.
here are the test codes

import getpass
import os
from langchain_openai import ChatOpenAI
from typing import Annotated, List, Tuple, TypedDict
from typing import Optional
from langchain_core.pydantic_v1 import BaseModel, Field

os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_ENDPOINT"] = "https://api.smith.langchain.com"
os.environ["LANGCHAIN_API_KEY"] = "your_langsmith_api_key"

class Joke(BaseModel):
    """Joke to tell user."""

    setup: str = Field(description="The setup of the joke")
    punchline: str = Field(description="The punchline to the joke")
    rating: Optional[int] = Field(description="How funny the joke is, from 1 to 10")


class planer(BaseModel):
    """Based on user question, plan a list of steps"""

    steps: List[str] = Field(description="List of steps")

llm = ChatOpenAI(model="majx13/test",base_url="http://localhost:11434/v1",api_key="1", temperature=0)

structured_llm = llm.with_structured_output(planer)

print(structured_llm.invoke("I want to put the elephant in the fridge."))
<!-- gh-comment-id:2266375502 --> @jiandandema commented on GitHub (Aug 3, 2024): I also tested the `llm.with_structured_output()` method in Langchain and was able to obtain the expected results. This is fantastic! ![760fb4f4197d02d951877faf2737757](https://github.com/user-attachments/assets/a56721a6-17db-4f4f-957c-e182abc3bcf2) I've been aiming for this result for the past month or two, and I was able to achieve it using OpenAI's API earlier on. I will continue to test other methods in Langchain going forward. here are the test codes ```python import getpass import os from langchain_openai import ChatOpenAI from typing import Annotated, List, Tuple, TypedDict from typing import Optional from langchain_core.pydantic_v1 import BaseModel, Field os.environ["LANGCHAIN_TRACING_V2"] = "true" os.environ["LANGCHAIN_ENDPOINT"] = "https://api.smith.langchain.com" os.environ["LANGCHAIN_API_KEY"] = "your_langsmith_api_key" class Joke(BaseModel): """Joke to tell user.""" setup: str = Field(description="The setup of the joke") punchline: str = Field(description="The punchline to the joke") rating: Optional[int] = Field(description="How funny the joke is, from 1 to 10") class planer(BaseModel): """Based on user question, plan a list of steps""" steps: List[str] = Field(description="List of steps") llm = ChatOpenAI(model="majx13/test",base_url="http://localhost:11434/v1",api_key="1", temperature=0) structured_llm = llm.with_structured_output(planer) print(structured_llm.invoke("I want to put the elephant in the fridge.")) ```
GiteaMirror commented 2026-04-28 14:54:36 -05:00
Author
Owner
Copy Link

@zhanghuanhuanlive commented on GitHub (Aug 14, 2024):

Thanks,it works on Ollama0.3.4 with Qwen2:7b.
How can I make it support Qwen2:72b?

<!-- gh-comment-id:2288040748 --> @zhanghuanhuanlive commented on GitHub (Aug 14, 2024): Thanks,it works on Ollama0.3.4 with Qwen2:7b. How can I make it support Qwen2:72b?
GiteaMirror commented 2026-04-28 14:54:38 -05:00
Author
Owner
Copy Link

@jiandandema commented on GitHub (Aug 15, 2024):

Thanks,it works on Ollama0.3.4 with Qwen2:7b. How can I make it support Qwen2:72b?

I think it can be done. I think updating the prompt template is enough. The path in Linux is:/usr/share/ollama/. ollama/models/Blobs/. You can take a detailed look at the files in this directory

<!-- gh-comment-id:2291421778 --> @jiandandema commented on GitHub (Aug 15, 2024): > Thanks,it works on Ollama0.3.4 with Qwen2:7b. How can I make it support Qwen2:72b? I think it can be done. I think updating the prompt template is enough. The path in Linux is:`/usr/share/ollama/. ollama/models/Blobs/`. You can take a detailed look at the files in this directory
GiteaMirror commented 2026-04-28 14:54:39 -05:00
Author
Owner
Copy Link

@codefromthecrypt commented on GitHub (Sep 2, 2024):

@jiandandema I wonder if you add the label "model request", this might become closer to solved? I tried majx13/test as well, and I think main thing is to get the right folks aware of this issue.

<!-- gh-comment-id:2323555665 --> @codefromthecrypt commented on GitHub (Sep 2, 2024): @jiandandema I wonder if you add the label "model request", this might become closer to solved? I tried majx13/test as well, and I think main thing is to get the right folks aware of this issue.
GiteaMirror commented 2026-04-28 14:54:42 -05:00
Author
Owner
Copy Link

@jiandandema commented on GitHub (Sep 2, 2024):

@jiandandema I wonder if you add the label "model request", this might become closer to solved? I tried majx13/test as well, and I think main thing is to get the right folks aware of this issue.

sorry, I don’t know which “model request” label you mean. In LangChain or Ollama?

<!-- gh-comment-id:2323587894 --> @jiandandema commented on GitHub (Sep 2, 2024): > @jiandandema I wonder if you add the label "model request", this might become closer to solved? I tried majx13/test as well, and I think main thing is to get the right folks aware of this issue. sorry, I don’t know which “model request” label you mean. In LangChain or Ollama?
GiteaMirror commented 2026-04-28 14:54:44 -05:00
Author
Owner
Copy Link

@codefromthecrypt commented on GitHub (Sep 2, 2024):

On this issue here in ollama.. I don't know if you have the right to add the label, but maybe try

Screenshot 2024-09-02 at 9 06 48 AM
<!-- gh-comment-id:2323591467 --> @codefromthecrypt commented on GitHub (Sep 2, 2024): On this issue here in ollama.. I don't know if you have the right to add the label, but maybe try <img width="226" alt="Screenshot 2024-09-02 at 9 06 48 AM" src="https://github.com/user-attachments/assets/470990ef-236e-46b3-905a-d9f4d8476943">
GiteaMirror commented 2026-04-28 14:54:44 -05:00
Author
Owner
Copy Link

@codefromthecrypt commented on GitHub (Sep 2, 2024):

For example, you can look at https://github.com/ollama/ollama/issues/6564 which has the label "model request"
Screenshot 2024-09-02 at 9 32 50 AM instead of what this issue has, which is "feature request"
image

By adding the label "model request" to this issue #6007, there is a chance someone may look at it who currently isn't yet.

<!-- gh-comment-id:2323610512 --> @codefromthecrypt commented on GitHub (Sep 2, 2024): For example, you can look at https://github.com/ollama/ollama/issues/6564 which has the label "model request" <img width="186" alt="Screenshot 2024-09-02 at 9 32 50 AM" src="https://github.com/user-attachments/assets/00e0ffb3-b946-4608-95ca-44f523a56866"> instead of what this issue has, which is "feature request" ![image](https://github.com/user-attachments/assets/ec4b3946-850b-4ac3-a866-d9e87d83a7a7) By adding the label "model request" to this issue #6007, there is a chance someone may look at it who currently isn't yet.
GiteaMirror commented 2026-04-28 14:54:46 -05:00
Author
Owner
Copy Link

@jiandandema commented on GitHub (Sep 2, 2024):

For example, you can look at #6564 which has the label "model request" Screenshot 2024-09-02 at 9 32 50 AM instead of what this issue has, which is "feature request" image

By adding the label "model request" to this issue #6007, there is a chance someone may look at it who currently isn't yet.

I'm sorry, I misunderstood what you meant. I'll add that label in a moment. Thank you for your reminding!

<!-- gh-comment-id:2323625013 --> @jiandandema commented on GitHub (Sep 2, 2024): > For example, you can look at #6564 which has the label "model request" <img alt="Screenshot 2024-09-02 at 9 32 50 AM" width="186" src="https://private-user-images.githubusercontent.com/64215/363534365-00e0ffb3-b946-4608-95ca-44f523a56866.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyNDE3NDIsIm5iZiI6MTcyNTI0MTQ0MiwicGF0aCI6Ii82NDIxNS8zNjM1MzQzNjUtMDBlMGZmYjMtYjk0Ni00NjA4LTk1Y2EtNDRmNTIzYTU2ODY2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAyVDAxNDQwMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE3ZDM1NjAxODZmNDMyMmU0OTNhYmU2MWQ4ZDNjNWRlNTIxM2I1ZGJjZTZhZGFmNDdiZjQwZjk4ODg5MmY0ZGImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.izGtznFBpIUvOzyl9adupCcfDdI8hsmpYj-L3pmygIE"> instead of what this issue has, which is "feature request" ![image](https://private-user-images.githubusercontent.com/64215/363534389-ec4b3946-850b-4ac3-a866-d9e87d83a7a7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyNDE3NDIsIm5iZiI6MTcyNTI0MTQ0MiwicGF0aCI6Ii82NDIxNS8zNjM1MzQzODktZWM0YjM5NDYtODUwYi00YWMzLWE4NjYtZDllODdkODNhN2E3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAyVDAxNDQwMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE0ZGE5MDE1YzliNjM3OTFkNmQyZDQ4MDgyYjg5MDNlNjRhOTI4MzE5NzdmODgwODFkYjQ1Zjg2ZmE1NjJkNTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.2S7IsARG0kEigpdNB60-ql_f2WSltONxMu61hS2Zb-4) > > By adding the label "model request" to this issue #6007, there is a chance someone may look at it who currently isn't yet. I'm sorry, I misunderstood what you meant. I'll add that label in a moment. Thank you for your reminding!
GiteaMirror commented 2026-04-28 14:54:47 -05:00
Author
Owner
Copy Link

@donkey566788 commented on GitHub (Sep 3, 2024):

I just made a tool call template for qwen2:7b: https://ollama.com/majx13/test

However, the latest install.sh installs Ollama version 0.3.0 for me (which is weird), not 0.3.1 that includes the tool call functionality. Therefore, I haven't tested if the template is correct. I'm just sharing this and hope it helps.

I'll test it later once I have 0.3.1 installed.

@JianxinMa 在LangGraph上测试可用，感谢，
请问qwen 对齐tool call的正式版什么时候会发布

<!-- gh-comment-id:2325965270 --> @donkey566788 commented on GitHub (Sep 3, 2024): > I just made a tool call template for `qwen2:7b`: https://ollama.com/majx13/test > > However, the latest install.sh installs Ollama version 0.3.0 for me (which is weird), not 0.3.1 that includes the tool call functionality. Therefore, I haven't tested if the template is correct. I'm just sharing this and hope it helps. > > I'll test it later once I have 0.3.1 installed. @JianxinMa 在LangGraph上测试可用，感谢， 请问qwen 对齐tool call的正式版什么时候会发布
GiteaMirror commented 2026-04-28 14:54:48 -05:00
Author
Owner
Copy Link

@JianxinMa commented on GitHub (Sep 3, 2024):

I just made a tool call template for qwen2:7b: https://ollama.com/majx13/test
However, the latest install.sh installs Ollama version 0.3.0 for me (which is weird), not 0.3.1 that includes the tool call functionality. Therefore, I haven't tested if the template is correct. I'm just sharing this and hope it helps.
I'll test it later once I have 0.3.1 installed.

@JianxinMa 在LangGraph上测试可用，感谢， 请问qwen 对齐tool call的正式版什么时候会发布

The Qwen2 repository on Ollama is maintained by the Ollama team and not by the Qwen team. We reached out to them several weeks ago, but we haven't received any updates from Ollama recently.

Our plan is to at least introduce support for tool calls in Qwen2.5, which should be happening soon.

<!-- gh-comment-id:2325997794 --> @JianxinMa commented on GitHub (Sep 3, 2024): > > I just made a tool call template for `qwen2:7b`: https://ollama.com/majx13/test > > However, the latest install.sh installs Ollama version 0.3.0 for me (which is weird), not 0.3.1 that includes the tool call functionality. Therefore, I haven't tested if the template is correct. I'm just sharing this and hope it helps. > > I'll test it later once I have 0.3.1 installed. > > @JianxinMa 在LangGraph上测试可用，感谢， 请问qwen 对齐tool call的正式版什么时候会发布 The Qwen2 repository on Ollama is maintained by the Ollama team and not by the Qwen team. We reached out to them several weeks ago, but we haven't received any updates from Ollama recently. Our plan is to at least introduce support for tool calls in Qwen2.5, which should be happening soon.
GiteaMirror commented 2026-04-28 14:54:53 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Sep 4, 2024):

@JianxinMa thank you! Sorry for delay, we will look at this asap (and in the near future make it easier to collaborate/contribute to this repository)

<!-- gh-comment-id:2327871782 --> @jmorganca commented on GitHub (Sep 4, 2024): @JianxinMa thank you! Sorry for delay, we will look at this asap (and in the near future make it easier to collaborate/contribute to this repository)
GiteaMirror commented 2026-04-28 14:54:57 -05:00
Author
Owner
Copy Link

@lxysl commented on GitHub (Sep 6, 2024):

Is there any plan to introduce the tool-use instruction tuning model of qwen2 into ollama?

<!-- gh-comment-id:2333064271 --> @lxysl commented on GitHub (Sep 6, 2024): Is there any plan to introduce the [tool-use instruction tuning model of qwen2](https://huggingface.co/modelscope/qwen2-7b-agent-instruct) into ollama?
GiteaMirror commented 2026-04-28 14:55:00 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Sep 11, 2024):

Hi folks sorry for the delay. Qwen2 has been updated with the recommended system prompt, as well as the nous-style tool calling from the Qwen-Agent repo https://ollama.com/library/qwen2

<!-- gh-comment-id:2342418239 --> @jmorganca commented on GitHub (Sep 11, 2024): Hi folks sorry for the delay. Qwen2 has been updated with the recommended system prompt, as well as the nous-style tool calling from the Qwen-Agent repo https://ollama.com/library/qwen2
GiteaMirror commented 2026-04-28 14:55:00 -05:00
Author
Owner
Copy Link

@codefromthecrypt commented on GitHub (Sep 11, 2024):

@jmorganca great! I can verify qwen2. I often use qwen2:0.5b and qwen2 qwen2:1.5b in tests as they are a lot smaller. Is there any chance you can update the prompts on the other tags?

<!-- gh-comment-id:2342461060 --> @codefromthecrypt commented on GitHub (Sep 11, 2024): @jmorganca great! I can verify qwen2. I often use qwen2:0.5b and qwen2 qwen2:1.5b in tests as they are a lot smaller. Is there any chance you can update the prompts on the other tags?
GiteaMirror commented 2026-04-28 14:55:01 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (Sep 11, 2024):

@codefromthecrypt I'm not sure qwen2:0.5b and qwen2:1.5b support tool calling. I tried with these models but it didn't seem to produce the intended output.

<!-- gh-comment-id:2342530439 --> @jmorganca commented on GitHub (Sep 11, 2024): @codefromthecrypt I'm not sure qwen2:0.5b and qwen2:1.5b support tool calling. I tried with these models but it didn't seem to produce the intended output.
GiteaMirror commented 2026-04-28 14:55:03 -05:00
Author
Owner
Copy Link

@codefromthecrypt commented on GitHub (Sep 11, 2024):

@jklj077 do you mind verifying the expectations on qwen2:0.5b and qwen2:1.5b vs qwen2 (7b) on this template? https://ollama.com/library/qwen2/blobs/77c91b422cc9

Ack I recall 2.5 will more formally support tools.

<!-- gh-comment-id:2342542728 --> @codefromthecrypt commented on GitHub (Sep 11, 2024): @jklj077 do you mind verifying the expectations on qwen2:0.5b and qwen2:1.5b vs qwen2 (7b) on this template? https://ollama.com/library/qwen2/blobs/77c91b422cc9 Ack I recall 2.5 will more formally support tools.
GiteaMirror commented 2026-04-28 14:55:06 -05:00
Author
Owner
Copy Link

@jklj077 commented on GitHub (Sep 12, 2024):

@codefromthecrypt Both Qwen2-0.5B-Instruct and Qwen2-1.5B-Instruct were indeed trained using function/tool calling data. While the original template might not be entirely feasible with Go, it seems that the new template (nous-style) works largely thanks to the models' ability to follow instructions. However, the smaller models, especially after quantization, don't always follow instructions as accurately. I found that the tags were often missing in the generated text, which I would say is expected.

<!-- gh-comment-id:2346043556 --> @jklj077 commented on GitHub (Sep 12, 2024): @codefromthecrypt Both Qwen2-0.5B-Instruct and Qwen2-1.5B-Instruct were indeed trained using function/tool calling data. While the original template might not be entirely feasible with Go, it seems that the new template (nous-style) works largely thanks to the models' ability to follow instructions. However, the smaller models, especially after quantization, don't always follow instructions as accurately. I found that the tags were often missing in the generated text, which I would say is expected.
GiteaMirror commented 2026-04-28 14:55:07 -05:00
Author
Owner
Copy Link

@michaelneale commented on GitHub (Sep 13, 2024):

@jklj077 this makes sense - I did see with Qwen2 it fail at tool calling syntax at one point - ebem 7B - 72B seemed potentialy better (but a little slow in my current hardware to fully experience) so makes sense.

<!-- gh-comment-id:2347977445 --> @michaelneale commented on GitHub (Sep 13, 2024): @jklj077 this makes sense - I did see with Qwen2 it fail at tool calling syntax at one point - ebem 7B - 72B seemed potentialy better (but a little slow in my current hardware to fully experience) so makes sense.
GiteaMirror commented 2026-04-28 14:55:07 -05:00
Author
Owner
Copy Link

@ivanstepanovftw commented on GitHub (Oct 12, 2024):

Does Qwen have ability to respond with special token <tool_call>?

<!-- gh-comment-id:2408283930 --> @ivanstepanovftw commented on GitHub (Oct 12, 2024): Does Qwen have ability to respond with special token `<tool_call>`?
GiteaMirror commented 2026-04-28 14:55:08 -05:00
Author
Owner
Copy Link

@jklj077 commented on GitHub (Oct 14, 2024):

@ivanstepanovftw technically yes for all models from all gens. There are examples in that pattern/template in training Qwen2.5 instruct models; for other gens, prompt engineering should also work. but since 2.5, <tool_call> is a special token and due to implementation differences, it may or may not be filtered by the framework in the output.

<!-- gh-comment-id:2409937297 --> @jklj077 commented on GitHub (Oct 14, 2024): @ivanstepanovftw technically yes for all models from all gens. There are examples in that pattern/template in training Qwen2.5 instruct models; for other gens, prompt engineering should also work. but since 2.5, <tool_call> is a special token and due to implementation differences, it may or may not be filtered by the framework in the output.
GiteaMirror commented 2026-04-28 14:55:08 -05:00
Author
Owner
Copy Link

@ivanstepanovftw commented on GitHub (Oct 14, 2024):

@jklj077 I have troubles in llama.cpp to get <tool_call> token 😞, though I do not remember if it is filtered somewhere...

<!-- gh-comment-id:2409972279 --> @ivanstepanovftw commented on GitHub (Oct 14, 2024): @jklj077 I have troubles in llama.cpp to get <tool_call> token :disappointed:, though I do not remember if it is filtered somewhere...
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#50268
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?