github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-17 05:36:16 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #6704] ollama model not support tool calling #50731

New Issue
Closed
opened 2026-04-28 16:56:20 -05:00 by GiteaMirror · 10 comments
GiteaMirror commented 2026-04-28 16:56:20 -05:00
Owner
Copy Link

Originally created by @sunshine19870316 on GitHub (Sep 9, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6704

Originally assigned to: @jmorganca on GitHub.

I use ollama model in langgraph multi-agent SupervisorAgent framework,
when I use API llm, that is give actual key and url, it can run successfully, but after changing to ollama server, can't call tools.

my code:
def get_qwen7b():
model = ChatOpenAI(
model_name="qwen2:7b",
openai_api_base="http://localhost:11434/v1",
openai_api_key="none",
streaming=True,
temperature=0.01
)
return model

llm = get_qwen7b()

class SupervisorAgent(BaseAgent):
def init(self, members, llm, **kwargs):
system_prompt = (
"""You are a supervisor tasked with managing a conversation between the
following workers: {members}. Given the following user request,
respond with the worker to act next. Each worker will perform a
task and respond with their results and status. When finished,
respond with FINISH."""
)
# supervisor is an LLM node. It just picks the next agent to process
options = ["FINISH"] + members

    function_def = {
        "name": "route",
        "description": "Select the next role.",
        "parameters": {
            "title": "routeSchema",
            "type": "object",
            "properties": {
                "next": {
                    "title": "Next",
                    "anyOf": [
                        {"enum": options},
                    ],
                }
            },
            "required": ["next"],
        },
    }

    prompt = ChatPromptTemplate.from_messages(
        [
            ("system", system_prompt),
            MessagesPlaceholder(variable_name="messages"),
            ("system",
             "Given the messages above, who should act next? Or should we FINISH? If get the final result, we should FINISH. Select one of: {options}"),
        ]
    ).partial(options=str(options), members=", ".join(members))
    tools = [convert_to_openai_tool(f) for f in [function_def]]
    model = llm.bind(tools=tools)
    self.agent = prompt | model | OutputParser()
Originally created by @sunshine19870316 on GitHub (Sep 9, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6704 Originally assigned to: @jmorganca on GitHub. I use ollama model in langgraph multi-agent SupervisorAgent framework, when I use API llm, that is give actual key and url, it can run successfully, but after changing to ollama server, can't call tools. my code: def get_qwen7b(): model = ChatOpenAI( model_name="qwen2:7b", openai_api_base="http://localhost:11434/v1", openai_api_key="none", streaming=True, temperature=0.01 ) return model llm = get_qwen7b() class SupervisorAgent(BaseAgent): def __init__(self, members, llm, **kwargs): system_prompt = ( """You are a supervisor tasked with managing a conversation between the following workers: {members}. Given the following user request, respond with the worker to act next. Each worker will perform a task and respond with their results and status. When finished, respond with FINISH.""" ) # supervisor is an LLM node. It just picks the next agent to process options = ["FINISH"] + members function_def = { "name": "route", "description": "Select the next role.", "parameters": { "title": "routeSchema", "type": "object", "properties": { "next": { "title": "Next", "anyOf": [ {"enum": options}, ], } }, "required": ["next"], }, } prompt = ChatPromptTemplate.from_messages( [ ("system", system_prompt), MessagesPlaceholder(variable_name="messages"), ("system", "Given the messages above, who should act next? Or should we FINISH? If get the final result, we should FINISH. Select one of: {options}"), ] ).partial(options=str(options), members=", ".join(members)) tools = [convert_to_openai_tool(f) for f in [function_def]] model = llm.bind(tools=tools) self.agent = prompt | model | OutputParser()
GiteaMirror added the feature request label 2026-04-28 16:56:20 -05:00
GiteaMirror commented 2026-04-28 16:56:22 -05:00
Author
Owner
Copy Link

@sunshine19870316 commented on GitHub (Sep 9, 2024):

that is, my agent framework is https://github.com/langchain-ai/langgraph/blob/main/examples/multi_agent/agent_supervisor.ipynb
I want to replace the llm from API to local ollama server. but failed in tool calling.

<!-- gh-comment-id:2337235066 --> @sunshine19870316 commented on GitHub (Sep 9, 2024): that is, my agent framework is https://github.com/langchain-ai/langgraph/blob/main/examples/multi_agent/agent_supervisor.ipynb I want to replace the llm from API to local ollama server. but failed in tool calling.
GiteaMirror commented 2026-04-28 16:56:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 9, 2024):

You need to use a model that supports tools. Alternatively, if you want to use qwen2,modify the template to handle tool calls.

<!-- gh-comment-id:2337359380 --> @rick-github commented on GitHub (Sep 9, 2024): You need to use a model that [supports tools](https://ollama.com/search?c=tools). Alternatively, if you want to use qwen2,[modify the template](https://ollama.com/sam4096/qwen2tools/blobs/c27bf43fc4ff) to handle tool calls.
GiteaMirror commented 2026-04-28 16:56:26 -05:00
Author
Owner
Copy Link

@sunshine19870316 commented on GitHub (Sep 10, 2024):

You need to use a model that supports tools. Alternatively, if you want to use qwen2,modify the template to handle tool calls.

if I use models who support tools, it also doesn't work but with other error, my framework is langgraph.
LLM.invoke(input_messages).content
^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'invoke'

<!-- gh-comment-id:2340225995 --> @sunshine19870316 commented on GitHub (Sep 10, 2024): > You need to use a model that [supports tools](https://ollama.com/search?c=tools). Alternatively, if you want to use qwen2,[modify the template](https://ollama.com/sam4096/qwen2tools/blobs/c27bf43fc4ff) to handle tool calls. if I use models who support tools, it also doesn't work but with other error, my framework is langgraph. LLM.invoke(input_messages).content ^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'invoke'
GiteaMirror commented 2026-04-28 16:56:30 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 10, 2024):

This would seem to be a langgraph issue. If you could provide a simple, standalone script that demonstrates your problem, it might be easier to diagnose the issue you are having.

<!-- gh-comment-id:2340304154 --> @rick-github commented on GitHub (Sep 10, 2024): This would seem to be a langgraph issue. If you could provide a simple, standalone script that demonstrates your problem, it might be easier to diagnose the issue you are having.
GiteaMirror commented 2026-04-28 16:56:33 -05:00
Author
Owner
Copy Link

@danib08 commented on GitHub (Sep 10, 2024):

Hi @rick-github, I would like to use my own fine-tuned model for tool calling in Ollama.

Do you happen to know if updating the template is enough for it to support tool calls? I've been having trouble finding information on how to make this possible with a custom model so I am kind of lost.

Is having a Jinja template like this enough for the Ollama chat completion to return a valid response_message.tool_calls? Or does the model need to generate a special format?
Thank you in advance

<!-- gh-comment-id:2342302967 --> @danib08 commented on GitHub (Sep 10, 2024): Hi @rick-github, I would like to use my own fine-tuned model for tool calling in Ollama. Do you happen to know if updating the template is enough for it to support tool calls? I've been having trouble finding information on how to make this possible with a custom model so I am kind of lost. Is having a Jinja template like [this](https://ollama.com/sam4096/qwen2tools/blobs/c27bf43fc4ff) enough for the Ollama chat completion to return a valid `response_message.tool_calls`? Or does the model need to generate a special format? Thank you in advance
GiteaMirror commented 2026-04-28 16:56:41 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 10, 2024):

If you are fine-tuning using a base model that supports tool use, then using the template from the original model should work.

If the base model wasn't trained for tool use it's a bit trickier. Before ollama supported it, tool use was done by explicitly injecting a system message, see https://github.com/ollama/ollama/issues/5793#issuecomment-2238938100. With tool support, tool calls are a bit simpler because you can use the template to construct the message sent to the model as you point out. Note that the template will need to be modified to account for tokens specific to a model, message layout, etc. The new problem is trying to convert the result from the model into a tool_calls response. If the response is well-formed JSON, then there's a good chance that it will be converted into a tool_calls structure. So this really relies on two things: a very clear system message with examples to allow the model to properly format the response, and the ability of the model to follow instructions without adding extra verbiage.

<!-- gh-comment-id:2342329196 --> @rick-github commented on GitHub (Sep 10, 2024): If you are fine-tuning using a base model that supports tool use, then using the template from the original model should work. If the base model wasn't trained for tool use it's a bit trickier. Before ollama supported it, tool use was done by explicitly injecting a `system` message, see https://github.com/ollama/ollama/issues/5793#issuecomment-2238938100. With tool support, tool calls are a bit simpler because you can use the template to construct the message sent to the model as you [point out](https://ollama.com/sam4096/qwen2tools/blobs/c27bf43fc4ff). Note that the template will need to be modified to account for tokens specific to a model, message layout, etc. The new problem is trying to convert the result from the model into a `tool_calls` response. If the response is well-formed JSON, then there's a good chance that it will be converted into a `tool_calls` structure. So this really relies on two things: a very clear system message with examples to allow the model to properly format the response, and the ability of the model to follow instructions without adding extra verbiage.
GiteaMirror commented 2026-04-28 16:56:43 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Sep 10, 2024):

Note that the templating language is not Jinja, it's Go templating

<!-- gh-comment-id:2342334259 --> @rick-github commented on GitHub (Sep 10, 2024): Note that the templating language is not Jinja, it's [Go templating](https://pkg.go.dev/text/template)
GiteaMirror commented 2026-04-28 16:56:44 -05:00
Author
Owner
Copy Link

@danib08 commented on GitHub (Sep 11, 2024):

Thank you so much for your help and clarifying the templating language, I now understand everything better.

Indeed, my model is based on Gemma so it doesn't support tool calling on its own. My main concern is with the model response: if it's a valid JSON, Ollama will handle the conversion to a tool_calls structure, correct? Is there anything else I need to implement?

<!-- gh-comment-id:2342369408 --> @danib08 commented on GitHub (Sep 11, 2024): Thank you so much for your help and clarifying the templating language, I now understand everything better. Indeed, my model is based on Gemma so it doesn't support tool calling on its own. My main concern is with the model response: if it's a valid JSON, Ollama will handle the conversion to a `tool_calls` structure, correct? Is there anything else I need to implement?
GiteaMirror commented 2026-04-28 16:56:44 -05:00
Author
Owner
Copy Link

@sunshine19870316 commented on GitHub (Sep 11, 2024):

This would seem to be a langgraph issue. If you could provide a simple, standalone script that demonstrates your problem, it might be easier to diagnose the issue you are having.

Thanks so much! it is indeed what you said!

<!-- gh-comment-id:2342787596 --> @sunshine19870316 commented on GitHub (Sep 11, 2024): > This would seem to be a langgraph issue. If you could provide a simple, standalone script that demonstrates your problem, it might be easier to diagnose the issue you are having. Thanks so much! it is indeed what you said!
GiteaMirror commented 2026-04-28 16:56:45 -05:00
Author
Owner
Copy Link

@sahanams2702 commented on GitHub (Nov 17, 2025):

this is my route, here i am using the llama3-groq-tool-use:latest model , and i am calling the tool , though i give the ai for the simple response it will through me an error saying an error ocuured if i remove the code related to the tool calling it will give me the reponse ? whta is the problem with the tool calling and also i tried using these models llama3.1:latest and llama3.1:8b still the response is same?

import { streamText } from "ai";
import { createOllama } from "ollama-ai-provider";

type CanonicalNode = {
id: string;
type: string;
position: { x: number; y: number };
data: Record<string, any>;
parent_id?: string | null;
next?: string[];
};

let canonicalFlow: CanonicalNode[] = [];

const SYSTEM_CONTEXT = `
You are Flow Studio Copilot — an assistant that modifies a flow graph via tools.

RULES:

  1. When the user requests modifications (add/update/delete), ALWAYS use tool calls.
  2. NEVER invent node IDs. Always call listAllFlowNodes first.
  3. Tool calls must be valid JSON.
  4. When responding with a tool call, return ONLY the tool call JSON.
  5. If the user asks for info, respond normally.

Available tools:

  • addEntryNode
  • addExitNode
  • addPageNode
  • addComponentNode
  • addActionNode
  • addNavigationNode
  • addAgentNode
  • listAllFlowNodes
  • updateNode
  • deleteNode
    `;

function buildDynamicContext(flow: CanonicalNode[]) {
if (!flow || flow.length === 0) return "";

const lines = flow.map((n) => {
const label =
n.data?.label ||
n.data?.pageName ||
n.data?.componentName ||
n.data?.agentName ||
${n.type}_${n.id};

return `- [${n.type}] ${label} (id=${n.id})`;

});

const connections = flow
.filter((n) => n.next && n.next.length)
.map((n) => ${n.id} → ${n.next!.join(", ")})
.join("\n");

return Current Flow Nodes:\n${lines.join("\n")}\n\nConnections:\n${connections};
}
const ollama = createOllama();
const model = ollama("llama3-groq-tool-use:latest");

export async function POST(req: Request) {
try {
const body = await req.json().catch(() => ({}));

let messages = body?.messages;
const singleMessage = body?.message ?? body?.text ?? null;

if (body?.canonicalFlow) {
  canonicalFlow = body.canonicalFlow;
  console.log("Received canonical flow:", canonicalFlow.length, "nodes");
}

if (!messages || !Array.isArray(messages)) {
  if (singleMessage?.trim()) {
    messages = [{ role: "user", content: singleMessage }];
  } else {
    return new Response(JSON.stringify({ error: "Missing input message" }), {
      status: 400,
    });
  }
}

const dynamicContext = buildDynamicContext(canonicalFlow);

console.log("Dynamic Context:\n", dynamicContext);

const response = await streamText({
  model,
  messages: [
    {
      role: "system",
      content: SYSTEM_CONTEXT + "\n" + dynamicContext,
    },
    ...messages,
  ],
  tools: {
    addEntryNode: {
      description: "Add an entry node",
      parameters: {
        type: "object",
        properties: {
          label: { type: "string" },
        },
        required: [],
      },
    },
    addExitNode: {
      description: "Add an exit node",
      parameters: {
        type: "object",
        properties: {
          label: { type: "string" },
        },
        required: [],
      },
    },
    addPageNode: {
      description: "Add a page node",
      parameters: {
        type: "object",
        properties: {
          pageName: { type: "string" },
        },
        required: ["pageName"],
      },
    },
    addComponentNode: {
      description: "Add a UI component node",
      parameters: {
        type: "object",
        properties: {
          componentName: { type: "string" },
          pageId: { type: "string" },
        },
        required: ["componentName", "pageId"],
      },
    },
    addActionNode: {
      description: "Add an action node",
      parameters: {
        type: "object",
        properties: {
          actionType: { type: "string" },
        },
        required: ["actionType"],
      },
    },
    addNavigationNode: {
      description: "Add navigation",
      parameters: {
        type: "object",
        properties: {
          fromPageId: { type: "string" },
          toPageId: { type: "string" },
        },
        required: ["fromPageId", "toPageId"],
      },
    },
    addAgentNode: {
      description: "Add an agent node",
      parameters: {
        type: "object",
        properties: {
          agentName: { type: "string" },
        },
        required: ["agentName"],
      },
    },
    listAllFlowNodes: {
      description: "List all nodes",
      parameters: { type: "object", properties: {} },
    },
    updateNode: {
      description: "Update a node",
      parameters: {
        type: "object",
        properties: {
          nodeId: { type: "string" },
          updates: { type: "object" },
        },
        required: ["nodeId", "updates"],
      },
    },
    deleteNode: {
      description: "Delete a node",
      parameters: {
        type: "object",
        properties: { nodeId: { type: "string" } },
        required: ["nodeId"],
      },
    },
  },

  toolChoice: "auto",
});
console.log("response",response)

return response.toDataStreamResponse();

} catch (err: any) {
console.error("chat route error:", err);
return new Response(JSON.stringify({ error: err?.message || "Unknown error" }), {
status: 500,
});
}
}

<!-- gh-comment-id:3540202069 --> @sahanams2702 commented on GitHub (Nov 17, 2025): this is my route, here i am using the llama3-groq-tool-use:latest model , and i am calling the tool , though i give the ai for the simple response it will through me an error saying an error ocuured if i remove the code related to the tool calling it will give me the reponse ? whta is the problem with the tool calling and also i tried using these models llama3.1:latest and llama3.1:8b still the response is same? import { streamText } from "ai"; import { createOllama } from "ollama-ai-provider"; type CanonicalNode = { id: string; type: string; position: { x: number; y: number }; data: Record<string, any>; parent_id?: string | null; next?: string[]; }; let canonicalFlow: CanonicalNode[] = []; const SYSTEM_CONTEXT = ` You are Flow Studio Copilot — an assistant that modifies a flow graph via tools. RULES: 1. When the user requests modifications (add/update/delete), ALWAYS use tool calls. 2. NEVER invent node IDs. Always call listAllFlowNodes first. 3. Tool calls must be valid JSON. 4. When responding with a tool call, return ONLY the tool call JSON. 5. If the user asks for info, respond normally. Available tools: - addEntryNode - addExitNode - addPageNode - addComponentNode - addActionNode - addNavigationNode - addAgentNode - listAllFlowNodes - updateNode - deleteNode `; function buildDynamicContext(flow: CanonicalNode[]) { if (!flow || flow.length === 0) return ""; const lines = flow.map((n) => { const label = n.data?.label || n.data?.pageName || n.data?.componentName || n.data?.agentName || `${n.type}_${n.id}`; return `- [${n.type}] ${label} (id=${n.id})`; }); const connections = flow .filter((n) => n.next && n.next.length) .map((n) => `${n.id} → ${n.next!.join(", ")}`) .join("\n"); return `Current Flow Nodes:\n${lines.join("\n")}\n\nConnections:\n${connections}`; } const ollama = createOllama(); const model = ollama("llama3-groq-tool-use:latest"); export async function POST(req: Request) { try { const body = await req.json().catch(() => ({})); let messages = body?.messages; const singleMessage = body?.message ?? body?.text ?? null; if (body?.canonicalFlow) { canonicalFlow = body.canonicalFlow; console.log("Received canonical flow:", canonicalFlow.length, "nodes"); } if (!messages || !Array.isArray(messages)) { if (singleMessage?.trim()) { messages = [{ role: "user", content: singleMessage }]; } else { return new Response(JSON.stringify({ error: "Missing input message" }), { status: 400, }); } } const dynamicContext = buildDynamicContext(canonicalFlow); console.log("Dynamic Context:\n", dynamicContext); const response = await streamText({ model, messages: [ { role: "system", content: SYSTEM_CONTEXT + "\n" + dynamicContext, }, ...messages, ], tools: { addEntryNode: { description: "Add an entry node", parameters: { type: "object", properties: { label: { type: "string" }, }, required: [], }, }, addExitNode: { description: "Add an exit node", parameters: { type: "object", properties: { label: { type: "string" }, }, required: [], }, }, addPageNode: { description: "Add a page node", parameters: { type: "object", properties: { pageName: { type: "string" }, }, required: ["pageName"], }, }, addComponentNode: { description: "Add a UI component node", parameters: { type: "object", properties: { componentName: { type: "string" }, pageId: { type: "string" }, }, required: ["componentName", "pageId"], }, }, addActionNode: { description: "Add an action node", parameters: { type: "object", properties: { actionType: { type: "string" }, }, required: ["actionType"], }, }, addNavigationNode: { description: "Add navigation", parameters: { type: "object", properties: { fromPageId: { type: "string" }, toPageId: { type: "string" }, }, required: ["fromPageId", "toPageId"], }, }, addAgentNode: { description: "Add an agent node", parameters: { type: "object", properties: { agentName: { type: "string" }, }, required: ["agentName"], }, }, listAllFlowNodes: { description: "List all nodes", parameters: { type: "object", properties: {} }, }, updateNode: { description: "Update a node", parameters: { type: "object", properties: { nodeId: { type: "string" }, updates: { type: "object" }, }, required: ["nodeId", "updates"], }, }, deleteNode: { description: "Delete a node", parameters: { type: "object", properties: { nodeId: { type: "string" } }, required: ["nodeId"], }, }, }, toolChoice: "auto", }); console.log("response",response) return response.toDataStreamResponse(); } catch (err: any) { console.error("chat route error:", err); return new Response(JSON.stringify({ error: err?.message || "Unknown error" }), { status: 500, }); } }
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#50731
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?