github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8345] llama3.2-vision does not support tools #51861

New Issue
Closed
opened 2026-04-28 21:04:48 -05:00 by GiteaMirror · 18 comments
GiteaMirror commented 2026-04-28 21:04:48 -05:00
Owner
Copy Link

Originally created by @kwaa on GitHub (Jan 8, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8345

Originally assigned to: @ParthSareen on GitHub.

What is the issue?

I get above error when I use llama3.2-vision:11b with tool calling.

I think llama3.2-vision:11b should support tool calls: https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/vision_prompt_format.md#builtin-and-zero-shot-tool-calling

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.5.0

Originally created by @kwaa on GitHub (Jan 8, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8345 Originally assigned to: @ParthSareen on GitHub. ### What is the issue? I get above error when I use `llama3.2-vision:11b` with tool calling. I think `llama3.2-vision:11b` should support tool calls: https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/vision_prompt_format.md#builtin-and-zero-shot-tool-calling - [llama3.2:3b template](https://ollama.com/library/llama3.2/blobs/966de95ca8a6) - [llama3.2-vision:11b template](https://ollama.com/library/llama3.2-vision/blobs/715415638c9c) ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.5.0
GiteaMirror added the bug label 2026-04-28 21:04:48 -05:00
GiteaMirror commented 2026-04-28 21:04:51 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 8, 2025):

https://github.com/ollama/ollama/issues/7611#issuecomment-2468749305

<!-- gh-comment-id:2577458074 --> @rick-github commented on GitHub (Jan 8, 2025): https://github.com/ollama/ollama/issues/7611#issuecomment-2468749305
GiteaMirror commented 2026-04-28 21:04:51 -05:00
Author
Owner
Copy Link

@kwaa commented on GitHub (Jan 8, 2025):

#7611 (comment)

Can the template be updated on ollama.com/models?

<!-- gh-comment-id:2577466928 --> @kwaa commented on GitHub (Jan 8, 2025): > [#7611 (comment)](https://github.com/ollama/ollama/issues/7611#issuecomment-2468749305) Can the template be updated on ollama.com/models?
GiteaMirror commented 2026-04-28 21:04:51 -05:00
Author
Owner
Copy Link

@x0wllaar commented on GitHub (Jan 9, 2025):

The issue is that

the vision models don’t support tool-calling with text+image inputs.

According to https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_2/#-code-interpreter-and-tool-calling-

So I seriously doubt the usefulness here.

<!-- gh-comment-id:2578929370 --> @x0wllaar commented on GitHub (Jan 9, 2025): The issue is that > the vision models don’t support tool-calling with text+image inputs. According to https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_2/#-code-interpreter-and-tool-calling- So I seriously doubt the usefulness here.
GiteaMirror commented 2026-04-28 21:04:52 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 9, 2025):

If you want a model that can do tool calls and also interpret images, as in the use case in #7611, it can be useful. You just cant ask the model to interpret an image and also do a tool call at the same time. For example, "read this business card and use the add_contact() tool to add this person to my contacts list" won't work, but "read this business card" and "use the details from the previous prompt with the add_contact() tool to add this person to my contact list" should work.

<!-- gh-comment-id:2578980184 --> @rick-github commented on GitHub (Jan 9, 2025): If you want a model that can do tool calls and also interpret images, as in the use case in #7611, it can be useful. You just cant ask the model to interpret an image and also do a tool call at the same time. For example, "read this business card and use the add_contact() tool to add this person to my contacts list" won't work, but "read this business card" and "use the details from the previous prompt with the add_contact() tool to add this person to my contact list" should work.
GiteaMirror commented 2026-04-28 21:04:52 -05:00
Author
Owner
Copy Link

@x0wllaar commented on GitHub (Jan 9, 2025):

Honestly, I always interpreted it as "tool calls break whenever there's an image in the context somewhere", but I might be wrong. I guess it can still be useful if you want to do 2 different prompts (remove image in the second one) and avoid unnecessary model reloads.

<!-- gh-comment-id:2578989523 --> @x0wllaar commented on GitHub (Jan 9, 2025): Honestly, I always interpreted it as "tool calls break whenever there's an image in the context somewhere", but I might be wrong. I guess it can still be useful if you want to do 2 different prompts (remove image in the second one) and avoid unnecessary model reloads.
GiteaMirror commented 2026-04-28 21:04:52 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 9, 2025):

I said "should work" so thought I'd better test to see if I was right. Had to manually add the tool since that's not a feature of the CLI but image+tool seems to work fine. I guess you can't rely on it though.

$ ollama run llama3.2-vision-tools
>>> Given the following functions, please respond with a JSON for a function call with its proper argum
... ents that best answers the given prompt. Respond in the format {"name": function name, "parameters"
... : dictionary of argument name and its value}. Do not use variables.  {"type":"function","function":
... {"name":"add_to_contact","parameters":{"name":"details"}}} ./business.card.jpg
Added image './business.card.jpg'
{"name":"add_to_contact","parameters": {"name":"VanArsdel, Ltd.", "address": "5678 Main St. New 
York, NY 90210", "phone_number": "212-555-0199", "email": "jordan@vanarsdelltd.com"}}
<!-- gh-comment-id:2579021418 --> @rick-github commented on GitHub (Jan 9, 2025): I said "should work" so thought I'd better test to see if I was right. Had to manually add the tool since that's not a feature of the CLI but image+tool seems to work fine. I guess you can't rely on it though. ```console $ ollama run llama3.2-vision-tools >>> Given the following functions, please respond with a JSON for a function call with its proper argum ... ents that best answers the given prompt. Respond in the format {"name": function name, "parameters" ... : dictionary of argument name and its value}. Do not use variables. {"type":"function","function": ... {"name":"add_to_contact","parameters":{"name":"details"}}} ./business.card.jpg Added image './business.card.jpg' {"name":"add_to_contact","parameters": {"name":"VanArsdel, Ltd.", "address": "5678 Main St. New York, NY 90210", "phone_number": "212-555-0199", "email": "jordan@vanarsdelltd.com"}} ```
GiteaMirror commented 2026-04-28 21:04:53 -05:00
Author
Owner
Copy Link

@ParthSareen commented on GitHub (Jan 15, 2025):

@x0wllaar @kwaa We usually try to follow whatever the spec of the model provider. In this case it's probably more useful to do structured outputs or create a copy of the model as mentioned by @rick-github

<!-- gh-comment-id:2593968459 --> @ParthSareen commented on GitHub (Jan 15, 2025): @x0wllaar @kwaa We usually try to follow whatever the spec of the model provider. In this case it's probably more useful to do structured outputs or create a copy of the model as mentioned by @rick-github
GiteaMirror commented 2026-04-28 21:04:53 -05:00
Author
Owner
Copy Link

@kwaa commented on GitHub (Jan 16, 2025):

We usually try to follow whatever the spec of the model provider.

But llama3.2-vision officially supports tool calls, as shown in the link above.

In this case it's probably more useful to do structured outputs or create a copy of the model

I don't think so.

<!-- gh-comment-id:2594349436 --> @kwaa commented on GitHub (Jan 16, 2025): > We usually try to follow whatever the spec of the model provider. But `llama3.2-vision` **officially supports tool calls**, as shown in the link above. > In this case it's probably more useful to do structured outputs or create a copy of the model I don't think so.
GiteaMirror commented 2026-04-28 21:04:54 -05:00
Author
Owner
Copy Link

@kwaa commented on GitHub (Jan 16, 2025):

https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/vision_prompt_format.md#builtin-and-zero-shot-tool-calling

<!-- gh-comment-id:2594352994 --> @kwaa commented on GitHub (Jan 16, 2025): https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/vision_prompt_format.md#builtin-and-zero-shot-tool-calling
GiteaMirror commented 2026-04-28 21:04:54 -05:00
Author
Owner
Copy Link

@kwaa commented on GitHub (Jan 16, 2025):

As mentioned above, llama3.2-vision should support both vision and tools instead of requiring users to create copies.

<!-- gh-comment-id:2594355901 --> @kwaa commented on GitHub (Jan 16, 2025): As mentioned above, `llama3.2-vision` should support both vision and tools instead of requiring users to create copies.
GiteaMirror commented 2026-04-28 21:04:54 -05:00
Author
Owner
Copy Link

@x0wllaar commented on GitHub (Jan 16, 2025):

Why do you want to use 3.2-vision for text-only tool calls when there already is 3.1 with literally the same weights in the text-only part that already supports tools?

<!-- gh-comment-id:2594603257 --> @x0wllaar commented on GitHub (Jan 16, 2025): Why do you want to use `3.2-vision` for text-only tool calls when there already is 3.1 with literally the same weights in the text-only part that already supports tools?
GiteaMirror commented 2026-04-28 21:04:54 -05:00
Author
Owner
Copy Link

@kwaa commented on GitHub (Jan 16, 2025):

Why do you want to use 3.2-vision for text-only tool calls when there already is 3.1 with literally the same weights in the text-only part that already supports tools?

Why not? I'm writing tests for an AI SDK so I don't have to repeatedly download and run similar models separately for vision and tool call testing, also easy for collaborators to reproduce.

<!-- gh-comment-id:2594797938 --> @kwaa commented on GitHub (Jan 16, 2025): > Why do you want to use `3.2-vision` for text-only tool calls when there already is 3.1 with literally the same weights in the text-only part that already supports tools? Why not? I'm writing tests for an AI SDK so I don't have to repeatedly download and run similar models separately for vision and tool call testing, also easy for collaborators to reproduce.
GiteaMirror commented 2026-04-28 21:04:55 -05:00
Author
Owner
Copy Link

@kwaa commented on GitHub (Jan 16, 2025):

meta-llama/llama-models@main/models/llama3_2/vision_prompt_format.md#builtin-and-zero-shot-tool-calling

@ParthSareen Would you mind taking a look at this?

<!-- gh-comment-id:2594817917 --> @kwaa commented on GitHub (Jan 16, 2025): > [meta-llama/llama-models@`main`/models/llama3_2/vision_prompt_format.md#builtin-and-zero-shot-tool-calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/vision_prompt_format.md#builtin-and-zero-shot-tool-calling) @ParthSareen Would you mind taking a look at this?
GiteaMirror commented 2026-04-28 21:04:55 -05:00
Author
Owner
Copy Link

@ParthSareen commented on GitHub (Jan 17, 2025):

Hey @kwaa I think in this case - I'd recommend trying out llama3.2-vision-tools as it has the updated template. You could also try it with structured outputs instead of doing tool calls. Lmk what you end up trying out and what works for you!

<!-- gh-comment-id:2599249849 --> @ParthSareen commented on GitHub (Jan 17, 2025): Hey @kwaa I think in this case - I'd recommend trying out `llama3.2-vision-tools` as it has the updated template. You could also try it with structured outputs instead of doing tool calls. Lmk what you end up trying out and what works for you!
GiteaMirror commented 2026-04-28 21:04:55 -05:00
Author
Owner
Copy Link

@kwaa commented on GitHub (Jan 18, 2025):

I think in this case - I'd recommend trying out llama3.2-vision-tools as it has the updated template. You could also try it with structured outputs instead of doing tool calls. Lmk what you end up trying out and what works for you!

@ParthSareen The main problem is that I can't get llama3.2-vision-tools directly from https://ollama.com/models, it has to be modified manually. (https://github.com/ollama/ollama/issues/8345#issuecomment-2577466928)

<!-- gh-comment-id:2599598184 --> @kwaa commented on GitHub (Jan 18, 2025): > I think in this case - I'd recommend trying out `llama3.2-vision-tools` as it has the updated template. You could also try it with structured outputs instead of doing tool calls. Lmk what you end up trying out and what works for you! @ParthSareen The main problem is that I can't get `llama3.2-vision-tools` directly from https://ollama.com/models, it has to be modified manually. (https://github.com/ollama/ollama/issues/8345#issuecomment-2577466928)
GiteaMirror commented 2026-04-28 21:04:56 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 18, 2025):

$ echo FROM llama3.2-vision > Modelfile
$ echo 'TEMPLATE """'"$(ollama show --template llama3.2)"'"""' >> Modelfile
$ ollama create llama3.2-vision-tools
<!-- gh-comment-id:2599643562 --> @rick-github commented on GitHub (Jan 18, 2025): ```console $ echo FROM llama3.2-vision > Modelfile $ echo 'TEMPLATE """'"$(ollama show --template llama3.2)"'"""' >> Modelfile $ ollama create llama3.2-vision-tools ```
GiteaMirror commented 2026-04-28 21:04:56 -05:00
Author
Owner
Copy Link

@kwaa commented on GitHub (Jan 18, 2025):

$ echo FROM llama3.2-vision > Modelfile
$ echo 'TEMPLATE """'"$(ollama show --template llama3.2)"'"""' >> Modelfile
$ ollama create llama3.2-vision-tools

I saw that, I just thought it should be available in https://ollama.com/models instead of needing to be manually modified by the user.

<!-- gh-comment-id:2599645970 --> @kwaa commented on GitHub (Jan 18, 2025): > $ echo FROM llama3.2-vision > Modelfile > $ echo 'TEMPLATE """'"$(ollama show --template llama3.2)"'"""' >> Modelfile > $ ollama create llama3.2-vision-tools I saw that, I just thought it should be available in https://ollama.com/models instead of needing to be manually modified by the user.
GiteaMirror commented 2026-04-28 21:04:56 -05:00
Author
Owner
Copy Link

@tstojecki commented on GitHub (Mar 3, 2025):

We have hit this issue by using openai client with instructor for structured output. Our code doesn't use tools but I suppouse tool use is implicit via Instructor's client specifc implementation.
To avoid this error with llama3.2-vision, we had to take a different approach for ollama using Ollama client from every other client.

I understand the registry is based on docs, at the same time I am not sure I see too much harm in letting users evaluate tool use for this model rather than blocking it.

<!-- gh-comment-id:2693521829 --> @tstojecki commented on GitHub (Mar 3, 2025): We have hit this issue by using openai client with instructor for structured output. Our code doesn't use tools but I suppouse tool use is implicit via Instructor's client specifc implementation. To avoid this error with llama3.2-vision, we had to take a different approach for ollama using Ollama client from every other client. I understand the registry is based on docs, at the same time I am not sure I see too much harm in letting users evaluate tool use for this model rather than blocking it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#51861
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?