github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #7611] with_structured_output support for ollama.chat() #51368

New Issue
Closed
opened 2026-04-28 19:42:15 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-28 19:42:15 -05:00
Owner
Copy Link

Originally created by @ChmHsm on GitHub (Nov 11, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7611

Thank you for the llama 3.2 vision integration!

I was using llama3.2-3b with ChatOllama(model="llama3.2:latest").with_structured_output() to get a structured response from the model and I was hoping to be able to do the same with llama3.2-vision.

But it turns out, at least to my knowledge, I can't for now since i can't use with_structured_output directly with ollama.chat(model='llama3.2-vision'). (when I do that I get a "llama vision doesn't support tools" error)

Is this still a pending feature or am I missing something?

It's very important to be able to get a structured response from llama vision. A workaround is I'm now having to use a chain containing llama3.2-vision AND llama3.2 to process an image + get a structured response, which is a bit of an overkill...

Originally created by @ChmHsm on GitHub (Nov 11, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7611 Thank you for the llama 3.2 vision integration! I was using llama3.2-3b with ChatOllama(model="llama3.2:latest").with_structured_output() to get a structured response from the model and I was hoping to be able to do the same with llama3.2-vision. But it turns out, at least to my knowledge, I can't for now since i can't use with_structured_output directly with ollama.chat(model='llama3.2-vision'). (when I do that I get a "llama vision doesn't support tools" error) Is this still a pending feature or am I missing something? It's very important to be able to get a structured response from llama vision. A workaround is I'm now having to use a chain containing llama3.2-vision AND llama3.2 to process an image + get a structured response, which is a bit of an overkill...
GiteaMirror added the feature request label 2026-04-28 19:42:15 -05:00
GiteaMirror commented 2026-04-28 19:42:19 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 11, 2024):

ChatOllama is a langchain project.

For the specific issue of tool support, you can try adding the tool related section of the llama3.2 template to the llama3.2 vision template. However, if the model hasn't been trained in tool use, the results may be poor.

<!-- gh-comment-id:2468076601 --> @rick-github commented on GitHub (Nov 11, 2024): ChatOllama is a [langchain](https://github.com/langchain-ai/langchain) project. For the specific issue of tool support, you can try adding the tool related section of the [llama3.2 template](https://ollama.com/library/llama3.2/blobs/966de95ca8a6) to the [llama3.2 vision template](https://ollama.com/library/llama3.2-vision/blobs/715415638c9c). However, if the model hasn't been trained in tool use, the results may be poor.
GiteaMirror commented 2026-04-28 19:42:20 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 11, 2024):

Just replacing the template in llama3.2-vision with the one in llama3.2 seems to work.

$ ollama run llama3.2-vision-tools 
>>> describe this picture: ./puppy.jpg
Added image './puppy.jpg'
The image features a small, white puppy with a fluffy coat and floppy ears. The puppy is sitting on a stone surface, facing to the right, with its head turned slightly towards the camera. A red 
collar adorns the puppy's neck, complete with a gold bell. The background of the image is blurred, but it appears to be an outdoor setting, possibly a patio or a garden.
>>> /bye
$ ./tool-test.py --model llama3.2-vision-tools --prompt 'what time is it?'
calling get_datetime({})
It is currently 6:38 PM on Monday, November 11, 2024.
$ ./tool-test.py --model llama3.2-vision-tools --prompt 'what is 1.34 ^ 10?'
calling power({'y': 10, 'x': 1.34})
The result of 1.34 raised to the power of 10 is approximately 18.67.
$ ./tool-test.py --model llama3.2-vision-tools --prompt 'how many files are in the current directory?'
calling run_shell_command({'command': 'ls -1 | wc -l'})
There are 58 files in the current directory.
$ ./tool-test.py --model llama3.2-vision-tools --prompt 'what operating system are you?'
calling get_operating_system({})
Based on the tool call response, I am running a Linux operating system with version 5.15.0-118-generic-x86_64 and glibc 2.35.
$ ./tool-test.py --model llama3.2-vision-tools --prompt 'what is in the file "./data.txt"?'
calling read_file({'filename': './data.txt'})
The file "./data.txt" contains the following lines:

1. hi
2. ok
3. hello
4. hi there
5. how are you
<!-- gh-comment-id:2468749305 --> @rick-github commented on GitHub (Nov 11, 2024): Just replacing the template in llama3.2-vision with the one in llama3.2 seems to work. ```console $ ollama run llama3.2-vision-tools >>> describe this picture: ./puppy.jpg Added image './puppy.jpg' The image features a small, white puppy with a fluffy coat and floppy ears. The puppy is sitting on a stone surface, facing to the right, with its head turned slightly towards the camera. A red collar adorns the puppy's neck, complete with a gold bell. The background of the image is blurred, but it appears to be an outdoor setting, possibly a patio or a garden. >>> /bye $ ./tool-test.py --model llama3.2-vision-tools --prompt 'what time is it?' calling get_datetime({}) It is currently 6:38 PM on Monday, November 11, 2024. $ ./tool-test.py --model llama3.2-vision-tools --prompt 'what is 1.34 ^ 10?' calling power({'y': 10, 'x': 1.34}) The result of 1.34 raised to the power of 10 is approximately 18.67. $ ./tool-test.py --model llama3.2-vision-tools --prompt 'how many files are in the current directory?' calling run_shell_command({'command': 'ls -1 | wc -l'}) There are 58 files in the current directory. $ ./tool-test.py --model llama3.2-vision-tools --prompt 'what operating system are you?' calling get_operating_system({}) Based on the tool call response, I am running a Linux operating system with version 5.15.0-118-generic-x86_64 and glibc 2.35. $ ./tool-test.py --model llama3.2-vision-tools --prompt 'what is in the file "./data.txt"?' calling read_file({'filename': './data.txt'}) The file "./data.txt" contains the following lines: 1. hi 2. ok 3. hello 4. hi there 5. how are you ```
GiteaMirror commented 2026-04-28 19:42:21 -05:00
Author
Owner
Copy Link

@srao-cipher commented on GitHub (Jan 10, 2025):

Where are you getting llama3.2-vision-tools?

<!-- gh-comment-id:2583600073 --> @srao-cipher commented on GitHub (Jan 10, 2025): Where are you getting llama3.2-vision-tools?
GiteaMirror commented 2026-04-28 19:42:22 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jan 11, 2025):

$ echo FROM llama3.2-vision > Modelfile
$ echo 'TEMPLATE """'"$(ollama show --template llama3.2)"'"""' >> Modelfile
$ ollama create llama3.2-vision-tools
<!-- gh-comment-id:2585009100 --> @rick-github commented on GitHub (Jan 11, 2025): ```console $ echo FROM llama3.2-vision > Modelfile $ echo 'TEMPLATE """'"$(ollama show --template llama3.2)"'"""' >> Modelfile $ ollama create llama3.2-vision-tools ```
GiteaMirror commented 2026-04-28 19:42:22 -05:00
Author
Owner
Copy Link

@lemassykoi commented on GitHub (Apr 12, 2025):

is it possible too with gemma3 ? adding tools capabilities

<!-- gh-comment-id:2798832867 --> @lemassykoi commented on GitHub (Apr 12, 2025): is it possible too with gemma3 ? adding tools capabilities
GiteaMirror commented 2026-04-28 19:42:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Apr 12, 2025):

Any model can have tool capability added, but if the model is not trained in tool use, the results can be poor. Google has explicitly not given Gemma tools support but there have been a few examples from the community.

<!-- gh-comment-id:2798836240 --> @rick-github commented on GitHub (Apr 12, 2025): Any model can have tool capability added, but if the model is not trained in tool use, the results can be poor. Google has explicitly not given Gemma [tools support](https://huggingface.co/google/gemma-3-12b-it/discussions/11#67d467df3589638cc558a649) but there have been a few examples from the [community](https://ollama.com/search?q=gemma+tools).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#51368
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?