github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #1559] Pass in prompt as arguments is broken for multimodal models #47364

New Issue
Closed
opened 2026-04-28 03:37:36 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-28 03:37:36 -05:00
Owner
Copy Link

Originally created by @elsatch on GitHub (Dec 16, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1559

Originally assigned to: @jmorganca on GitHub.

I have been trying to evaluate the performance of the multimodal models passing the prompt as argument, as stated in the README.md section. Whenever I pass the image as an argument for ollama cli, it hallucinates the whole response. Asking about the same image using the regular chat, works without problem.

This is a sample image I am using:

image

These are the responses I am getting when passing the image as argument:

image

Note: original image taken created Leonid Mamchenkov, using the Carbon website to style the code.

Originally created by @elsatch on GitHub (Dec 16, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1559 Originally assigned to: @jmorganca on GitHub. I have been trying to evaluate the performance of the multimodal models passing the prompt as argument, as stated in the README.md section. Whenever I pass the image as an argument for ollama cli, it hallucinates the whole response. Asking about the same image using the regular chat, works without problem. This is a sample image I am using: ![image](https://github.com/jmorganca/ollama/assets/653433/11de3f68-0613-447d-8c3b-23bdd546dc08) These are the responses I am getting when passing the image as argument: ![image](https://github.com/jmorganca/ollama/assets/653433/5d3ae67c-bca0-45d0-b62b-f22074aa1450) Note: original image taken created Leonid Mamchenkov, using the Carbon website to style the code.
GiteaMirror commented 2026-04-28 03:37:38 -05:00
Author
Owner
Copy Link

@technovangelist commented on GitHub (Dec 19, 2023):

What do you mean by regular chat? The multi modal models do a pretty good job with OCR, but they aren't going to be as good as a full OCR engine. You will probably get better results using an OCR engine and then using the model on that.

<!-- gh-comment-id:1863351509 --> @technovangelist commented on GitHub (Dec 19, 2023): What do you mean by regular chat? The multi modal models do a pretty good job with OCR, but they aren't going to be as good as a full OCR engine. You will probably get better results using an OCR engine and then using the model on that.
GiteaMirror commented 2026-04-28 03:37:38 -05:00
Author
Owner
Copy Link

@markab21 commented on GitHub (Jan 4, 2024):

When using Ollama's multimodal capabilities, initiating the model with a direct command to analyze an image (e.g., "ollama run llava:13b 'What is in this image /tmp/test_image.jpg'") leads to hallucination outputs (as the original report stated, I see Japanese characters output).

However, starting the model normally and then entering the image query as a dialogue yields correct results. This suggests an inconsistency in how the model handles direct image processing commands versus interactive dialogue prompts.

<!-- gh-comment-id:1876208201 --> @markab21 commented on GitHub (Jan 4, 2024): When using Ollama's multimodal capabilities, initiating the model with a direct command to analyze an image (e.g., "ollama run llava:13b 'What is in this image /tmp/test_image.jpg'") leads to hallucination outputs (as the original report stated, I see Japanese characters output). However, starting the model normally and then entering the image query as a dialogue yields correct results. This suggests an inconsistency in how the model handles direct image processing commands versus interactive dialogue prompts.
GiteaMirror commented 2026-04-28 03:37:39 -05:00
Author
Owner
Copy Link

@elsatch commented on GitHub (Jan 4, 2024):

@markab21 thanks for the succint and clear explanation.

When I talked about regular chat, I meant the interactive chat that launches when you run ollama in the command line without any prompts (as opposed to passing the prompt as in the example provided by Mark)

<!-- gh-comment-id:1876890544 --> @elsatch commented on GitHub (Jan 4, 2024): @markab21 thanks for the succint and clear explanation. When I talked about regular chat, I meant the interactive chat that launches when you run ollama in the command line without any prompts (as opposed to passing the prompt as in the example provided by Mark)
GiteaMirror commented 2026-04-28 03:37:39 -05:00
Author
Owner
Copy Link

@sam1am commented on GitHub (Jan 13, 2024):

I would also like to know how to pass the image when running from the command line or from python. I can only get it to work in the interactive chat.

<!-- gh-comment-id:1890295358 --> @sam1am commented on GitHub (Jan 13, 2024): I would also like to know how to pass the image when running from the command line or from python. I can only get it to work in the interactive chat.
GiteaMirror commented 2026-04-28 03:37:40 -05:00
Author
Owner
Copy Link

@antonme commented on GitHub (Jan 19, 2024):

I second that, right now it seems impossible to use ollama's multimodal capabilities from command line for scripting or something like that

<!-- gh-comment-id:1900783291 --> @antonme commented on GitHub (Jan 19, 2024): I second that, right now it seems impossible to use ollama's multimodal capabilities from command line for scripting or something like that
GiteaMirror commented 2026-04-28 03:37:41 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Mar 11, 2024):

Sorry for the slow response. This did get fixed a while back but the issue never got updated. Here's an example:

% ./ollama run llava:13b "Describe this image: /Users/pdevine/Pictures/steve.png"
Added image '/Users/pdevine/Pictures/steve.png'
 The image is a black and white sketch of a person wearing a hard hat. The individual appears to be smiling or laughing,
with a relaxed posture, indicating a sense of joy or amusement. On the head of the figure, there's text that reads "PASCO,"
which could possibly be related to the brand of the hard hat or it might simply be part of the name of the person depicted.
The drawing style is somewhat reminiscent of comic strip illustrations with bold lines and a simplistic form. Below the
figure, there's another line of text that reads "Sam," which could either be the name of the artist or the name of the
character being drawn.

You can see from the Added image '/Users/pdevine/Pictures/steve.png' that the image was added.

@sam1am To make this work from Python, it depends on if you are using the Chat function or the Generate function. Both functions take base64 encoded images (you'll have to convert the image yourself) and then you can pass that as a string along with the request in the images field.

Here's an example:

import ollama
response = ollama.chat(model='llava:13b', messages=[
  {
    'role': 'user',
    'content': 'What's in this image?',
    'images': [ <list of base64 encoded images> ],
  },
])
print(response['message']['content'])

Hopefully that answers everyone's questions. I'm going to go ahead and close the issue.

<!-- gh-comment-id:1989664371 --> @pdevine commented on GitHub (Mar 11, 2024): Sorry for the slow response. This did get fixed a while back but the issue never got updated. Here's an example: ``` % ./ollama run llava:13b "Describe this image: /Users/pdevine/Pictures/steve.png" Added image '/Users/pdevine/Pictures/steve.png' The image is a black and white sketch of a person wearing a hard hat. The individual appears to be smiling or laughing, with a relaxed posture, indicating a sense of joy or amusement. On the head of the figure, there's text that reads "PASCO," which could possibly be related to the brand of the hard hat or it might simply be part of the name of the person depicted. The drawing style is somewhat reminiscent of comic strip illustrations with bold lines and a simplistic form. Below the figure, there's another line of text that reads "Sam," which could either be the name of the artist or the name of the character being drawn. ``` You can see from the `Added image '/Users/pdevine/Pictures/steve.png'` that the image was added. @sam1am To make this work from Python, it depends on if you are using the `Chat` function or the `Generate` function. Both functions take base64 encoded images (you'll have to convert the image yourself) and then you can pass that as a string along with the request in the `images` field. Here's an example: ``` import ollama response = ollama.chat(model='llava:13b', messages=[ { 'role': 'user', 'content': 'What's in this image?', 'images': [ <list of base64 encoded images> ], }, ]) print(response['message']['content']) ``` Hopefully that answers everyone's questions. I'm going to go ahead and close the issue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#47364
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?