github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #746] Support multi-modal models #62390

New Issue
Closed
opened 2026-05-03 08:45:32 -05:00 by GiteaMirror · 21 comments
GiteaMirror commented 2026-05-03 08:45:32 -05:00
Owner
Copy Link

Originally created by @arian81 on GitHub (Oct 10, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/746

Originally assigned to: @pdevine on GitHub.

This is one of the best open source multi modals based on llama 7 currently. It would nice to be able to host it in ollama.
https://llava-vl.github.io/

Originally created by @arian81 on GitHub (Oct 10, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/746 Originally assigned to: @pdevine on GitHub. This is one of the best open source multi modals based on llama 7 currently. It would nice to be able to host it in ollama. https://llava-vl.github.io/
GiteaMirror added the feature request label 2026-05-03 08:45:32 -05:00
GiteaMirror commented 2026-05-03 08:45:57 -05:00
Author
Owner
Copy Link

@ryansereno commented on GitHub (Oct 10, 2023):

Came here looking for this, to see if the discussion had begun surrounding this.
Curious to see what will be required to make this happen.

Edit: Progress is being made upstream in llama.cpp to support this.

<!-- gh-comment-id:1755941917 --> @ryansereno commented on GitHub (Oct 10, 2023): Came here looking for this, to see if the discussion had begun surrounding this. Curious to see what will be required to make this happen. Edit: Progress is being made [upstream](https://github.com/ggerganov/llama.cpp/pull/3436) in llama.cpp to support this.
GiteaMirror commented 2026-05-03 08:45:58 -05:00
Author
Owner
Copy Link

@spielhoelle commented on GitHub (Oct 13, 2023):

The PR @ryansereno mentioned is merged and in master now. How can we run this in ollama?

<!-- gh-comment-id:1761157688 --> @spielhoelle commented on GitHub (Oct 13, 2023): The PR @ryansereno mentioned is [merged and in master](https://github.com/ggerganov/llama.cpp/tree/master/examples/llava) now. How can we run this in ollama?
GiteaMirror commented 2026-05-03 08:45:59 -05:00
Author
Owner
Copy Link

@marscod commented on GitHub (Oct 15, 2023):

I could successfully run llava-v1.5-7b and it is available at: https://ollama.ai/marscod/llava but I have to map an image parameter to llama.cpp's image parameter. Maybe within the prompt?

<!-- gh-comment-id:1763518683 --> @marscod commented on GitHub (Oct 15, 2023): I could successfully run `llava-v1.5-7b` and it is available at: https://ollama.ai/marscod/llava but I have to map an `image` parameter to `llama.cpp`'s image parameter. Maybe within the prompt?
GiteaMirror commented 2026-05-03 08:46:00 -05:00
Author
Owner
Copy Link

@chigkim commented on GitHub (Oct 16, 2023):

It would be good to have file reader command in the prompt like /read file.jpg for this.

<!-- gh-comment-id:1763658789 --> @chigkim commented on GitHub (Oct 16, 2023): It would be good to have file reader command in the prompt like /read file.jpg for this.
GiteaMirror commented 2026-05-03 08:46:01 -05:00
Author
Owner
Copy Link

@hugh-min commented on GitHub (Oct 17, 2023):

I could successfully run llava-v1.5-7b and it is available at: https://ollama.ai/marscod/llava but I have to map an image parameter to llama.cpp's image parameter. Maybe within the prompt?

Could you elaborate on how to map an image within ollama?

<!-- gh-comment-id:1765866173 --> @hugh-min commented on GitHub (Oct 17, 2023): > I could successfully run `llava-v1.5-7b` and it is available at: https://ollama.ai/marscod/llava but I have to map an `image` parameter to `llama.cpp`'s image parameter. Maybe within the prompt? Could you elaborate on how to map an image within ollama?
GiteaMirror commented 2026-05-03 08:46:02 -05:00
Author
Owner
Copy Link

@Bortus-AI commented on GitHub (Oct 28, 2023):

I could successfully run llava-v1.5-7b and it is available at: https://ollama.ai/marscod/llava but I have to map an image parameter to llama.cpp's image parameter. Maybe within the prompt?

Could you elaborate on how to map an image within ollama?

I would like to know as well. Thanks

<!-- gh-comment-id:1783894513 --> @Bortus-AI commented on GitHub (Oct 28, 2023): > > I could successfully run `llava-v1.5-7b` and it is available at: https://ollama.ai/marscod/llava but I have to map an `image` parameter to `llama.cpp`'s image parameter. Maybe within the prompt? > > Could you elaborate on how to map an image within ollama? I would like to know as well. Thanks
GiteaMirror commented 2026-05-03 08:46:03 -05:00
Author
Owner
Copy Link

@tmc commented on GitHub (Oct 29, 2023):

it seems a couple of interface design decisions are are play: 1) how to represent this in the http api and 2) what the user/cli interface should be.

I want to note/highlight that the folks hacking on iTerm2 have done some work that may be relevant in the cli context here: https://iterm2.com/documentation-images.html

For the HTTP interface I'd suggest taking some inspiration of how OpenAI is folding in image data may be useful. I did a bit of protocol decoding and the TL;DR of how they do it is upload to blob store then include a special message type in the completion message list.

There's also a/the consideration of if it's an ollama concern to allow annotation of an incoming image to support highlighting part of the image. That feels a bit out of scope to start but perhaps the design should keep that in mind.

<!-- gh-comment-id:1784017191 --> @tmc commented on GitHub (Oct 29, 2023): it seems a couple of interface design decisions are are play: 1) how to represent this in the http api and 2) what the user/cli interface should be. I want to note/highlight that the folks hacking on iTerm2 have done some work that may be relevant in the cli context here: https://iterm2.com/documentation-images.html For the HTTP interface I'd suggest taking some inspiration of how OpenAI is folding in image data may be useful. I did a bit of protocol decoding and the TL;DR of how they do it is upload to blob store then include a special message type in the completion message list. There's also a/the consideration of if it's an ollama concern to allow annotation of an incoming image to support highlighting part of the image. That feels a bit out of scope to start but perhaps the design should keep that in mind.
GiteaMirror commented 2026-05-03 08:46:03 -05:00
Author
Owner
Copy Link

@sausheong commented on GitHub (Nov 3, 2023):

I could successfully run llava-v1.5-7b and it is available at: https://ollama.ai/marscod/llava but I have to map an image parameter to llama.cpp's image parameter. Maybe within the prompt?

Could you elaborate on how to map an image within ollama?

I would like to know as well. Thanks

Me too, can explain how to map an image within ollama?

<!-- gh-comment-id:1792573242 --> @sausheong commented on GitHub (Nov 3, 2023): > > > I could successfully run `llava-v1.5-7b` and it is available at: https://ollama.ai/marscod/llava but I have to map an `image` parameter to `llama.cpp`'s image parameter. Maybe within the prompt? > > > > > > Could you elaborate on how to map an image within ollama? > > I would like to know as well. Thanks Me too, can explain how to map an image within ollama?
GiteaMirror commented 2026-05-03 08:46:04 -05:00
Author
Owner
Copy Link

@itsPreto commented on GitHub (Nov 11, 2023):

Love that this is marked as closed but everyone still clueless over here lol

<!-- gh-comment-id:1806618166 --> @itsPreto commented on GitHub (Nov 11, 2023): Love that this is marked as closed but everyone still clueless over here lol
GiteaMirror commented 2026-05-03 08:46:05 -05:00
Author
Owner
Copy Link

@orkutmuratyilmaz commented on GitHub (Nov 18, 2023):

@marscod thanks for importing the model. Can you type an example of API call, on the model page?

<!-- gh-comment-id:1817577260 --> @orkutmuratyilmaz commented on GitHub (Nov 18, 2023): @marscod thanks for importing the model. Can you type an example of API call, on [the model page](https://ollama.ai/marscod/llava)?
GiteaMirror commented 2026-05-03 08:46:06 -05:00
Author
Owner
Copy Link

@mangiucugna commented on GitHub (Nov 21, 2023):

So I figured how to use it, here's the code snippet:

with open("image.jpg", "rb") as f:
      encoded_string = base64.b64encode(f.read()).decode('utf-8')
  data = {"model": "marscod/llava", "prompt": f"USER: {encoded_string} {prompt}\nASSISTANT:", }
  try:
    response = requests.post(url="http://127.0.0.1:11434/api/generate", headers={"Content-Type": "application/json"}, json=data, stream=True)
  except Exception as e:
   # manage exception
  output = ""
  for chunk in response.text.split('\n'):
    chunk = json_repair.loads(chunk)
    if isinstance(chunk, dict):
      output += chunk.get("response") or ""

However it also throws this error: {"error":"error reading llm response: bufio.Scanner: token too long"}

For reference, I prefer using llama.cpp directly with bakllava-1 (way more precise) and the syntax there looks like this:

with open("image.jpg", "rb") as f:
      encoded_string = base64.b64encode(f.read()).decode('utf-8')
  image_data = [{"data": encoded_string, "id": 42}]
  data = {"prompt": f"USER:[img-42] {prompt}.\nASSISTANT:", "n_predict": 4000, "image_data": image_data, "stream": True}
  try:
    response = requests.post(url="http://localhost:8080/completion", headers={"Content-Type": "application/json"}, json=data, stream=True)
  except Exception as e:
    # Manage exception
  output = ""
  for chunk in response.iter_content(chunk_size=128):
    content = chunk.decode().strip().split('\n\n')[0]
    try:
        content_split = content.split('data: ')
        if len(content_split) > 1:
            content_json = json_repair.loads(content_split[1])
            output += content_json["content"]
            yield output
    except Exception as e:
       # Manage exception

This is taken from: https://github.com/mangiucugna/local_multimodal_ai

Hope this helps!

<!-- gh-comment-id:1820714015 --> @mangiucugna commented on GitHub (Nov 21, 2023): So I figured how to use it, here's the code snippet: ``` with open("image.jpg", "rb") as f: encoded_string = base64.b64encode(f.read()).decode('utf-8') data = {"model": "marscod/llava", "prompt": f"USER: {encoded_string} {prompt}\nASSISTANT:", } try: response = requests.post(url="http://127.0.0.1:11434/api/generate", headers={"Content-Type": "application/json"}, json=data, stream=True) except Exception as e: # manage exception output = "" for chunk in response.text.split('\n'): chunk = json_repair.loads(chunk) if isinstance(chunk, dict): output += chunk.get("response") or "" ``` However it also throws this error: `{"error":"error reading llm response: bufio.Scanner: token too long"}` For reference, I prefer using llama.cpp directly with bakllava-1 (way more precise) and the syntax there looks like this: ``` with open("image.jpg", "rb") as f: encoded_string = base64.b64encode(f.read()).decode('utf-8') image_data = [{"data": encoded_string, "id": 42}] data = {"prompt": f"USER:[img-42] {prompt}.\nASSISTANT:", "n_predict": 4000, "image_data": image_data, "stream": True} try: response = requests.post(url="http://localhost:8080/completion", headers={"Content-Type": "application/json"}, json=data, stream=True) except Exception as e: # Manage exception output = "" for chunk in response.iter_content(chunk_size=128): content = chunk.decode().strip().split('\n\n')[0] try: content_split = content.split('data: ') if len(content_split) > 1: content_json = json_repair.loads(content_split[1]) output += content_json["content"] yield output except Exception as e: # Manage exception ``` This is taken from: https://github.com/mangiucugna/local_multimodal_ai Hope this helps!
GiteaMirror commented 2026-05-03 08:46:07 -05:00
Author
Owner
Copy Link

@ryansereno commented on GitHub (Nov 21, 2023):

@mangiucugna thank you, will give it a try.
Hadn't heard of Bakllava before, very excited to try it.

<!-- gh-comment-id:1820815186 --> @ryansereno commented on GitHub (Nov 21, 2023): @mangiucugna thank you, will give it a try. Hadn't heard of Bakllava before, very excited to try it.
GiteaMirror commented 2026-05-03 08:46:08 -05:00
Author
Owner
Copy Link

@mangiucugna commented on GitHub (Nov 21, 2023):

I imported bakllava-1 locally and did some tests and it performs so badly when compared to the llama.cpp implementation that is unusable.
I suspect that something is going wrong and the data arriving to the model is corrupted and that somehow {"error":"error reading llm response: bufio.Scanner: token too long"} is related.

Happy to share my Modelfile and link to the gguf for anyone to try to reproduce

<!-- gh-comment-id:1821370238 --> @mangiucugna commented on GitHub (Nov 21, 2023): I imported bakllava-1 locally and did some tests and it performs so badly when compared to the llama.cpp implementation that is unusable. I suspect that something is going wrong and the data arriving to the model is corrupted and that somehow `{"error":"error reading llm response: bufio.Scanner: token too long"}` is related. Happy to share my Modelfile and link to the gguf for anyone to try to reproduce
GiteaMirror commented 2026-05-03 08:46:09 -05:00
Author
Owner
Copy Link

@Kreijstal commented on GitHub (Dec 9, 2023):

https://github.com/Mozilla-Ocho/llamafile llamafile supports llava-1.5 it would be nice if ollama supported it too

<!-- gh-comment-id:1848395530 --> @Kreijstal commented on GitHub (Dec 9, 2023): https://github.com/Mozilla-Ocho/llamafile llamafile supports llava-1.5 it would be nice if ollama supported it too
GiteaMirror commented 2026-05-03 08:46:11 -05:00
Author
Owner
Copy Link

@mak448a commented on GitHub (Dec 15, 2023):

Since this is now added, I can't figure out how to upload an image to the model. When I follow the instructions at: https://github.com/jmorganca/ollama/releases/tag/v0.1.15, it describes something completely different than what was in the picture. I'm on Linux.

<!-- gh-comment-id:1857156901 --> @mak448a commented on GitHub (Dec 15, 2023): Since this is now added, I can't figure out how to upload an image to the model. When I follow the instructions at: https://github.com/jmorganca/ollama/releases/tag/v0.1.15, it describes something completely different than what was in the picture. I'm on Linux.
GiteaMirror commented 2026-05-03 08:46:13 -05:00
Author
Owner
Copy Link

@arian81 commented on GitHub (Dec 15, 2023):

Since this is now added, I can't figure out how to upload an image to the model. When I follow the instructions at: https://github.com/jmorganca/ollama/releases/tag/v0.1.15, it describes something completely different than what was in the picture. I'm on Linux.

You probably haven't updated to the latest version of Ollama if you're getting a bunch of Chinese characters as the output.

<!-- gh-comment-id:1857193654 --> @arian81 commented on GitHub (Dec 15, 2023): > Since this is now added, I can't figure out how to upload an image to the model. When I follow the instructions at: https://github.com/jmorganca/ollama/releases/tag/v0.1.15, it describes something completely different than what was in the picture. I'm on Linux. You probably haven't updated to the latest version of Ollama if you're getting a bunch of Chinese characters as the output.
GiteaMirror commented 2026-05-03 08:46:16 -05:00
Author
Owner
Copy Link

@orkutmuratyilmaz commented on GitHub (Dec 16, 2023):

I guess that we can consider this issue as completed :)

<!-- gh-comment-id:1858747283 --> @orkutmuratyilmaz commented on GitHub (Dec 16, 2023): I guess that we can consider this issue as completed :)
GiteaMirror commented 2026-05-03 08:46:19 -05:00
Author
Owner
Copy Link

@prologic commented on GitHub (Dec 26, 2023):

When I try this I get:

$ ollama run llama2
>>> What's in this image? /Users/prologic/Downloads/IMG_1325.png

I cannot directly view or analyze the image you provided as it is a personal file located on a local computer. However, I can provide some general
information about images and how they can be analyzed.
...

And I'm using the l atest version of ollama:

$ ollama --version
ollama version is 0.1.17
<!-- gh-comment-id:1869247153 --> @prologic commented on GitHub (Dec 26, 2023): When I try this I get: ``` $ ollama run llama2 >>> What's in this image? /Users/prologic/Downloads/IMG_1325.png I cannot directly view or analyze the image you provided as it is a personal file located on a local computer. However, I can provide some general information about images and how they can be analyzed. ... ``` And I'm using the l atest version of ollama: ``` $ ollama --version ollama version is 0.1.17 ```
GiteaMirror commented 2026-05-03 08:46:22 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Dec 26, 2023):

@prologic llama2 isn't a multimodal model. You should try:

$ ollama run llava
<!-- gh-comment-id:1869291054 --> @pdevine commented on GitHub (Dec 26, 2023): @prologic llama2 isn't a multimodal model. You should try: ``` $ ollama run llava ```
GiteaMirror commented 2026-05-03 08:46:24 -05:00
Author
Owner
Copy Link

@prologic commented on GitHub (Dec 26, 2023):

Ahh! Thanks. When I tried to search for multimodel models the search turend up empty. This is why I wasn't able to figure this out so easily :/ There should be a way to list for and search for multimodel models, even with ollama search (does this sub-command exist?)

<!-- gh-comment-id:1869297313 --> @prologic commented on GitHub (Dec 26, 2023): Ahh! Thanks. When I tried to search for [multimodel models](https://ollama.ai/library?q=multimodal) the search turend up empty. This is why I wasn't able to figure this out so easily :/ There should be a way to list for and search for multimodel models, even with `ollama search` (does this sub-command exist?)
GiteaMirror commented 2026-05-03 08:46:25 -05:00
Author
Owner
Copy Link

@schuster-rainer commented on GitHub (Mar 6, 2024):

if you want to use it with langchain. here is what you need to add to the HumanMessage:

 HumanMessage(
                content=[
                    {"type": "text", "text": prompt},
                    {
                        "type": "image_url",
                        "image_url": f"data:image/jpeg;base64,{img_base64}",
                    },
                ]
            )
<!-- gh-comment-id:1980846963 --> @schuster-rainer commented on GitHub (Mar 6, 2024): if you want to use it with langchain. here is what you need to add to the HumanMessage: ```python HumanMessage( content=[ {"type": "text", "text": prompt}, { "type": "image_url", "image_url": f"data:image/jpeg;base64,{img_base64}", }, ] ) ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#62390
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?