[GH-ISSUE #5462] Inaccurate reasoning of multimodal (llava) using HTTP #3418

New Issue

GiteaMirror · 2026-04-12T14:03:40-05:00

GiteaMirror commented

2026-04-12 14:03:40 -05:00

Originally created by @peanutpaste on GitHub (Jul 3, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5462

What is the issue?

I passed the image path directly to the command line through cmd and asked llava to describe the image, which was very accurate. But when I called llava through http (http://127.0.0.1:11434/v1), its description was incorrect. How can I make it maintain the same inference effect as cmd? Is it because of my base64 conversion problem?

this is cmd：

this is http：

this is my img2base64 code：
        with open(image_path, "rb") as f:
            encoded_string = base64.b64encode(f.read()).decode("utf-8")
            return encoded_string

OS

No response

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @peanutpaste on GitHub (Jul 3, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5462 ### What is the issue? I passed the image path directly to the command line through cmd and asked llava to describe the image, which was very accurate. But when I called llava through http (http://127.0.0.1:11434/v1), its description was incorrect. How can I make it maintain the same inference effect as cmd? Is it because of my base64 conversion problem? <img width="183" alt="dog" src="https://github.com/ollama/ollama/assets/62008312/fac417e1-68f2-46cb-b2e8-259a22594b7b"> this is cmd： <img width="729" alt="aaaa" src="https://github.com/ollama/ollama/assets/62008312/4aff72bc-395e-4256-822d-fd0512811826"> this is http： <img width="726" alt="Snipaste_2024-07-03_21-07-39" src="https://github.com/ollama/ollama/assets/62008312/baab806c-268d-4d97-b294-0db66127988e"> ``` this is my img2base64 code： with open(image_path, "rb") as f: encoded_string = base64.b64encode(f.read()).decode("utf-8") return encoded_string ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_

GiteaMirror added the bug label 2026-04-12 14:03:40 -05:00

GiteaMirror closed this issue

2026-04-12 14:03:40 -05:00

GiteaMirror commented

2026-04-12 14:03:41 -05:00

@rick-github commented on GitHub (Jul 3, 2024):

What was the payload (prompt, messages, images array, etc) that you sent via http?

@rick-github commented on GitHub (Jul 3, 2024): What was the payload (`prompt`, `messages`, `images` array, etc) that you sent via http?

GiteaMirror commented

2026-04-12 14:03:42 -05:00

@rick-github commented on GitHub (Jul 3, 2024):

After some experimentation, what I think you are experiencing is the lack of image support for /v1 endpoints. You can either use the ollama /api/chat endpoint or wait until https://github.com/ollama/ollama/pull/5208 is merged.

$ echo '{"model": "llava","messages":[{"role":"user","content":"please describe this image","images": ["'"$(base64 puppy.png)"'"]}],"format": "","options": {},"stream":false}' | curl http://ollama:11434/api/chat -d @-
{"model":"llava","created_at":"2024-07-03T21:48:16.61858311Z","message":{"role":"assistant","content":" In the image, there is a small puppy with white fur sitting on a step. The puppy has its head tilted to the side and appears to be looking at something off-camera. It's wearing a red collar with a bell. Behind the puppy, there's a building with a dark-colored facade that contrasts with the light color of the dog's fur. The image is clear and well-lit, allowing for sharp details of the puppy and its surroundings. The puppy seems to be indoors, possibly in an urban or residential area, as indicated by the concrete step and the building structure in the background. "},"done_reason":"stop","done":true,"total_duration":2035306349,"load_duration":5257921,"prompt_eval_count":1,"prompt_eval_duration":286873000,"eval_count":137,"eval_duration":1591851000}

@rick-github commented on GitHub (Jul 3, 2024): After some experimentation, what I think you are experiencing is the lack of image support for `/v1` endpoints. You can either use the ollama `/api/chat` endpoint or wait until https://github.com/ollama/ollama/pull/5208 is merged. ``` $ echo '{"model": "llava","messages":[{"role":"user","content":"please describe this image","images": ["'"$(base64 puppy.png)"'"]}],"format": "","options": {},"stream":false}' | curl http://ollama:11434/api/chat -d @- {"model":"llava","created_at":"2024-07-03T21:48:16.61858311Z","message":{"role":"assistant","content":" In the image, there is a small puppy with white fur sitting on a step. The puppy has its head tilted to the side and appears to be looking at something off-camera. It's wearing a red collar with a bell. Behind the puppy, there's a building with a dark-colored facade that contrasts with the light color of the dog's fur. The image is clear and well-lit, allowing for sharp details of the puppy and its surroundings. The puppy seems to be indoors, possibly in an urban or residential area, as indicated by the concrete step and the building structure in the background. "},"done_reason":"stop","done":true,"total_duration":2035306349,"load_duration":5257921,"prompt_eval_count":1,"prompt_eval_duration":286873000,"eval_count":137,"eval_duration":1591851000} ```

GiteaMirror commented

2026-04-12 14:03:43 -05:00

@peanutpaste commented on GitHub (Jul 4, 2024):

I've tried all the endpoints I can find, but I still can't get the same output as in cmd.
Here's my code：

# MULTIMODEL
MUMULTIMODAL_URL=http://10.66.78.38:11434/api/chat/
（I've tried all the endpoints I can find ）
MUMULTIMODAL_KEY=ollama

# -*- coding: utf-8 -*-
import base64
import os
from io import BytesIO
from typing import Optional
from dotenv import load_dotenv
from openai import OpenAI


def image_to_base64(image_path: str) -> Optional[str]:
    """image2b64
    Args:
        image_path (_type_): image path
    """
    try:
        with open(image_path, "rb") as f:
            encoded_string = base64.b64encode(f.read()).decode("utf-8")
            return encoded_string

    except IOError as e:
        print(f"can not open or read images: {e}")
        return None

class multiModalSummary(object):
    """multiModal summary images
    Args:
        object (_type_): base64 style iamge
    """

    def __init__(self) -> None:
        load_dotenv()
        self.base_url: str = os.getenv("MUMULTIMODAL_URL")
        self.api_key: str = os.getenv("MUMULTIMODAL_KEY")
        self.model: str = "llava:latest"
        self.client = OpenAI(base_url=self.base_url, api_key=self.api_key)

    def getMessage(self, b64_image: str) -> list:
        message = [
            {"role": "user", "content": "please describe this image", "images": [b64_image]},
        ]
        return message

    def getAnswer(self, b64_image: str) -> str:
        chat_completion = self.client.chat.completions.create(model=self.model, messages=self.getMessage(b64_image))
        return chat_completion.choices[0].message.content


if __name__ == "__main__":
    # example
    image_path = "C:/Users/peanut/Desktop/dog.png"
    b64Image = image_to_base64(image_path)
    imgSummaryUtils = multiModalSummary()
    imageSummary = imgSummaryUtils.getAnswer(b64Image)
    print(imageSummary)

@rick-github

@peanutpaste commented on GitHub (Jul 4, 2024): I've tried all the endpoints I can find, but I still can't get the same output as in cmd. Here's my code： ``` # MULTIMODEL MUMULTIMODAL_URL=http://10.66.78.38:11434/api/chat/ （I've tried all the endpoints I can find ） MUMULTIMODAL_KEY=ollama # -*- coding: utf-8 -*- import base64 import os from io import BytesIO from typing import Optional from dotenv import load_dotenv from openai import OpenAI def image_to_base64(image_path: str) -> Optional[str]: """image2b64 Args: image_path (_type_): image path """ try: with open(image_path, "rb") as f: encoded_string = base64.b64encode(f.read()).decode("utf-8") return encoded_string except IOError as e: print(f"can not open or read images: {e}") return None class multiModalSummary(object): """multiModal summary images Args: object (_type_): base64 style iamge """ def __init__(self) -> None: load_dotenv() self.base_url: str = os.getenv("MUMULTIMODAL_URL") self.api_key: str = os.getenv("MUMULTIMODAL_KEY") self.model: str = "llava:latest" self.client = OpenAI(base_url=self.base_url, api_key=self.api_key) def getMessage(self, b64_image: str) -> list: message = [ {"role": "user", "content": "please describe this image", "images": [b64_image]}, ] return message def getAnswer(self, b64_image: str) -> str: chat_completion = self.client.chat.completions.create(model=self.model, messages=self.getMessage(b64_image)) return chat_completion.choices[0].message.content if __name__ == "__main__": # example image_path = "C:/Users/peanut/Desktop/dog.png" b64Image = image_to_base64(image_path) imgSummaryUtils = multiModalSummary() imageSummary = imgSummaryUtils.getAnswer(b64Image) print(imageSummary) ``` @rick-github

GiteaMirror commented

2026-04-12 14:03:44 -05:00

@rick-github commented on GitHub (Jul 4, 2024):

The problem is that the OpenAI module adds stuff to the base url. It's easier to use the Ollama module instead:

# -*- coding: utf-8 -*-
import base64
import os
from io import BytesIO
from typing import Optional
from dotenv import load_dotenv
from ollama import Client


def image_to_base64(image_path: str) -> Optional[str]:
    """image2b64
    Args:
        image_path (_type_): image path
    """
    try:
        with open(image_path, "rb") as f:
            encoded_string = base64.b64encode(f.read()).decode("utf-8")
            return encoded_string

    except IOError as e:
        print(f"can not open or read images: {e}")
        return None

class multiModalSummary(object):
    """multiModal summary images
    Args:
        object (_type_): base64 style iamge
    """

    def __init__(self) -> None:
        load_dotenv()
        self.base_url: str = os.getenv("MUMULTIMODAL_URL")
        self.api_key: str = os.getenv("MUMULTIMODAL_KEY")
        self.model: str = "llava:latest"
        self.client = Client(host=self.base_url)

    def getMessage(self, b64_image: str) -> list:
        message = [
            {"role": "user", "content": "please describe this image", "images": [b64_image]},
        ]
        return message

    def getAnswer(self, b64_image: str) -> str:
        chat_completion = self.client.chat(model=self.model, messages=self.getMessage(b64_image))
        return chat_completion["message"]["content"]


if __name__ == "__main__":
    # example
#image_path = "C:/Users/peanut/Desktop/dog.png"
    image_path = "dog.png"
    b64Image = image_to_base64(image_path)
    imgSummaryUtils = multiModalSummary()
    imageSummary = imgSummaryUtils.getAnswer(b64Image)
    print(imageSummary)

$ MUMULTIMODAL_URL="http://localhost:11434/" MUMULTIMODAL_KEY="ollama" ./x.py  
 The image shows a small white puppy sitting on the edge of a stone step. The puppy appears to be looking directly at the camera, with its ears perked up in curiosity or alertness. It is wearing what seems to be a red collar around its neck, and there is a small bell attached to it, which is common for pet safety and identification. The puppy has a fluffy coat, suggesting that it might have long hair.

In the background, you can see a blurred view of what looks like an outdoor area with a building or structure, possibly indicating that this photo was taken in an urban environment or a residential area with some architectural features visible. The focus is on the puppy itself, with the background being out of focus to emphasize the subject. There are no texts or distinguishable brands in the image.

@rick-github commented on GitHub (Jul 4, 2024): The problem is that the OpenAI module adds stuff to the base url. It's easier to use the Ollama module instead: ``` # -*- coding: utf-8 -*- import base64 import os from io import BytesIO from typing import Optional from dotenv import load_dotenv from ollama import Client def image_to_base64(image_path: str) -> Optional[str]: """image2b64 Args: image_path (_type_): image path """ try: with open(image_path, "rb") as f: encoded_string = base64.b64encode(f.read()).decode("utf-8") return encoded_string except IOError as e: print(f"can not open or read images: {e}") return None class multiModalSummary(object): """multiModal summary images Args: object (_type_): base64 style iamge """ def __init__(self) -> None: load_dotenv() self.base_url: str = os.getenv("MUMULTIMODAL_URL") self.api_key: str = os.getenv("MUMULTIMODAL_KEY") self.model: str = "llava:latest" self.client = Client(host=self.base_url) def getMessage(self, b64_image: str) -> list: message = [ {"role": "user", "content": "please describe this image", "images": [b64_image]}, ] return message def getAnswer(self, b64_image: str) -> str: chat_completion = self.client.chat(model=self.model, messages=self.getMessage(b64_image)) return chat_completion["message"]["content"] if __name__ == "__main__": # example #image_path = "C:/Users/peanut/Desktop/dog.png" image_path = "dog.png" b64Image = image_to_base64(image_path) imgSummaryUtils = multiModalSummary() imageSummary = imgSummaryUtils.getAnswer(b64Image) print(imageSummary) ``` ``` $ MUMULTIMODAL_URL="http://localhost:11434/" MUMULTIMODAL_KEY="ollama" ./x.py The image shows a small white puppy sitting on the edge of a stone step. The puppy appears to be looking directly at the camera, with its ears perked up in curiosity or alertness. It is wearing what seems to be a red collar around its neck, and there is a small bell attached to it, which is common for pet safety and identification. The puppy has a fluffy coat, suggesting that it might have long hair. In the background, you can see a blurred view of what looks like an outdoor area with a building or structure, possibly indicating that this photo was taken in an urban environment or a residential area with some architectural features visible. The focus is on the puppy itself, with the background being out of focus to emphasize the subject. There are no texts or distinguishable brands in the image. ```

GiteaMirror commented

2026-04-12 14:03:45 -05:00

@jmorganca commented on GitHub (Jul 4, 2024):

Hi there, sorry you hit this. As others mentioned the OpenAI-compatible endpoint doesn't support vision models but it's being worked on: https://github.com/ollama/ollama/pull/5208

Closing for https://github.com/ollama/ollama/issues/3690

@jmorganca commented on GitHub (Jul 4, 2024): Hi there, sorry you hit this. As others mentioned the OpenAI-compatible endpoint doesn't support vision models but it's being worked on: https://github.com/ollama/ollama/pull/5208 Closing for https://github.com/ollama/ollama/issues/3690

GiteaMirror commented

2026-04-12 14:03:46 -05:00

@peanutpaste commented on GitHub (Jul 8, 2024):

It actually worked, thank you very much！
@rick-github

@peanutpaste commented on GitHub (Jul 8, 2024): It actually worked, thank you very much！ @rick-github