github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-08 00:51:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #2384] [Bug]: Not able to make call to Ollama multimodal model in the cookbook #1384

New Issue
Closed
opened 2026-04-12 11:13:00 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-12 11:13:00 -05:00
Owner
Copy Link

Originally created by @AutoAPEAI on GitHub (Feb 7, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/2384

Bug Description

When I trying to follow the steps int the cookbook, I was getting an error saying that there's a ResponseError: calling Ollama. However, I have installed Ollama successfully and was able to make model call via curl command. I suspect that this has something to do with http://localhost:11434.

Version

llama-index 0.9.44
ollama version is 0.1.23

Steps to Reproduce

Following the steps in the cookbook

from pydantic import BaseModel

class Restaurant(BaseModel):
    """Data model for an restaurant."""

    restaurant: str
    food: str
    discount: str
    price: str
    rating: str
    review: str

from llama_index.program import MultiModalLLMCompletionProgram
from llama_index.output_parsers import PydanticOutputParser

prompt_template_str = """\
{query_str}

Return the answer as a Pydantic object. The Pydantic schema is given below:

"""
mm_program = MultiModalLLMCompletionProgram.from_defaults(
    output_parser=PydanticOutputParser(Restaurant),
    image_documents=image_documents,
    prompt_template_str=prompt_template_str,
    multi_modal_llm=mm_model,
    verbose=True,
)

response = mm_program(query_str="Can you summarize what is in the image?")
for res in response:
    print(res)

Relevant Logs/Tracbacks

{
"name": "ResponseError",
"message": "",
"stack": "---------------------------------------------------------------------------
ResponseError Traceback (most recent call last)
Cell In[13], line 1
----> 1 response = mm_program(query_str="Can you summarize what is in the image?")
2 for res in response:
3 print(res)

File /usr/local/lib/python3.11/site-packages/llama_index/program/multi_modal_llm_program.py:85, in MultiModalLLMCompletionProgram.call(self, llm_kwargs, *args, **kwargs)
82 llm_kwargs = llm_kwargs or {}
83 formatted_prompt = self._prompt.format(llm=self._multi_modal_llm, **kwargs)
---> 85 response = self._multi_modal_llm.complete(
86 formatted_prompt,
87 image_documents=self._image_documents,
88 **llm_kwargs,
89 )
91 raw_output = response.text
92 if self._verbose:

File /usr/local/lib/python3.11/site-packages/llama_index/multi_modal_llms/ollama.py:158, in OllamaMultiModal.complete(self, prompt, image_documents, formatted, **kwargs)
155 """Complete."""
156 import ollama
--> 158 response = ollama.generate(
159 model=self.model,
160 prompt=prompt,
161 images=image_documents_to_base64(image_documents),
162 stream=False,
163 options=self._model_kwargs,
164 )
165 return CompletionResponse(
166 text=response["response"],
167 raw=response,
168 additional_kwargs=get_additional_kwargs(response, ("response",)),
169 )

File /usr/local/lib/python3.11/site-packages/ollama/_client.py:111, in Client.generate(self, model, prompt, system, template, context, stream, raw, format, images, options, keep_alive)
108 if not model:
109 raise RequestError('must provide a model')
--> 111 return self._request_stream(
112 'POST',
113 '/api/generate',
114 json={
115 'model': model,
116 'prompt': prompt,
117 'system': system,
118 'template': template,
119 'context': context or [],
120 'stream': stream,
121 'raw': raw,
122 'images': [_encode_image(image) for image in images or []],
123 'format': format,
124 'options': options or {},
125 'keep_alive': keep_alive,
126 },
127 stream=stream,
128 )

File /usr/local/lib/python3.11/site-packages/ollama/_client.py:82, in Client._request_stream(self, stream, *args, **kwargs)
76 def _request_stream(
77 self,
78 *args,
79 stream: bool = False,
80 **kwargs,
81 ) -> Union[Mapping[str, Any], Iterator[Mapping[str, Any]]]:
---> 82 return self._stream(*args, **kwargs) if stream else self._request(*args, **kwargs).json()

File /usr/local/lib/python3.11/site-packages/ollama/_client.py:58, in Client._request(self, method, url, **kwargs)
56 response.raise_for_status()
57 except httpx.HTTPStatusError as e:
---> 58 raise ResponseError(e.response.text, e.response.status_code) from None
60 return response

ResponseError: "
}

Originally created by @AutoAPEAI on GitHub (Feb 7, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/2384 ### Bug Description When I trying to follow the steps int the [cookbook](https://docs.llamaindex.ai/en/latest/examples/multi_modal/ollama_cookbook.html#), I was getting an error saying that there's a ResponseError: calling Ollama. However, I have installed Ollama successfully and was able to make model call via curl command. I suspect that this has something to do with http://localhost:11434. ### Version llama-index 0.9.44 ollama version is 0.1.23 ### Steps to Reproduce Following the steps in the [cookbook](https://docs.llamaindex.ai/en/latest/examples/multi_modal/ollama_cookbook.html#) ``` from pydantic import BaseModel class Restaurant(BaseModel): """Data model for an restaurant.""" restaurant: str food: str discount: str price: str rating: str review: str from llama_index.program import MultiModalLLMCompletionProgram from llama_index.output_parsers import PydanticOutputParser prompt_template_str = """\ {query_str} Return the answer as a Pydantic object. The Pydantic schema is given below: """ mm_program = MultiModalLLMCompletionProgram.from_defaults( output_parser=PydanticOutputParser(Restaurant), image_documents=image_documents, prompt_template_str=prompt_template_str, multi_modal_llm=mm_model, verbose=True, ) response = mm_program(query_str="Can you summarize what is in the image?") for res in response: print(res) ``` ### Relevant Logs/Tracbacks { "name": "ResponseError", "message": "", "stack": "--------------------------------------------------------------------------- ResponseError Traceback (most recent call last) Cell In[13], line 1 ----> 1 response = mm_program(query_str=\"Can you summarize what is in the image?\") 2 for res in response: 3 print(res) File /usr/local/lib/python3.11/site-packages/llama_index/program/multi_modal_llm_program.py:85, in MultiModalLLMCompletionProgram.__call__(self, llm_kwargs, *args, **kwargs) 82 llm_kwargs = llm_kwargs or {} 83 formatted_prompt = self._prompt.format(llm=self._multi_modal_llm, **kwargs) ---> 85 response = self._multi_modal_llm.complete( 86 formatted_prompt, 87 image_documents=self._image_documents, 88 **llm_kwargs, 89 ) 91 raw_output = response.text 92 if self._verbose: File /usr/local/lib/python3.11/site-packages/llama_index/multi_modal_llms/ollama.py:158, in OllamaMultiModal.complete(self, prompt, image_documents, formatted, **kwargs) 155 \"\"\"Complete.\"\"\" 156 import ollama --> 158 response = ollama.generate( 159 model=self.model, 160 prompt=prompt, 161 images=image_documents_to_base64(image_documents), 162 stream=False, 163 options=self._model_kwargs, 164 ) 165 return CompletionResponse( 166 text=response[\"response\"], 167 raw=response, 168 additional_kwargs=get_additional_kwargs(response, (\"response\",)), 169 ) File /usr/local/lib/python3.11/site-packages/ollama/_client.py:111, in Client.generate(self, model, prompt, system, template, context, stream, raw, format, images, options, keep_alive) 108 if not model: 109 raise RequestError('must provide a model') --> 111 return self._request_stream( 112 'POST', 113 '/api/generate', 114 json={ 115 'model': model, 116 'prompt': prompt, 117 'system': system, 118 'template': template, 119 'context': context or [], 120 'stream': stream, 121 'raw': raw, 122 'images': [_encode_image(image) for image in images or []], 123 'format': format, 124 'options': options or {}, 125 'keep_alive': keep_alive, 126 }, 127 stream=stream, 128 ) File /usr/local/lib/python3.11/site-packages/ollama/_client.py:82, in Client._request_stream(self, stream, *args, **kwargs) 76 def _request_stream( 77 self, 78 *args, 79 stream: bool = False, 80 **kwargs, 81 ) -> Union[Mapping[str, Any], Iterator[Mapping[str, Any]]]: ---> 82 return self._stream(*args, **kwargs) if stream else self._request(*args, **kwargs).json() File /usr/local/lib/python3.11/site-packages/ollama/_client.py:58, in Client._request(self, method, url, **kwargs) 56 response.raise_for_status() 57 except httpx.HTTPStatusError as e: ---> 58 raise ResponseError(e.response.text, e.response.status_code) from None 60 return response ResponseError: " }
GiteaMirror commented 2026-04-12 11:13:01 -05:00
Author
Owner
Copy Link

@LucasSchelkes-BA commented on GitHub (Feb 18, 2024):

Im behind a corporate firewall and Im having a similar error message.

What I did was:

  1. Install ollama into environment via pip install ollama
  2. Run ollama serve in one terminal, and ollama pull mistral in another

Then the error message

Error: pull model manifest: Get "https://registry.ollama.ai/v2/library/help/manifests/latest": authenticationrequired

appears.

Adding the host via OLLAMA_HOST="127.0.0.1:11434"`` or running ollama pull mistral --insecure` both didn't help either.

I've already researched about "authenticationrequired" errors, but doesn't seem to be occur often. Im guessing I might have to add some token or password, but I cannot find any insturctions on this. What can I do?

<!-- gh-comment-id:1951353585 --> @LucasSchelkes-BA commented on GitHub (Feb 18, 2024): Im behind a corporate firewall and Im having a similar error message. What I did was: 1. Install ollama into environment via `pip install ollama` 2. Run `ollama serve` in one terminal, and `ollama pull mistral` in another Then the error message ```pulling manifest Error: pull model manifest: Get "https://registry.ollama.ai/v2/library/help/manifests/latest": authenticationrequired ``` appears. Adding the host via `OLLAMA_HOST="127.0.0.1:11434"`` or running `ollama pull mistral --insecure` both didn't help either. I've already researched about "authenticationrequired" errors, but doesn't seem to be occur often. Im guessing I might have to add some token or password, but I cannot find any insturctions on this. What can I do?
GiteaMirror commented 2026-04-12 11:13:01 -05:00
Author
Owner
Copy Link

@bmizerany commented on GitHub (Mar 11, 2024):

These seems like a dup of https://github.com/ollama/ollama/issues/2424#issuecomment-1961173682

It seems using the official Ollama python client fixes the issue.

Closing as a dup of https://github.com/ollama/ollama/issues/2424#issuecomment-1961173682

<!-- gh-comment-id:1989431239 --> @bmizerany commented on GitHub (Mar 11, 2024): These seems like a dup of https://github.com/ollama/ollama/issues/2424#issuecomment-1961173682 It seems using the official Ollama python client fixes the issue. Closing as a dup of https://github.com/ollama/ollama/issues/2424#issuecomment-1961173682
GiteaMirror commented 2026-04-12 11:13:01 -05:00
Author
Owner
Copy Link

@zhangjy328 commented on GitHub (Dec 5, 2024):

ImportError: cannot import name '_encode_image' from 'ollama._client'

<!-- gh-comment-id:2519748757 --> @zhangjy328 commented on GitHub (Dec 5, 2024): ImportError: cannot import name '_encode_image' from 'ollama._client'
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1384
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?