github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 08:30:05 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #9845] Deploying using ollama has poor performance,compared to other deployment methods #6446

New Issue
Open
opened 2026-04-12 18:00:10 -05:00 by GiteaMirror · 15 comments
GiteaMirror commented 2026-04-12 18:00:10 -05:00
Owner
Copy Link

Originally created by @WR-CREATOR on GitHub (Mar 18, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/9845

What is the issue?

Recently, after trying out the official demo provided, I deployed gemma3:27b-it-fp16 and qwq: 32b-fp16 on Ollama. However, I found that the same prompt words did not work well on Ollama deployment as it could not understand my instructions and did not output in the format I needed. Later, I tried to deploy using Alibaba Cloud and was able to answer my questions with high quality, just like on the official website. This question makes me very confused. I am not using a quantitative version, why is the effect so poor.

deploy using Alibaba Cloud Result
Image
deployed using Ollama Result
Image

Relevant log output

OS

No response

GPU

No response

CPU

No response

Ollama version

No response

Originally created by @WR-CREATOR on GitHub (Mar 18, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/9845 ### What is the issue? Recently, after trying out the official demo provided, I deployed gemma3:27b-it-fp16 and qwq: 32b-fp16 on Ollama. However, I found that the same prompt words did not work well on Ollama deployment as it could not understand my instructions and did not output in the format I needed. Later, I tried to deploy using Alibaba Cloud and was able to answer my questions with high quality, just like on the official website. This question makes me very confused. I am not using a quantitative version, why is the effect so poor. deploy using Alibaba Cloud Result ![Image](https://github.com/user-attachments/assets/66a188b1-1afd-49bd-a889-1c62b613cf27) deployed using Ollama Result ![Image](https://github.com/user-attachments/assets/c3eb92ea-1e1a-4f49-a596-057821c76155) ### Relevant log output ```shell ``` ### OS _No response_ ### GPU _No response_ ### CPU _No response_ ### Ollama version _No response_
GiteaMirror added the needs more infobug labels 2026-04-12 18:00:10 -05:00
GiteaMirror commented 2026-04-12 18:00:11 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 18, 2025):

Since few people on this site read Chinese, could you summarize what is poor about the generated results? What prompt did you use, and were there additional documents?

<!-- gh-comment-id:2734102725 --> @rick-github commented on GitHub (Mar 18, 2025): Since few people on this site read Chinese, could you summarize what is poor about the generated results? What prompt did you use, and were there additional documents?
GiteaMirror commented 2026-04-12 18:00:11 -05:00
Author
Owner
Copy Link

@pdevine commented on GitHub (Mar 18, 2025):

I'm wondering if this is an issue with tokenizing? There is a similar problem that was reported w/ Korean hangul.

<!-- gh-comment-id:2734642177 --> @pdevine commented on GitHub (Mar 18, 2025): I'm wondering if this is an issue with tokenizing? There is a similar problem that was reported w/ Korean hangul.
GiteaMirror commented 2026-04-12 18:00:12 -05:00
Author
Owner
Copy Link

@tneQpx commented on GitHub (Mar 28, 2025):

I have a very similar Issue:
I give gemma3 an image and a relatively long prompt where I tell it to extract certain values from the image (with descriptions) and output it in a specific json format.
Most of the time it just completly hallucinates, or gives example values like "John Doe".

If I adjust the prompt and just ask it to extract all the text from the image it works pretty well (minor mistakes for reading numbers etc, but as expected).

If I use transformers / torch with the same prompt and image, everything works perfectly. So to me it seems that either the template or the tokenizer is suboptimal.

<!-- gh-comment-id:2760569227 --> @tneQpx commented on GitHub (Mar 28, 2025): I have a very similar Issue: I give gemma3 an image and a relatively long prompt where I tell it to extract certain values from the image (with descriptions) and output it in a specific json format. Most of the time it just completly hallucinates, or gives example values like "John Doe". If I adjust the prompt and just ask it to extract all the text from the image it works pretty well (minor mistakes for reading numbers etc, but as expected). If I use transformers / torch with the same prompt and image, everything works perfectly. So to me it seems that either the template or the tokenizer is suboptimal.
GiteaMirror commented 2026-04-12 18:00:13 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 28, 2025):

If you can provide information (logs, console runs, sample program, images, etc) it will be easier to diagnose.

<!-- gh-comment-id:2760579960 --> @rick-github commented on GitHub (Mar 28, 2025): If you can provide information (logs, console runs, sample program, images, etc) it will be easier to diagnose.
GiteaMirror commented 2026-04-12 18:00:13 -05:00
Author
Owner
Copy Link

@tneQpx commented on GitHub (Mar 28, 2025):

Image

prompt:
You are an advanced financial data extractor specialized in analyzing transaction documents.
Analyze the following images from the document. Extract the relevant information.
Only extract the fields that are provided in the json schema.
Look into the description of the fields to find the right information.
Map the extracted data to the corresponding field.

As your response will be directly passed to json.load(s), it is very important
that no other text, aside from a valid json string, is present in your response.
Do not enclose the response in backticks and do not include the tag either.

IMPORTANT: DO NOT ASK QUESTION! DO NOT OUTPUT ANYTHING EXCEPT THE JSON!

Instructions:

  1. Extract exact values for each field according to the json properties. Extract dates in this format 'yyyy-mm-dd'.
  2. List all criteria completely. If a info is not available set it to none.
  3. Output ONLY the extracted information as a valid JSON string!
  4. Follow the following JSON format structure strictly!
  5. Return only a valid JSON string following the format.
  6. The response must not contain any explanations, additional text, or formatting errors. - ONLY JSON OUTPUT!

Extraction fields: External ID - Description: A unique identifier for this dividend payment (order number / transaction number), assigned by the brokerage system. Useful for cross-referencing and tracking.
Due Date - Description: The ex date of the dividend. This is before the pay date.
Dividend Payment Date - Description: The pay date of the dividend. This is after the due date.
Dividend Record Date - Description: The date by which investors must be on the company's books to be eligible to receive the upcoming dividend. Use the same value as the Due Date!
Quantity - Description: The number of shares held (or eligible for dividend) for this dividend. Typically a whole number representing the shareholder's position, sometimes abbreviated in the documents.
Dividend per Share - Description: The per-share dividend amount paid to the shareholder. Usually quoted in the currency of the stock or market.
Dividend Amount - Description: The total price of the dividend before reducing it by fees and taxes
ISIN - Description: The International Securities Identification Number (ISIN) for the traded security. This 12-character alphanumeric code uniquely identifies tradable securities.
Bank Account Number - Description: The bank or brokerage cash account number associated with this dividend payment. This account can sometimes be found close to the value date. The document will mention something like 'payed to'.
Security Name - Description: The name of the stock or security issuing the dividend. Often the company's official name or a common identifier.
Custodian ID - Description: An identifier for the custodian, can help identify an account in aholding or bank, usually named 'depot number'. This value is usually longer than 5 characters.
Legal Entity ID - Description: An external reference or identifier for the legal entity holding the shares and receiving the dividend.
Legal Entity Name - Description: The official or recognized name of the legal entity that is entitled to receive the dividend payout.
The output should be formatted as a JSON instance that conforms to the JSON schema below.

As an example, for the schema {"properties": {"foo": {"title": "Foo", "description": "a list of strings", "type": "array", "items": {"type": "string"}}}, "required": ["foo"]}
the object {"foo": ["bar", "baz"]} is a well-formatted instance of the schema. The object {"properties": {"foo": ["bar", "baz"]}} is not well-formatted.

Here is the output schema:

{"additionalProperties": true, "properties": {"ExternalId": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "A unique identifier for this dividend payment (order number / transaction number), assigned by the brokerage system. Useful for cross-referencing and tracking.", "title": "External ID"}, "DueDate": {"anyOf": [{"format": "date", "type": "string"}, {"type": "string"}, {"type": "null"}], "default": null, "description": "The ex date of the dividend. This is before the pay date.", "title": "Due Date"}, "DividendPaymentDate": {"anyOf": [{"format": "date", "type": "string"}, {"type": "string"}, {"type": "null"}], "default": null, "description": "The pay date of the dividend. This is after the due date.", "title": "Dividend Payment Date"}, "DividendRecordDate": {"anyOf": [{"format": "date", "type": "string"}, {"type": "string"}, {"type": "null"}], "default": null, "description": "The date by which investors must be on the company's books to be eligible to receive the upcoming dividend. Use the same value as the Due Date!", "title": "Dividend Record Date"}, "Quantity": {"anyOf": [{"type": "integer"}, {"type": "null"}], "default": null, "description": "The number of shares held (or eligible for dividend) for this dividend. Typically a whole number representing the shareholder's position, sometimes abbreviated in the documents.", "title": "Quantity"}, "DividendPerShare": {"anyOf": [{"type": "number"}, {"type": "null"}], "default": null, "description": "The per-share dividend amount paid to the shareholder. Usually quoted in the currency of the stock or market.", "title": "Dividend per Share"}, "DividendAmount": {"anyOf": [{"type": "number"}, {"type": "null"}], "default": null, "description": "The total price of the dividend before reducing it by fees and taxes", "title": "Dividend Amount"}, "Security.Isin": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "The International Securities Identification Number (ISIN) for the traded security. This 12-character alphanumeric code uniquely identifies tradable securities.", "title": "ISIN"}, "BankSecurity.AccountNumber": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "The bank or brokerage cash account number associated with this dividend payment. This account can sometimes be found close to the value date. The document will mention something like 'payed to'.", "title": "Bank Account Number"}, "Security.Name": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "The name of the stock or security issuing the dividend. Often the company's official name or a common identifier.", "title": "Security Name"}, "Custodian.ExternalId": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "An identifier for the custodian, can help identify an account in aholding or bank, usually named 'depot number'. This value is usually longer than 5 characters.", "title": "Custodian ID"}, "LegalEntity.ExternalId": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "An external reference or identifier for the legal entity holding the shares and receiving the dividend.", "title": "Legal Entity ID"}, "LegalEntity.Name": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "The official or recognized name of the legal entity that is entitled to receive the dividend payout.", "title": "Legal Entity Name"}}}

`payload = {
"model": "gemma3:12b",
"messages": [
{
"role": "user",
"content": prompt,
"images": [image],
}
],
"add_generation_prompt": True,
"stream": False,
"options": {
"num_ctx": 32768,
}
}

headers = {"Content-Type": "application/json"}
response = requests.post("http://localhost:11434/api/chat", json=payload, headers=headers)`

This is how I call the api/chat endpoint.

Thank you for the fast response!

<!-- gh-comment-id:2760631665 --> @tneQpx commented on GitHub (Mar 28, 2025): ![Image](https://github.com/user-attachments/assets/acb6f018-002a-4d60-a458-4a0726048542) prompt: You are an advanced financial data extractor specialized in analyzing transaction documents. Analyze the following images from the document. Extract the relevant information. Only extract the fields that are provided in the json schema. Look into the description of the fields to find the right information. Map the extracted data to the corresponding field. As your response will be directly passed to json.load(s), it is very important that no other text, aside from a valid json string, is present in your response. Do not enclose the response in backticks and do not include the <json> tag either. IMPORTANT: DO NOT ASK QUESTION! DO NOT OUTPUT ANYTHING EXCEPT THE JSON! Instructions: 1. Extract exact values for each field according to the json properties. Extract dates in this format 'yyyy-mm-dd'. 2. List all criteria completely. If a info is not available set it to none. 3. Output ONLY the extracted information as a valid JSON string! 4. Follow the following JSON format structure strictly! 5. Return only a valid JSON string following the format. 6. The response must not contain any explanations, additional text, or formatting errors. - ONLY JSON OUTPUT! Extraction fields: External ID - Description: A unique identifier for this dividend payment (order number / transaction number), assigned by the brokerage system. Useful for cross-referencing and tracking. Due Date - Description: The ex date of the dividend. This is before the pay date. Dividend Payment Date - Description: The pay date of the dividend. This is after the due date. Dividend Record Date - Description: The date by which investors must be on the company's books to be eligible to receive the upcoming dividend. Use the same value as the Due Date! Quantity - Description: The number of shares held (or eligible for dividend) for this dividend. Typically a whole number representing the shareholder's position, sometimes abbreviated in the documents. Dividend per Share - Description: The per-share dividend amount paid to the shareholder. Usually quoted in the currency of the stock or market. Dividend Amount - Description: The total price of the dividend before reducing it by fees and taxes ISIN - Description: The International Securities Identification Number (ISIN) for the traded security. This 12-character alphanumeric code uniquely identifies tradable securities. Bank Account Number - Description: The bank or brokerage cash account number associated with this dividend payment. This account can sometimes be found close to the value date. The document will mention something like 'payed to'. Security Name - Description: The name of the stock or security issuing the dividend. Often the company's official name or a common identifier. Custodian ID - Description: An identifier for the custodian, can help identify an account in aholding or bank, usually named 'depot number'. This value is usually longer than 5 characters. Legal Entity ID - Description: An external reference or identifier for the legal entity holding the shares and receiving the dividend. Legal Entity Name - Description: The official or recognized name of the legal entity that is entitled to receive the dividend payout. The output should be formatted as a JSON instance that conforms to the JSON schema below. As an example, for the schema {"properties": {"foo": {"title": "Foo", "description": "a list of strings", "type": "array", "items": {"type": "string"}}}, "required": ["foo"]} the object {"foo": ["bar", "baz"]} is a well-formatted instance of the schema. The object {"properties": {"foo": ["bar", "baz"]}} is not well-formatted. Here is the output schema: ``` {"additionalProperties": true, "properties": {"ExternalId": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "A unique identifier for this dividend payment (order number / transaction number), assigned by the brokerage system. Useful for cross-referencing and tracking.", "title": "External ID"}, "DueDate": {"anyOf": [{"format": "date", "type": "string"}, {"type": "string"}, {"type": "null"}], "default": null, "description": "The ex date of the dividend. This is before the pay date.", "title": "Due Date"}, "DividendPaymentDate": {"anyOf": [{"format": "date", "type": "string"}, {"type": "string"}, {"type": "null"}], "default": null, "description": "The pay date of the dividend. This is after the due date.", "title": "Dividend Payment Date"}, "DividendRecordDate": {"anyOf": [{"format": "date", "type": "string"}, {"type": "string"}, {"type": "null"}], "default": null, "description": "The date by which investors must be on the company's books to be eligible to receive the upcoming dividend. Use the same value as the Due Date!", "title": "Dividend Record Date"}, "Quantity": {"anyOf": [{"type": "integer"}, {"type": "null"}], "default": null, "description": "The number of shares held (or eligible for dividend) for this dividend. Typically a whole number representing the shareholder's position, sometimes abbreviated in the documents.", "title": "Quantity"}, "DividendPerShare": {"anyOf": [{"type": "number"}, {"type": "null"}], "default": null, "description": "The per-share dividend amount paid to the shareholder. Usually quoted in the currency of the stock or market.", "title": "Dividend per Share"}, "DividendAmount": {"anyOf": [{"type": "number"}, {"type": "null"}], "default": null, "description": "The total price of the dividend before reducing it by fees and taxes", "title": "Dividend Amount"}, "Security.Isin": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "The International Securities Identification Number (ISIN) for the traded security. This 12-character alphanumeric code uniquely identifies tradable securities.", "title": "ISIN"}, "BankSecurity.AccountNumber": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "The bank or brokerage cash account number associated with this dividend payment. This account can sometimes be found close to the value date. The document will mention something like 'payed to'.", "title": "Bank Account Number"}, "Security.Name": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "The name of the stock or security issuing the dividend. Often the company's official name or a common identifier.", "title": "Security Name"}, "Custodian.ExternalId": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "An identifier for the custodian, can help identify an account in aholding or bank, usually named 'depot number'. This value is usually longer than 5 characters.", "title": "Custodian ID"}, "LegalEntity.ExternalId": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "An external reference or identifier for the legal entity holding the shares and receiving the dividend.", "title": "Legal Entity ID"}, "LegalEntity.Name": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null, "description": "The official or recognized name of the legal entity that is entitled to receive the dividend payout.", "title": "Legal Entity Name"}}} ``` `payload = { "model": "gemma3:12b", "messages": [ { "role": "user", "content": prompt, "images": [image], } ], "add_generation_prompt": True, "stream": False, "options": { "num_ctx": 32768, } } headers = {"Content-Type": "application/json"} response = requests.post("http://localhost:11434/api/chat", json=payload, headers=headers)` This is how I call the api/chat endpoint. Thank you for the fast response!
GiteaMirror commented 2026-04-12 18:00:14 -05:00
Author
Owner
Copy Link

@tneQpx commented on GitHub (Mar 28, 2025):

Now of course after anonymizing the image it works in that one case...
I hope this still helps to get a general idea of the issue. I will try to replicate it with the anonymized image

<!-- gh-comment-id:2760703519 --> @tneQpx commented on GitHub (Mar 28, 2025): Now of course after anonymizing the image it works in that one case... I hope this still helps to get a general idea of the issue. I will try to replicate it with the anonymized image
GiteaMirror commented 2026-04-12 18:00:14 -05:00
Author
Owner
Copy Link

@tneQpx commented on GitHub (Mar 28, 2025):

from the logs:
level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256

might this be an issue if the images are a little bit larger?

<!-- gh-comment-id:2760761639 --> @tneQpx commented on GitHub (Mar 28, 2025): from the logs: level=WARN source=ggml.go:149 msg="key not found" key=gemma3.mm_tokens_per_image default=256 might this be an issue if the images are a little bit larger?
GiteaMirror commented 2026-04-12 18:00:14 -05:00
Author
Owner
Copy Link

@tneQpx commented on GitHub (Mar 28, 2025):

Image

I drew a little bit on it and now it outputs dummy data again. So its extremly unstable

If I use transformed it still gives me the right json

<!-- gh-comment-id:2760826704 --> @tneQpx commented on GitHub (Mar 28, 2025): ![Image](https://github.com/user-attachments/assets/51437743-edd8-467e-a993-b61a72be2f8c) I drew a little bit on it and now it outputs dummy data again. So its extremly unstable If I use transformed it still gives me the right json
GiteaMirror commented 2026-04-12 18:00:15 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Mar 28, 2025):

A lot of the prompt is spent proscribing the model, have you considered using structured outputs? I get fairly consistent results. However, the prompt is minimal and the output would be better with a more nuanced prompt.

#!/usr/bin/env python3

from pydantic import BaseModel, Field
from typing import Optional

import ollama
import json
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("-m", "--model", default="gemma3:12b")
parser.add_argument("files", nargs='*')
args = parser.parse_args()

class FinancialData(BaseModel):
  ExternalId:                 Optional[str] = Field(..., description="A unique identifier for this dividend payment (order number / transaction number), assigned by the brokerage system. Useful for cross-referencing and tracking.")
  DueDate:                    Optional[str] = Field(..., description="The ex date of the dividend. This is before the pay date.")
  DividendPayment:            Optional[str] = Field(..., description="The pay date of the dividend. This is after the due date.")
  DividendRecordDate:         Optional[str] = Field(..., description="The date by which investors must be on the company's books to be eligible to receive the upcoming dividend. Use the same value as the Due Date!")
  Quantity:                   Optional[str] = Field(..., description="The number of shares held (or eligible for dividend) for this dividend. Typically a whole number representing the shareholder's position, sometimes abbreviated in the documents.")
  DividendPerShare:           Optional[str] = Field(..., description="The per-share dividend amount paid to the shareholder. Usually quoted in the currency of the stock or market.")
  DividendAmount:             Optional[str] = Field(..., description="The total price of the dividend before reducing it by fees and taxes")
  Security_Isin:              Optional[str] = Field(..., description="The International Securities Identification Number (ISIN) for the traded security. This 12-character alphanumeric code uniquely identifies tradable securities.")
  BankSecurity_AccountNumber: Optional[str] = Field(..., description="The bank or brokerage cash account number associated with this dividend payment. This account can sometimes be found close to the value date. The document will mention something like 'payed to'.")
  Security_Name:              Optional[str] = Field(..., description="The name of the stock or security issuing the dividend. Often the company's official name or a common identifier.")
  Custodian_ExternalId:       Optional[str] = Field(..., description="An identifier for the custodian, can help identify an account in aholding or bank, usually named 'depot number'. This value is usually longer than 5 characters.")
  LegalEntity_ExternalId:     Optional[str] = Field(..., description="An external reference or identifier for the legal entity holding the shares and receiving the dividend.")
  LegalEntity_Name:           Optional[str] = Field(..., description="The official or recognized name of the legal entity that is entitled to receive the dividend payout.")

def extract(file):
  response = ollama.Client().chat(
      model=args.model,
      messages=[
        {
          "role":"user",
          "content":"Extract the text from this image",
          "images":[file]
        }
      ],
      stream=False,
      options={
        "num_ctx": 32768,
      },
      format=FinancialData.model_json_schema(),
    )
  financial_data = FinancialData.model_validate_json(response.message.content)
  print(json.dumps(vars(financial_data), indent=4))

if __name__ == "__main__":
  for f in args.files:
    extract(f)

$ ./9845.py ing.jpg 
{
    "ExternalId": "1",
    "DueDate": "2024-03-28",
    "DividendPayment": "1.39",
    "DividendRecordDate": "2024-03-13",
    "Quantity": "5.00",
    "DividendPerShare": "0.278",
    "DividendAmount": "1.39",
    "Security_Isin": "IE0059YS762",
    "BankSecurity_AccountNumber": "123400001234123",
    "Security_Name": "Linde plc Registered Shares EO -001",
    "Custodian_ExternalId": "12345123",
    "LegalEntity_ExternalId": "123400001234123",
    "LegalEntity_Name": "Max Mustermann"
}
$ ./9845.py ing-scribble.jpg 
{
    "ExternalId": "0",
    "DueDate": "2024-03-28",
    "DividendPayment": "6.95",
    "DividendRecordDate": "2024-03-28",
    "Quantity": "5.00",
    "DividendPerShare": "1.38",
    "DividendAmount": "6.95",
    "Security_Isin": "IE00095Y5762",
    "BankSecurity_AccountNumber": "DE12 2340 0000 1234 1231",
    "Security_Name": "Linde plc Registered Shares EO",
    "Custodian_ExternalId": "1234123",
    "LegalEntity_ExternalId": "Max Mustermann",
    "LegalEntity_Name": "Max Mustermann"
}

Of course, that doesn't explain why you get better performance with transformers and the original prompt. Can you share the code you use with transformers?

<!-- gh-comment-id:2761156033 --> @rick-github commented on GitHub (Mar 28, 2025): A lot of the prompt is spent proscribing the model, have you considered using structured outputs? I get fairly consistent results. However, the prompt is minimal and the output would be better with a more nuanced prompt. ```python #!/usr/bin/env python3 from pydantic import BaseModel, Field from typing import Optional import ollama import json import argparse parser = argparse.ArgumentParser() parser.add_argument("-m", "--model", default="gemma3:12b") parser.add_argument("files", nargs='*') args = parser.parse_args() class FinancialData(BaseModel): ExternalId: Optional[str] = Field(..., description="A unique identifier for this dividend payment (order number / transaction number), assigned by the brokerage system. Useful for cross-referencing and tracking.") DueDate: Optional[str] = Field(..., description="The ex date of the dividend. This is before the pay date.") DividendPayment: Optional[str] = Field(..., description="The pay date of the dividend. This is after the due date.") DividendRecordDate: Optional[str] = Field(..., description="The date by which investors must be on the company's books to be eligible to receive the upcoming dividend. Use the same value as the Due Date!") Quantity: Optional[str] = Field(..., description="The number of shares held (or eligible for dividend) for this dividend. Typically a whole number representing the shareholder's position, sometimes abbreviated in the documents.") DividendPerShare: Optional[str] = Field(..., description="The per-share dividend amount paid to the shareholder. Usually quoted in the currency of the stock or market.") DividendAmount: Optional[str] = Field(..., description="The total price of the dividend before reducing it by fees and taxes") Security_Isin: Optional[str] = Field(..., description="The International Securities Identification Number (ISIN) for the traded security. This 12-character alphanumeric code uniquely identifies tradable securities.") BankSecurity_AccountNumber: Optional[str] = Field(..., description="The bank or brokerage cash account number associated with this dividend payment. This account can sometimes be found close to the value date. The document will mention something like 'payed to'.") Security_Name: Optional[str] = Field(..., description="The name of the stock or security issuing the dividend. Often the company's official name or a common identifier.") Custodian_ExternalId: Optional[str] = Field(..., description="An identifier for the custodian, can help identify an account in aholding or bank, usually named 'depot number'. This value is usually longer than 5 characters.") LegalEntity_ExternalId: Optional[str] = Field(..., description="An external reference or identifier for the legal entity holding the shares and receiving the dividend.") LegalEntity_Name: Optional[str] = Field(..., description="The official or recognized name of the legal entity that is entitled to receive the dividend payout.") def extract(file): response = ollama.Client().chat( model=args.model, messages=[ { "role":"user", "content":"Extract the text from this image", "images":[file] } ], stream=False, options={ "num_ctx": 32768, }, format=FinancialData.model_json_schema(), ) financial_data = FinancialData.model_validate_json(response.message.content) print(json.dumps(vars(financial_data), indent=4)) if __name__ == "__main__": for f in args.files: extract(f) ``` ```console $ ./9845.py ing.jpg { "ExternalId": "1", "DueDate": "2024-03-28", "DividendPayment": "1.39", "DividendRecordDate": "2024-03-13", "Quantity": "5.00", "DividendPerShare": "0.278", "DividendAmount": "1.39", "Security_Isin": "IE0059YS762", "BankSecurity_AccountNumber": "123400001234123", "Security_Name": "Linde plc Registered Shares EO -001", "Custodian_ExternalId": "12345123", "LegalEntity_ExternalId": "123400001234123", "LegalEntity_Name": "Max Mustermann" } $ ./9845.py ing-scribble.jpg { "ExternalId": "0", "DueDate": "2024-03-28", "DividendPayment": "6.95", "DividendRecordDate": "2024-03-28", "Quantity": "5.00", "DividendPerShare": "1.38", "DividendAmount": "6.95", "Security_Isin": "IE00095Y5762", "BankSecurity_AccountNumber": "DE12 2340 0000 1234 1231", "Security_Name": "Linde plc Registered Shares EO", "Custodian_ExternalId": "1234123", "LegalEntity_ExternalId": "Max Mustermann", "LegalEntity_Name": "Max Mustermann" } ``` Of course, that doesn't explain why you get better performance with transformers and the original prompt. Can you share the code you use with transformers?
GiteaMirror commented 2026-04-12 18:00:16 -05:00
Author
Owner
Copy Link

@tneQpx commented on GitHub (Mar 28, 2025):

I have considered structured outputs aswell as tools. The reason why I am not doing it in my current pipeline is that it was originally build using langchain and I think also started before ollama supported structured outputs.

In general I am also fine with using tools or structured outputs, but I feel like the performance difference shouldn't be that significant between Ollama and transformers, even with a suboptimal prompt.

the transformers code looks like this:

import torch
from transformers import TorchAoConfig, AutoProcessor, Gemma3ForConditionalGeneration

model_id = "google/gemma-3-12b-it"

quantization_config = TorchAoConfig("int4_weight_only")
model = Gemma3ForConditionalGeneration.from_pretrained(
	model_id,
	device_map=0,
	# device_map="auto",
	quantization_config=quantization_config,
	torch_dtype=torch.bfloat16
).eval()

processor = AutoProcessor.from_pretrained(model_id)
messages = [
	{
		"role": "system",
		"content": [{"type": "text", "text": ""}]
	},
	{
		"role": "user",
		"content": []  # Initialize content as an empty list to append image and text
	}
]

# Append images to the user's content
for image_encoded in images_encoded:
	messages[1]["content"].append(
		{"type": "image", "image": f"data:image/jpeg;base64,{image_encoded}"})

# Append the text to the user's content
messages[1]["content"].append({"type": "text", "text": prompt})

inputs = processor.apply_chat_template(
	messages, add_generation_prompt=True, tokenize=True,
	return_dict=True, return_tensors="pt", pad_to_multiple_of=8, padding="longest", padding_side="left"
).to(model.device, dtype=torch.bfloat16)

input_len = inputs["input_ids"].shape[-1]

with torch.inference_mode():
	generation = model.generate(**inputs, max_new_tokens=1000, do_sample=False)
	generation = generation[0][input_len:]

plain_llm_result = processor.decode(generation, skip_special_tokens=True)

the padding I added because there was an issue with german special characters that I send when extracting the image text and sending it in the prompt
Thank you very much for looking at this. I am still new to using llms locally and try to understand the differences :).

<!-- gh-comment-id:2761194226 --> @tneQpx commented on GitHub (Mar 28, 2025): I have considered structured outputs aswell as tools. The reason why I am not doing it in my current pipeline is that it was originally build using langchain and I think also started before ollama supported structured outputs. In general I am also fine with using tools or structured outputs, but I feel like the performance difference shouldn't be that significant between Ollama and transformers, even with a suboptimal prompt. the transformers code looks like this: ``` import torch from transformers import TorchAoConfig, AutoProcessor, Gemma3ForConditionalGeneration model_id = "google/gemma-3-12b-it" quantization_config = TorchAoConfig("int4_weight_only") model = Gemma3ForConditionalGeneration.from_pretrained( model_id, device_map=0, # device_map="auto", quantization_config=quantization_config, torch_dtype=torch.bfloat16 ).eval() processor = AutoProcessor.from_pretrained(model_id) messages = [ { "role": "system", "content": [{"type": "text", "text": ""}] }, { "role": "user", "content": [] # Initialize content as an empty list to append image and text } ] # Append images to the user's content for image_encoded in images_encoded: messages[1]["content"].append( {"type": "image", "image": f"data:image/jpeg;base64,{image_encoded}"}) # Append the text to the user's content messages[1]["content"].append({"type": "text", "text": prompt}) inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", pad_to_multiple_of=8, padding="longest", padding_side="left" ).to(model.device, dtype=torch.bfloat16) input_len = inputs["input_ids"].shape[-1] with torch.inference_mode(): generation = model.generate(**inputs, max_new_tokens=1000, do_sample=False) generation = generation[0][input_len:] plain_llm_result = processor.decode(generation, skip_special_tokens=True) ``` the padding I added because there was an issue with german special characters that I send when extracting the image text and sending it in the prompt Thank you very much for looking at this. I am still new to using llms locally and try to understand the differences :).
GiteaMirror commented 2026-04-12 18:00:17 -05:00
Author
Owner
Copy Link

@tneQpx commented on GitHub (Mar 31, 2025):

Even with structured outputs it still hallucinates a lot compared to the transformers version with the long prompt.

Are the 4bit model in ollama and transformers the same?

<!-- gh-comment-id:2765331802 --> @tneQpx commented on GitHub (Mar 31, 2025): Even with structured outputs it still hallucinates a lot compared to the transformers version with the long prompt. Are the 4bit model in ollama and transformers the same?
GiteaMirror commented 2026-04-12 18:00:17 -05:00
Author
Owner
Copy Link

@codearranger commented on GitHub (Apr 2, 2025):

What version of ollama are you using? I'm doing something similar but it breaks if I upgrade to 0.6.3 It only works reliably with 0.6.1

<!-- gh-comment-id:2773515279 --> @codearranger commented on GitHub (Apr 2, 2025): What version of ollama are you using? I'm doing something similar but it breaks if I upgrade to 0.6.3 It only works reliably with 0.6.1
GiteaMirror commented 2026-04-12 18:00:18 -05:00
Author
Owner
Copy Link

@tneQpx commented on GitHub (Apr 3, 2025):

@codearranger Used 0.6.2 and 0.6.3
I think 0.6.1 gemma didnt work with multiple images?

<!-- gh-comment-id:2774587981 --> @tneQpx commented on GitHub (Apr 3, 2025): @codearranger Used 0.6.2 and 0.6.3 I think 0.6.1 gemma didnt work with multiple images?
GiteaMirror commented 2026-04-12 18:00:18 -05:00
Author
Owner
Copy Link

@mmb78 commented on GitHub (Apr 3, 2025):

One thing that you should check is if your prompt is not over 2048 tokens. The image itself takes some tokens and on top your long prompt. If this prompt goes over the limit (2048, unless you changed the Ollama default), your prompt will be cut and you get garbage back. Test if running it with 8k context window would not help.

<!-- gh-comment-id:2774766166 --> @mmb78 commented on GitHub (Apr 3, 2025): One thing that you should check is if your prompt is not over 2048 tokens. The image itself takes some tokens and on top your long prompt. If this prompt goes over the limit (2048, unless you changed the Ollama default), your prompt will be cut and you get garbage back. Test if running it with 8k context window would not help.
GiteaMirror commented 2026-04-12 18:00:19 -05:00
Author
Owner
Copy Link

@tneQpx commented on GitHub (Apr 3, 2025):

@mmb78 I added the num_ctx, is that enough? I am also running ollama in a docker container and set the context setting. Sadly both things didnt help

<!-- gh-comment-id:2775136829 --> @tneQpx commented on GitHub (Apr 3, 2025): @mmb78 I added the num_ctx, is that enough? I am also running ollama in a docker container and set the context setting. Sadly both things didnt help
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug needs more info
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#6446
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?