github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8162] StructuredOutputs Schema Missing in Prompt [Unlike OpenAI API Default Behavior] #30972

New Issue
Open
opened 2026-04-22 11:01:08 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2026-04-22 11:01:08 -05:00
Owner
Copy Link

Originally created by @ikot-humanoid on GitHub (Dec 18, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/8162

When using StructuredOutputs, I noticed that the model's outputs were nonsensical and didn't align with expectations.

After debugging, I discovered that the output schema isn't included in the prompt, leaving the model unaware of its options and what it should generate. While developers could manually add the schema to the prompt, this isn't a common practice. For instance, OpenAI's API automatically includes the schema (easily verified by counting input prompt tokens), and I believe this behavior should be standard for all inference engines.

I tested Ollama in three different ways, and all of them exhibited the same behavior. Below is the code to reproduce the issue:

import aiohttp
from pydantic import BaseModel, Field
from typing import Union, List, Literal

from typing import Annotated


class Action1(BaseModel):
    type: Literal["action1"]
    __doc__: str = "store in memory"

class Action2(BaseModel):
    type: Literal["action2"]
    __doc__: str = "call someone"

class Action3(BaseModel):
    type: Literal["action3"]
    __doc__: str = "move to a location"

class Response(BaseModel):
    steps: Annotated[
        List[Union[Action1, Action2, Action3]],
        Field(..., description='Sequence of steps to perform', discriminator='type')
    ]

messages = [
     {"role": "system", "content": "Decompose input request into sequence of steps. Possible steps are listed in the response schema."},
     {"role": "user", "content": "I want to go to the park."},
]

payload = {
    "model": 'qwen2.5-coder:7b',
    "messages": messages,
    "stream": False,
    "options": {"temperature" : 0.0},
    "format": Response.model_json_schema()
}

async def get_response(payload):
    async with aiohttp.ClientSession() as session:
        async with session.post('http://localhost:11434/api/chat', json=payload) as response:
            json_response = await response.json()
            raw_string = json_response['message']['content']
            res = Response.model_validate_json(raw_string)
            return json_response, res

json_response, res = await get_response(payload)

# json_response['prompt_eval_count'] shows 40 prompt tokens
# res.model_dump() shows `{'steps': [{'type': 'action1'}, {'type': 'action2'}]}`, which doesn't make any sense
# the correct answer for "I want to go to the park." is Action 3 ("move to a location"), not 1 or 2

messages_w_schema = [
     {
         "role": "system",
         "content":
          "Decompose input request into sequence of steps. Possible steps are listed in the response schema."
          + f'\nSCHEMA:\n{Response.model_json_schema()}'
     },
     {"role": "user", "content": "I want to go to the park."},
]

payload_new = {
    "model": 'qwen2.5-coder:7b',
    "messages": messages_w_schema,
    "stream": False,
    "options": {"temperature" : 0.0},
    "format": Response.model_json_schema()
}

json_response_2, res_2 = await get_response(payload_new)

# json_response_2['prompt_eval_count'] shows 328 prompt tokens
# res_2.model_dump() correclty suggests action3

#============

from ollama import chat

chat_res_1 = chat(
    model=payload['model'],
    messages=messages,
    format=Response.model_json_schema(),
    options={'temperature': 0}
)

chat_res_2 = chat(
    model=payload['model'],
    messages=messages_w_schema,
    format=Response.model_json_schema(),
    options={'temperature': 0}
)

# chat_res_1.prompt_eval_count = 40, chat_res_2.prompt_eval_count = 328
# Response.model_validate_json(chat_res_1.message.content) again points to actions 1 & 2, not 3

#================
from openai import OpenAI
client = OpenAI(
    base_url = 'http://localhost:11434/v1',
    api_key='ollama',
)

oai_client_1 = client.beta.chat.completions.parse(
    model=payload['model'],
    messages=payload['messages'],
    response_format=Response,
    temperature=0
)

oai_client_2 = client.beta.chat.completions.parse(
    model=payload_new['model'],
    messages=payload_new['messages'],
    response_format=Response,
    temperature=0
)

# same here: oai_client_1.choices[0].message.parsed contains actions 1 & 2, not 3
# and input prompt token counts are the same

As you might see, we have 3 actions: memory, call and navigation. We ask an llm to decompose input query into sequence of actions. For this example (I want to go to the park.) we expect model to output 3rd action. There is no way for the model to understand which action to choose other than parsing that info from the output schema. Without it, model just randomly guesses, and is not able to produce anything meaningful.

====
The biggest issue I see is that even though Ollama is OpenAI compatible, it doesn't have the same logic under the hood, which might surprise a lot of developers when switching from proprietary LLMs to local ones.

Proposal: by default, add output schema to inputs, and maybe allow to disable that behaviour with a flag. Given that the functionality was added not so long ago, I believe it is important to fix it RN, before the workaround logic with system prompt patching will be widely spread.

===

My library versions:

  • pip: ollama==0.4.4
  • ollama CLI version is 0.5.1
  • pip: openai=='1.57.3'
Originally created by @ikot-humanoid on GitHub (Dec 18, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/8162 When using StructuredOutputs, I noticed that the model's outputs were nonsensical and didn't align with expectations. After debugging, I discovered that the output schema isn't included in the prompt, leaving the model unaware of its options and what it should generate. While developers could manually add the schema to the prompt, this isn't a common practice. For instance, OpenAI's API automatically includes the schema (easily verified by counting input prompt tokens), and I believe this behavior should be standard for all inference engines. I tested Ollama in three different ways, and all of them exhibited the same behavior. Below is the code to reproduce the issue: ```python import aiohttp from pydantic import BaseModel, Field from typing import Union, List, Literal from typing import Annotated class Action1(BaseModel): type: Literal["action1"] __doc__: str = "store in memory" class Action2(BaseModel): type: Literal["action2"] __doc__: str = "call someone" class Action3(BaseModel): type: Literal["action3"] __doc__: str = "move to a location" class Response(BaseModel): steps: Annotated[ List[Union[Action1, Action2, Action3]], Field(..., description='Sequence of steps to perform', discriminator='type') ] messages = [ {"role": "system", "content": "Decompose input request into sequence of steps. Possible steps are listed in the response schema."}, {"role": "user", "content": "I want to go to the park."}, ] payload = { "model": 'qwen2.5-coder:7b', "messages": messages, "stream": False, "options": {"temperature" : 0.0}, "format": Response.model_json_schema() } async def get_response(payload): async with aiohttp.ClientSession() as session: async with session.post('http://localhost:11434/api/chat', json=payload) as response: json_response = await response.json() raw_string = json_response['message']['content'] res = Response.model_validate_json(raw_string) return json_response, res json_response, res = await get_response(payload) # json_response['prompt_eval_count'] shows 40 prompt tokens # res.model_dump() shows `{'steps': [{'type': 'action1'}, {'type': 'action2'}]}`, which doesn't make any sense # the correct answer for "I want to go to the park." is Action 3 ("move to a location"), not 1 or 2 messages_w_schema = [ { "role": "system", "content": "Decompose input request into sequence of steps. Possible steps are listed in the response schema." + f'\nSCHEMA:\n{Response.model_json_schema()}' }, {"role": "user", "content": "I want to go to the park."}, ] payload_new = { "model": 'qwen2.5-coder:7b', "messages": messages_w_schema, "stream": False, "options": {"temperature" : 0.0}, "format": Response.model_json_schema() } json_response_2, res_2 = await get_response(payload_new) # json_response_2['prompt_eval_count'] shows 328 prompt tokens # res_2.model_dump() correclty suggests action3 #============ from ollama import chat chat_res_1 = chat( model=payload['model'], messages=messages, format=Response.model_json_schema(), options={'temperature': 0} ) chat_res_2 = chat( model=payload['model'], messages=messages_w_schema, format=Response.model_json_schema(), options={'temperature': 0} ) # chat_res_1.prompt_eval_count = 40, chat_res_2.prompt_eval_count = 328 # Response.model_validate_json(chat_res_1.message.content) again points to actions 1 & 2, not 3 #================ from openai import OpenAI client = OpenAI( base_url = 'http://localhost:11434/v1', api_key='ollama', ) oai_client_1 = client.beta.chat.completions.parse( model=payload['model'], messages=payload['messages'], response_format=Response, temperature=0 ) oai_client_2 = client.beta.chat.completions.parse( model=payload_new['model'], messages=payload_new['messages'], response_format=Response, temperature=0 ) # same here: oai_client_1.choices[0].message.parsed contains actions 1 & 2, not 3 # and input prompt token counts are the same ``` As you might see, we have 3 actions: memory, call and navigation. We ask an llm to decompose input query into sequence of actions. For this example (`I want to go to the park.`) we expect model to output 3rd action. There is no way for the model to understand which action to choose other than parsing that info from the output schema. Without it, model just randomly guesses, and is not able to produce anything meaningful. ==== The biggest issue I see is that even though Ollama is OpenAI compatible, it doesn't have the same logic under the hood, which might surprise a lot of developers when switching from proprietary LLMs to local ones. Proposal: by default, add output schema to inputs, and maybe allow to disable that behaviour with a flag. Given that the functionality was added not so long ago, I believe it is important to fix it RN, before the workaround logic with system prompt patching will be widely spread. === My library versions: - pip: ollama==0.4.4 - ollama CLI version is 0.5.1 - pip: openai=='1.57.3'
GiteaMirror added the feature request label 2026-04-22 11:01:08 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#30972
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?