github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-30 01:17:39 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #8444] Ollama not respecting structured outputs with some ordering of refs #31190

New Issue
Open
opened 2026-04-22 11:25:14 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-22 11:25:14 -05:00
Owner
Copy Link

Originally created by @andygill on GitHub (Jan 16, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/8444

What is the issue?

The following program fails, but should work.

Changing the schema = ... line to use schema_works, which is a slightly different schema, works. The two schemas should parse the same JSON - the only difference is the ordering in "$defs".

I also have examples of pydantic-generated schemas which fail because of this bug.

It looks like the json_schema_to_grammar implementation has a "binding problem", that is it binds a def in _refs before visiting it, expecting a "input#... prefix, and consequently generates an incorrect grammar.

from ollama import chat
import json
from jsonschema import validate, ValidationError

#
# Schema originally generated by pydantic. Depending
# on the alphabetical order of the names of the classes,
# both failing and working schemas can be generated.
# This is the smallest example I could find, and
# I'm using client version is 0.5.5.
#
# The *only* difference is the order of "C" and "A" in "$defs",
# This should not make any difference: both are supposed to be the
# same schema.
schema_fails = {
    "$defs": {
        "C": {
            "properties": {"payload": {"$ref": "#/$defs/A"}},
            "required": ["payload"],
            "title": "C",
            "type": "object",
        },
        "A": {
            "properties": {"payload": {"$ref": "#/$defs/E"}},
            "required": ["payload"],
            "title": "A",
            "type": "object",
        },
        "E": {"type": "boolean"},
    },
    "properties": {"payload": {"$ref": "#/$defs/C"}},
    "required": ["payload"],
    "title": "B",
    "type": "object",
}

schema_works = {
    "$defs": {
        "A": {
            "properties": {"payload": {"$ref": "#/$defs/E"}},
            "required": ["payload"],
            "title": "A",
            "type": "object",
        },
        "C": {
            "properties": {"payload": {"$ref": "#/$defs/A"}},
            "required": ["payload"],
            "title": "C",
            "type": "object",
        },
        "E": {"type": "boolean"},
    },
    "properties": {"payload": {"$ref": "#/$defs/C"}},
    "required": ["payload"],
    "title": "B",
    "type": "object",
}

schema = schema_fails  # or schema_works
# schema_fails generates {"payload": {"payload": {"payload": {"payload": "true"}}}}
# schema_works generates {"payload": {"payload": {"payload": true}}}

response = chat(
    model="mistral-nemo:latest",
    messages=[
        {
            "role": "user",
            "content": "I want to choose true, inside A, inside C, inside B",
        }
    ],
    format=schema,
    options={"temperature": 0},  # Make responses more deterministic
)
valid_json = json.loads(response.message.content)
print(json.dumps(valid_json))
validate(instance=valid_json, schema=schema)
print("validates!")

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

0.5.5

Originally created by @andygill on GitHub (Jan 16, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/8444 ### What is the issue? The following program fails, but should work. Changing the `schema = ...` line to use `schema_works`, which is a slightly different schema, works. The two schemas should parse the same JSON - the only difference is the ordering in `"$defs"`. I also have examples of pydantic-generated schemas which fail because of this bug. It looks like the `json_schema_to_grammar` implementation has a "binding problem", that is it binds a def in `_refs` before visiting it, expecting a `"input#...` prefix, and consequently generates an incorrect grammar. ```python from ollama import chat import json from jsonschema import validate, ValidationError # # Schema originally generated by pydantic. Depending # on the alphabetical order of the names of the classes, # both failing and working schemas can be generated. # This is the smallest example I could find, and # I'm using client version is 0.5.5. # # The *only* difference is the order of "C" and "A" in "$defs", # This should not make any difference: both are supposed to be the # same schema. schema_fails = { "$defs": { "C": { "properties": {"payload": {"$ref": "#/$defs/A"}}, "required": ["payload"], "title": "C", "type": "object", }, "A": { "properties": {"payload": {"$ref": "#/$defs/E"}}, "required": ["payload"], "title": "A", "type": "object", }, "E": {"type": "boolean"}, }, "properties": {"payload": {"$ref": "#/$defs/C"}}, "required": ["payload"], "title": "B", "type": "object", } schema_works = { "$defs": { "A": { "properties": {"payload": {"$ref": "#/$defs/E"}}, "required": ["payload"], "title": "A", "type": "object", }, "C": { "properties": {"payload": {"$ref": "#/$defs/A"}}, "required": ["payload"], "title": "C", "type": "object", }, "E": {"type": "boolean"}, }, "properties": {"payload": {"$ref": "#/$defs/C"}}, "required": ["payload"], "title": "B", "type": "object", } schema = schema_fails # or schema_works # schema_fails generates {"payload": {"payload": {"payload": {"payload": "true"}}}} # schema_works generates {"payload": {"payload": {"payload": true}}} response = chat( model="mistral-nemo:latest", messages=[ { "role": "user", "content": "I want to choose true, inside A, inside C, inside B", } ], format=schema, options={"temperature": 0}, # Make responses more deterministic ) valid_json = json.loads(response.message.content) print(json.dumps(valid_json)) validate(instance=valid_json, schema=schema) print("validates!") ``` ### OS macOS ### GPU Apple ### CPU Apple ### Ollama version 0.5.5
GiteaMirror added the bug label 2026-04-22 11:25:14 -05:00
GiteaMirror commented 2026-04-22 11:25:15 -05:00
Author
Owner
Copy Link

@andygill commented on GitHub (Jan 16, 2025):

Two quick solutions.

Solution 1:

In json-schema-to-grammar.cpp, adding a visit_refs before the assignment into _refs fixes the problem.

The issue is that

  • target might not have been visited yet, and
  • target is a copy of what is being walked by the visitor pattern, because of the value semantics of json,
  • schema, what is being walked, is getting refs updated from "#..." to "input#...".
visit_refs(target); // Resolve any refs in the referenced schema
_refs[ref] = target;

This fix might result in needless walks over the JSON schema, but should do no harm, and there is a deep copy happening anyway.

Solution 2:

Another fix is to remove the use of "input" as the input URI (which I think is non-standard anyway).

 converter.resolve_refs(copy, ""); // was "input"
<!-- gh-comment-id:2594510700 --> @andygill commented on GitHub (Jan 16, 2025): Two quick solutions. # Solution 1: In `json-schema-to-grammar.cpp`, adding a `visit_refs` before the assignment into `_refs` fixes the problem. The issue is that * `target` might not have been visited yet, and * `target` is a copy of what is being walked by the visitor pattern, because of the value semantics of `json`, * `schema`, what is being walked, is getting refs updated from "#..." to "input#...". ```cpp visit_refs(target); // Resolve any refs in the referenced schema _refs[ref] = target; ``` This fix might result in needless walks over the JSON schema, but should do no harm, and there is a deep copy happening anyway. # Solution 2: Another fix is to remove the use of "input" as the input URI (which I think is non-standard anyway). ``` converter.resolve_refs(copy, ""); // was "input" ```
GiteaMirror commented 2026-04-22 11:25:16 -05:00
Author
Owner
Copy Link

@paztis commented on GitHub (Jul 25, 2025):

any update on this ?

<!-- gh-comment-id:3116977100 --> @paztis commented on GitHub (Jul 25, 2025): any update on this ?
GiteaMirror commented 2026-04-22 11:25:16 -05:00
Author
Owner
Copy Link

@ItzCrazyKns commented on GitHub (Nov 5, 2025):

Any update on this?

<!-- gh-comment-id:3492736468 --> @ItzCrazyKns commented on GitHub (Nov 5, 2025): Any update on this?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#31190
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?