github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #11500] Structured output breaks with string literals containing double-quotes #33357

New Issue
Open
opened 2026-04-22 15:56:05 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-22 15:56:05 -05:00
Owner
Copy Link

Originally created by @PladsElsker on GitHub (Jul 23, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11500

What is the issue?

Problematic schema

String literals containing escaped double-quotes (\") at any position in schemas seem to break the api at the moment. I also tested directly with the requests lib to make sure it has nothing to do with the python api. Adding spaces or indentation to the schema doesn't seem to help (#10799).

from pydantic import BaseModel
from ollama import chat
from typing import Literal


class BadModel(BaseModel):
    bad_attr: Literal["\""]


chat(
    model="gemma3:27b",
    messages=[{"role": "user", "content": "Hi!"}],
    format=BadModel.model_json_schema()
)

The schema of the model looks like this:

{
    "properties": {
        "bad_attr": {
            "const": "\"", 
            "title": "Bad Attr", 
            "type": "string"
        }
    }, 
    "required": [
        "bad_attr"
    ], 
    "title": "BadSchema", 
    "type": "object"
}

Literals usually work

In contrast, this model works:

class GoodModel(BaseModel):
    good_attr: Literal["Hello world!"]

Relevant log output

  File "/**/venv/lib/python3.12/site-packages/ollama/_client.py", line 122, in _request_raw
    raise ResponseError(e.response.text, e.response.status_code) from None
ollama._types.ResponseError: failed to load model vocabulary required for format
 (status code: 500)

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.9.6

Originally created by @PladsElsker on GitHub (Jul 23, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11500 ### What is the issue? ## Problematic schema String literals containing escaped double-quotes (`\"`) at any position in schemas seem to break the api at the moment. I also tested directly with the `requests` lib to make sure it has nothing to do with the python api. Adding spaces or indentation to the schema doesn't seem to help (#10799). ```py from pydantic import BaseModel from ollama import chat from typing import Literal class BadModel(BaseModel): bad_attr: Literal["\""] chat( model="gemma3:27b", messages=[{"role": "user", "content": "Hi!"}], format=BadModel.model_json_schema() ) ``` The schema of the model looks like this: ```json { "properties": { "bad_attr": { "const": "\"", "title": "Bad Attr", "type": "string" } }, "required": [ "bad_attr" ], "title": "BadSchema", "type": "object" } ``` ## Literals usually work In contrast, this model works: ``` class GoodModel(BaseModel): good_attr: Literal["Hello world!"] ``` ### Relevant log output ```shell File "/**/venv/lib/python3.12/site-packages/ollama/_client.py", line 122, in _request_raw raise ResponseError(e.response.text, e.response.status_code) from None ollama._types.ResponseError: failed to load model vocabulary required for format (status code: 500) ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.9.6
GiteaMirror added the bug label 2026-04-22 15:56:05 -05:00
GiteaMirror commented 2026-04-22 15:56:06 -05:00
Author
Owner
Copy Link

@PladsElsker commented on GitHub (Jul 23, 2025):

My use-case for this is that I'm passing python programs as optional string literals in the schema. I don't think I'll be stuck on this bug for too long, since it seems possible to make a small parser that converts all double quotes in python programs to single quotes. Python programs are a special case, though. Not every schema can be patched like that.

<!-- gh-comment-id:3105444560 --> @PladsElsker commented on GitHub (Jul 23, 2025): My use-case for this is that I'm passing python programs as optional string literals in the schema. I don't think I'll be stuck on this bug for too long, since it seems possible to make a small parser that converts all double quotes in python programs to single quotes. Python programs are a special case, though. Not every schema can be patched like that.
GiteaMirror commented 2026-04-22 15:56:07 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jul 23, 2025):

This is a bug in the schema to grammar conversion in llama.cpp.

$ build/bin/llama-cli -m test.gguf -no-cnv -j '{"type":"string","const":"\""}' 
build: 5544 (b49a8ff9) with cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0 for x86_64-linux-gnu
...

parse: error parsing grammar: expecting newline or end at \"" space
space ::= | " " | "\n"{1,2} [ \t]{0,20}


root ::= "\"\\"\"" space
space ::= | " " | "\n"{1,2} [ \t]{0,20}

llama_grammar_init_impl: failed to parse grammar
<!-- gh-comment-id:3108647727 --> @rick-github commented on GitHub (Jul 23, 2025): This is a bug in the schema to grammar conversion in llama.cpp. ``` $ build/bin/llama-cli -m test.gguf -no-cnv -j '{"type":"string","const":"\""}' build: 5544 (b49a8ff9) with cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0 for x86_64-linux-gnu ... parse: error parsing grammar: expecting newline or end at \"" space space ::= | " " | "\n"{1,2} [ \t]{0,20} root ::= "\"\\"\"" space space ::= | " " | "\n"{1,2} [ \t]{0,20} llama_grammar_init_impl: failed to parse grammar ```
GiteaMirror commented 2026-04-22 15:56:07 -05:00
Author
Owner
Copy Link

@PladsElsker commented on GitHub (Jul 23, 2025):

It's fixable in llama.cpp by installing LLGuidance:

cmake -B build -DLLAMA_LLGUIDANCE=ON -DGGML_CUDA=ON
make -C build -j

build/bin/llama-cli -m models/gemma-3-12b-it-Q2_K.gguf -no-cnv -j '{"type":"string","const":"\""}' 

"\"" [end of text]
<!-- gh-comment-id:3109909836 --> @PladsElsker commented on GitHub (Jul 23, 2025): It's fixable in llama.cpp by installing [LLGuidance](https://github.com/ggml-org/llama.cpp/blob/master/docs/llguidance.md): ```shell cmake -B build -DLLAMA_LLGUIDANCE=ON -DGGML_CUDA=ON make -C build -j build/bin/llama-cli -m models/gemma-3-12b-it-Q2_K.gguf -no-cnv -j '{"type":"string","const":"\""}' "\"" [end of text] ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#33357
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?