github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #7881] OpenAI-compatible API tool calls have no index #5041

New Issue
Closed
opened 2026-04-12 16:07:49 -05:00 by GiteaMirror · 3 comments
GiteaMirror commented 2026-04-12 16:07:49 -05:00
Owner
Copy Link

Originally created by @jackmpcollins on GitHub (Nov 29, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/7881

Originally assigned to: @ParthSareen on GitHub.

What is the issue?

What is the issue?

The streamed chat-completion response from ollama's openai-compatible API does not populate the .choices[].delta.tool_calls[].index field. This is different to OpenAI's API where this is populated on all tool call chunks and enumerates the tool calls. This breaks compatibility with the client.beta.chat.completions.stream helper from the openai package. It also breaks compatibility with https://github.com/pydantic/logfire which uses the same underlying code from openai. Please add the index field to the tool calls to match openai.

OpenAI chunks: tool call index is present starting at 0

from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is the weather like in Boston?"}],
    stream=True,
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        },
                        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                    },
                    "required": ["location"],
                },
            },
        },
    ],
)
for chunk in response:
    print(chunk.model_dump_json(exclude_none=True))
{"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"role":"assistant","tool_calls":[{"index":0,"id":"call_2dlDZrcl0VQMiDtzJzwjIN5j","function":{"arguments":"","name":"get_current_weather"},"type":"function"}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"}
{"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\""}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"}
{"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"location"}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"}
{"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":\""}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"}
{"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"Boston"}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"}
{"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":","}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"}
{"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":" MA"}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"}
{"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"}"}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"}
{"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{},"finish_reason":"tool_calls","index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"}

Ollama chunks: tool call index is not present

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",
)
response = client.chat.completions.create(
    model="llama3.1",
    # model="gpt-4o",
    messages=[{"role": "user", "content": "What is the weather like in Boston?"}],
    stream=True,
    # stream_options={"include_usage": True},
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        },
                        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                    },
                    "required": ["location"],
                },
            },
        },
    ],
)
for chunk in response:
    print(chunk.model_dump_json(exclude_none=True))
{"id":"chatcmpl-914","choices":[{"delta":{"content":"","role":"assistant","tool_calls":[{"id":"call_rn5g1z57","function":{"arguments":"{\"location\":\"Boston, MA\",\"unit\":\"fahrenheit\"}","name":"get_current_weather"},"type":"function"}]},"index":0}],"created":1732871553,"model":"llama3.1","object":"chat.completion.chunk","system_fingerprint":"fp_ollama"}
{"id":"chatcmpl-914","choices":[{"delta":{"content":"","role":"assistant"},"finish_reason":"stop","index":0}],"created":1732871553,"model":"llama3.1","object":"chat.completion.chunk","system_fingerprint":"fp_ollama"}

Using client.beta.chat.completions.stream with ollama results in an exception due to None value for tool call index.
openai docs for this function: 646a579cdb/helpers.md (chat-completions-api)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",
)
with client.beta.chat.completions.stream(
    model="llama3.1",
    messages=[{"role": "user", "content": "What is the weather like in Boston?"}],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        },
                        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                    },
                    "required": ["location"],
                },
            },
        },
    ],
) as stream:
    for event in stream:
        pass

print(stream.get_final_completion().model_dump_json(indent=2))
...
openai/lib/streaming/chat/_completions.py:505, in ChatCompletionStreamState._build_events(self, chunk, completion_snapshot)
    502 assert tool_calls is not None
    504 for tool_call_delta in choice.delta.tool_calls:
--> 505     tool_call = tool_calls[tool_call_delta.index]
    507     if tool_call.type == "function":
    508         assert tool_call_delta.function is not None
TypeError: list indices must be integers or slices, not NoneType

OS

macOS

GPU

No response

CPU

Apple

Ollama version

0.4.6

Originally created by @jackmpcollins on GitHub (Nov 29, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/7881 Originally assigned to: @ParthSareen on GitHub. ### What is the issue? ### What is the issue? The streamed chat-completion response from ollama's openai-compatible API does not populate the `.choices[].delta.tool_calls[].index` field. This is different to OpenAI's API where this is populated on all tool call chunks and enumerates the tool calls. This breaks compatibility with the `client.beta.chat.completions.stream` helper from the openai package. It also breaks compatibility with https://github.com/pydantic/logfire which uses the same underlying code from openai. Please add the index field to the tool calls to match openai. --- OpenAI chunks: tool call `index` is present starting at 0 ```python from openai import OpenAI client = OpenAI() response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "What is the weather like in Boston?"}], stream=True, tools=[ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, }, ], ) for chunk in response: print(chunk.model_dump_json(exclude_none=True)) {"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"role":"assistant","tool_calls":[{"index":0,"id":"call_2dlDZrcl0VQMiDtzJzwjIN5j","function":{"arguments":"","name":"get_current_weather"},"type":"function"}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"} {"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\""}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"} {"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"location"}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"} {"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":\""}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"} {"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"Boston"}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"} {"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":","}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"} {"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":" MA"}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"} {"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"}"}}]},"index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"} {"id":"chatcmpl-AYrJepetKGmoEh6pqPHCwPydZRPI3","choices":[{"delta":{},"finish_reason":"tool_calls","index":0}],"created":1732871462,"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_7f6be3efb0"} ``` Ollama chunks: tool call `index` is not present ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:11434/v1", api_key="ollama", ) response = client.chat.completions.create( model="llama3.1", # model="gpt-4o", messages=[{"role": "user", "content": "What is the weather like in Boston?"}], stream=True, # stream_options={"include_usage": True}, tools=[ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, }, ], ) for chunk in response: print(chunk.model_dump_json(exclude_none=True)) {"id":"chatcmpl-914","choices":[{"delta":{"content":"","role":"assistant","tool_calls":[{"id":"call_rn5g1z57","function":{"arguments":"{\"location\":\"Boston, MA\",\"unit\":\"fahrenheit\"}","name":"get_current_weather"},"type":"function"}]},"index":0}],"created":1732871553,"model":"llama3.1","object":"chat.completion.chunk","system_fingerprint":"fp_ollama"} {"id":"chatcmpl-914","choices":[{"delta":{"content":"","role":"assistant"},"finish_reason":"stop","index":0}],"created":1732871553,"model":"llama3.1","object":"chat.completion.chunk","system_fingerprint":"fp_ollama"} ``` Using `client.beta.chat.completions.stream` with ollama results in an exception due to `None` value for tool call index. openai docs for this function: https://github.com/openai/openai-python/blob/646a579cdb305a9d3fba6c5f9a96011c5e2c2882/helpers.md#chat-completions-api ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:11434/v1", api_key="ollama", ) with client.beta.chat.completions.stream( model="llama3.1", messages=[{"role": "user", "content": "What is the weather like in Boston?"}], tools=[ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, }, ], ) as stream: for event in stream: pass print(stream.get_final_completion().model_dump_json(indent=2)) ... openai/lib/streaming/chat/_completions.py:505, in ChatCompletionStreamState._build_events(self, chunk, completion_snapshot) 502 assert tool_calls is not None 504 for tool_call_delta in choice.delta.tool_calls: --> 505 tool_call = tool_calls[tool_call_delta.index] 507 if tool_call.type == "function": 508 assert tool_call_delta.function is not None TypeError: list indices must be integers or slices, not NoneType ``` ### OS macOS ### GPU _No response_ ### CPU Apple ### Ollama version 0.4.6
GiteaMirror added the bug label 2026-04-12 16:07:49 -05:00
GiteaMirror commented 2026-04-12 16:07:51 -05:00
Author
Owner
Copy Link

@gregnr commented on GitHub (Nov 29, 2024):

Experiencing this too using Vercel's AI SDK. The SDK expects that each tool call has an index, and validation fails with Ollama:

Error: Type validation failed: Value: {
  "id": "chatcmpl-763",
  "object": "chat.completion.chunk",
  "created": 1732922184,
  "model": "qwen2.5:7b",
  "system_fingerprint": "fp_ollama",
  "choices": [
    {
      "index": 0,
      "delta": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "call_rcja46yu",
            "type": "function",
            "function": { "name": "<redacted>", "arguments": "<redacted>" }
          }
        ]
      },
      "finish_reason": null
    }
  ]
}
.
Error message: [
  {
    "code": "invalid_union",
    "unionErrors": [
      {
        "issues": [
          {
            "code": "invalid_type",
            "expected": "number",
            "received": "undefined",
            "path": [
              "choices",
              0,
              "delta",
              "tool_calls",
              0,
              "index"
            ],
            "message": "Required"
          }
        ],
        "name": "ZodError"
      },
      {
        "issues": [
          {
            "code": "invalid_type",
            "expected": "object",
            "received": "undefined",
            "path": [
              "error"
            ],
            "message": "Required"
          }
        ],
        "name": "ZodError"
      }
    ],
    "path": [],
    "message": "Invalid input"
  }
]
<!-- gh-comment-id:2508729728 --> @gregnr commented on GitHub (Nov 29, 2024): Experiencing this too using Vercel's AI SDK. The SDK expects that each tool call has an index, and validation fails with Ollama: ``` Error: Type validation failed: Value: { "id": "chatcmpl-763", "object": "chat.completion.chunk", "created": 1732922184, "model": "qwen2.5:7b", "system_fingerprint": "fp_ollama", "choices": [ { "index": 0, "delta": { "role": "assistant", "content": "", "tool_calls": [ { "id": "call_rcja46yu", "type": "function", "function": { "name": "<redacted>", "arguments": "<redacted>" } } ] }, "finish_reason": null } ] } . Error message: [ { "code": "invalid_union", "unionErrors": [ { "issues": [ { "code": "invalid_type", "expected": "number", "received": "undefined", "path": [ "choices", 0, "delta", "tool_calls", 0, "index" ], "message": "Required" } ], "name": "ZodError" }, { "issues": [ { "code": "invalid_type", "expected": "object", "received": "undefined", "path": [ "error" ], "message": "Required" } ], "name": "ZodError" } ], "path": [], "message": "Invalid input" } ] ```
GiteaMirror commented 2026-04-12 16:07:54 -05:00
Author
Owner
Copy Link

@ParthSareen commented on GitHub (Nov 29, 2024):

Sorry about that! Looking into it now!

<!-- gh-comment-id:2508734744 --> @ParthSareen commented on GitHub (Nov 29, 2024): Sorry about that! Looking into it now!
GiteaMirror commented 2026-04-12 16:07:55 -05:00
Author
Owner
Copy Link

@ParthSareen commented on GitHub (Dec 2, 2024):

@gregnr @jackmpcollins just released - https://github.com/ollama/ollama/releases/tag/v0.4.7

<!-- gh-comment-id:2510519787 --> @ParthSareen commented on GitHub (Dec 2, 2024): @gregnr @jackmpcollins just released - https://github.com/ollama/ollama/releases/tag/v0.4.7
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#5041
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?