When a streamed reply from a Responses API connection stopped early, for example at the token limit or through a content filter, its token counts were lost. The chat showed no usage for that reply and the admin analytics recorded zero tokens for it. Such replies now keep the usage the provider reported, the same as replies that finished normally.
Fixes#33615
Since a recent refactor on dev, the prompt and completion token counts of a reply that used tools added up every model call in the loop, even though each later call already re-sends the whole conversation. A reply whose context was 50k tokens showed 90k, the context gauge read over 100% and the next message triggered compaction long before the chat reached the threshold. Prompt and completion tokens show the latest call again, while total tokens and cost still add up across all calls.
Outlet filters see the finished reply with its content and token usage, but had no way to tell whether the model ended on its own, ran into the token limit or stopped to call a tool, short of reading every chunk in a stream filter. For OpenAI-compatible and Ollama models, the assistant message handed to outlet now carries finish_reason with the value the provider reported on the last model call of the turn, for streaming and non-streaming replies. Ollama replies cut off by the token limit now report length as well, where they always said stop before.
Swap the JSON encoder/decoder used across the backend from stdlib json to
orjson when ENABLE_ORJSON is set — HTTP request bodies, JSONResponse
bodies, upstream provider responses, SSE chunks, and socket.io/Redis
payloads.
The flag defaults to off, in which case the app uses stdlib json and
engineio's codec verbatim, so default behaviour is unchanged.
- json_codec exports JSONCodec (stdlib json or the orjson codec) and
SOCKETIO_JSON (engineio's codec or the orjson codec); call sites import
JSONCodec and stay implementation-agnostic
- apply_orjson_http_json() is a no-op when the flag is off, leaving
starlette's Request.json / JSONResponse.render untouched
- the orjson codec falls back to the stdlib for inputs orjson rejects
(non-str dict keys, ints beyond 64 bits, NaN literals)
- orjson is imported only when the flag is on
- FastAPI(default_response_class=...) is deliberately not used: an
explicit default disables the Pydantic direct-to-bytes fast path for
response_model routes
Classic298andGitHub
Timothy Jaeryang Baek
EntropyYue
henry
Simone
Timothy J. Baek