* mlxrunner: add Phonon text-to-speech support
Add Phonon text-to-speech through the existing MLX runner, with an
OpenAI-compatible speech endpoint and CLI playback.
Include model loading, text preprocessing, streaming audio, and
regression tests.
* docs: add speech models
* fix min version
---------
Co-authored-by: Daniel Hiltgen <daniel@ollama.com>
* Deduplicate MLX CUDA runtime payloads
Move CUDA libraries shared by the GGML and MLX backends into the sibling CUDA payload, and teach in-process and subprocess MLX loading to resolve that packaged dependency directory.
Keep MLX-private libraries and JIT headers in the MLX overlay, include the full OpenBLAS Fortran runtime closure, and verify those dependencies in release packaging.
* address comments
* win: prevent orphaned runners
Assign runner processes to a singleton Windows Job Object configured with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. This lets Windows terminate owned llama-server and MLX subprocesses when Ollama exits, including forced termination where graceful cleanup cannot run.
* address comments
Implements the EmbeddingGemma2Model architecture on the MLX runner: 24-layer
bidirectional text encoder with PLE, shared gemma4 vision/audio towers,
mean-pool + L2 output. /api/embed accepts per-item media via input dicts.
A format on a thinking model ran two generations: an unconstrained one,
cancelled once the parser reported content, then a re-rendered prompt with
the parsed thinking under the grammar. The restart cost a second prefill,
dropped the chunk that crossed the boundary, needed a harmony prompt hack,
stitched metrics across the two requests, and on MLX could leak a stray
first token into the JSON. The generate endpoint never deferred at all, so
its JSON was forced inside the thinking.
Both handlers now make one completion request that names the strings
ending the response's thinking, from the builtin parser or the generic
thinking parser, and the runner constrains only the content after them in
a single generation. The prompt is evaluated once and metrics pass straight
through. A raw generate prompt names no strings, since nothing says where
its response starts, and its format applies from the first token as
before. A format now applies to whatever follows the thinking, so a tool
call can no longer take the place of formatted content, which was already
the case with thinking off; harmony is the exception, since its tool calls
precede the final message.
The per-token metrics flag both runners carried for the cancelled first
pass has no caller left and goes with the two-pass code and its tests.
Fixes#18441Fixes#17544Fixes#14196Fixes#10929
The MLX runner applies a format's grammar from the first sampled token,
so a thinking model asked for a format cannot think first, and the server
has to run two generations to get both the thinking and the formatted
content.
A completion request now carries the strings that end the thinking its
response begins with, and the MLX client builds from them a structural
tag: free text that cannot contain any of them, then one of them, then
the schema. The tail is optional so a response may still end inside its
thinking, as an unconstrained one can. Without a closing string the tag
is the plain schema, as before. The server does not send the strings yet.
The runner and its weight loader read a model's manifest through
x/imagegen/manifest, the last piece of the removed image generation
engine. It was a hand-rolled copy of the manifest package: its own model
name parser with the default registry and namespace spelled out, its own
blob path builder, and re-spelled media types, plus a model_index.json
reader and other helpers that nothing has called since the engine went.
The manifest package gains the three lookups the runner needs, a config
layer by path, its contents, and the tensor layers, and ReadConfigJSON is
built on the second of them. The weight loader resolves the model name
with the shared parser, which fills in the same defaults the copy did, and
locates blobs with BlobsPath. The architectures' calls to read their
config.json compile unchanged. x/imagegen is gone.
The MLX runner is the only Go inference runner left and is no longer
experimental, so its packages leave x/. The bindings become a top-level
mlx package beside the carried patches in mlx/compat, mirroring how
llama/ holds the llama.cpp integration, and the runner becomes mlxrunner
with the architectures nested under the package they implement.
Subpackages move with their parent unless listed.
x/mlxrunner/mlx mlx
x/internal/mlxthread mlx/mlxthread
x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest
x/internal/mlxtest mlx/mlxtest
x/quant mlx/quant
mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx)
x/mlxrunner mlxrunner
x/models/nn mlxrunner/nn
x/models/<arch> mlxrunner/model/<arch>
x/mlxrunner/imports.go mlxrunner/model/architectures (new package)
x/create create
x/safetensors fs/safetensors
x/tokenizer mlxrunner/tokenizer
Every package keeps its name, so the Go changes are the import path
rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift
check and Darwin payload script follow the new paths. Four edits are not
paths: the runner's blank architecture imports become the package
mlxrunner/model/architectures, so the list to extend for a new model sits
beside the architecture directories; a depguard rule keeps the two test
harnesses out of non-test code, as the x/internal placement used to; the
CI change filter's two entries for the long-deleted x/imagegen/mlx now
name the bindings' CMake project and the carried patches, so a change to
either builds the payload; and the tokenizer parity test reads its
fixtures from its own testdata instead of walking out of x/.
x/server and x/imagegen/manifest stay for the next two commits.
Daniel HiltgenandGitHub
Patrick DevineandGitHub
Jesse Gross