* MLX: reduce Qwen3.8 Flash Next memory use
For NVFP4 creation, retain source precision for sensitive QSA projections
and non-expert MTP weights. Keep routed experts and PLE at NVFP4, and
promote other eligible weights to MXFP8.
Keep NVFP4 PLE tables host-backed, transferring only selected packed rows
and their scales to MLX. Honor scalar and per-expert NVFP4 global scales
in MoE, and per-shard global scales in host-backed PLE.
Use compressed QSA block selections in fused indexed Metal attention for
long prefill batches, retaining the graph fallback. Co-locate component
loading and forward methods, with explicit quantization and cache ownership.
* mlx: skip materializing host-backed tensors
Let models exclude tensors already owned by host storage from MLX loading.
Avoid allocating and discarding the full Qwen PLE tables at startup while
preserving the resident fallback.
* address comments
MLX's argpartition sorts the whole row, and a row longer than one block
of the GPU sort takes a multi-pass merge sort. TopKIndices sorts each
2048-value chunk in a single block, then only the chunks' top k, and
returns the same indices in the same order.
The sampler's top_k and top_logprobs and the DFlash2 candidate selector
use it in place of a full-vocabulary argpartition.
Fixes for running LFM3 GGUFs on the llama-server runner, plus GGUF
text-to-speech that turns on once llama.cpp is bumped past b11901.
- BOS: honor an explicit tokenizer.ggml.add_bos_token. LFM3 GGUFs (and
lfm2:24b) set it to false and expect the template's BOS, which the
lfm2 default was stripping. The lfm2 default still applies when the
key is absent. On lfm2:24b, WikiText PPL goes from 81.3 to 72.0.
- repeat_last_n -1 now maps to the context size; llama-server rejects
negative values.
- Audio-only projectors (LFM3-Audio) no longer advertise vision, so
clients don't send images the model can't take.
- /v1/audio/speech on GGUF models calls llama-server /tts chunk by
chunk. Renderers opt in through a new SpeechPrompter (voices,
voice parsing, chunking), so other speech models can add their own.
The LFM3 voice and chunking helpers move from the MLX runner into
model/renderers and are shared by both runners.
DFlash2 adds block-local dynamic convolutions and a candidate selector
to DFlash. The selector proposes each token conditioned on its
predecessor and returns each position's candidate distribution, which
verification uses to keep the output exact. Like the reference, the
draft samples with the request's temperature only.
A DFlash2 session drafts seven tokens every round on the existing
snapshot verify path instead of following the depth controller, whose
shallower picks lose more accepted tokens than they save in verify time.
qwen3.5-family targets now expose the layer taps, token embeddings and
raw logits a DFlash draft needs.
* MLX, MLX-C: version bump
The MLX-C pin now provides its own bindings for gather_qmm global_scale
and fast::gated_delta_update, so those two carried patches drop out.
Carry 0001-mlx-c-sdpa-window-size instead: sliding-window attention
added a window_size argument to fast::scaled_dot_product_attention
(ml-explore/mlx#4734) that MLX-C has not bound yet, to be regenerated
from MLX-C once the upstream binding PR merges. The SDPA wrappers gain
a window size argument; all call sites default to the full context.
* mlxrunner: dispatch sliding-window prefills to the native SDPA window
Sliding layers no longer materialize a causal∩sliding mask for prefill
chunks past the window: resolve() dispatches {causal, window} natively
when the rotating cache's L>1 concat has linearized the ring and the
mask is otherwise pure causal, via a new MaskApplier.NativeWindow hint.
Vision relaxations, padding, model arrays, and decode keep the
materialized path. No model changes; sinks compose natively.
Dynamic sizing can floor the smallest image grid below its intended 70-token budget, losing detail in small images. Round that grid up for the compact tower, capped at the next supported budget.
Keep explicit budgets and the existing sizing for larger and unified towers. Update geometry and media-expansion tests.
* mlxrunner: spell out symbols, numbers and links for Phonon
* docs: describe speech text normalization
* mlxrunner: fix Phonon misreadings inherited from xn-ptts
* mlxrunner: fix remaining Phonon misreadings
* mlxrunner: fold Phonon letters from the vocab, read #, _ and lists
* mlxrunner: add xn-ptts license notice for Phonon
* mlxrunner: move the xn-ptts license notice into normalize.go
* mlxrunner: fix remaining Phonon readings and port elongations
* docs: speech reads times and numeric dates
* mlxrunner: fix Phonon readings found in a listening pass
* docs: speech reads times and dates as written
* renderers: add gemma4-nemotron renderer
Add a Gemma 4 renderer variant that renders assistant content before
its tool calls rather than after the tool responses. Models opt in with
RENDERER gemma4-nemotron; gemma4 auto-detection is unchanged.
Adds the corresponding chat template as a test fixture for Jinja parity
checks.
* mlx: support Gemma 4 targets with DFlash drafts
Load a draft-owned output head when its config requires one, while preserving the target-head path used by existing DFlash models.
Expose Gemma 4 hidden-state taps and raw logits for speculative decoding. Report safetensors target precision independently of quantized draft weights.
Replace the prerelease checkpoint path with Gradium's GA format, including default-voice import and speech conditioning. Set the 0.40.9 minimum, update the speech docs, and test import, inference setup, and rejection of the old checkpoint.
* mlxrunner: add Phonon text-to-speech support
Add Phonon text-to-speech through the existing MLX runner, with an
OpenAI-compatible speech endpoint and CLI playback.
Include model loading, text preprocessing, streaming audio, and
regression tests.
* docs: add speech models
* fix min version
---------
Co-authored-by: Daniel Hiltgen <daniel@ollama.com>
Preserve source tool JSON alongside decoded API values so Flint can retain
numeric types and schema order in prompts. Existing API marshaling is unchanged.
Assemble ModelOpt expert banks during safetensors import, and reject direct
Flint NVFP4 creation before conversion. Other models retain their existing
import and quantization paths.
Add optional learned sinks and device causal masks to shared MLX attention.
Existing callers retain the no-sink and host-mask paths; Flint opts into the
new behavior.
* Deduplicate MLX CUDA runtime payloads
Move CUDA libraries shared by the GGML and MLX backends into the sibling CUDA payload, and teach in-process and subprocess MLX loading to resolve that packaged dependency directory.
Keep MLX-private libraries and JIT headers in the MLX overlay, include the full OpenBLAS Fortran runtime closure, and verify those dependencies in release packaging.
* address comments
* win: prevent orphaned runners
Assign runner processes to a singleton Windows Job Object configured with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. This lets Windows terminate owned llama-server and MLX subprocesses when Ollama exits, including forced termination where graceful cleanup cannot run.
* address comments
* ci: shorten Windows Vulkan build path
Keep the nested shader-generator compiler probes below Windows path limits in checkouts with longer repository names.
* mlx: fix Nemotron bugs and support global scales
Accept published Nemotron layer names while retaining legacy aliases, and
fix layer-count inference and detached MTP quantization. Stabilize Mamba
softplus, preserve FP32 vision normalization, and compile Mamba gate and
norm scaling.
Support per-expert F32 global scales and preserve activation-scale
companions in ModelOpt and compressed-tensors NVFP4 imports. Normalize
reciprocal globals and reject invalid scales. Keep activation scales
separate from weight-only inference.
* mlx: add Nemotron 3.5 Super
* minor adjustments
A failure during decoding can trigger deferred drain and cache close, which may panic again and hide the original error. Skip stateful cleanup while a panic is unwinding so the runner reports the initial failure.
Add regression tests for the masked panic paths and normal cache saving.
* Add Kev decision scoring to MLX
Load Kev models with their FP32 decision head and release metadata.
Support System One scoring with shared-state caching and Kev-specific limits.
* address comments
* mlx: compact restored recurrent state after speculative rollback
Per-token recurrent captures can leave a restored state pointing into a larger buffer. If a request ends before another model step, the live cache retains that buffer even though it accounts for only the selected state.
Track when a restored state may share that backing buffer. At request close, compact marked states in one batch before retaining them in the prefix cache. Clear the marker when a normal model step replaces the state, and cover restoration and buffer detachment in tests.
* address comments
Implements the EmbeddingGemma2Model architecture on the MLX runner: 24-layer
bidirectional text encoder with PLE, shared gemma4 vision/audio towers,
mean-pool + L2 output. /api/embed accepts per-item media via input dicts.
* Reduce model lookup and MLX decision request overhead
Avoid decoding unrelated manifests when resolving existing model names.
Reuse small Metal scratch buffers between decision requests, releasing
larger working sets and clearing scratch after scoring errors.
* address comments
* mlxrunner: restore System One scoring
Carry the MLX implementation and its tests from the original System One
commit. The API/compiler and llama-server implementation already exist on
the release branch, so retain those versions. Adapt the optional model test
to the renamed decision package and its explicit rendering/context setup.
(cherry picked from commit 26e546ddb3)
* Enable MLX System One and add regression coverage
* mlxrunner: add Laya support for System One
Import Laya checkpoints and implement bidirectional decision scoring.
* mlxrunner: release cached buffers after System One scoring
* decision: match Tev1 and Nimble publisher prompt formats
Select Tev1 prompts using renderer metadata and fix JSON formatting, with regression coverage for GGUF and MLX.
* mlx: support Clef and Clef Flash decision models
Add model import, quantization safeguards, and text/image scoring with the Clef joint decision head.
* support vision capabilities for decision models
* Make Clef scoring cancellable between prefill chunks
Preserve joint-head inputs and test cancellation, cleanup, and recovery.
* mlx: separate decision preparation and readout from forward
Let serial and batched scoring share model-owned preparation and readout
without duplicating architecture-specific logic in the runner.
* mlx: add Strands Decider support
* implement caching
---------
Co-authored-by: ParthSareen <parth.sareen@ollama.com>
Move the ollama_xgrammar target into mlxrunner/xgrammar/native so it
can be configured on its own against an installed xgrammar. cmake/mlx
now adds it as a subdirectory and still uses the pinned xgrammar.
* mlx: speed up Qwen 3.8 prompt processing
Use MLX's gated-delta kernel for long scans and fold dense MLP global scales into SwiGLU.
* address comments
Replace the fixed checkpoint image budget with per-image selection across
the supported 70, 140, 280, 560, and 1120 budgets. Choose the publisher resize
grid closest to the input resolution, accounting for aspect ratio.
This preserves more detail in high-resolution documents while allowing
smaller images to use fewer tokens, without adding an API parameter.
Cover budget boundaries, extreme dimensions, position limits, and media
expansion for both vision architectures.
A format on a thinking model ran two generations: an unconstrained one,
cancelled once the parser reported content, then a re-rendered prompt with
the parsed thinking under the grammar. The restart cost a second prefill,
dropped the chunk that crossed the boundary, needed a harmony prompt hack,
stitched metrics across the two requests, and on MLX could leak a stray
first token into the JSON. The generate endpoint never deferred at all, so
its JSON was forced inside the thinking.
Both handlers now make one completion request that names the strings
ending the response's thinking, from the builtin parser or the generic
thinking parser, and the runner constrains only the content after them in
a single generation. The prompt is evaluated once and metrics pass straight
through. A raw generate prompt names no strings, since nothing says where
its response starts, and its format applies from the first token as
before. A format now applies to whatever follows the thinking, so a tool
call can no longer take the place of formatted content, which was already
the case with thinking off; harmony is the exception, since its tool calls
precede the final message.
The per-token metrics flag both runners carried for the cancelled first
pass has no caller left and goes with the two-pass code and its tests.
Fixes#18441Fixes#17544Fixes#14196Fixes#10929
The MLX runner applies a format's grammar from the first sampled token,
so a thinking model asked for a format cannot think first, and the server
has to run two generations to get both the thinking and the formatted
content.
A completion request now carries the strings that end the thinking its
response begins with, and the MLX client builds from them a structural
tag: free text that cannot contain any of them, then one of them, then
the schema. The tail is optional so a response may still end inside its
thinking, as an unconstrained one can. Without a closing string the tag
is the plain schema, as before. The server does not send the strings yet.
* nemotron_h: add MLX vision support
Implement the RADIO vision encoder and projector on the shared MLX media pipeline, including dynamic-resolution preprocessing, deterministic placeholder expansion, chunked feature scattering, and MTP offsets.
Expose source-advertised Nemotron vision while continuing to suppress unsupported audio, and preserve both modality towers at source precision during create.
Harden Nemotron streaming parser termination and add focused coverage for vision configuration, media placement, capability reporting, and tool-call parsing.
* review comments
* address comments
The decode loop releases MLX's pool of freed buffers every 256 generated
tokens, which is also how often the KV cache grows and drops its previous,
smaller buffers. The check fires only when the token count lands exactly on
a multiple of 256. Speculative decoding emits several tokens per round, so
most rounds step over the boundary and the pool is never released. Each
growth at a long context leaves several GB of buffers that no later
allocation can reuse, so the runner's footprint keeps climbing over a long
generation until the system runs out of memory.
We now release the pool whenever a round crosses a multiple of 256 tokens,
which is what a single-token round already did. With qwen3.8:27b-mlx at a
98k-token context on a 128 GB machine, a long speculative generation
previously grew the runner past 90 GB and panicked the kernel; it now stays
flat at 30 GB.
model is one package with three jobs: the contract between the runner
and the architectures, the opened checkpoint, and building nn layers
from checkpoint tensors. Its files did not say which was which. base.go
carried the folded package's name over the interfaces and the registry,
root.go held the safetensors header scan next to Root, and quant.go
mixed the nvfp4 global-scale helpers with quant parameter resolution.
base.go becomes model.go, named for what it holds. root.go keeps Root
and Open; TensorQuantInfo and the header scan join quant.go, so
everything the checkpoint says about quantization is read and resolved
in one file. The global-scale helpers move to globalscale.go with their
tests. Root.Close, a no-op with one caller, goes. No code changes
otherwise.
nn.go held every layer type in the package apart from attention,
recurrence and rope: the Linear and Embedding interfaces with their dense
and quantized types, Conv1d, RMSNorm, LayerNorm and MultiLinear, with one
test file to match. Finding a layer meant scanning the file named after
the package.
Each layer kind gets its own file: linear.go and embedding.go hold the
interface and the dense and quantized types, conv.go, norm.go and
multilinear.go take the rest, and nn_test.go splits the same way. The
Layer and MultiLinearLayer interfaces go; nothing implemented or accepted
them. No code changes otherwise.
The runner and its weight loader read a model's manifest through
x/imagegen/manifest, the last piece of the removed image generation
engine. It was a hand-rolled copy of the manifest package: its own model
name parser with the default registry and namespace spelled out, its own
blob path builder, and re-spelled media types, plus a model_index.json
reader and other helpers that nothing has called since the engine went.
The manifest package gains the three lookups the runner needs, a config
layer by path, its contents, and the tensor layers, and ReadConfigJSON is
built on the second of them. The weight loader resolves the model name
with the shared parser, which fills in the same defaults the copy did, and
locates blobs with BlobsPath. The architectures' calls to read their
config.json compile unchanged. x/imagegen is gone.
model/base held the Model interface and the architecture registry while
model held weight loading and quant parameters, and every architecture
imported both. create/client was the CLI side of safetensors imports,
with the create command as its only caller, a duplicate of the command's
adapter error, and a name that read like a second API client. The
safetensors show helpers had their own package under x/ although it
already declared package server.
base merges into model, so base.Model and base.Register become
model.Model and model.Register; nothing in the two overlapped. The create
client's two files and their tests join package cmd, and the five names
the command called are no longer exported. The show helpers join the
server package, and the three entry points routes.go calls become
unexported like the helpers around them.
The MLX runner is the only Go inference runner left and is no longer
experimental, so its packages leave x/. The bindings become a top-level
mlx package beside the carried patches in mlx/compat, mirroring how
llama/ holds the llama.cpp integration, and the runner becomes mlxrunner
with the architectures nested under the package they implement.
Subpackages move with their parent unless listed.
x/mlxrunner/mlx mlx
x/internal/mlxthread mlx/mlxthread
x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest
x/internal/mlxtest mlx/mlxtest
x/quant mlx/quant
mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx)
x/mlxrunner mlxrunner
x/models/nn mlxrunner/nn
x/models/<arch> mlxrunner/model/<arch>
x/mlxrunner/imports.go mlxrunner/model/architectures (new package)
x/create create
x/safetensors fs/safetensors
x/tokenizer mlxrunner/tokenizer
Every package keeps its name, so the Go changes are the import path
rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift
check and Darwin payload script follow the new paths. Four edits are not
paths: the runner's blank architecture imports become the package
mlxrunner/model/architectures, so the list to extend for a new model sits
beside the architecture directories; a depguard rule keeps the two test
harnesses out of non-test code, as the x/internal placement used to; the
CI change filter's two entries for the long-deleted x/imagegen/mlx now
name the bindings' CMake project and the carried patches, so a change to
either builds the payload; and the tokenizer parity test reads its
fixtures from its own testdata instead of walking out of x/.
x/server and x/imagegen/manifest stay for the next two commits.
Daniel HiltgenandGitHub
Patrick DevineandGitHub
Jeffrey MorganandGitHub
Jesse Gross