* MLX: reduce Qwen3.8 Flash Next memory use
For NVFP4 creation, retain source precision for sensitive QSA projections
and non-expert MTP weights. Keep routed experts and PLE at NVFP4, and
promote other eligible weights to MXFP8.
Keep NVFP4 PLE tables host-backed, transferring only selected packed rows
and their scales to MLX. Honor scalar and per-expert NVFP4 global scales
in MoE, and per-shard global scales in host-backed PLE.
Use compressed QSA block selections in fused indexed Metal attention for
long prefill batches, retaining the graph fallback. Co-locate component
loading and forward methods, with explicit quantization and cache ownership.
* mlx: skip materializing host-backed tensors
Let models exclude tensors already owned by host storage from MLX loading.
Avoid allocating and discarding the full Qwen PLE tables at startup while
preserving the resident fallback.
* address comments
MLX's argpartition sorts the whole row, and a row longer than one block
of the GPU sort takes a multi-pass merge sort. TopKIndices sorts each
2048-value chunk in a single block, then only the chunks' top k, and
returns the same indices in the same order.
The sampler's top_k and top_logprobs and the DFlash2 candidate selector
use it in place of a full-vocabulary argpartition.
Fixes for running LFM3 GGUFs on the llama-server runner, plus GGUF
text-to-speech that turns on once llama.cpp is bumped past b11901.
- BOS: honor an explicit tokenizer.ggml.add_bos_token. LFM3 GGUFs (and
lfm2:24b) set it to false and expect the template's BOS, which the
lfm2 default was stripping. The lfm2 default still applies when the
key is absent. On lfm2:24b, WikiText PPL goes from 81.3 to 72.0.
- repeat_last_n -1 now maps to the context size; llama-server rejects
negative values.
- Audio-only projectors (LFM3-Audio) no longer advertise vision, so
clients don't send images the model can't take.
- /v1/audio/speech on GGUF models calls llama-server /tts chunk by
chunk. Renderers opt in through a new SpeechPrompter (voices,
voice parsing, chunking), so other speech models can add their own.
The LFM3 voice and chunking helpers move from the MLX runner into
model/renderers and are shared by both runners.
DFlash2 adds block-local dynamic convolutions and a candidate selector
to DFlash. The selector proposes each token conditioned on its
predecessor and returns each position's candidate distribution, which
verification uses to keep the output exact. Like the reference, the
draft samples with the request's temperature only.
A DFlash2 session drafts seven tokens every round on the existing
snapshot verify path instead of following the depth controller, whose
shallower picks lose more accepted tokens than they save in verify time.
qwen3.5-family targets now expose the layer taps, token embeddings and
raw logits a DFlash draft needs.
* MLX, MLX-C: version bump
The MLX-C pin now provides its own bindings for gather_qmm global_scale
and fast::gated_delta_update, so those two carried patches drop out.
Carry 0001-mlx-c-sdpa-window-size instead: sliding-window attention
added a window_size argument to fast::scaled_dot_product_attention
(ml-explore/mlx#4734) that MLX-C has not bound yet, to be regenerated
from MLX-C once the upstream binding PR merges. The SDPA wrappers gain
a window size argument; all call sites default to the full context.
* mlxrunner: dispatch sliding-window prefills to the native SDPA window
Sliding layers no longer materialize a causal∩sliding mask for prefill
chunks past the window: resolve() dispatches {causal, window} natively
when the rotating cache's L>1 concat has linearized the ring and the
mask is otherwise pure causal, via a new MaskApplier.NativeWindow hint.
Vision relaxations, padding, model arrays, and decode keep the
materialized path. No model changes; sinks compose natively.
Dynamic sizing can floor the smallest image grid below its intended 70-token budget, losing detail in small images. Round that grid up for the compact tower, capped at the next supported budget.
Keep explicit budgets and the existing sizing for larger and unified towers. Update geometry and media-expansion tests.
* mlxrunner: spell out symbols, numbers and links for Phonon
* docs: describe speech text normalization
* mlxrunner: fix Phonon misreadings inherited from xn-ptts
* mlxrunner: fix remaining Phonon misreadings
* mlxrunner: fold Phonon letters from the vocab, read #, _ and lists
* mlxrunner: add xn-ptts license notice for Phonon
* mlxrunner: move the xn-ptts license notice into normalize.go
* mlxrunner: fix remaining Phonon readings and port elongations
* docs: speech reads times and numeric dates
* mlxrunner: fix Phonon readings found in a listening pass
* docs: speech reads times and dates as written
* renderers: add gemma4-nemotron renderer
Add a Gemma 4 renderer variant that renders assistant content before
its tool calls rather than after the tool responses. Models opt in with
RENDERER gemma4-nemotron; gemma4 auto-detection is unchanged.
Adds the corresponding chat template as a test fixture for Jinja parity
checks.
* mlx: support Gemma 4 targets with DFlash drafts
Load a draft-owned output head when its config requires one, while preserving the target-head path used by existing DFlash models.
Expose Gemma 4 hidden-state taps and raw logits for speculative decoding. Report safetensors target precision independently of quantized draft weights.
Replace the prerelease checkpoint path with Gradium's GA format, including default-voice import and speech conditioning. Set the 0.40.9 minimum, update the speech docs, and test import, inference setup, and rejection of the old checkpoint.
* mlxrunner: add Phonon text-to-speech support
Add Phonon text-to-speech through the existing MLX runner, with an
OpenAI-compatible speech endpoint and CLI playback.
Include model loading, text preprocessing, streaming audio, and
regression tests.
* docs: add speech models
* fix min version
---------
Co-authored-by: Daniel Hiltgen <daniel@ollama.com>
Preserve source tool JSON alongside decoded API values so Flint can retain
numeric types and schema order in prompts. Existing API marshaling is unchanged.
Assemble ModelOpt expert banks during safetensors import, and reject direct
Flint NVFP4 creation before conversion. Other models retain their existing
import and quantization paths.
Add optional learned sinks and device causal masks to shared MLX attention.
Existing callers retain the no-sink and host-mask paths; Flint opts into the
new behavior.
* ci: shorten Windows Vulkan build path
Keep the nested shader-generator compiler probes below Windows path limits in checkouts with longer repository names.
* mlx: fix Nemotron bugs and support global scales
Accept published Nemotron layer names while retaining legacy aliases, and
fix layer-count inference and detached MTP quantization. Stabilize Mamba
softplus, preserve FP32 vision normalization, and compile Mamba gate and
norm scaling.
Support per-expert F32 global scales and preserve activation-scale
companions in ModelOpt and compressed-tensors NVFP4 imports. Normalize
reciprocal globals and reject invalid scales. Keep activation scales
separate from weight-only inference.
* mlx: add Nemotron 3.5 Super
* minor adjustments
* Add Kev decision scoring to MLX
Load Kev models with their FP32 decision head and release metadata.
Support System One scoring with shared-state caching and Kev-specific limits.
* address comments
Implements the EmbeddingGemma2Model architecture on the MLX runner: 24-layer
bidirectional text encoder with PLE, shared gemma4 vision/audio towers,
mean-pool + L2 output. /api/embed accepts per-item media via input dicts.
* mlxrunner: restore System One scoring
Carry the MLX implementation and its tests from the original System One
commit. The API/compiler and llama-server implementation already exist on
the release branch, so retain those versions. Adapt the optional model test
to the renamed decision package and its explicit rendering/context setup.
(cherry picked from commit 26e546ddb3)
* Enable MLX System One and add regression coverage
* mlxrunner: add Laya support for System One
Import Laya checkpoints and implement bidirectional decision scoring.
* mlxrunner: release cached buffers after System One scoring
* decision: match Tev1 and Nimble publisher prompt formats
Select Tev1 prompts using renderer metadata and fix JSON formatting, with regression coverage for GGUF and MLX.
* mlx: support Clef and Clef Flash decision models
Add model import, quantization safeguards, and text/image scoring with the Clef joint decision head.
* support vision capabilities for decision models
* Make Clef scoring cancellable between prefill chunks
Preserve joint-head inputs and test cancellation, cleanup, and recovery.
* mlx: separate decision preparation and readout from forward
Let serial and batched scoring share model-owned preparation and readout
without duplicating architecture-specific logic in the runner.
* mlx: add Strands Decider support
* implement caching
---------
Co-authored-by: ParthSareen <parth.sareen@ollama.com>
* mlx: speed up Qwen 3.8 prompt processing
Use MLX's gated-delta kernel for long scans and fold dense MLP global scales into SwiGLU.
* address comments
Replace the fixed checkpoint image budget with per-image selection across
the supported 70, 140, 280, 560, and 1120 budgets. Choose the publisher resize
grid closest to the input resolution, accounting for aspect ratio.
This preserves more detail in high-resolution documents while allowing
smaller images to use fewer tokens, without adding an API parameter.
Cover budget boundaries, extreme dimensions, position limits, and media
expansion for both vision architectures.
* nemotron_h: add MLX vision support
Implement the RADIO vision encoder and projector on the shared MLX media pipeline, including dynamic-resolution preprocessing, deterministic placeholder expansion, chunked feature scattering, and MTP offsets.
Expose source-advertised Nemotron vision while continuing to suppress unsupported audio, and preserve both modality towers at source precision during create.
Harden Nemotron streaming parser termination and add focused coverage for vision configuration, media placement, capability reporting, and tool-call parsing.
* review comments
* address comments
model is one package with three jobs: the contract between the runner
and the architectures, the opened checkpoint, and building nn layers
from checkpoint tensors. Its files did not say which was which. base.go
carried the folded package's name over the interfaces and the registry,
root.go held the safetensors header scan next to Root, and quant.go
mixed the nvfp4 global-scale helpers with quant parameter resolution.
base.go becomes model.go, named for what it holds. root.go keeps Root
and Open; TensorQuantInfo and the header scan join quant.go, so
everything the checkpoint says about quantization is read and resolved
in one file. The global-scale helpers move to globalscale.go with their
tests. Root.Close, a no-op with one caller, goes. No code changes
otherwise.
The runner and its weight loader read a model's manifest through
x/imagegen/manifest, the last piece of the removed image generation
engine. It was a hand-rolled copy of the manifest package: its own model
name parser with the default registry and namespace spelled out, its own
blob path builder, and re-spelled media types, plus a model_index.json
reader and other helpers that nothing has called since the engine went.
The manifest package gains the three lookups the runner needs, a config
layer by path, its contents, and the tensor layers, and ReadConfigJSON is
built on the second of them. The weight loader resolves the model name
with the shared parser, which fills in the same defaults the copy did, and
locates blobs with BlobsPath. The architectures' calls to read their
config.json compile unchanged. x/imagegen is gone.
model/base held the Model interface and the architecture registry while
model held weight loading and quant parameters, and every architecture
imported both. create/client was the CLI side of safetensors imports,
with the create command as its only caller, a duplicate of the command's
adapter error, and a name that read like a second API client. The
safetensors show helpers had their own package under x/ although it
already declared package server.
base merges into model, so base.Model and base.Register become
model.Model and model.Register; nothing in the two overlapped. The create
client's two files and their tests join package cmd, and the five names
the command called are no longer exported. The show helpers join the
server package, and the three entry points routes.go calls become
unexported like the helpers around them.
The MLX runner is the only Go inference runner left and is no longer
experimental, so its packages leave x/. The bindings become a top-level
mlx package beside the carried patches in mlx/compat, mirroring how
llama/ holds the llama.cpp integration, and the runner becomes mlxrunner
with the architectures nested under the package they implement.
Subpackages move with their parent unless listed.
x/mlxrunner/mlx mlx
x/internal/mlxthread mlx/mlxthread
x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest
x/internal/mlxtest mlx/mlxtest
x/quant mlx/quant
mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx)
x/mlxrunner mlxrunner
x/models/nn mlxrunner/nn
x/models/<arch> mlxrunner/model/<arch>
x/mlxrunner/imports.go mlxrunner/model/architectures (new package)
x/create create
x/safetensors fs/safetensors
x/tokenizer mlxrunner/tokenizer
Every package keeps its name, so the Go changes are the import path
rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift
check and Darwin payload script follow the new paths. Four edits are not
paths: the runner's blank architecture imports become the package
mlxrunner/model/architectures, so the list to extend for a new model sits
beside the architecture directories; a depguard rule keeps the two test
harnesses out of non-test code, as the x/internal placement used to; the
CI change filter's two entries for the long-deleted x/imagegen/mlx now
name the bindings' CMake project and the carried patches, so a change to
either builds the payload; and the tokenizer parity test reads its
fixtures from its own testdata instead of walking out of x/.
x/server and x/imagegen/manifest stay for the next two commits.
Daniel HiltgenandGitHub
Patrick DevineandGitHub
Jeffrey MorganandGitHub
Jesse Gross