These architectures were forced to num_parallel=1 because the older Go
engine's recurrent caches held one sequence. llama-server supports
multiple slots for all of them so we can allow parallelism now.
Fixes for running LFM3 GGUFs on the llama-server runner, plus GGUF
text-to-speech that turns on once llama.cpp is bumped past b11901.
- BOS: honor an explicit tokenizer.ggml.add_bos_token. LFM3 GGUFs (and
lfm2:24b) set it to false and expect the template's BOS, which the
lfm2 default was stripping. The lfm2 default still applies when the
key is absent. On lfm2:24b, WikiText PPL goes from 81.3 to 72.0.
- repeat_last_n -1 now maps to the context size; llama-server rejects
negative values.
- Audio-only projectors (LFM3-Audio) no longer advertise vision, so
clients don't send images the model can't take.
- /v1/audio/speech on GGUF models calls llama-server /tts chunk by
chunk. Renderers opt in through a new SpeechPrompter (voices,
voice parsing, chunking), so other speech models can add their own.
The LFM3 voice and chunking helpers move from the MLX runner into
model/renderers and are shared by both runners.
* decision: allow empty state for vllm-d3
* server: reuse decoded state in System One tests
---------
Co-authored-by: Patrick Devine <patrick@ollama.com>
* renderers: add gemma4-nemotron renderer
Add a Gemma 4 renderer variant that renders assistant content before
its tool calls rather than after the tool responses. Models opt in with
RENDERER gemma4-nemotron; gemma4 auto-detection is unchanged.
Adds the corresponding chat template as a test fixture for Jinja parity
checks.
* mlx: support Gemma 4 targets with DFlash drafts
Load a draft-owned output head when its config requires one, while preserving the target-head path used by existing DFlash models.
Expose Gemma 4 hidden-state taps and raw logits for speculative decoding. Report safetensors target precision independently of quantized draft weights.
* mlxrunner: add Phonon text-to-speech support
Add Phonon text-to-speech through the existing MLX runner, with an
OpenAI-compatible speech endpoint and CLI playback.
Include model loading, text preprocessing, streaming audio, and
regression tests.
* docs: add speech models
* fix min version
---------
Co-authored-by: Daniel Hiltgen <daniel@ollama.com>
* server: read loaded runner models without refMu
Concurrent lookups of a warm model failed TryLock against each other and
fell back to a manifest reload, defeating the cache and making
TestLoadedModelLookupConcurrent flaky. Make runnerRef.model an
atomic.Pointer so lookups never take refMu.
* address comments
ShowResponse's renderer and parser fields existed but the server never
set them; they now carry the effective values (renderer resolves the
gemma4 legacy alias by model size). A new template_source names what
actually renders chat prompts: the configured renderer/parser (leaving
any packaged template inert), a Go template layer, the GGUF's embedded
jinja, harmony, or none.
The CLI prints renderer/parser/template source rows only when set, so
ollama show output is unchanged for models without rendering config.
Behind a proxy, defer redirect-target DNS to the proxy; without one, keep the resolve-and-pin path. Trust default-registry/CDN hostnames with no local DNS. Fix a nil CancelFunc crash in blobDownload.release.
* Add Kev decision scoring to MLX
Load Kev models with their FP32 decision head and release metadata.
Support System One scoring with shared-state caching and Kev-specific limits.
* address comments
* server: reuse verified manifests and loaded models in inference routes
Resolve named manifests and runner selection in one verified cold lookup.
When a runner is loaded, reuse its model after checking that the v2 tag
still points to the same digest. Fully verify regular-file manifests,
including Windows copies.
Share the lookup across generate, chat, embed, and System One. Keep parser
selection request-local so reusing a model does not mutate its config.
Cover tag changes, runner selection, concurrent unloads, and existing
missing-model responses with regression tests and a lookup benchmark.
* address comments
When a legacy GGUF which requires our llama.cpp patch set is loaded, we do a
lazy conversion on disk to make it llama.cpp compatible. To support
downgrades, we temporarily hold both the new and old GGUFs and create a shadow
v1 manifest tag to prevent the downgraded server from deleting the new blobs.
Showing these in the list output is confusing to users and API consumers. The
legacy copies will be removed in a future release to recover the disk space.
Forward GET /api/usage and GET /api/balance through the local server
using the signed-in account.
Add API documentation and tests for proxying, signing, and
disabled cloud access.
Implements the EmbeddingGemma2Model architecture on the MLX runner: 24-layer
bidirectional text encoder with PLE, shared gemma4 vision/audio towers,
mean-pool + L2 output. /api/embed accepts per-item media via input dicts.
* Reduce model lookup and MLX decision request overhead
Avoid decoding unrelated manifests when resolving existing model names.
Reuse small Metal scratch buffers between decision requests, releasing
larger working sets and clearing scratch after scoring errors.
* address comments
* mlxrunner: restore System One scoring
Carry the MLX implementation and its tests from the original System One
commit. The API/compiler and llama-server implementation already exist on
the release branch, so retain those versions. Adapt the optional model test
to the renamed decision package and its explicit rendering/context setup.
(cherry picked from commit 26e546ddb3)
* Enable MLX System One and add regression coverage
* mlxrunner: add Laya support for System One
Import Laya checkpoints and implement bidirectional decision scoring.
* mlxrunner: release cached buffers after System One scoring
* decision: match Tev1 and Nimble publisher prompt formats
Select Tev1 prompts using renderer metadata and fix JSON formatting, with regression coverage for GGUF and MLX.
* mlx: support Clef and Clef Flash decision models
Add model import, quantization safeguards, and text/image scoring with the Clef joint decision head.
* support vision capabilities for decision models
* Make Clef scoring cancellable between prefill chunks
Preserve joint-head inputs and test cancellation, cleanup, and recovery.
* mlx: separate decision preparation and readout from forward
Let serial and batched scoring share model-owned preparation and readout
without duplicating architecture-specific logic in the runner.
* mlx: add Strands Decider support
* implement caching
---------
Co-authored-by: ParthSareen <parth.sareen@ollama.com>
Show and list now return just "decision" for models declaring it, so clients don't offer them for general chat, tools, or thinking. The show response also omits the thinking field for these models. Scheduling and serving still check the full capability set.
Add CAPABILITY declarations to Modelfiles and an additive capabilities
field to create requests. Preserve declarations across GGUF and safetensors
creation, inheritance, and Modelfile export.
Require decision capability before scheduling System One requests instead
of matching Qwen architecture/renderer metadata. Retain main's GGUF-only
scoring restriction until the separate MLX runtime work lands.
Extracted from the capability foundation in 36d46a0c3 on system_one_mlx;
MLX scoring and manifest-list changes are intentionally separate.
* transfer: bound stall retries and let the watchdog interrupt them
A download that stalls could never recover. copy derived its own
cancellable context, but the request was built from the parent, so
cancelling on a stall never reached the in-flight body and the read
stayed parked. The stall watchdog could detect a stall it had no way
to stop.
Stall retries were also unbounded: they never incremented attempt, so
the retry loop had no limit, and because backoff was gated on
attempt > 0 they retried with no delay at all.
The request now owns the cancellable context and passes its cancel
down to copy, so a stall aborts the read. Stalls get their own retry
budget before counting against the limit, and backoff is keyed to
every retry rather than only the counted ones.
* move out of x/
Daniel HiltgenandGitHub
Parth SareenandGitHub
Eva HandGitHub
Jeffrey MorganandGitHub
Devon RifkinandGitHub
J BergandGitHub