* mlxrunner: add Phonon text-to-speech support
Add Phonon text-to-speech through the existing MLX runner, with an
OpenAI-compatible speech endpoint and CLI playback.
Include model loading, text preprocessing, streaming audio, and
regression tests.
* docs: add speech models
* fix min version
---------
Co-authored-by: Daniel Hiltgen <daniel@ollama.com>
ShowResponse's renderer and parser fields existed but the server never
set them; they now carry the effective values (renderer resolves the
gemma4 legacy alias by model size). A new template_source names what
actually renders chat prompts: the configured renderer/parser (leaving
any packaged template inert), a Go template layer, the GGUF's embedded
jinja, harmony, or none.
The CLI prints renderer/parser/template source rows only when set, so
ollama show output is unchanged for models without rendering config.
Add manifest-list storage so runner-specific manifests can coexist under one tag while preserving existing v1 tags as best-effort downgrade anchors. Show/list/copy/remove/pull/push now understand runner and digest selection and transfer referenced child manifests and layers.
Add lazy local compatibility migration for legacy Ollama GGUFs into llama.cpp-compatible children, covering the patched model families and preserving parser/renderer, templates, projectors, media metadata, split GGUFs, and MTP/draft tensors where applicable.
Extend create/import/conversion paths for the same compatibility rules, harden manifest-list combine validation, and add focused manifest, migration, transfer, and integration coverage for first-load conversion plus chat/tools/vision/audio/embedding behavior.
Switch from rejecting API requests to logging a warning. Model creation still rejects the deprecated param.
Follow up from #18448 to remove the hard API failure.
model/base held the Model interface and the architecture registry while
model held weight loading and quant parameters, and every architecture
imported both. create/client was the CLI side of safetensors imports,
with the create command as its only caller, a duplicate of the command's
adapter error, and a name that read like a second API client. The
safetensors show helpers had their own package under x/ although it
already declared package server.
base merges into model, so base.Model and base.Register become
model.Model and model.Register; nothing in the two overlapped. The create
client's two files and their tests join package cmd, and the five names
the command called are no longer exported. The show helpers join the
server package, and the three entry points routes.go calls become
unexported like the helpers around them.
The MLX runner is the only Go inference runner left and is no longer
experimental, so its packages leave x/. The bindings become a top-level
mlx package beside the carried patches in mlx/compat, mirroring how
llama/ holds the llama.cpp integration, and the runner becomes mlxrunner
with the architectures nested under the package they implement.
Subpackages move with their parent unless listed.
x/mlxrunner/mlx mlx
x/internal/mlxthread mlx/mlxthread
x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest
x/internal/mlxtest mlx/mlxtest
x/quant mlx/quant
mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx)
x/mlxrunner mlxrunner
x/models/nn mlxrunner/nn
x/models/<arch> mlxrunner/model/<arch>
x/mlxrunner/imports.go mlxrunner/model/architectures (new package)
x/create create
x/safetensors fs/safetensors
x/tokenizer mlxrunner/tokenizer
Every package keeps its name, so the Go changes are the import path
rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift
check and Darwin payload script follow the new paths. Four edits are not
paths: the runner's blank architecture imports become the package
mlxrunner/model/architectures, so the list to extend for a new model sits
beside the architecture directories; a depguard rule keeps the two test
harnesses out of non-test code, as the x/internal placement used to; the
CI change filter's two entries for the long-deleted x/imagegen/mlx now
name the bindings' CMake project and the carried patches, so a change to
either builds the payload; and the tokenizer parity test reads its
fixtures from its own testdata instead of walking out of x/.
x/server and x/imagegen/manifest stay for the next two commits.
The runner package used to pick an engine from the first argument of the
runner subcommand. Only the MLX engine is left, so the dispatcher has a
single arm, its README still describes the removed Go runner's flags and
endpoints, and the standalone cmd/runner binary exists only to invoke it.
We call mlxrunner.Execute directly from the hidden runner subcommand and
drop the --mlx-engine argument from the command line the MLX client
spawns. Both sides ship in the same binary, so nothing has to accept both
forms. The runner package and cmd/runner are removed.
The subcommand's help hook hands the runner a bare --help. cobra calls the
hook with no arguments for `ollama help runner`, which used to index past
the end of the slice.
* create: add server-side MLX imports and drop GGUF conversion
Support safetensors imports through the MLX create pipeline both locally and on the server, including remote upload/staging, draft layer handling, cancellation propagation, transfer limits, and shared manifest/blob writing.
Limit GGUF create to wrapping existing GGUF inputs into Ollama manifests. Remove the in-tree safetensors-to-GGUF converter, server quantization path, and converter-only dependencies so GGUF conversion and quantization stay in llama.cpp tooling.
Keep the MLX path focused on supported safetensors model creation with validation before MLX work, and expose that flow without the --experimental CLI gate.
* address comments
* add client side gguf create fast path
* address comments
* rebase adjustments
Remove the x/imagegen tree (MLX image generation engine, Flux2/zimage
models, cache, C bindings) and all imagegen integration points:
- server: drop imagegen routes, scheduling, and generate handling
- api/cmd/docs: remove image generation API surface and docs
- middleware/openai: remove image endpoint support
- integration: remove imagegen test suites
- x/create: adopt the rewritten create pipeline from main; drop
imagegen create path (CreateImageGenModel, IsTensorModelDir,
model_index.json detection, Flux2KleinPipeline vision hack)
- retain x/imagegen/manifest (Ollama-store safetensors manifest
loader), still used by x/mlxrunner and x/create/client
- fix Windows MLX dl.dll install, MLX CMake version path, and the
show command after removing safetensors models
The bare `ollama` command (and `ollama launch` with no integration) used a
bespoke `ensureServerRunning` that forked `ollama serve` directly and polled
its heartbeat forever (no timeout, no platform-aware launch). Every other
subcommand (`ollama run`, `ollama pull`, `ollama launch <integration>`, ...)
goes through `checkServerHeartbeat` -> `startApp`, so the root command behaved
differently and could hang indefinitely.
Route `runInteractiveTUI` through `checkServerHeartbeat(cmd, nil)` — the same
path `ollama launch <thing>` uses — so the root command is consistent and no
longer runs an unbounded server-spawn loop. `ensureServerRunning` and its
`backgroundServerSysProcAttr` helpers (only it referenced them) are removed,
along with the now-unused `os/exec` import.
The platform `startApp`/`waitForServer` paths are unchanged, so behavior on
macOS/Windows is identical to the other subcommands, and on Linux the root
command now errors the same way the subcommands already do when no server is
running.
* broad lint fixes to sidestep CI scope glitch
* runner: Remove CGO engines, use llama-server exclusively for GGML models
Remove the vendored GGML and llama.cpp backend, CGO runner, Go model
implementations, and sample. llama-server (built from upstream llama.cpp via
FetchContent) is now the sole inference engine for GGUF-based models.
(Safetensor based models continue to run on the new MLX engine.) This allows
us to more rapidly pick up new capabilities and fixes from llama.cpp as they
come out.
On windows this now requires recent AMD driver versions to support ROCm v7 as
llama.cpp currently does not support building against v6.
* llama/compat: load Ollama-format GGUFs in llama-server
Squashed from upstream/jmorganca/llama-compat on 2026-04-29.
Source tip: 0c33775d37.
Original source commits:
- 25223160d llama/compat: add in-memory shim so llama-server can load Ollama-format GGUFs
- 7449b539a llm,server: route Ollama-format gemma3 blobs through llama/compat
- 436f2e2b1 llama/compat: make patch-apply idempotent
- 8c2c9d4c8 llama/compat: extend gemma3 handler to cover 1B and 270M blobs
- 021389f7b llama/compat: shrink clip.cpp injection from 18 lines to 1
- 61b367ec2 llama/compat: shrink patch to pure call-site hooks (34 -> 20 lines)
- 36049361c llama/compat: simplify shim (gemma3-tested)
- 8fa664865 llama/compat: add qwen35moe text handler
- db0c74530 llama/compat: add qwen35moe vision (clip) support
- 2a388da77 llama/compat: split shared infra into a util TU
- 9a69a17dc llama/compat: document non-public API dependencies
- d0f38a915 llama/compat: add gpt-oss and lfm2 handlers
- 086071822 llama/compat: add mistral3 text handler (vision TODO)
- 63bde9ff7 llama/compat: add mistral3 vision (clip) support
- 3a57b89d5 llama/compat: apply LLaMA RoPE permute to mistral3 vision Q/K
- 99cb87439 llama/compat: add qwen35, gemma4, deepseek-ocr handlers
- 2c7850dba llama/compat: add nemotron_h_moe handler (latent FFN + MTP skip)
- 9e3b54225 llama/compat: add llama4 text + clip handlers
- 034fee349 llama/compat: add gemma4 clip handler (gemma4v projector)
- 9945c5a93 server: remove dhiltgen/* compat redirect table
- 5d4539101 llama/compat: rewrite gemma4 tokenizer model to BPE
- 7e0765327 llama/compat: add glm-ocr text handler + text-loader load-op hook
- f1bd1a25a llama/compat: add glm-ocr clip handler (glm4v projector)
- 4b5cf3420 llama/compat: collapse text-loader hook back to one new patch line
- eb4ecf4fc llama/compat: extend gemma4 clip handler to gemma4a (audio)
- a23a5e76f llama/compat: fix gemma4a per-block norm tensor mapping
- cd2dcaff4 llama/compat: add embeddinggemma handler
- 1ce8a6b26 llama/compat: add qwen3-vl + qwen2.5-vl handlers
- fd98ffa1e llama/compat: add gemma3n + glm4moelite handlers
- cc7bdf0bc llama/compat: handle null buft in maybe_load_tensor
- 0c33775d3 llama/compat: disable mmap when load_op transforms text-side tensors
* refine implementation
* ci: fix windows MLX build
* ci: fix windows llama-server build
* ci: fix windows rocm build
* ci: windows mlx tuning
Shorten long-tail on build, and get OllamaSetup.exe back under 2g limit
* ci: fix windows dependencies
* win: fix dependency gathering
* disable openmp
* win: arm64 cross-compile build
also DRY out CI steps
* scheduler improvements
* ci: improvements from #15982
* win: favor ninja for faster developer builds
* win: fix build
* win: fix arm64 cross-compile
* win: avoid spaces in compiler path
* misc discovery fixes, and bos handling
* lint fixes
* win: fix arm cross-compile build/CI bugs
* llama.cpp update
* win: handle multiple CRT dirs
* vulkan: add windows iGPU detection
* fix creation bugs for patched models, other refactoring work
* tune batch size for better performance
* ci and lint fixes
* fix repeat_last_n bug
* build: revamp build for better developer UX
* amd, sampler, qwen3next fixes
* version bump
* fix mlx build
* revamp GPU discovery
Scanning the output of llama-server is turning out to be too error prone across
llama.cpp updates, so this switches to a thin dynamic library load against the
bundled GGML libraries so more details can be gathered from the API.
* version bump
* missing file
* ci: fix cache miss on rocm build
* refine vulkan dep handling
* fix ps reporting bug on full GPU load
* improve cmake wiring for customized local builds
* version bump
* docker build arg cleanup
* improve windows exit error logs
* fix community gemma4 support and ci flakes
* fix mlx unit test
* tighten up ps logic to avoid double counting fit log lines
* version bump
* fix ps view for full gpu layer offload
* add MTP wiring for llama-server and create with GGUFs
* pick best template by capabilities
* version bump
* ci: harden apt repos
* remove unused cpu core discovery
* adjust batch default logic to reduce OOMs
* support larger tool calls
* fix audio support, template show
* qwen35 mtp patch support
* flesh out dtypes
* rocm deps
* version bump
* lint fix
* block broken gfx1150 on windows
* fix qwen3.5 moe mtp tensors in patch
* mmproj oom fallback and vulkan on by default
* qwen MTP compat fix
* version bump
* ci: fix WoA cross-compile
* ci: workaround ui tool in cross-compile
* version bump
* win: enable OpenMP for CPU builds
* build: improve developer UX
* ci: windows path workaround for CPU build
* win: fix WoA dependencies
* win: fix large offset reads for mmproj patched loads
* version bump
* fix vulkan dup detection
* add OLLAMA_IGPU_ENABLE and largely disable iGPUs by default
* opt-in MTP, win large offset, integraton fixes
* fix unit test scheduler interaction hang
* fix multi-gpu filtering
* version bump
* review comments
* fix thinking level
* fix linux rocm ordering and granite 3.3 template
* version bump
* ci fix - non-shallow MLX checkout
* bypass linux sysfs unit test on windows
---------
Co-authored-by: jmorganca <jmorganca@gmail.com>
* mlx: refined model push behavior
Refine the algorithm for parallel push of safetensors based models to get
better reliability and throughput.
* review comments, hardening, and performance tuning for slow links
* review comments
This change adds support for MTP (multi-token prediction) speculative decoding for the
gemma4 model family.
It includes:
* support for importing safetensors based gemma4 draft models with `ollama create`
* a new DRAFT command in the Modelfile for specifying draft models
* a --quantize-draft flag for the ollama create command to quantize the draft model
* cache support for speculation
* changes to the rotating cache to be able to handle MTP correctly
* sampling support for draft model token prediction
---------
Co-authored-by: Daniel Hiltgen <daniel@ollama.com>
launchInteractiveModel was introduced in PR #14609 without the
client.Show() capability-detection block that RunHandler uses.
This left opts.MultiModal always false in the TUI path, causing
image/audio file paths to always be treated as unknown commands
instead of being loaded as multimodal attachments.
Mirror the Show() call, pull-on-404 fallback, cloud auth handling,
and MultiModal/Think population from RunHandler into
launchInteractiveModel.
Fixes#15711
This change fixes two issues with Modelfiles:
1. If a user uses `ollama show --modelfile` to show a safetensors based
model, the Model would leave the "FROM" field blank which won't allow
a user to recreate the model. This change adds the model's current
canonical short name to the FROM field.
2. If a user uses the `/save` command in the CLI any messages which were
saved in a previous model wouldn't get saved (only the set of messages
from the current session).
* create: Clean up experimental paths
This cleans up the experimental features, and adds both unit and integration test coverage to verify no regressions.
* create: preserve config and layer names when creating from safetensors models
When creating a model FROM an existing safetensors model, ModelFormat,
Capabilities, and layer Name fields were lost. ModelFormat stayed empty
because it's only set from GGML layers (which safetensors models lack),
and layer names weren't copied in parseFromModel. This caused derived
models to fail loading ("config.json not found in manifest").
* review comments
* bench: add prompt calibration, context size flag, and NumCtx reporting
Add --num-ctx flag to set context size, and report NumCtx in model info
header. Calibrate tokens-per-word ratio during warmup using actual
tokenization metrics from the model, replacing the fixed 1.3 heuristic.
This produces more accurate prompt token counts for --prompt-tokens.
Also add fetchContextLength() to query running model context via /api/ps.
* integration: improve vision test robustness and add thinking tests
Add skipIfNoVisionOverride() to skip vision tests when OLLAMA_TEST_MODEL
is set to a non-vision model. Add Think:false to context exhaustion test
to prevent thinking models from using all context before the test can
measure it. Add third test image (ollama homepage) and replace OCR test
with ImageDescription test using it. Relax match strings for broader
model compatibility. Add TestThinkingEnabled and TestThinkingSuppressed
to verify thinking output and channel tag handling.
* gemma4: add Gemma 4 GGML model support
Add full Gemma 4 model family support (E2B, E4B, 26B MoE, 31B Dense)
for the GGML backend including text, vision, converter, parser, and
renderer.
Text model features:
- Sliding window + full attention with per-layer patterns
- KV sharing across layers with donor map
- Per-layer embeddings (PLE) with learned projections
- MoE routing with RMSNorm + learned scale
- Proportional RoPE with freq_factors for global attention
- Final logit softcapping
Vision model features:
- SigLIP vision encoder with 2D RoPE
- ClippableLinear with input/output clamping via packed v.clamp_data
- Adaptive average pooling with nMerge kernel
- Multi-modal projection with unweighted RMSNorm
Converter:
- Safetensors to GGUF with vision tensor renaming
- Fused MoE gate_up_proj splitting
- Vision patch embedding reshape (HF to Conv2D layout)
- Packed clamp data tensor for ClippableLinear bounds
- Proportional RoPE freq_factors generation
Also includes:
- BackendGet() on ml.Tensor for reading weight tensor data
- Q6_K CUDA get_rows kernel support
- MoE-aware ffn_down quantization layer counting
- Gemma4 parser with tool calling and thinking support
- Gemma4 renderer with structured tool format
- Architecture-based auto-detection of renderer/parser/stop tokens
- Integration test gemma4 model list additions
* gemma4: add audio support with USM conformer encoder
Add audio encoding for Gemma 4 using the USM conformer architecture:
- Converter: audio tensor mapping, SSCP/conformer/embedder name replacements,
softplus repacker for per_dim_scale, F32 enforcement for conv weights
- GGML backend: Conv1DDW and PadExt tensor ops
- Audio encoder: SSCP Conv2D, 12 conformer blocks (FFW + block-local
attention with relative position embeddings + LightConv1d + FFW),
output projection, audio-to-text embedding projector
- Audio preprocessing: WAV decode, mel spectrogram, FFT (pure Go)
- Model wiring: WAV detection, audio token handling, unified PostTokenize
Correctly transcribes "why is the sky blue" from test audio.
* integration: add gemma4 audio tests including OpenAI API coverage
Test audio transcription and response via the Ollama native API, plus
two new tests exercising the OpenAI-compatible endpoints:
- /v1/audio/transcriptions (multipart form upload)
- /v1/chat/completions with input_audio content type
All tests use capability checks and skip models without audio support.
* gemma4: add OpenAI audio API support and capability detection
- Add CapabilityAudio and detect from audio.block_count in GGUF
- Add /v1/audio/transcriptions endpoint with TranscriptionMiddleware
- Add input_audio content type support in /v1/chat/completions
- Add TranscriptionRequest/Response types in openai package
* gemma4: add audio input support for run command
- /audio toggle in interactive mode for voice chat
- Platform-specific microphone recording (AVFoundation on macOS,
PulseAudio/ALSA on Linux, WASAPI on Windows)
- Space to start/stop recording, automatic chunking for long audio
* gemma4: add transcribe command (ollama transcribe MODEL)
- Interactive mode with readline prompt and slash commands
- Non-interactive mode for piped audio or record-until-Ctrl+C
- Chunked streaming transcription for long recordings
- Word-wrapped output matching run command style
* gemma4: add parser, renderer, and integration test plumbing
* gemma4: fix renderer to emit BOS token
* gemma4: add OpenAI audio transcription API and input_audio support
* gemma4: update converter for new weight drop naming
* gemma4: add per_expert_scale to MoE router and fix moe_intermediate_size config
* gemma4: rewrite renderer to match HF Jinja2 template exactly
Fix 8 bugs found by building 55 reference tests verified against the
HF Jinja2 chat template (VERIFY_JINJA2=1 shells out to Python):
- Tool responses use separate <|turn>tool turns (not inline tags)
- Tool calls emitted before content in assistant messages
- Thinking content stripped from assistant history (strip_thinking)
- User, tool, and system content trimmed (template does | trim)
- Empty system message still emits system turn (check role, not content)
- Nested object properties rendered recursively with required field
- Array items specification rendered for array-type properties
- OBJECT/ARRAY type-specific rendering comma logic matches template
Also adds Required field to api.ToolProperty for nested object schemas,
replaces old gemma4_test.go with comprehensive gemma4_reference_test.go,
and commits the Jinja2 template as testdata for verification.
* gemma4: fix MoE fused gate_up split and multiline tool-call arg parsing
- Text MoE: split `ffn_gate_up_exps` into contiguous `[gate|up]` halves instead of stride-2 slices.
- Parser: escape control characters in `<|"|>...<|"|>` string literals when converting tool-call args to JSON.
- Fixes warnings like `invalid character '\n' in string literal` for multiline tool arguments.
- Add Gemma4 parser regressions for multiline tool-call args and `gemma4ArgsToJSON`.
* cmd: simplify audio input to dropped file attachments
* gemma4: use full SWA memory for better cache reuse
* gemma4: initialize clamps after backend load
* convert: align gemma4 audio tensor renames with llama.cpp
* Remove redundant comments in gemma4 vision model
* Format Gemma4 MoE block field alignment
* use 4096 kvcache.NewSWAMemCache
* convert: support new Gemma4 audio_tower tensor naming (#15221)
Co-authored-by: jmorganca <jmorganca@gmail.com>
* fix integration test defaults for audio
* review comments and lint fixes
* remove unused audio/video files
---------
Co-authored-by: jmorganca <jmorganca@gmail.com>
Eva HandGitHub
Daniel HiltgenandGitHub
Jesse Gross
Devon RifkinandGitHub
Michael VerrilliandGitHub
Patrick DevineandGitHub