Abandoning a """ block left the REPL in multiline mode, so the next
line was swallowed or saved as the system message. Reset the multiline
state along with the buffer whenever readline returns an error.
Fixes#19340
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mlxrunner: batch speculative verification
Keep drafting active for concurrent generation by sharing target verification while preserving per-request sampling and cache rollback. Learn draft depth for each concurrent workload separately from isolated requests, and preserve probes that short requests never execute.
Materialize owned target features before a reader can pause, and cover acceptance, cancellation, handoffs, mixed decoding and retained-memory boundaries.
* address comments
Let packagers supply MLX-C and XGrammar binaries independently. Stage
MLX-C in the runner directory, build Ollama's XGrammar wrapper against
the supplied library, and allow additional Go linker flags.
On Windows, pinned host memory for an integrated GPU is charged to the
same WDDM shared-memory budget as its device buffers. llama-server turns
off mmap for iGPUs and places CPU-side weights in pinned host memory,
which --fit does not count, so a partially offloaded model overflows the
budget and the KV cache allocation fails. Pass --no-host for Windows
iGPUs so CPU-side weights stay in pageable memory.
* mlx: batch requests through a shared scheduler
Use one request queue and scheduler for generation, decisions, embeddings
and speech, including parallelism one. Share compatible model operations
while keeping request caches, sampling state and response streams independent.
Interleave prompt chunks with decoding and retain speculative decoding when
generation runs alone. Cover isolation, cancellation, slow readers, cache
reuse and mixed requests with unit and integration tests. Remove the stale
llama-server architecture restriction on parallel requests.
* mlx: prevent cache snapshots from retaining batch peers
Copy each request’s initial batched K/V row so a sliding-cache snapshot cannot retain the whole batch allocation while accounting for only one row.
Exclude grammar compilation waits from prompt timing and reuse row-selected attention masks across layers.
Return errors for requests rejected before admission and strengthen numerical and compiled-speech regression checks.
* address comments
Non-mmap loads put CPU-resident tensors in the Vulkan host buffer, which is
drawn from the same device budget as the weights. On a shared-memory iGPU that
roughly doubles the footprint and the load fails with "Failed to allocate
pinned memory" or, on Windows AMD, 0xe06d7363.
Direct I/O does not avoid the host buffer, so Vulkan uses mmap rather than
reusing the CUDA/ROCm dio policy.
Make --restore confirm before changing an active profile, leave a closed
app closed, and preserve existing settings and chat history. Let interactive
users finish ChatGPT's own quit prompt instead of force-quitting it.
Carry cancellation and timeouts through status checks, quit requests and
exit polling. Check cancellation before profile writes, and keep legacy
PID and singleton probes read-only until shutdown is confirmed, including
when a legacy client starts after preflight.
Keep context checks around process queries so a canceled probe cannot be
mistaken for a confirmed exit. Publish the legacy-quit test marker with an
atomic rename so cancellation cannot interrupt its write.
Validation: 591 launch tests, focused race checks, CLI build, and 100 repeated
legacy-quit cancellation checks with the race detector.
* Fix ChatGPT restore CI setup
Stub installed-app discovery in the late legacy-start consent test so it
runs without ChatGPT installed on the CI host.
Document the containedctx exception for the context implementation that
observes spinner cancellation synchronously. This preserves the existing
cancellation behavior while satisfying the repository lint configuration.
Add the Ripple model, renderer, parser and import registration.
Support per-layer NoPE in DFlash drafts.
Fuse MLP activation, value blending and attention gating.
Add reference fixtures and model/runner regression tests.
* MLX: reduce Qwen3.8 Flash Next memory use
For NVFP4 creation, retain source precision for sensitive QSA projections
and non-expert MTP weights. Keep routed experts and PLE at NVFP4, and
promote other eligible weights to MXFP8.
Keep NVFP4 PLE tables host-backed, transferring only selected packed rows
and their scales to MLX. Honor scalar and per-expert NVFP4 global scales
in MoE, and per-shard global scales in host-backed PLE.
Use compressed QSA block selections in fused indexed Metal attention for
long prefill batches, retaining the graph fallback. Co-locate component
loading and forward methods, with explicit quantization and cache ownership.
* mlx: skip materializing host-backed tensors
Let models exclude tensors already owned by host storage from MLX loading.
Avoid allocating and discarding the full Qwen PLE tables at startup while
preserving the resident fallback.
* address comments
These architectures were forced to num_parallel=1 because the older Go
engine's recurrent caches held one sequence. llama-server supports
multiple slots for all of them so we can allow parallelism now.
NVIDIA's libcusolver looks for its dependencies only in its own
directory, so libcublas and libnvJitLink resolved to the host's copies
and MLX failed to load when those didn't match the bundled ones. Preload
the bundled copies by full path before loading MLX.
The OpenAI-compatible endpoints filled in sampler options the client
left out: temperature and top_p fell back to 1.0, and /v1/completions
always sent frequency_penalty and presence_penalty as 0. Request
options are applied last, so these values replaced the model's packaged
defaults. /api/chat honored those defaults; /v1 did not.
Omitted options are now left unset on /v1/chat/completions,
/v1/completions and /v1/responses, so the model's parameters apply, the
same as on /api/chat. Values the client sends still win, including an
explicit 0. On /v1/completions, top_p and the penalties are now
pointers so "omitted" and "0" can be told apart.
Clients that send no sampler options now get the model's defaults
instead of 1.0/1.0. A model with no sampler parameters gets Ollama's
defaults (temperature 0.8, top_p 0.9), as on /api/chat.
Fixes#17744Fixes#18690
MLX's argpartition sorts the whole row, and a row longer than one block
of the GPU sort takes a multi-pass merge sort. TopKIndices sorts each
2048-value chunk in a single block, then only the chunks' top k, and
returns the same indices in the same order.
The sampler's top_k and top_logprobs and the DFlash2 candidate selector
use it in place of a full-vocabulary argpartition.
Fixes for running LFM3 GGUFs on the llama-server runner, plus GGUF
text-to-speech that turns on once llama.cpp is bumped past b11901.
- BOS: honor an explicit tokenizer.ggml.add_bos_token. LFM3 GGUFs (and
lfm2:24b) set it to false and expect the template's BOS, which the
lfm2 default was stripping. The lfm2 default still applies when the
key is absent. On lfm2:24b, WikiText PPL goes from 81.3 to 72.0.
- repeat_last_n -1 now maps to the context size; llama-server rejects
negative values.
- Audio-only projectors (LFM3-Audio) no longer advertise vision, so
clients don't send images the model can't take.
- /v1/audio/speech on GGUF models calls llama-server /tts chunk by
chunk. Renderers opt in through a new SpeechPrompter (voices,
voice parsing, chunking), so other speech models can add their own.
The LFM3 voice and chunking helpers move from the MLX runner into
model/renderers and are shared by both runners.