6007 Commits
Author SHA1 Message Date
Gaurav JainandGitHub 42ac2c0c1e Merge pull request #19748 from ollama/claude/sched-shift-no-reload-18950
server, llm: share one runner across `shift` settings
2026-11-18 14:53:02 -08:00
Daniel HiltgenandGitHub 508d8be9c8 test: stabilize speech slow-reader coverage (#19746)
This removes dependence on socket buffering and large conversion work under the
race detector which should address a flaky CI test.

Block the response writer explicitly instead of generating 32 MiB of
audio to fill socket buffers. Verify generation finishes before reading
begins and all audio is delivered afterward.
2026-11-18 08:30:00 -08:00
Daniel HiltgenandGitHub b8db8d5b5c mlxrunner: reuse saved prefixes across request slots (#19677)
Reuse completed prefix checkpoints from another slot, including one still
serving a request. Restore into recipient-owned state and copy full-attention
history directly without materializing an intermediate donor snapshot.

Cover donor lifetime, pending captures, cancellation, media and drafting
boundaries, merged history, and cached evaluation.
2026-11-16 20:54:35 -08:00
Gaurav JainandGitHub 312c3c8d0c Merge pull request #19731 from ollama/fix/repl-ctrl-c-multiline
cmd: leave multiline mode when Ctrl+C or Ctrl+G discards input
2026-11-16 19:33:27 -08:00
Gaurav JainandClaude Opus 5.5 169626b403 cmd: leave multiline mode when Ctrl+C or Ctrl+G discards input
Abandoning a """ block left the REPL in multiline mode, so the next
line was swallowed or saved as the system message. Reset the multiline
state along with the buffer whenever readline returns an error.

Fixes #19340

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:20:35 -08:00
Daniel HiltgenandGitHub db5266a71d mlxrunner: batch speculative verification (#19663)
* mlxrunner: batch speculative verification

Keep drafting active for concurrent generation by sharing target verification while preserving per-request sampling and cache rollback. Learn draft depth for each concurrent workload separately from isolated requests, and preserve probes that short requests never execute.

Materialize owned target features before a reader can pause, and cover acceptance, cancellation, handoffs, mixed decoding and retained-memory boundaries.

* address comments
v0.40.17-rc1
2026-11-16 08:43:54 -08:00
Daniel HiltgenandGitHub 9a4b136efe cmake: support compatible binary MLX-C and XGrammar (#19688)
Let packagers supply MLX-C and XGrammar binaries independently. Stage
MLX-C in the runner directory, build Ollama's XGrammar wrapper against
the supplied library, and allow additional Go linker flags.
2026-11-16 07:11:09 -08:00
saman-amdandGitHub 15af2b261c llm: keep weights out of pinned host memory on Windows iGPUs (#19636)
On Windows, pinned host memory for an integrated GPU is charged to the
same WDDM shared-memory budget as its device buffers. llama-server turns
off mmap for iGPUs and places CPU-side weights in pinned host memory,
which --fit does not count, so a partially offloaded model overflows the
budget and the KV cache allocation fails. Pass --no-host for Windows
iGPUs so CPU-side weights stay in pageable memory.
2026-11-15 10:46:15 -08:00
J BergandGitHub a5864fcb33 api: preserve unmodeled tool schema keywords (#19686) 2026-11-13 16:00:07 -08:00
J BergandGitHub 5632619189 Map disabled thinking to the models lowest supported level (#19684) 2026-11-12 15:03:07 -08:00
Parth SareenandGitHub 9da7dfd5c3 cmd: show help for noninteractive bare invocations (#19557) 2026-11-12 14:35:02 -08:00
J BergandGitHub 361b3e17fd Support deferred tool loading in claude code (#19578) 2026-11-12 13:24:56 -08:00
Anna RahnandGitHub 4a7b194120 Merge pull request #18972 from ollama/docs_restructuring
docs: restructuring to separate out cloud features
2026-11-12 10:28:01 -08:00
Eva HandGitHub 8a89b76cc7 launch: refresh Pi metadata after sign-in and honor prompt args (#19575) 2026-11-10 17:43:58 -05:00
Daniel HiltgenandGitHub 6140670fd1 MLX, MLX-C: version bump (#19647) v0.40.17-rc0 2026-11-10 13:42:11 -08:00
Daniel HiltgenandGitHub 9b390157f2 llama.cpp: version bump b12257 (#19646) 2026-11-10 13:41:48 -08:00
Daniel HiltgenandGitHub ab9295d03c mlx: support concurrent generation (#19197)
* mlx: batch requests through a shared scheduler

Use one request queue and scheduler for generation, decisions, embeddings
and speech, including parallelism one. Share compatible model operations
while keeping request caches, sampling state and response streams independent.

Interleave prompt chunks with decoding and retain speculative decoding when
generation runs alone. Cover isolation, cancellation, slow readers, cache
reuse and mixed requests with unit and integration tests. Remove the stale
llama-server architecture restriction on parallel requests.

* mlx: prevent cache snapshots from retaining batch peers

Copy each request’s initial batched K/V row so a sliding-cache snapshot cannot retain the whole batch allocation while accounting for only one row.

Exclude grammar compilation waits from prompt timing and reuse row-selected attention masks across layers.

Return errors for requests rejected before admission and strengthen numerical and compiled-speech regression checks.

* address comments
2026-11-10 13:24:12 -08:00
Daniel HiltgenandGitHub 3f3fa58f7f llm: use mmap for Vulkan integrated GPUs (#19640)
Non-mmap loads put CPU-resident tensors in the Vulkan host buffer, which is
drawn from the same device budget as the weights. On a shared-memory iGPU that
roughly doubles the footprint and the load fails with "Failed to allocate
pinned memory" or, on Windows AMD, 0xe06d7363.

Direct I/O does not avoid the host buffer, so Vulkan uses mmap rather than
reusing the CUDA/ROCm dio policy.
2026-11-10 10:42:12 -08:00
Eva HandGitHub f1ee5679fe cmd/launch: use thinking metadata for OpenCode settings (#19409) 2026-11-09 17:29:01 -05:00
Eva HandGitHub 1f23c7232e Improve ChatGPT CLI restore and graceful restart (#19583)
Make --restore confirm before changing an active profile, leave a closed
app closed, and preserve existing settings and chat history. Let interactive
users finish ChatGPT's own quit prompt instead of force-quitting it.

Carry cancellation and timeouts through status checks, quit requests and
exit polling. Check cancellation before profile writes, and keep legacy
PID and singleton probes read-only until shutdown is confirmed, including
when a legacy client starts after preflight.

Keep context checks around process queries so a canceled probe cannot be
mistaken for a confirmed exit. Publish the legacy-quit test marker with an
atomic rename so cancellation cannot interrupt its write.

Validation: 591 launch tests, focused race checks, CLI build, and 100 repeated
legacy-quit cancellation checks with the race detector.

* Fix ChatGPT restore CI setup

Stub installed-app discovery in the late legacy-start consent test so it
runs without ChatGPT installed on the CI host.

Document the containedctx exception for the context implementation that
observes spinner cancellation synchronously. This preserves the existing
cancellation behavior while satisfying the repository lint configuration.
2026-11-09 12:29:51 -05:00
Patrick DevineandGitHub d63b702609 Add ripple preview support to the MLX runner (#19592)
Add the Ripple model, renderer, parser and import registration.
Support per-layer NoPE in DFlash drafts.
Fuse MLP activation, value blending and attention gating.
Add reference fixtures and model/runner regression tests.
v0.40.16 v0.40.16-rc0
2026-11-07 16:36:53 -08:00
Eva HandGitHub 5e983d4f80 launch: fix Claude Code routing and retired model errors (#19587) 2026-11-06 21:28:13 -05:00
anra8571 04494de5b2 docs: merging to remove retired model recommendations 2026-11-06 15:21:50 -08:00
Daniel HiltgenandGitHub c053d1e985 MLX: make Qwen3.8 Flash Next memory efficient (#18078)
* MLX: reduce Qwen3.8 Flash Next memory use

For NVFP4 creation, retain source precision for sensitive QSA projections
and non-expert MTP weights. Keep routed experts and PLE at NVFP4, and
promote other eligible weights to MXFP8.

Keep NVFP4 PLE tables host-backed, transferring only selected packed rows
and their scales to MLX. Honor scalar and per-expert NVFP4 global scales
in MoE, and per-shard global scales in host-backed PLE.

Use compressed QSA block selections in fused indexed Metal attention for
long prefill batches, retaining the graph fallback. Co-locate component
loading and forward methods, with explicit quantization and cache ownership.

* mlx: skip materializing host-backed tensors

Let models exclude tensors already owned by host storage from MLX loading.
Avoid allocating and discarding the full Qwen PLE tables at startup while
preserving the resident fallback.

* address comments
2026-11-06 13:39:21 -08:00
Daniel HiltgenandGitHub eaf3f28586 windows: add CUDA 13.5 and SM 103 for ARM64 (#19580)
Install the separate ARM64 cross package in CI and retain SM 121
support. Other platforms are unchanged.
2026-11-06 13:33:36 -08:00
Matteo CelaniandGitHub 51e7030f94 README: add Quock to Mobile community integrations (#19361) 2026-11-06 12:41:45 -08:00
Daniel HiltgenandGitHub 7ed1eb9c01 llama.cpp: version bump b12177 (#19576) 2026-11-06 12:00:00 -08:00
liangliang205andGitHub 6362e0a308 openai: return an empty array for /v1/models when no models are installed (#19539) 2026-11-05 16:57:40 -08:00
e968efb60a openai: forward truncation option in responses requests (#19501)
Co-authored-by: Eva Ho <hoyyeva@gmail.com>
2026-11-05 16:56:45 -08:00
J BergandGitHub 02480ca2a4 cmd/launch: add Kimi Code CLI support (#19506) v0.40.15-rc2 v0.40.15 2026-11-05 13:41:21 -08:00
Patrick DevineandGitHub 8e264f8989 mlxrunner: close generation cache on shutdown (#19550) 2026-11-05 12:59:06 -08:00
f60dd2060f cmd: restore the no-argument launcher (#19478)
Co-authored-by: Jeffrey Morgan <jmorganca@gmail.com>
2026-11-05 13:41:22 -05:00
Parth SareenandGitHub 32ce4379cc cmd/launch: show gateway model names in Claude Desktop's model picker (#19533) 2026-11-05 10:40:54 -08:00
Daniel HiltgenandGitHub 84dc81db1f llama-server: allow parallel requests for recurrent and vision models (#19545)
These architectures were forced to num_parallel=1 because the older Go
engine's recurrent caches held one sequence. llama-server supports
multiple slots for all of them so we can allow parallelism now.
2026-11-05 09:22:54 -08:00
Daniel HiltgenandGitHub a6d3325162 llama.cpp: version bump b12111 (#19530)
Drop 002-metal-unary-cast-constexpr.patch: superseded by upstream fix
v0.40.15-rc1
2026-11-04 17:31:38 -08:00
Patrick DevineandGitHub cdd4b55b43 mlx: support YaRN scaling in Qwen3 and Qwen3.5 (#19231) 2026-11-04 14:14:32 -08:00
Daniel HiltgenandGitHub 2e7b870fbc mlx: reject unsupported CUDA devices (#19521)
* mlx: bump for CUDA architecture metadata

* mlx: reject unsupported CUDA devices

* address comments
2026-11-04 09:57:25 -08:00
Daniel HiltgenandGitHub d666cdb246 Add AutoTrust decision model to MLX (#19399)
* Add AutoTrust decision scoring to MLX

* address comments

* Support AutoTrust 35B decision models in MLX

* Resolve AutoTrust choice limits during config loading

Apply publisher-specific defaults and validate requests against the resolved limit.
2026-11-04 09:47:37 -08:00
Eva HandGitHub e9fde36d75 launch: populate Pi models automatically (#19460) 2026-11-03 15:26:02 -08:00
Daniel HiltgenandGitHub 355eaee353 model: add MiMo V2.6 renderer for Qwen3.5 (#19400) v0.40.15-rc0 2026-11-03 07:09:06 -08:00
Daniel HiltgenandGitHub ddf22489b6 mlx: load bundled CUDA libraries in-process on Linux (#19433)
NVIDIA's libcusolver looks for its dependencies only in its own
directory, so libcublas and libnvJitLink resolved to the host's copies
and MLX failed to load when those didn't match the bundled ones. Preload
the bundled copies by full path before loading MLX.
2026-11-03 06:45:30 -08:00
Parth SareenandGitHub 9a3e817776 server: decode cloud web search responses (#19476) v0.40.14-rc0 v0.40.14 2026-11-02 16:59:50 -08:00
Eva HandGitHub 0c85e0afbe launch: check ChatGPT routing before configuring (#19470) 2026-11-02 13:39:49 -08:00
Daniel HiltgenandGitHub c1db5f9ef2 openai: don't override model sampler defaults (#19432)
The OpenAI-compatible endpoints filled in sampler options the client
left out: temperature and top_p fell back to 1.0, and /v1/completions
always sent frequency_penalty and presence_penalty as 0. Request
options are applied last, so these values replaced the model's packaged
defaults. /api/chat honored those defaults; /v1 did not.

Omitted options are now left unset on /v1/chat/completions,
/v1/completions and /v1/responses, so the model's parameters apply, the
same as on /api/chat. Values the client sends still win, including an
explicit 0. On /v1/completions, top_p and the penalties are now
pointers so "omitted" and "0" can be told apart.

Clients that send no sampler options now get the model's defaults
instead of 1.0/1.0. A model with no sampler parameters gets Ollama's
defaults (temperature 0.8, top_p 0.9), as on /api/chat.

Fixes #17744
Fixes #18690
2026-10-31 15:46:17 -07:00
Parth SareenandGitHub f9323416f0 feat(launch): use model output limits for Claude Code (#19410) 2026-10-30 14:34:54 -07:00
Eva HandGitHub 174cb1dcca app: make ChatGPT disconnect failures retryable (#19336) 2026-10-30 17:34:24 -04:00
Daniel HiltgenandGitHub 4d22e65d49 mlx: sort long top-k rows in chunks (#19392)
MLX's argpartition sorts the whole row, and a row longer than one block
of the GPU sort takes a multi-pass merge sort. TopKIndices sorts each
2048-value chunk in a single block, then only the chunks' top k, and
returns the same indices in the same order.

The sampler's top_k and top_logprobs and the DFlash2 candidate selector
use it in place of a full-vocabulary argpartition.
2026-10-29 18:53:49 -07:00
Eva HandGitHub 8861adaa77 openai: preserve readable Responses input content (#19192) 2026-10-28 23:46:40 -04:00
Daniel HiltgenandGitHub 6db7be24cc ci: pin buildx to workaround deadlock (#19373)
Recent buildx updates are causing frequent deadlocks in our CI builds.
Until this is resolved upstream, pin to an older version.
2026-10-28 16:44:29 -07:00
Daniel HiltgenandGitHub 04088e7c9d llama.cpp: workaround Metal unary crash (#19372)
This temporary patch works around a regression on older versions of MacOS
ggml_metal_library_compile_all: failed to build 'unary' library
v0.40.13-rc1 v0.40.13
2026-10-28 15:20:04 -07:00