Files
ollama/mlxrunner
Daniel HiltgenandGitHub db5266a71d mlxrunner: batch speculative verification (#19663)
* mlxrunner: batch speculative verification

Keep drafting active for concurrent generation by sharing target verification while preserving per-request sampling and cache rollback. Learn draft depth for each concurrent workload separately from isolated requests, and preserve probes that short requests never execute.

Materialize owned target features before a reader can pause, and cover acceptance, cancellation, handoffs, mixed decoding and retained-memory boundaries.

* address comments
2026-11-16 08:43:54 -08:00
..
2026-10-03 13:17:21 -07:00
…