mirror of
https://github.com/ollama/ollama.git
synced 2026-11-10 15:02:13 -06:00
Daniel HiltgenandGitHub
c053d1e985
* MLX: reduce Qwen3.8 Flash Next memory use For NVFP4 creation, retain source precision for sensitive QSA projections and non-expert MTP weights. Keep routed experts and PLE at NVFP4, and promote other eligible weights to MXFP8. Keep NVFP4 PLE tables host-backed, transferring only selected packed rows and their scales to MLX. Honor scalar and per-expert NVFP4 global scales in MoE, and per-shard global scales in host-backed PLE. Use compressed QSA block selections in fused indexed Metal attention for long prefill batches, retaining the graph fallback. Co-locate component loading and forward methods, with explicit quantization and cache ownership. * mlx: skip materializing host-backed tensors Let models exclude tensors already owned by host storage from MLX loading. Avoid allocating and discarding the full Qwen PLE tables at startup while preserving the resident fallback. * address comments