oMLX 0.7.0.dev1

This is a development release. Please report any bugs through GitHub Issues. Thank you to everyone who keeps the issues updated with reports, follow-ups, and testing results.

This development release speeds up Qwen3.8-Flash-Next generation and prefill, improves responsiveness under concurrent requests, rebuilds distributed serving around Cluster v2, and adds IFM/K2-Horizon support and local usage history.

Qwen3.8-Flash-Next (tok/s)

• Faster Qwen3.8-Flash-Next generation, with and without Lightning MTP. Reduced host-side decode overhead improved generation throughput by 49.0–55.1% with MTP off and 39.9–47.6% with MTP on across 4K–64K contexts. At 4K, throughput increased from 34.8 to 53.9 tok/s without MTP and 56.2 to 83.0 tok/s with MTP. By @Rascal in #3469.

• Faster long-context Qwen3.8 generation. Gathered QSA attention now also accelerates eligible text decode and Lightning MTP verification. With MTP enabled, generation increased from 49.6 to 59.2 tok/s at 128K (+19.4%) and 46.4 to 50.1 tok/s at 200K (+8.0%), while prefill stayed close to baseline. A follow-up preserves this path on full text-prefix cache hits. Greedy outputs can differ from dense attention; the maintainer's extended-context evaluations found no clear quality regression. By @Rascal in #3520.

• 8–20% faster Qwen3.8 prefill with SSD-backed PLE. Host-side PLE gathering, prefetch, and compiled mean operations improve prompt processing across 16K–128K contexts while retaining the existing prefill normalization. At 16K, throughput increased from 925.6 to 1,113.3 tok/s with MTP off and 906.6 to 1,085.4 tok/s with it on. These figures reflect the combination retained after the maintainer follow-up. By @Rascal in #3534. Cold-page PLE reads also gain concurrent prefetch by @scott-kim-7, with the cold-page gate contributed by @williamxie1989, in #3287.

  These three Qwen3.8 comparisons were measured separately on an M3 Ultra using Qwen3.8-Flash-Next-oQ4e-mtp, Code Python prompts, and no prefix-cache reuse. Each uses its own pre-PR baseline; the gains are not additive or a combined comparison against 0.6.4.

Rebuilt distributed serving - By @jonathan308

• Rebuilt distributed serving around Cluster v2. A major overhaul spanning device pairing, model deployment, and the serving runtime makes running one model across multiple Macs a guided workflow. Discover and pair Macs with a code, plan tensor or pipeline placement from live memory budgets, automatically stage missing model files, and launch or reload saved deployments from the redesigned Cluster dashboard. Underneath, the runtime adds authenticated worker control, coordinated startup and shutdown, watchdogs and worker recovery, and cancellation of individual requests while others continue generating. Prompt snapshots survive worker restarts, with per-request performance monitoring and cluster-wide RAM/SSD cache clearing built into the same flow. By @jonathan308 in #3155, #3256, #3257, #3258, and #3516.

Other Improvements and Fixes

• Shorter output stalls during concurrent prefill. With three 16K prefills running, a short request's time to first token fell from about 5.99s to 4.96s, and its longest output gap fell from 1.64–4.57s to 0.58s. Measured aggregate throughput decreased by at most 1.8% across the tested workloads. Measured on an M3 Ultra with Qwen3.6-35B-A3B 4-bit, cache, MTP, and ANE disabled. By @chenqianhe in #3487.

• Added IFM/K2-Horizon and MoVA support. Includes serving, oQ quantization, native reasoning and tool-call parsing, and optional ANE prefill with tuning controls in the dashboard and macOS app. By @mbarnson in #3486.

• Improved long-context memory handling and recovery. Corrected Qwen4 prefill estimates and reservations, preserved progress across eviction pauses, bounded Lightning MTP reconciliation chunks, and fixed rollback through VLM adapters. ANE bank release, DFlash memory aborts, and emergency-unload retries also recover more reliably. Contributions by @wolfyy970, @mvdbos, @williamxie1989, @pedroberaldo87, @andyoneal, @Cianidos, @saintlits, and @AmirF194 in #3351, #3461, #3465, #3455, #3414, #3415, #3416, #3340, #3364, #3365, #3424, and #3547.

• Improved prompt-cache reuse and streaming. Removed duplicate restore work, exposed precise stream TTFT, restored hybrid GDN boundary storage, and fixed Qwen4 cache joins and gathered-path eligibility. Reasoning output remains reusable when conversation history preserves it. By @jonathan308, @apcooley, @astro-15-alive, @ddark-il, @ivaniguarans, @contrapuntal, and @Rascal in #3327, #3288, #3369, #3355, #3428, #3367, #3447, and #3525.

• Added local usage history. The dashboard and macOS app now show requests, tokens, cache usage, per-model output speed, and daily/hourly heatmaps. History stays on the machine without storing prompts or responses, and recording can be disabled. Follow-up changes eliminate idle database work between maintenance runs and reduce response size. By @thatcherstudio in #3532.

• Refined the macOS interface. Controls, switches, buttons, spacing, and footers now follow consistent native conventions across screens. By @Lwrless in #3376.

• Fixed ModernBERT padded-batch embeddings. A finite attention mask prevents NaN vectors when batching inputs of different lengths (#3507). The API also rejects non-finite embeddings with structured errors, including after keepalive begins, by @L4XB in #3545, with a maintainer follow-up.

• Fixed Claude launch settings and model compatibility. The launcher preserves saved Claude model tiers and uses a shared context limit that fits the selected models. Gemma4 loads when global KV-head fields need to be derived. By @L4XB in #3546 and #3549, with maintainer follow-ups. Hugging Face model-size estimates now account for U32-packed weights by @aywengo in #3419, and the macOS app rejects custom kernels built for the wrong bundled Python ABI by @PhilipJohnBasile in #3558.

New Contributors

A special thanks to the new contributors who helped make this release possible (in alphabetical order):

• @AmirF194: Retry handling that lets incomplete emergency memory recovery continue.
• @Anai-Guo: Restored three engine-pool tests hidden by a duplicate class name.
• @andyoneal: Proper cleanup of cached state when releasing ANE banks.
• @astro-15-alive: More reliable Qwen4 cache joins for mixed-length batches.
• @Cianidos: More accurate ANE memory accounting and better prefill memory reclamation.
• @ddark-il: Restored gathered QSA acceleration and fixes for M5 gather and NAX kernels.
• @L4XB: Fixes for Claude model selection, non-finite embedding responses, and Gemma4 loading.
• @Lwrless: A more consistent macOS interface with carefully aligned controls and native styling.
• @MaxFreedomPollard: A GSM8K parsing fix so commas no longer invalidate correct answers.
• @mbarnson: IFM/K2-Horizon and MoVA support, including oQ, tool parsing, and ANE controls.
• @pedroberaldo87: Fixes across model swaps, prefill recovery, Lightning MTP rollback, and MoE performance.
• @Rascal: Substantial Qwen3.8 generation and prefill speedups, backed by careful benchmark follow-ups.
• @saintlits: Cleaner DFlash request handling under memory pressure.
• @scott-kim-7: Concurrent cold-page PLE reads that make better use of SSD parallelism.
• @thatcherstudio: Local usage history with an intuitive per-model heatmap.

Thank you to all contributors for the code, bug reports, testing, and feedback that helped shape this release.

Full Changelog: v0.6.4...v0.7.0.dev1