oMLX 0.7.0rc1

The release candidate for oMLX 0.7.0 brings faster Qwen prefill and decoding, new multimodal models, and the improvements introduced throughout the 0.7.0 development releases.

Download: macOS 26 / 27 | macOS 15 Sequoia

Faster Qwen Prefill and Generation

Qwen gains faster prompt processing, batched DFlash decoding, and improved Lightning MTP. DFlash now serves concurrent Qwen3.5-family requests together, while Lightning MTP adds an adaptive depth selector. #3797, #3903, 62171bd.

Measured on an M5 Max with 128 GB unified memory. Each comparison uses its respective pre-PR main baseline. Prefill figures are single runs with paged caching disabled.
*The GDN prefill kernels are adapted from mlx-serve by David Dalcu.

New Model Support

• Ternary Bonsai 2. Added text and vision support through updated mlx-vlm, including vision feature caching and prefix-cache reuse. #3782.
• MiMo V2.6 multimodal support. Compatible checkpoints with bundled multimodal sidecars now accept images, sampled-frame video, and audio through the OpenAI-compatible API, with Lightning MTP and DFlash for faster generation. By @sayyidfareed in #3860, @ashhart in #3821, with MTP contributions from @sayyidfareed and @williamxie1989 integrated in #3877.

Less Waiting Between Messages - Partial Block Caching

Previously, even a cached conversation could leave thousands of tokens to reprocess simply because they did not fill a complete cache block. oMLX now reuses that final partial block too, so the next reply starts sooner. In a 13.4K-token Qwen3.6-35B-A3B test, next-turn prefill dropped from 1,174 tokens to 37, cutting time to first token from 0.83s to 0.42s. #3835.

New Features and Improvements

• MCDMA RDMA stage links. Supported Mac + CUDA deployments can carry the Mac-facing pipeline activation edge over a verified MCDMA RDMA link. By @ashhart in #3869.
• Lightning MTP with expert offload. DeepSeek V4.1 and GLM-5.3-Flash can keep their draft heads resident while offloading main-model experts. By @williamxie1989 in #3784, #3841.
• Broader MoE expert offload. Added compatible Qwen3.5 MoE, GLM DSA, DeepSeek V4, and GLM-5.3 layouts, plus DeepSeek V4.1 MLX affine source checkpoints. By @LXD-8 in #3717, #3713; @beatakouchnir in #3696; @williamxie1989 in #3720.
• Configurable expert residency. Set the resident expert fraction from 5% to 95%, including fractional percentages. By @LXD-8 in #3714.
• Lightning MTP for text-only Gemma 4. Checkpoints with merged MTP heads now use the existing mlx-vlm draft path while remaining text-only models. By @andyoneal in #3561.
• Faster Qwen GDN decoding. Automatically fuse compatible FP16/BF16 single-token GDN work on Metal, extending the original M1 Max optimization through a maintainer follow-up. By @samfenwick in #3853.
• Extended Qwen4 fused decode. Added an opt-in model setting for compatible community quantization recipes; fused projections can use additional resident memory. By @ddark-il in #3890.
• Faster MiMo and GLM prefill. Use larger model-specific prefill blocks for MiMo V2 hybrid MoE and GLM-5.3 hybrids. By @williamxie1989 in #3872, #3889.
• Reusable SpecPrefill draft state. Preserve recurrent draft state at cache boundaries and restore positions from attention layers to enable hybrid draft-prefix hits. By @tc3oliver in #3842, #3840.
• Earlier first-token streaming. Release the first generated chunk before continuing a decode burst. By @samfenwick in #3854.
• Optional unauthenticated inference. Added an explicit opt-in for inference without API keys while keeping administrative access authenticated. #3859.

Bug Fixes

• Lower prefill memory peaks. Bound DeepSeek V4.1 and low-bit GLM-5.3 prefill allocations and correct V4.1 index-cache estimates. By @williamxie1989 in #3808, #3807.
• More accurate prefill admission. Account for fixed reclaim costs and refresh stale memory samples before sizing chunks. By @samfenwick in #3852; @Chuhan1112 in #3866.
• Bounded attention memory. Keep SDPA256 prefill on the bounded route and use tiled attention for FP32 prefill. By @tc3oliver in #3685. Also addresses #3795.
• Bounded expert-offload buffers. Prevent oversized prefill staging allocations when expert offload is enabled. #3783.
• Restored Qwen and VLM MTP speed. Recover Qwen VLM decoding, external VLM MTP, and Qwen Lightning MTP throughput after dependency changes. #3760, #3789, #3791.
• Faster Qwen4 MTP verification. Restore fused GDN verify prework with the correct normalization variant. By @williamxie1989 in #3776.
• MTP checkpoint loading. Load MLX-format MTP head norms as stored instead of applying an incompatible conversion. #3788.
• TurboQuant batch correctness. Align rollback and attention masks for batched speculative decoding. Addresses #3767.
• DeepSeek reasoning and batch caches. Render echoed reasoning in its native form and remove padding from restored V4.1 batched caches. Addresses #3758.
• Thinking and tool-call output. Keep truncated thinking out of answer content, decode untyped Qwen tool parameters, and recognize the VLM Qwen parser. By @arnavprabhu in #3809. Also addresses #3796 and #3780.
• ANE and MTP resource cleanup. Release ANE prefill state, drain outstanding work before unloading, and drop retained VLM MTP target references. By @rockminster in #3157.
• Clearer ANE warnings. Report the actual minimum sequence length and account for padded chunks. By @tc3oliver in #3746.
• Safer automatic SSD-cache sizing. Base automatic capacity on available disk space and include offline GDN sidecars in cache maintenance. Addresses #3829 and #3883.
• Quiet hot-cache-only operation. Stop disk-usage warnings when SSD caching is disabled. By @williamxie1989 in #3781.
• Preserved cache settings. Saving unchanged cache settings no longer unloads models or rewrites existing values. By @zviratko in #3764.
• Recovered legacy server startup. Handle older unauthenticated network settings without preventing the server from starting. #3735.
• GLM checkpoint compatibility. Accept vision_tower weight names and correctly match quantization overrides after sanitization. By @cropduster in #3843.
• Correct oQ conversion settings. Fix GLM-5.3 text-only sanitization and restrict DeepSeek V4.1 options to supported settings. Addresses #3845 and #3846.
• Model discovery. Resolve Qwen4 model_file checkpoints correctly and use Hugging Face repository names for cache-name heuristics. By @pierou in #3752; @maxpfingsthorn in #3881.
• Distributed worker compatibility. Keep HTTP dependencies out of worker imports and agree on rank-zero sampling before enabling it. By @ashhart in #3855, #3857.
• Chat settings and uploads. Preserve thinking-budget edits during autosave and honor the configured audio upload limit. By @LXD-8 in #3778; @aywengo in #3160.
• Mobile usage history. Keep usage-history controls and charts readable on smaller screens. By @samfenwick in #3739.

Highlights from Dev Releases

• Cluster v2. Discover and pair Macs, plan and stage model deployments, and manage distributed serving with request cancellation, worker recovery, persistent prompt snapshots, and cluster-wide cache controls. By @jonathan308 in #3155, #3256, #3257, #3258, #3516.
• One-click optimal settings and recipes. Apply benchmarked settings for your Mac and model, or import a compatible architecture recipe from omlx.ai. #3701.
• DeepSeek V4.1 Flash with DSpark MTP. Added text and vision inference, tool calling, oQ quantization, and Engram SSD offload to reduce resident memory. #3574.
• Faster DeepSeek V4.1 CED prefill. Optional bounded replay improved prefill by up to 79% in the published M3 Ultra benchmark; this approximate mode is off by default and can change outputs. By @williamxie1989 in #3607.
• Multi-request Lightning MTP. Concurrent requests can use Lightning MTP on supported Qwen, DeepSeek V4.1, and GLM-5.3-Flash adapters. #3695. Builds on the fused-batch approach by @gaineyllc.
• Faster Qwen3.8-Flash-Next generation. Reduced host overhead and gathered QSA attention accelerate ordinary decoding and long-context MTP verification. By @Rascal in #3469, #3520.
• Faster Qwen3.8 prefill with SSD-backed PLE. Host-side gathering, prefetch, and compiled mean operations delivered 8-20% faster prefill in published 16K-128K tests. By @Rascal in #3534.
• Parallel cold-page PLE reads. Concurrent SSD reads improve cold-page lookup throughput. By @scott-kim-7 in #3287. Includes the cold-page gate by @williamxie1989.
• M5 INT8-activation prefill. Optional Qwen3.5/3.6/3.8 kernels improved 32K prefill by 34.4% in the published M5 Max test; activation quantization can change outputs. By @PowerSpy in #3548.
• MoE expert SSD offload. Run supported MoE checkpoints with fewer resident expert weights by fetching other experts on demand from existing safetensors files. By @beatakouchnir in #2595. Includes fixes by @fxd0h and maintainer integration for additional models.
• Customizable dashboard. Move, resize, hide, and restore status blocks, with saved layouts and adjustable dashboard width. #3694.
• Local usage history. View request, token, cache, speed, and activity history in the dashboard and macOS app without storing prompts or responses. By @thatcherstudio in #3532.
• IFM/K2-Horizon and MoVA. Added serving, oQ quantization, reasoning and tool-call parsing, and optional ANE prefill controls. By @mbarnson in #3486.
• Spark-X2.5. Added model support with reasoning output and structured tool calls. By @osviel91 in #3606.
• GLM-5.3-Flash Lightning MTP. Added MTP-head loading, speculative rollback, and model-specific runtime integration. By @davidtorcivia in #3228.
• Layer-streaming oQe calibration. Calibrate supported MiniMax-M3 and Qwen3.8-Flash-Next models larger than RAM one layer at a time. By @builtbycnob in #2161.
• More responsive concurrent requests. Yield between contended prefill chunks and prevent queued requests from starving during active decoding. By @chenqianhe in #3487; #3726.

Full changes and release notes from the development releases are available below.

• 0.7.0.dev4 release notes
• 0.7.0.dev2 release notes
• 0.7.0.dev1 release notes

New Contributors

@astro-15-alive in #3369, @pedroberaldo87 in #3299, @Lwrless in #3376, @saintlits in #3424, @andyoneal in #3340, @Cianidos in #3364, @ddark-il in #3428, @MaxFreedomPollard in #3484, @Rascal in #3469, @scott-kim-7 in #3287, @mbarnson in #3486, @Anai-Guo in #3544, @L4XB in #3546, @AmirF194 in #3547, @thatcherstudio in #3532, @arnavprabhu in #3568, @beatakouchnir in #2595, @ernestas-poskus in #3633, @mensaprodigy in #3659, @nisargnegi in #3661, @pierre427 in #3627, @ccdsky in #3624, @osviel91 in #3606, @djtech42 in #3512, @joergbirkhold in #2873, @LittleYier in #3617, @astutegrogan in #3383, @X-Wei in #3530, @tc3oliver in #3664, @luken in #3710, @pierou in #3752, @zviratko in #3764, @cropduster in #3843, @sayyidfareed in #3860, @maxpfingsthorn in #3881.

Full changelog from 0.6.4