oMLX 0.6.3rc1

This is the release candidate for oMLX 0.6.3. After a short testing period, this release will be followed by either rc2 or the final release. Thank you, as always, to everyone who has patiently waited for this release!

This release introduces a faster Qwen ANE split tuner, extends ANE prefill to additional quantization formats, and adds DFlash 2 runtime support. It also substantially improves distributed-cluster reliability, restores prefix-cache reuse in long Claude Code sessions, reports Responses API truncation correctly, and adds bit-exact Laguna decode optimizations.

Qwen ANE Tuner V2 and Wider Quantization Support

• Made the ANE tuner much faster. Instead of preparing and testing many complete model configurations, the tuner first measures a few representative layers, predicts the best ANE/GPU split, and then verifies it with the full model. This reduces full-model setup runs from 11 to at most 3 while still basing the final recommendation on real prompt performance. By @onthehub97 in #2891.

• Added affine Q5, Q6, and Q8 ANE prefill support. Eligible Qwen3.5/3.6/3.8 MLP and GDN projections with group sizes 64 or 128 can now use the hybrid ANE/GPU path. Q4 behavior remains unchanged. By @GordoAR and @onthehub97 in #2833 and #2889.

• Improved ANE prefill for longer prompts. Large prompt chunks can now be divided into blocks that fit the compiled ANE program, while any remaining tokens continue on the GPU. This allows the server to keep efficient prompt chunk sizes instead of shrinking every chunk to the ANE block size. #2890.

• Made ANE tuning results easier to trust and diagnose. The tuner no longer recommends a configuration when the ANE was compiled but never actually used. Benchmark logs also show how much work ran on the ANE and GPU, making it easier to understand why a candidate succeeded or failed. By @beaglemoo in #2829.

Local 2,048-token Qwen3.8-27B measurements on M3 Ultra reported:

These are local serial measurements. Performance and the best split depend on the Mac, checkpoint, quantization, and prompt shape.

DFlash 2 Support

• Added end-to-end DFlash 2 support. Compatible checkpoints can use checkpoint-derived sliding windows and per-model block-size overrides. DFlash now matches the regular batched engine's sampling behavior, including min_p and seeded requests. The default sink size is now 0, SSD prefix-cache writes remain functional with that setting, and versioned DFlash2 checkpoints appear in the draft-model picker. Initial support by @liang2kl in #2840, finalized in #2850, with picker support by @williamxie1989 in #2879.

Measured on an M3 Ultra with Qwen3.8-27B-oQ4e-mtp, the z-lab/Qwen3.8-27B-DFlash2 draft, temperature 0.7, and a 128-token generation tail:

Exact repeated prompts continue to use the in-memory prefix cache. SSD snapshot writes also work with the new sink-size default.

Distributed Serving and Cluster Reliability

• Added a persistent cluster incident feed. Activation, staging, and peer-health failures are recorded by the server and remain visible after dashboard refreshes. Incidents have stable sequence numbers, machine-readable guidance codes, severity, source, and explicit dismissal state. By @alytaphoenix in #2866.

• Improved two-Mac activation and planning reliability. Dashboard polling no longer clears a valid plan or changes tensor topology during activation. Cross-user clusters resolve model paths in each peer's own home directory, and unsupported VLM pipeline plans are no longer offered. By @alytaphoenix in #2819.

• Stopped remote rank processes more reliably. Teardown now validates each peer's deployment marker and process identity, sends SIGTERM, escalates to SIGKILL when necessary, and reports whether the rank was actually reaped. Failure markers remain available as diagnostic evidence. By @xunlinkx in #2722.

• Reduced repeated planner work. Cluster autoconfiguration caches model layouts and shard metadata until the relevant files change, avoiding repeated safetensors scans and monkey-patch installation during dashboard polling. By @alytaphoenix in #2883.

• Added tensor-parallel support for quantized Nemotron-H at TP=2. Uneven but group-aligned routed-expert shards, corrected quantized Mamba slicing, and stricter planner checks allow compatible Nemotron-H checkpoints to load and generate across two Macs. By @alytaphoenix in #2844.

• Improved distributed request compatibility and failure handling. Unsupported reasoning_effort values are retried through the same bounded fallback used by local engines. Requests also check rank health before streaming begins, allowing a dead or partially failed cluster to return HTTP 503 instead of an empty HTTP 200 response. By @alytaphoenix in #2869.

• Preserved configured memory-guard settings during remote admission. Cluster probes now initialize local settings before falling back to defaults, so custom memory tiers and ceilings are honored. By @xunlinkx in #2835.

APIs, Agent Workflows, and Admin Fixes

• Restored prefix-cache reuse in long Claude Code sessions. The Anthropic adapter now strips changing <total_tokens> budget markers from system and developer content while preserving user-quoted markers. On the reported 112K-token workload, reused tokens increased from 4,096 to 112,640 and turn latency fell from minutes to seconds. By @Pietroski in #2882.

• Reported Responses API truncation correctly. Requests stopped by max_output_tokens now return status: "incomplete" with incomplete_details.reason: "max_output_tokens". Streaming responses terminate with response.incomplete instead of response.completed, allowing clients to continue truncated reasoning or generation. By @mvdbos in #2817.

• Fixed forwarded arguments in omlx launch. The oMLX forwarding separator is removed without discarding a second separator intended for the launched tool, so flags passed to Claude Code and similar CLIs remain flags instead of becoming prompt text. By @fparrav in #2818.

• Added a token-budget override for external accuracy benchmarks. The optional value acts as a floor over each benchmark's default, preventing thinking models from being truncated before producing an answer without reducing larger benchmark-native budgets. By @williamxie1989 in #2863.

• Fixed default-model state after unsetting a model. Removing the default flag now clears the server-side pointer and updates the dashboard immediately instead of leaving requests routed to the old default until restart. By @alytaphoenix in #2884.

Laguna Decode Performance

• Added default-on, bit-exact compiled fusions for Laguna XS 2.1. Compiled router renormalization, expert combination, per-head softplus gating, and SiLU gate/up work improved single-token decode by 3.96% on an M4 Max, from 66.4 to 69.1 tok/s. The verified greedy trajectory remained token-identical. By @0xClandestine in #2827.

• Experimental routed and shared-expert fusion banks remain opt-in because they were neutral or slower on the tested MLX runtime.

Upgrade Notes

• This is a release candidate. Please report regressions before the final 0.6.3 release.

• DFlash draft sink size now defaults to 0 when unset. Existing explicit non-negative values are preserved.

• Qwen ANE/GPU prefill remains experimental, opt-in, and dependent on private Apple runtime interfaces and the oMLX native custom kernels.

• Keep the configured ANE sequence_length at or below the scheduler's delivered prefill chunk width. With prefix caching enabled, 2,048 remains the safe default because cache boundaries split delivered chunks at 2,048 tokens.

• Streaming Responses API clients should handle response.incomplete as the terminal event when generation reaches max_output_tokens.

• Distributed inference remains experimental and disabled by default.

• No manual settings migration is otherwise required.

Thanks to @liang2kl, @GordoAR, @onthehub97, @beaglemoo, @xunlinkx, @alytaphoenix, @williamxie1989, @Pietroski, @fparrav, @mvdbos, and @0xClandestine for their contributions.

Full Changelog: v0.6.2...v0.6.3rc1