oMLX 0.6.2

oMLX 0.6.2 combines focused hotfixes for regressions and edge cases uncovered in 0.6.1 with a new built-in ANE/GPU split tuner. It fixes the TurboQuant KV + Lightning MTP verification crash, restores exact GDN SSD-cache snapshots as the default, and addresses several distributed, cluster, MCP, memory, and admin issues.

Thanks to an incredible contribution from @onthehub97, oMLX can now benchmark ANE/GPU splits directly on each Mac instead of relying on ratios tuned for another machine. Dedicated NAX GPU suffix kernels also allow M5-family Macs to find and retain measurable ANE/GPU prefill gains.

Built-in ANE Split Tuner and M5 NAX Support

• Added a built-in ANE/GPU split tuner. The macOS app and web dashboard can now benchmark a GPU-only baseline, several MLP-only splits, and combined MLP/GDN splits directly on the current Mac. Candidate settings remain temporary until the user explicitly applies the result. By @onthehub97 in #2814.

• Added NAX GPU suffix kernels for M5-family Macs. Supported Qwen3.5/3.6/3.8 quantized layers use dedicated NAX QMM kernels for the GPU portion of hybrid ANE/GPU prefill. This resolves the regression seen when early M5 testing used split ratios and GPU kernels tuned for previous generations.

• Removed the blanket M5 ANE disable. Experimental ANE prefill remains disabled by default and must still be enabled per model, but M5 systems can now use the tuner to determine whether a local ANE/GPU split outperforms GPU-only execution.

• Kept safe runtime fallbacks. If NAX support or its metallib is unavailable, oMLX falls back to the existing Metal path. OMLX_QWEN35_QMM_NAX=0 remains available as a NAX kill switch.

Community testing on M5 reported approximately a 5% improvement in a practical benchmark, with the best tuner trial approaching 1,000 prompt tokens/s. Results depend on the chip, model, quantization, and prompt shape, so the tuner recommends GPU-only execution when the best ANE candidate is less than 1% faster.

ANE program loading was also hardened for single-die chips by retrying with smaller program banks when the preferred layout exceeds the device window. Benchmark traces now report the actual compiled MLP and GDN layer counts for easier validation.

Qwen and Cache Hotfixes

• Fixed Lightning MTP generation crashing with TurboQuant KV enabled. The optimized Qwen verification-attention path now safely rejects TurboQuant proxy objects it cannot process and falls back to the compatible attention path. This fixes the regression reported in #2778 and #2780. By @aaramos in #2782.

• Restored exact GDN SSD-cache snapshots as the default. GDN sidecar state now uses FP32 unless reduced precision is explicitly selected. Existing settings carrying the v0.6.0 lossy default are migrated back to FP32, preventing reduced-precision cache restoration from unexpectedly changing greedy outputs or breaking bit-level reproducibility. Reported in #2775.

• Fixed boundary-snapshot arrays remaining alive through a Python closure cycle. The scheduler now traverses snapshot state without creating a recursive closure that could retain large Metal-backed arrays. This prevents latent memory growth for reallocating cache implementations, including affected TurboQuant workflows. By @hojin12312 in #2809.

• Fixed VLM MTP profile conflicts during profile application. Applying output-shaping settings such as presence_penalty now preserves the requested settings and disables the incompatible speed-only VLM MTP toggle, matching request-time behavior. Remaining invalid combinations return an actionable HTTP 400 instead of an internal server error. By @jakeuj in #2806.

Distributed Serving and Cluster Management

• Added distributed thinking_budget enforcement. Rank workers now apply the same per-request thinking-budget processor used by single-machine inference, allowing distributed Qwen deployments to place a hard bound on reasoning without disabling it entirely. By @xunlinkx in #2731.

• Made the distributed request read timeout configurable. Long prefills and cold model reloads can use OMLX_DISTRIBUTED_REQUEST_READ_TIMEOUT to override the existing 300-second default. Invalid, non-finite, or non-positive values now fail with a clear configuration error. By @xunlinkx in #2714.

• Stopped idle cluster polling from repeatedly running the planner. The cluster dashboard still creates its initial plan and responds to explicit model or control changes, but no longer posts /plan every polling cycle. By @xunlinkx in #2721.

• Preserved the selected role of the local coordinator. Dedicated coordinator Macs can now remain configured as headless instead of being reset to the workstation role and unnecessarily reserving 32 GiB. By @xunlinkx in #2723.

• Improved compatibility reporting for older or uninitialized workers. Cluster preflight now handles peers predating omlx._version, distinguishes a missing worker runtime from an actual version mismatch, and keeps unverified workers non-ready until compatibility has been established. By @hellodk in #2737 and #2738.

MCP, Memory, and Admin Fixes

• Restored remote MCP Streamable HTTP connections with MCP SDK 2.x. The client now follows the SDK's reader/writer return contract, and package constraints consistently require MCP 2.x. By @nnhanndakara in #2776, addressing #2520.

• Fixed wired-memory limit recommendations on 128 GiB Macs. Backend warnings, the admin API, and the displayed iogpu.wired_limit_mb command now use the same safe whole-MiB value, eliminating an impossible rounding loop. By @jakeuj in #2799.

Upgrade Notes

• Qwen ANE/GPU prefill remains experimental, opt-in, and dependent on private Apple runtime interfaces. It requires the oMLX native custom kernels and may increase model loading time and memory use.

• The ANE split tuner takes several minutes because it performs full prewarm and end-to-end measurements across multiple temporary configurations. Saved model settings do not change until the recommended result is explicitly applied.

• Existing GDN SSD-cache settings using the legacy gdn_sidecar_state_dtype key are reset to exact FP32 storage. Users who intentionally accept reduced precision for smaller SSD snapshots can select it again in the updated settings UI.

• The distributed request timeout remains 300 seconds unless OMLX_DISTRIBUTED_REQUEST_READ_TIMEOUT is configured.

Thanks to @onthehub97, @xunlinkx, @hellodk, @nnhanndakara, @jakeuj, @hojin12312, and @aaramos for their contributions, and to everyone who reported and validated the 0.6.1 issues.

Full Changelog: v0.6.1...v0.6.2