oMLX 0.6.1

oMLX 0.6.1 focuses on Qwen3.8 performance and fixes several compatibility regressions found after 0.6.0. It adds opt-in dual-ANE/GPU prefill on M3 Ultra (+18.9% throughput at 32K context), accelerates Lightning MTP generation (+34.0% decode throughput at 16K context), and restores reasoning-effort compatibility, Qwen vision loading, model discovery, and tool-heavy prefix-cache reuse.

Experimental Qwen3.8 Dual-ANE/GPU Prefill

• Added experimental prompt processing across both ANEs and the GPU. Eligible fixed-size Qwen3.8 MLP and GDN prefill blocks split work across two physical ANEs and Metal, while decode, verification, residual chunks, and unsupported layers continue using the existing GPU path. The feature supports compatible AWQ and oQ4e layouts and is disabled by default. By @onthehub97 in #2756, extended and hardened in #2760.

Measured on an M3 Ultra with Qwen3.8-27B-oQ4e-mtp, Lightning MTP disabled, a 2,048-token ANE block, and a 128-token generation tail:

Longer prompts benefit more because they contain more complete fixed-size blocks. The 4K case contains only one 2,048-token ANE block followed by a 2,047-token GPU tail.

This path uses private Apple runtime interfaces and approximate INT8 weights. On the reference system, enabling it increased peak memory by about 4.15 GB and model loading from 3.35 seconds to approximately 27–29 seconds. Decode remains on the GPU, so this is a prompt-processing optimization rather than a token-generation acceleration.

Qwen3.8 Lightning MTP and Serving Performance

• Sped up Lightning MTP verification and Qwen3.8 prefill. Fused target-verification attention and GDN prework reduce dispatch overhead, while high-memory systems use larger prefill chunks when memory permits. #2751.

Measured on an M3 Ultra with Qwen3.8-27B-oQ4e-mtp, Lightning MTP enabled, and a 128-token generation tail:

• Improved concurrent Qwen3.8 VLM throughput. When several requests are ready together, oMLX now prefers ordinary batching instead of mixing Lightning MTP and batched decoding on the same model. Batch-4 throughput improved from 69.34 to 108.14 tok/s for greedy decoding and from 64.68 to 103.79 tok/s for sampled decoding, while uncontended single-request Lightning MTP remains available. By @DiscoStew6082 in #2752.

• Fused Qwen MoE router top-k selection. The optimized Metal path improved Qwen3.6-35B-A3B decode throughput by 3.9% at 4K context and 2.5% at 16K in the reported workload. #2755.

Qwen3.8 VLM and Model Loading

• Fixed affected Qwen3.8 checkpoints falling back to text-only inference. Channels-first vision patch-embedding weights are normalized during MLX checkpoint loading, preserving full vision support and Lightning MTP. By @frank-beans in #2754.

• Fixed Qwen VLM checkpoints with root mtp.* weights. These weights are now mapped to the attached language-model MTP module instead of failing strict VLM loading and silently disabling vision. Reported in #2732.

• Restored intentionally filtered Hugging Face cache models to discovery. Auxiliary safetensors omitted by filtered downloads no longer cause otherwise loadable models to disappear from the model list. Reported in #2742.

Reasoning and Agent Compatibility

• Normalized reasoning_effort across model chat templates. Common client values are retried with compatible aliases or the model's native default instead of returning HTTP 400 when a template uses a different effort vocabulary. This fixes Qwen3.8 clients sending values such as high. Reported in #2740.

• Added explicit DeepSeek V4 reasoning-effort aliases. OpenAI- and Hermes-style values such as minimal, medium, xhigh, and ultra now map to the closest supported DeepSeek level. By @jonathan308 in #2724.

• Preserved prefix-cache reuse across tool-adjacent system messages. Claude Code system reminders that follow tool results can remain in place when the model template safely supports them, avoiding repeated large prefills in long tool-heavy sessions. Template-specific probing keeps the fallback safe for incompatible templates. By @q-p in #2753, addressing #2483.

Distributed Serving and Benchmarks

• Fixed false distributed-version mismatches. Cluster probes now compare local MLX and MLX-LM versions using the same source reported by the peer, so matching editable or nightly installations no longer fail enrollment because module and package metadata differ. By @hellodk in #2758.

• Improved community intelligence benchmark identity. Uploads now carry the organization-qualified model repository separately from the display name, allowing omlx.ai to distinguish checkpoints with the same leaf name without cluttering the model-name column.

Upgrade Notes

• Qwen ANE prefill remains experimental and disabled by default. It requires compatible native custom kernels and has currently been validated on M3 Ultra.

• Enabling or changing Qwen ANE settings reloads the model because fixed-shape ANE programs are compiled eagerly.

• Existing configurations remain compatible, and no manual settings migration is required.

Thanks to @onthehub97, @DiscoStew6082, @frank-beans, @q-p, @hellodk, and @jonathan308 for their contributions.

Full Changelog: v0.6.0...v0.6.1