oMLX 0.6.4

oMLX 0.6.4 substantially improves Qwen3.8-Flash-Next prefill and generation performance, while fixing continuous batching, prefix-cache reconstruction, Lightning MTP state handling, GLM-5.3 correctness, and several model-loading and settings issues.

Qwen3.8-Flash-Next Performance

• Added exact QSA prefill and decode acceleration. Eligible batch-one text workloads now use native FP32 QSA scoring, deterministic block selection, direct sparse-GQA Metal attention, and a selected-K/V decode path. Unsupported layouts and execution modes continue through the official fallback path.
• Accelerated resident PLE, GDN, and hyperconnection projections without replacing the checkpoint state used by fallback paths.
• Added warm-prefix restoration for Lightning MTP prompt history. Matching prefix-cache entries can restore the MTP sidecar instead of replaying the full reusable prompt head.
• Fixed Qwen vision grid compatibility with newer MLX releases.

By @jonathan308 in #3244.

Maintainer benchmark on an Apple M3 Ultra with 512 GB of unified memory, using Qwen3.8-Flash-Next-oQ4e-mtp, Code (Python) context, 128 generated tokens, temperature 1.0, top-p 1.0, and Lightning MTP with adaptive maximum depth 3. Each length was measured once with zero cached prompt tokens.

At 32K context, prompt processing improved by 33.5%, generation throughput improved by 14.6%, and total request time decreased by 24.2%.

Bug Fixes and Operational Improvements

• Isolated DFlash and Lightning MTP patches across loaded engines. Loading one engine no longer mutates shared model classes in a way that breaks resident or subsequently loaded engines, preventing repetition loops and Qwen4 Lightning MTP hidden-state shape failures. By @e-Evolution in #3285.

• Fixed GLM-5.3 affine prefill correctness for strided inputs. Split projection views are now made contiguous before entering native quantized tiles, preventing GLM-5.3-Flash from losing track of its prompt at longer input lengths. By @davidtorcivia in #3275.

• Fixed restored-prefix mRoPE state on Qwen3.8 ANE and NAX paths. The restored text-position seed is materialized before the prefill graph consumes it, preventing Metal timeouts, immediate EOS, stale tool-content continuation, and run-to-run output variance. By @hesara in #3279.

• Prevented Metal resource-count exhaustion during long decode. ArraysCache hybrid models such as GLM-5.3 now periodically materialize recurrent state, release boundary snapshots and paged-cache blocks on failures, and avoid retaining unnecessary hot-cache payloads. By @Pietroski in #3227.

• Fixed Qwen3.8-Flash-Next late-join continuous batching. Model-owned caches now convert to their batch representation correctly, ragged QSA positions remain aligned, and indexer arrays are trimmed with the rest of the cache. This removes join-time crashes and subsequent cache-recovery cycles. By @daveycodez in #3246.

• Fixed loading models symlinked from the Hugging Face cache. Qwen4 and MiniMax sanitization now recognizes symlinked checkpoint shards without accidentally matching unrelated safetensors files. By @ChildOTK in #3224.

• Fixed GLM-5.3 mixed text and image conversations. Registering glm5_next with mlx-vlm's prompt configuration prevents image placeholders from being dropped when an image or screenshot appears alongside plain conversation turns or tool results. By @BTankut in #3278.

• Restored Qwen4 PLE state after rejected Lightning MTP drafts. N-gram history and short-convolution state now roll back to the committed prefix together with QSA and GDN state, with fail-closed validation around incomplete snapshots. By @hojin12312 in #3232.

• Enabled RAM-safe Qwen4 oQe calibration. Calibration now initializes the Qwen4 compatibility runtime, preserves SSD-mapped PLE behavior, uses the intended lenient VLM loading contract, and accounts for the tensors actually resident during calibration. By @actualrealhuman in #3269.

• Reduced Qwen4 long-prefill memory growth. QSA boolean masks now use tiled attention instead of dense score slabs, and QSA boundary snapshots store incremental slices instead of complete prefixes. On the reported 64 GB system, the clean-prefill ceiling increased from approximately 43K to 96K-100K tokens. By @alytaphoenix in #3283.

• Fixed mixed text and mRoPE prefix-cache reconstruction. One-channel text positions are promoted to the three-channel representation when joined with multimodal cache blocks, avoiding silent reconstruction failure and unnecessary full re-prefill. By @rsnow in #3219.

• Allowed TurboQuant KV cache and Lightning MTP to be enabled together in the macOS app. The stale client-side conflict check is removed while the actual DFlash and VLM incompatibility checks remain intact. By @popfido in #3254.

• Fixed model-setting persistence through the admin API. mtp_num_draft_tokens, preserve_thinking, and turboquant_skip_last are no longer silently dropped. Lightning MTP draft depth is validated against the supported range, and unknown settings now fail explicitly. By @yanzhaohui1999 in #3280.

• Added hybrid-cache boundary diagnostics. Runtime cache observability now reports boundary capture, SSD fallback, restore, fail-closed store-skip, and prefix re-prefill reasons without exposing prompt content. By @MDX-Tom in #3249.

Contributors

Thanks to @jonathan308, @e-Evolution, @davidtorcivia, @hesara, @Pietroski, @daveycodez, @ChildOTK, @BTankut, @hojin12312, @actualrealhuman, @alytaphoenix, @rsnow, @MDX-Tom, @popfido, and @yanzhaohui1999 for their contributions to oMLX 0.6.4.

Full Changelog: v0.6.3...v0.6.4