This development release adds Meta Muse Glimmer 30B with DFlash speculative decoding, makes DeepSeek V4 prefill 23-28% faster across context lengths, persists GDN recurrent state as bounded SSD sidecars, and fixes Ling accuracy, grammar decoding overhead, and tool-call parsing edge cases.
- Added Meta Muse Glimmer 30B VLM support with DFlash speculative decoding. The model implementation (hybrid SWA/NoPE text backbone, ViT-G/14 encoder) is vendored from the upstream mlx-vlm port by @Blaizzy (Blaizzy/mlx-vlm#1838), plus the quantized embedding-norm fix by @johntdavies (Blaizzy/mlx-vlm#1839). Thank you both for the quick upstream turnaround. On the oMLX side this adds channel-scoped output parsing with ATEM tool calls, prefix and SSD cache integration, and oQ quantization (#2586), and DFlash speculative decoding is wired in #2587, reaching about 1.4x decode on the oQ4 pair.
- Faster DeepSeek V4 prefill, 23-28% across context lengths. Layered across four changes following the ablation in #2558: ratio-128 layers now route through the native attention kernel (#2559), the MXFP4 MoE block threshold matches 2048-token chunks (#2562), and redundant all-pooled indexer scoring is skipped (#2563), all by @DiscoStew6082. DeepSeek V4 cache blocks now target 2048 tokens so those kernels engage with the prompt cache enabled. On top of that, a fused windowed+pooled prefill kernel, skipping the discarded lm_head projection on intermediate chunks, in-place pooled cache appends, and a fused indexer mask, by @jonathan308 in #2568. Measured on an M3 Ultra (V4-Flash 4-bit, default server config, cold prompts, prompt processing tok/s):
| stage | pp4096 | pp16384 | pp32768 | pp65536 | pp131072 |
|---|---|---|---|---|---|
| 0.5.8.dev2 | 496.5 | 481.2 | – | 453.3 | 398.4 |
| + #2559 #2562 #2563 | 491.0 | 475.2 | – | 442.4 | 405.4 |
| + 2048-token cache blocks | 549.4 | 558.7 | 549.5 | 524.2 | 461.6 |
| + #2568 (= 0.5.8.dev3) | 631.1 | 604.9 | 589.3 | 559.1 | 508.5 |
The kernel PRs need 2048-token prefill chunks to pay off, which is why the middle row is flat: the block-alignment change is what unlocks them under the default cache-enabled config. Decode speed is unchanged at every context length. Note that temperature-0 outputs can differ from 0.5.8.dev2 for identical prompts: the new kernels change floating-point reduction order, which only flips tokens where the model is effectively indifferent, and quality is unaffected.
- Reduced long-context memory for GDN models when the SSD cache is enabled. GDN recurrent state is now persisted as bounded SSD sidecars instead of retaining one full in-memory state per historical boundary, making long-context Qwen3.6 serving practical on unified-memory Macs. Opt-in and backward-compatible. By @hojin12312 in #2569.
- Fixed Ling-3.0-flash code accuracy, +17pp on HumanEval. Its late layers were running without the trained SwiGLU clamp shipped in
config.json; applying it raises HumanEval from 71.3% to 88.4% at no runtime cost. By @True2456 in #2571. - Removed a per-token host sync from grammar-constrained decoding. Token acceptance is deferred to the top of the next step with bit-identical output, so the forward dispatch never blocks on host work. By @fxd0h in #2561.
- Improved adaptive prefill memory estimates at high context. The estimator now remembers process footprint released after a chunk and accounts for it when sizing the next one, closing an over-admission gap. By @mvdbos in #2573.
- Fixed tool calls whose arguments contain a literal close marker. These were dropped with raw markup leaking into the response; payload boundaries are now located by span scanning with JSON-aware decoding, in both the batch parser and the streaming filter. By @richgoodson in #2544, with follow-up fixes for prose around malformed envelopes at end of stream and for JSON array payloads.