This development release focuses on Lightning MTP speculative decoding, broader oMLX custom-kernel acceleration, and oQe imatrix-enhanced quantization, with new model support, admin UX improvements, and memory/build hardening after 0.4.5.dev1.
Highlights
- Lightning MTP. Depth-k native speculative decoding now covers Qwen3.6-27B, Qwen3.6-35B-A3B, DeepSeek-V4-Flash, and GLM-5.2 with adaptive depth, sharper draft sampling, and verify-shape Metal kernels. On M3 Ultra, greedy 400-token runs improved Qwen3.6-35B-A3B from 89.6 to 140.4 tok/s and Qwen3.6-27B from 35.0 to 55.1 tok/s. (#2113)
- oMLX Custom Kernels. Added DeepSeek V4 MXFP4 MoE/sparse-attention kernels, Qwen3.5/3.6 native prefill kernels, and GLM-5.2 Sparse MLA DSA kernels. DeepSeek-V4-Flash 64k prefill improved 313.3 -> 455.8 tok/s with 6.50 GiB lower peak memory; Qwen3.6-27B 64k prefill improved 262.2 -> 347.6 tok/s. (#2048, #2100, #1984)
- oQe imatrix enhanced quantization. oQ now has an activation-importance calibration pass, MoE expert coverage tracking, weighted affine quantization, cache reuse, and admin UI controls. In the PR benchmark, oQ4e improved average accuracy over oQ4 on Gemma-4-26B-A4B-it, Qwen3.5-9B, Qwen3.6-35B-A3B, and Qwen3.6-27B while keeping the same disk-size class. (#2057)
Performance Snapshot
| Area | Benchmark result |
|---|---|
| Lightning MTP | Qwen3.6-27B: 35.0 -> 55.1 tok/s, 1.57x; Qwen3.6-35B-A3B: 89.6 -> 140.4 tok/s, 1.57x; DeepSeek-V4-Flash: 29.5 -> 35.1 tok/s, 1.19x; GLM-5.2: 15.6 -> 17.9 tok/s, 1.15x. (#2113) |
| DeepSeek V4 kernels | DeepSeek-V4-Flash-oQ8 prefill speedup reached 1.45x at 64k context, with peak memory reduced from 151.25 GiB to 144.75 GiB. (#2048) |
| Qwen kernels | Qwen3.6-27B-oQ4e prefill improved up to +32.6% at 64k context; a 16k parity run generated identical token IDs and improved wall time 47.271s -> 42.505s. (#2100) |
| GLM-5.2 kernels | 16k PP improves from 128.1 to 178.9 tok/s, a 1.40x gain, while TG stays in the same band. 32k PP improves from 87.7 to 174.4 tok/s, a 1.99x gain, with 18.53 GB lower peak memory. (#1984) |
| oQe accuracy | oQ4e averages: Gemma-4-26B-A4B-it 85.27% vs oQ4 83.49%; Qwen3.5-9B 73.09% vs 70.01%; Qwen3.6-35B-A3B 83.88% vs 82.83%; Qwen3.6-27B 85.62% vs 84.99%. (#2057) |
New Features
- Added Lightning MTP depth-k native speculative decoding for Qwen3.6, DeepSeek-V4-Flash, and GLM-5.2. (#2113)
- Added bundled DeepSeek V4 MXFP4 MoE and sparse-attention custom kernels. (#2048)
- Added Qwen3.5/3.6 native prefill kernels for FA-256 attention, quantized qmm, GatedDeltaNet, and MoE weighted-sum paths. (#2100)
- Added GLM-5.2 32-head sparse-MLA custom-kernel support for tensor-sharded multi-device runs. by @aidiffuser in #2070
- Added oQe imatrix-enhanced quantization, calibration cache handling, MoE expert accumulation, and admin controls. (#2057)
- Added Tencent Hy3 / HunYuan V3 support with upstream-first mlx-lm patching and
:opensourcethinking tag handling. by @gilby in #2108 - Added Ornith model support for oQ conversion and inference, including per-expert MoE tensor stacking. by @kreeger in #2044
- Added admin model search, filtering, sortable model tables, persistent sort state, and model visibility controls. by @jasonpaulso in #2107
- Added Russian localization for the macOS app. by @DrMaks22 in #2028
- Added custom-kernel install documentation and a Homebrew custom-kernel option.
Bug Fixes
- Fixed custom-kernel source builds by pinning nanobind to the MLX 0.31.2 ABI-compatible version. by @GeorgeTheo99 in #2035
- Fixed macOS 27 beta Homebrew installation, source-build flags, and pip cache handling. by @heyparth1 in #2111; follow-up for #2110
- Fixed Homebrew spaCy model wheel installation and Kokoro torchless/runtime packaging gaps. (#2034, #2039)
- Fixed GLM-5.1 MXFP4 loading under the GLM DSA patch. (#2040)
- Fixed memory retention across embedding, reranker, VLM, failed-load, unload, and boundary snapshot writer paths. by @zwcf5200 in #2052, #2053, #2056, #2061, #2103; by @tbro0815 in #2087
- Fixed false 507s during back-to-back large-model swaps. by @thornad in #2021
- Fixed hard-pressure recovery and missing/corrupt model retry loops. (#2042, #2063)
- Fixed buffered rotating cache metadata and MTP boundary snapshot/cache-store skew. (#2060, #2113)
- Fixed streamed reasoning leakage and chunk-mode SSE keepalive role handling for LangChain.js/n8n tool calls. by @isaac-cf-wong in #2043; by @richgoodson in #2117, fixes #2074
- Fixed explicit
json_schema: nullhandling in JSON output paths. by @JimStenstrom in #2007 - Fixed duplicate model-id discovery shadowing by warning and keeping the first registration. by @JimStenstrom in #2014
- Fixed Kokoro model discovery and G2P language inference. by @ethannortharc in #2086
- Fixed Python/CI compatibility by pinning
transformers < 5.13, capping supported Python below 3.14, and updating related CI/development dependencies. by @ethannortharc in #2085; by @mxl in #2122; #2024, #2093, #2023, #1950 - Fixed admin/macOS UI copy for hot-cache sizing, model table tooltips, and pin OOM risk. by @mxl in #2095
New Contributors
Thank you to @GeorgeTheo99, @heyparth1, @mxl, @tbro0815, and @aidiffuser for their first contributions to oMLX in this release.
Full Changelog: v0.4.5.dev1...v0.5.0.dev1