oMLX 0.7.0.dev2
oMLX 0.7.0.dev2 adds DeepSeek V4.1 Flash with DSpark MTP and Engram SSD offload, faster Qwen prefill on M5 Macs, and experimental MoE expert SSD offload. This release also adds Lightning MTP for GLM-5.3-Flash and fixes model loading, structured output, thinking controls, and distributed serving.
DeepSeek V4.1 Flash with DSpark MTP and Engram SSD Offload
DeepSeek V4.1 Flash now supports text and vision inference, tool calling, and oQ quantization in oMLX. DSpark MTP, integrated through Lightning MTP, accelerates generation on supported -mtp checkpoints. Engram SSD offload keeps Engram tables on disk and reads the required pages on demand, reducing resident memory requirements.
Ready-to-use checkpoints:
- DeepSeek-V4.1-Flash-oQ4e-mtp
- DeepSeek-V4.1-Flash-oQ3e-mtp — designed to fit 256 GB machines with Engram SSD offload. Available context depends on remaining system memory.
Benchmarked on an M3 Ultra with 512 GiB of unified memory, using DeepSeek-V4.1-Flash-oQ4e-mtp with Engram in RAM, Code (Python) context, 128 generated tokens, temperature 1.0, and prefix caching disabled. Each configuration used a 4K warm-up; each row is one sampled run, excluding model loading.
| Context | Prefill: MTP OFF | Prefill: MTP ON | Generation: MTP OFF | Generation: MTP ON | Generation change |
|---|---|---|---|---|---|
| 4K | 457.99 tok/s | 452.15 tok/s | 20.23 tok/s | 32.06 tok/s | +58.5% |
| 16K | 459.12 tok/s | 454.77 tok/s | 20.00 tok/s | 34.72 tok/s | +73.6% |
| 32K | 452.19 tok/s | 447.73 tok/s | 19.80 tok/s | 31.51 tok/s | +59.1% |
| 64K | 439.39 tok/s | 435.56 tok/s | 19.67 tok/s | 39.66 tok/s | +101.6% |
At 64K context, DSpark MTP improved generation throughput by 101.6%, reaching 39.66 tok/s. Lightning MTP remains single-stream only.
See #3574 for implementation details and Engram SSD offload benchmarks.
INT8-Activation Prefill for Qwen on M5 Macs — +34.4% Prefill
New prefill kernels use M5 tensor units to accelerate eligible Qwen3.5, Qwen3.6, and Qwen3.8 projections with INT8 activations. The kernels read existing packed Q4/Q5 weights directly, without creating another weight copy.
The feature is off by default and available under per-model Experimental Features. It requires compatible affine Q4/Q5 weights with group size 64 and cannot be combined with ANE prefill. Activation quantization can change model outputs.
Maintainer benchmark on an M5 Max, using Qwen3.8-27B-oQ4e-mtp with thinking and Lightning MTP disabled. Speed figures are medians of three runs.
| Metric | INT8 OFF | INT8 ON |
|---|---|---|
| 32K prefill | 615.2 tok/s | 826.7 tok/s (+34.4%) |
| Generation, 128 tokens after 32K input | 24.6 tok/s | 22.2 tok/s |
| MMLU, 1,000 questions, 5-shot | 82.9% | 83.0% |
| HumanEval, 164 questions, pass@1 | 96.34% | 95.73% |
See the benchmark comment for details.
MoE Expert SSD Offload
Experimental expert offload lets supported MoE models run with fewer expert weights resident in unified memory. A configurable 12.5%–75% of experts per layer stays in memory; other experts are loaded on demand directly from the existing safetensors checkpoint. No converted checkpoint or additional weight copy is required.
The router still selects the model’s original experts. Lower residency saves memory at the cost of SSD reads, slower prefill, and lower generation throughput.
Supported models with compatible checkpoint layouts:
- DeepSeek V4.1 Flash, including original and oMLX-converted checkpoints.
- Qwen3.8-Flash-Next (
qwen4_exp), including original and oMLX-converted checkpoints. - Gemma 4 MoE.
- OLMoE.
Enable it in Model Settings → Experimental Features → MoE Expert Offload and choose the resident fraction. DeepSeek Engram SSD offload and Qwen PLE SSD offload can be enabled alongside expert offload. Lightning MTP, DSpark MTP, and DFlash must be disabled.
By @beatakouchnir, with fixes from @fxd0h, in #2595, followed by maintainer integration for DeepSeek V4.1 and Qwen3.8-Flash-Next. See the supported layouts and usage guide.
Other Improvements and Bug Fixes
-
Added Lightning MTP for GLM-5.3-Flash, including MTP-head loading, speculative cache rollback, and model-specific runtime patch isolation. By @davidtorcivia in #3228.
-
Fixed quantized GLM-5.3-Flash loading by remapping forget-gate scales and biases alongside their weights. By @arnavprabhu in #3570.
-
Fixed structured output on VLM engines. Grammar masks now use the language model’s vocabulary size, preventing repeated token rejection on composite model configurations. By @L4XB in #3551.
-
Fixed thinking-control handling. Top-level
enable_thinkingis preserved, contradictory controls are rejected, and a zero thinking budget no longer implicitly enables thinking. By @PhilipJohnBasile in #3559. -
Fixed DeepSeek V4 attention selection for mixed-bit checkpoints. Nested 2-bit and 3-bit overrides now select the appropriate reference path. By @PhilipJohnBasile in #3560.
-
Fixed an SSD prefix-cache write race that could discard recurrent boundary snapshots while another write was queued. By @L4XB in #3563.
-
Improved distributed serving. Control connections try direct transport first while preserving proxy fallback and deadlines, and progressive sharding releases temporary weight references earlier to reduce loading memory. By @xunlinkx in #3577 and #3578. Interrupted pairing sessions can also recover without restarting the setup process (#3565).
-
Fixed admin validation errors returning HTTP 500 instead of 422. By @arnavprabhu in #3568.
-
Corrected physical memory capacity display in the macOS app. By @xunlinkx in #3542.
-
Corrected oQ output dtype metadata to reflect the selected output precision. By @andyoneal in #3528.
New Contributors
- @arnavprabhu made their first contribution in #3568.
- @beatakouchnir made their first contribution in #2595.
Full Changelog: v0.7.0.dev1...v0.7.0.dev2