oMLX 0.7.3.dev1

This is a development release. Please report any bugs through GitHub Issues. If you need a stable release, please use oMLX 0.7.2.

oMLX 0.7.3.dev1 adds MiniMax M3.1 Flash with Lightning MTP, makes M1 and M5 Macs, DFlash on fp16 models and GLM-5.3 faster, and gives low-bit oQ quants a better split of their bits.

Download: macOS 26 / 27 | macOS 15 Sequoia

MiniMax M3.1 Flash

oMLX now serves MiniMax M3.1 Flash from the original NVFP4 checkpoint or from oQ builds. Its bundled DSpark draft runs as Lightning MTP and verifies a whole batch in one forward, and greedy output is the same with it on or off. #4777

oQ4e on an M3 Ultra, 1024-token code continuations with real-use sampling, measured before #4806.

On top of that, fused decode kernels make one-request decode 14% faster at short context and 21-23% faster from 5K to 64K tokens with bitwise-identical output, and a batched row now decodes exactly like the same request alone (@yohann-bearzi in #4806).

Faster M1 and M5 Macs, DFlash and GLM-5.3

M1 Macs get speculative verify and long-context attention routes of their own. They were tuned on an M1 Max and checked on an M1 Ultra by @jmmarkiewicz, where prefill runs about 50% faster with decode level and peak memory a little lower (@fneto1977 in #4496). On M5 Macs, DeepSeek V4's ratio-4 prefill attention now runs on the tensor units (@Phenom300 in #4808, cf91ba8).

A bf16 DFlash drafter next to an fp16 target used to run its whole forward in fp32. It now runs in fp16 with an fp32 residual stream (@fneto1977 in #4773). Batched GLM-5.3 decode scores short indexer queries without padded tiles (@jwiegley in #4801, 0b8ab34), and each GLM-5.3 KDA layer stops keeping its input projections twice, which frees 3.27 GB on GLM-5.3 Flash oQ4e (@junmo-kim in #4764).

Better Low-Bit oQ

oQ gave the output head and embeddings 8 bits at every level, which left too few bits for the attention and MLP layers in low-bit quants. The head now gets base + 3 bits, dense models below 3-bit move every attention and MLP tensor to 3-bit first, and MoE models raise the layers every token runs through before routed experts. Sizes stay within 0.07 GB, and DeepSeek V4.1 plans don't change. #4776

KL against the original model in fp32, lower is better. Qwen3.8-27B and Qwen3.6-35B-A3B are compared with the 0.7.1 builds.

oQ can also keep DeepSeek V4.1's Engram tables at their original FP8 precision with "Keep original Engram tables" in Advanced settings. The tables sit outside the bit budget and can be served from SSD, so they cost disk space but no resident memory (#4758).

New Features and Improvements

• Models: RecursiveSI Flint serving and oQ (@scaryrawr in #4763, 3df9460), KRAFTON Raon-1 mini without trust_remote_code (@Bannng in #4750, a0eebd0), MiniCPM-V 4.7 with OpenBMB's DSpark draft through DFlash, 72.7 -> 110 tok/s (+51%) on an M4 Max (@ChristianPraiss in #4737, a184dc1), and Muse Spark on mlx-lm's own model, with earlier oMLX conversions still loading (#4778).
• Lightning MTP: the batch park backoff now carries across requests joining and leaving, so MTP-on throughput with 8 clients on Qwen3.8-Flash-Next rises 11% on an M5 Max and 3% on an M3 Ultra (@yohann-bearzi in #4810).
• DeepSeek V4.1: with MoE expert offload, prefill reads the next chunk's experts while the current one runs, +4% when the experts come from SSD (@williamxie1989 in #4668, 1330571).
• Prompt cache: clients can send X-oMLX-Cache-Store: off to keep a one-shot request out of the prefix cache, and the request still reuses prefixes others stored (@BrownMark in #4815).
• Chat: images returned by MCP tools show as thumbnails and the model gets a one-line note instead of a huge base64 string (@wesbiggs in #4769), and full model names show on hover in the model picker (@sampe-bot in #4804).
• Downloads: Hugging Face model URLs, including /tree/main and file links, work in the download box (@shaanchander in #4807).
• Cluster: the plan preview says why the cluster runs the TCP ring instead of RDMA and how to fix a Thunderbolt Bridge address (6aba0dd), and a rank that exits shows its own last output lines (@Abelo9996 in #4753).
• Dependencies: mlx-lm 0d4d3d04 (#4778), and mlx-vlm 0.7.10, which fixes wrong output for shorter requests in Qwen2-VL, Qwen2.5-VL and LLaVA batches (#4783).

Bug Fixes

• Prompt-opened thinking is split from the answer on Chat Completions, Messages and Responses, and a Responses stream with thinking off no longer sends the answer twice (@jwiegley in #4798).
• /v1/messages honors tool_choice: none for client and MCP tools and in count_tokens (@richgoodson in #4818, 8c1b980).
• SpecPrefill keep rate and threshold from Model Settings apply on Responses, Messages, Completions and benchmarks, not only on chat completions (24740a3 with @popfido, #3056).
• An empty completion prompt gets a 400 instead of failing every other request on the engine (19178d0 with @jwiegley), and SSE keepalive chunks carry the request's model and a real timestamp (@jwiegley in #4797).
• VLM prompts keep each image next to its label when text sits between images (@bercedev in #4770, 017b0cc), and MiniCPM-V 4.7 loads after a Qwen3.5/3.6 VLM in the same server (#4783).
• Cache: a dirty hot-cache entry is written at shutdown even when an older file has the same hash (@billythegod in #4792), and prefix hit counters count only KV that was actually reused (@fqx in #4782).
• Cancelling a model load while the engine is still starting, such as cancelling a benchmark, no longer leaves the model in memory (@billythegod in #4788, dc91812).
• Cancelling a ModelScope download stops the transfer right away (8391858 with @billythegod), and a Hugging Face upload keeps the model's own README (9c4e981 with @billythegod).
• Restoring the Claude Desktop gateway twice no longer switches a third-party setup back to first-party (@billythegod in #4791).
• CUDA cluster workers no longer exit at startup when mlx-vlm is not installed (@bercedev in #4751).
• oQ: text-only Qwen3.5 checkpoints such as Hemmingway-1 quantize (@arnavprabhu in #4803), DiffusionGemma gets past sanitize (@Abelo9996 in #4754), and MiniMax M3's patch merger stays unquantized like the rest of its vision path (#4806).
• CMMLU, JMMLU and KMMLU find the answer letter when Chinese, Japanese or Korean text touches it (@MaxFreedomPollard in #4774).
• Web UI: the chat context meter shows the window the server enforces (@kathsucurry in #4795), the dashboard and chat run their startup once instead of twice (@kathsucurry in #4796), the Model Settings header fits on phones (@stellarsailor in #4752), and the duplicate readability switch leaves the chat settings sheet (@LXD-8 in #4700).

Upgrade Notes

• Source installs: run make install after updating. mlx-lm moves to 0d4d3d04, mlx-vlm to 0.7.10 and modelscope to 1.28 or newer.
• oQ: new quants at the same level get a different bit split (#4776). Existing quants load and run as before.
• SSD cache: blocks that earlier versions saved for gpt-oss, ERNIE 4.5 MoE, PhiMoE, PLaMo 3, FP8 Ministral 3 and Muse Spark are not reused, since mlx-lm changed their numerics (#4778).
• DeepSeek V4 on M5: the tensor-unit prefill kernel is not bit-identical to the old one, so greedy output can differ at near-ties (#4808).
• Cache stats: prefix hit rates can read lower than before, since an evicted or partial restore no longer counts as a full hit (#4782).
• Benchmarks: CMMLU, JMMLU and KMMLU scores can be higher than in earlier runs (#4774).
• API: an empty completion prompt now returns 400, and admin output token defaults must be 1 or more (19178d0).

New Contributors

@jwiegley in #4797, @kathsucurry in #4795, @wesbiggs in #4769, @Abelo9996 in #4753, @ChristianPraiss in #4737, @sampe-bot in #4804, @Phenom300 in #4808, @BrownMark in #4815.

Full changelog