oMLX 0.7.3rc1
This is a release candidate. Please report any bugs through GitHub Issues. If you need a stable release, please use oMLX 0.7.2.
The release candidate for oMLX 0.7.3 adds Ask AI, a question bar for your local models in any app, a GPU power limit for working on battery, and Ling 3.1 Flash with Lightning MTP. It also brings everything from the 0.7.3 development releases: MiniMax M3.1 Flash, faster M1 and M5 Macs, and better low-bit oQ.
Download: macOS 26 / 27 | macOS 15 Sequoia
Ask AI
Ask your local models from any app. Press Option-Space and a question bar floats over whatever you are working on without taking focus, and the answer streams in as Markdown. Slash presets set the model and system prompt per command, so /translate can run a small model with a fixed prompt and copy the result for you. Vision models also take images pasted with Command-V. Turn it on in the macOS app under App > Ask AI. #4854
GPU Power Limit
Working on battery? GPU Power Limit in Model Settings > Advanced caps the Mac's GPU power from 2 to 1000 W while that model generates, in both the web UI and the macOS app, and Ask AI has a limit of its own. The limit applies only while a capped request runs, comes off 15 s after the last one, and never applies to benchmarks. No root or helper app is needed. #4855
M5 Max, decode speed and CPU+GPU energy per token against no limit.
| Limit | 35B-A3B decode | Energy per token | 27B decode | Energy per token |
|---|---|---|---|---|
| 15 W | -6% | -25% | -7% | -29% |
| 10 W | -14% | -37% | -18% | -43% |
| 5 W | -36% | -49% | -48% | -52% |
The GPU power target it sets comes from m5gpu by @beamivalice.
Faster Qwen3.8-Flash-Next, Ling 3.1 Flash and More
Ling 3.1 Flash now loads straight from its original FP8 checkpoint, and Ling Flash 3.0 and 3.1 run their bundled nextn layer as Lightning MTP, which oQ keeps with Preserve MTP. On an M3 Ultra, Ling-3.1-flash oQ4 decodes 18.8 -> 34.3 tok/s (+82%) at 4K context and 18.1 -> 29.6 tok/s (+64%) at 32K, with real-use sampling and one request at a time. #4881
| Model | Change | Hardware |
|---|---|---|
| Qwen3.8-Flash-Next oQ4e | Decode 65.6 -> 81.5 tok/s (+24%), 4 requests 167.6 -> 186.5 tok/s (+11%) | M3 Ultra |
| GLM-5.3-Flash, Macs before M5 | One-request decode, FP8 22.6 -> 25.2 tok/s (+12%), oQ4e 38.6 -> 41.7 tok/s (+8%) | M3 Ultra |
| Flint oQ3e | Decode 91.0 -> 109.1 tok/s (+20%), with TurboQuant at 12K 23.5 -> 75.7 tok/s (3.2x) | M4 Max |
| DeepSeek-V4.1-Flash oQ2e, Engram on SSD | Prefill at 128K 1512 -> 1719 tok/s (+14%) | M5 Ultra |
Each figure is measured against the code right before the change. By @yohann-bearzi (#4880 from #4853, and #4843), @alan13367 (#4857, #4856) and @junmo-kim (#4745). On Macs before M5, GLM-5.3 now gives the same greedy output with Lightning MTP on and off (#4843). Also faster: DeepSeek V4.1 long prefill, +3.5% at 128K on an M3 Ultra (696d070 with @N1k1tung), and M1 5-bit verify, +2.7% with DFlash2 on an M1 Ultra (@fneto1977 in #4830, measured by @jmmarkiewicz).
New Features and Improvements
- Splash: Splash 1.4.1's Ornith-1.5-122B and Flint with their DFlash2 drafts, MoE GGUFs pruned to fewer experts (@arnavprabhu in #4870), Flash-Next GGUF repos with PLE variant folders (@arnavprabhu in #4821, c5127a2), and Delete in the dashboard's SSD cache panel clears Splash's cache too (@arnavprabhu in #4824). The settings now say Splash's SSD cache limit applies per model (#4852).
- API:
/v1/chat/completionsreturns OpenAIlogprobsandtop_logprobs(up to 20) for streaming and non-streaming requests, including reasoning and tool-call tokens and with Lightning MTP on. Closes #1549, taking over from @popfido's #1591 and @ashokgelal's #4380 (#4859). - M1 and M2 Macs: Float16 Activations in Model Settings > Advanced converts a checkpoint's bfloat16 scales, norms and embeddings to float16, which these GPUs run natively. Qwen3.6-35B-A3B oQ4e decode goes 75.9 -> 82.4 tok/s (+9%) on an M1 Max (@parasew in #4819 from @pedroberaldo87's #3277, d1545cc, @yohann-bearzi in #4871).
- MoE expert offload: Model Settings suggests a resident percent that still leaves room for a prompt, and clicking the hint fills it in (#4858, based on @beatakouchnir's #3676).
- Rerankers: a per-model Context Window in the web and Mac Model Settings (@zviratko in #4826, 9401674). Jina rerankers follow the reference block window and v3.5's query and document limits, so a long query no longer fails (79bd852 with @zviratko), and v3.5 runs its sliding layers in window-sized chunks (e4844fe).
- macOS app: updates download in the background and still ask before installing, and a failed install shows in the Status screen and the update sheet (6855520, c03d936 with @joshuabke, from #3144).
- Dependencies and builds: transformers up to 5.20 (#4834), markitdown 0.1.9 (#4833), wheels built with Xcode 26.5 so they include the NAX kernels (@XXXXRT666 in #4836), and dead code cleanups (@shaanchander in #4847, #4848).
Bug Fixes
- Lightning MTP: GLM-5.3 on Macs before M5 and DeepSeek V4 verify drafts with their decode arithmetic (@yohann-bearzi in #4843, @Phenom300 in #4813), the DSpark ring GEMM uses the Steel kernel only where it matches bit for bit (@williamxie1989 in #4612), and mRoPE VLMs keep the running request's positions when a text prompt joins (@popfido in #4837).
- A reply that ends inside an unclosed tool-call envelope returns an
incomplete_tool_callerror instead of the raw text withfinish_reason: stop(@HeinrichvH in #4869, #3153). /v1/messages/count_tokenscounts the prompt/v1/messagesactually renders, with tools, MCP and thinking settings (@richgoodson in #4875).- The memory guard re-reads
iogpu.wired_limit_mb, so raising it while the server runs takes effect (@Woody-Hu in #4860, #4849). - Macs before M5 get the driver's relaxed compute context-store timeout at start, a setting other MLX apps use against Metal
0000000eerrors in long prefills and image prompts (#4872). - Model directories on a disconnected drive stay in settings, and the server starts when none of them is available (0fd3475 with @isnaenihidayat, #4873). An unavailable SSD cache dir falls back to the default one until restart instead of stopping the server (28f50bb with @pmjpmj).
- Splash reports a missing draft or launcher version before memory admission, instead of a memory error sized from the GGUF files (#4876).
- SigLIP and SigLIP2 embed text and images through their own towers (#4850), and Unlimited-OCR takes the model card's
<image>prompt (@tc3oliver in #4822). omlx restartfrom a conda env under Homebrew's Caskroom no longer restarts the Homebrew service (009e75c with @ketpatil77, #3120).- Downloads: the speed readout no longer jumps above the real rate (@LXD-8 in #4844), and a refused retry keeps the row (@LXD-8 in #4845).
- Web UI: long model names in the chat model picker truncate instead of running under the reasoning pill (@bercedev in #4812).
Highlights from 0.7.3.dev2
- MiniMax M3.1 Flash. Served from NVFP4 or oQ with its DSpark draft as Lightning MTP, up to +87% for one request (#4777), plus fused decode kernels for another 14-23% (@yohann-bearzi in #4806).
- Faster M1 and M5 Macs, DFlash and GLM-5.3. About 50% faster long-context prefill on M1 Ultra (@fneto1977 in #4496), DeepSeek V4 prefill attention on the M5 tensor units (@Phenom300 in #4808), fp16 DFlash +33% (@fneto1977 in #4773), and batched GLM-5.3 decode +17% (@jwiegley in #4801).
- Better low-bit oQ. A new bit split lowers KL by up to 73% at the same size (#4776), and DeepSeek V4.1 Engram tables can stay FP8 (#4758).
Benchmarks, the full feature and fix lists, and every contributor credit are in the 0.7.3.dev2 release notes.
Upgrade Notes
- Source installs: run
make installafter updating. mlx-lm moves to 0d4d3d04, mlx-vlm to 0.7.10 and modelscope to 1.28 or newer, and transformers up to 5.20 is allowed (#4834). - oQ: new quants at the same level get a different bit split (#4776). Existing quants load and run as before.
- SSD cache: blocks that earlier versions saved for gpt-oss, ERNIE 4.5 MoE, PhiMoE, PLaMo 3, FP8 Ministral 3 and Muse Spark (#4778) and for Ling Flash (#4881) are not reused, since their numerics changed.
- DeepSeek V4 on M5: the tensor-unit prefill kernel is not bit-identical to the old one, so greedy output can differ at near-ties (#4808).
- Cache stats: prefix hit rates can read lower than before, since an evicted or partial restore no longer counts as a full hit (#4782).
- Benchmarks: CMMLU, JMMLU and KMMLU scores can be higher than in earlier runs (#4774).
- API: an empty completion prompt now returns 400, and admin output token defaults must be 1 or more (19178d0).
logprobson an engine that cannot produce them (cluster, Splash, diffusion) returns 400 (#4859), an unclosed tool-call envelope returnsincomplete_tool_callinstead ofstop(#4869), and/v1/messages/count_tokenscan read higher where tools or thinking settings apply (#4875). - macOS app: updates now download in the background by default and still ask before installing. Turn off Download Automatically under Updates on the Status screen to only get notified.
New Contributors
@jwiegley in #4797, @kathsucurry in #4795, @wesbiggs in #4769, @Abelo9996 in #4753, @ChristianPraiss in #4737, @sampe-bot in #4804, @Phenom300 in #4808, @BrownMark in #4815, @parasew in #4819, @HeinrichvH in #4869.