oMLX 0.7.4.dev1

This is a development release. Please report any bugs through GitHub Issues. If you need a stable release, please use oMLX 0.7.3.

oMLX 0.7.4.dev1 lets a second Mac with a fast GPU speed up prefill for a model that only fits on your big Mac, and makes DeepSeek V4.1 and Qwen3.8 Flash-Next faster.

Download: macOS 26 / 27 | macOS 15 Sequoia

Prefill Accelerator

Big MoE models fit on a Mac with lots of memory like an M3 Ultra 512GB, but long prompts prefill slowly there. Pair it with a Mac that has a faster GPU, like an M5 Max, and the fast Mac runs the first layers of each prompt chunk while the big Mac runs the rest, decodes and keeps the KV cache. Decode speed stays the same, and Thunderbolt or 5GbE is enough, no RDMA needed (#5065).

Prefill tok/s on a 64K prompt, M3 Ultra 512GB alone and with an M5 Max as the accelerator.

To set it up, turn on Global Settings > Advanced > Enable Distributed Inference on both Macs, pair them in the Cluster tab and choose Prefill Accelerator, then pick the Mac under Acceleration in the model's settings (guide).

Faster DeepSeek V4.1

Prefill now keeps a bounded buffer pool through the layer loop instead of handing every buffer back to Metal after each layer, which makes it 43-48% faster on an M5 Ultra (@junmo-kim in #5031). On M5, oQ4e builds with MXFP4 experts and MXFP8 dense weights also run INT8 Activation Prefill (#5058), and the first decode on a fresh shader cache no longer waits on hundreds of kernel compiles (@junmo-kim in #5025).

Faster Qwen3.8 Flash-Next

FP16 Qwen3.8 Flash-Next checkpoints now run the fused hyper-connection and GDN kernels that only BF16 used before, and their greedy output with Lightning MTP on matches MTP off (#5028 with @qoli). On 128 GB Macs, models like Qwen3.8 Flash-Next, GLM-5.3 and DeepSeek V4.1 keep wide prefill chunks under memory pressure, so back-to-back long prompts stop slowing down (@yohann-bearzi in #4902). Decode after long prompts is also 3-5% faster on M5 Max (@yohann-bearzi in #4901).

New Features and Improvements

• Integrations: omlx launch codex, omlx launch codex_app and omlx launch copilot hand the agent every chat model on the server, so /model switches between oMLX models without relaunching. Each model keeps its own context window, image support and reasoning effort levels (#5026 with @scaryrawr, @scaryrawr in #5066).
• Models: Ming-3.0-tts and Ming-3.0 Tiny load as TTS models with a compiled decode loop and fused DiT kernels. Real-time factor on an M5 Max goes 0.210 -> 0.101 (-52%), and 0.798 -> 0.258 (-68%) for Tiny. Voice clones use the checkpoint's CAM++ speaker embedding, and /v1/audio/voices lists the built-in voices (#5029). LFM3 models apply RoPE in one launch (b73ee56 with @alan13367).
• Batched decode: 6-row 8-bit layers read their weights once, so Qwen3.8-27B oQ8e with 6 requests goes 62.2 -> 72.4 tok/s (+16%) on an M5 Max (b5c7793 with @yohann-bearzi).
• macOS app: Ask AI can answer with an oMLX server on another Mac, with a Test connection button (@ianderse in #5050). The model library checks Hugging Face models for updates and shows the changed files before updating (@ngutech21 in #5046). German now covers the whole app (@christianschmidt in #5014, #5048), and Japanese and Korean counts use 万 and 만 units like the web console (@LXD-8 in #4980).
• Web UI: German for the admin dashboard (@alexandergreil in #4673), and chat sidebar items are more compact, with rename and delete in a ... menu (ce43187).
• API: /v1/messages continues a trailing assistant message as an Anthropic prefill, like partial on chat completions (@yanzhaohui1999 in #4995). The first plain-text request after a model load no longer builds the grammar compiler, which saves 0.31-0.37 s (ae659d3 with @LXD-8).

Bug Fixes

• Parallel tool results reach the chat template in the order of the tool calls on /v1/messages and /v1/chat/completions, so Qwen no longer pairs a result with the wrong call (@yanzhaohui1999 in #5012, 008987a, #4999).
• Cluster: agent conversations reuse the prompt cache across tool turns instead of prefilling the whole conversation again. Cached tokens on the fifth turn of a tool loop go 743 -> 9,266 (#5062, #3062).
• A long VLM prompt is tokenized once and off the event loop, so it no longer stalls other streams. On a 119K-token prompt, response headers arrive in 162 ms instead of 320 ms (@ska2704 in #5057, #5005).
• GLM-5.3 gets earlier reasoning as a reasoning_content field in history, as its chat template expects (3b2c355, #5006).
• Model Settings shows the saved reasoning_effort when it opens (debf5f7, #5053).
• DeepSeek V4.1 fused decode no longer keeps a small index array for every decode position (0de3b0b).
• A memory refusal says how much oMLX itself uses, so the rest of the gap reads as other apps' memory (e47101f).
• Attachments are accepted by their extension when the browser reports an unusual MIME type, so Firefox can attach .md files (@HamedShams in #4887, #3844).
• Splash: an unreadable directory such as /usr/local/bin with mode 0700 no longer makes /admin/api/models return 500 (@begcdn in #5018, #5007).
• Web UI: the Claude Code command on the status page masks the API key while the copy button still copies the full command, double-clicking Save sends one request (@billythegod in #5055), and the chat input shows a focus ring only with enhanced readability (e493d10, #5054).
• macOS app: Restart Server from the web dashboard no longer leaves the app without a server (@cgs4github in #5070, #5069), a slow update download keeps going and resumes after a stall instead of stopping after an hour (@HamedShams in #4890, #4199), and the menubar icon redraws only when it changes, so WindowServer no longer logs constraint errors on every stats poll on macOS 26 and 27 (f524399, #5049).

Upgrade Notes

• Source installs: run make install after updating. transformers up to 5.21 is allowed (#5004), and ddgs moves to 9.16.1 (#5002).
• Prefill Accelerator: both Macs need the same oMLX version (#5065).
• Integrations: omlx launch codex and omlx launch codex_app need Codex 0.148 or newer (#5026). omlx launch copilot writes the oMLX provider into Copilot's providers.json instead of environment variables and needs Copilot CLI 1.0.83 or newer (#5066).
• Cluster: existing deployments pick up the new prompt cache slot count when they are activated again (#5062).
• DeepSeek V4.1 on M5: oQ4e builds with INT8 Activation Prefill on now take the INT8 path, so greedy output can differ at near-ties (#5058).
• API: a trailing assistant message on /v1/messages is now continued instead of starting a new turn (#4995). An attachment with a supported extension is accepted whatever MIME type the client reports, except spreadsheets (#4887).

New Contributors

@alexandergreil in #4673, @christianschmidt in #5014, @begcdn in #5018, @ianderse in #5050, @ska2704 in #5057.

Full changelog