oMLX 0.7.1

oMLX 0.7.1 adds GGUF downloads served through Splash, much faster DeepSeek V4.1 decode, 4-bit and 8-bit KV cache formats, a fix for GPU hangs when several models run at once, and distributed inference for more models. The notes below cover changes since 0.7.1rc1, and earlier 0.7.1 highlights are summarized at the end.

Download: macOS 26 / 27 | macOS 15 Sequoia

GGUF Models Through Splash

The model downloader now filters Hugging Face repos by MLX or GGUF and lets you pick the files you want, and a GGUF variant brings its vision projector along. Downloaded GGUFs show up as models, and Qwen3.8-27B and Qwen3.6-35B-A3B GGUFs run on Splash, as do NVFP4 and FP8 checkpoints with Splash 1.3.2. #4679

When memory is tight, Splash now fits the context to its budget instead of failing to start with a 409 (@Flo5k5 in #4677).

Faster DeepSeek V4.1

DeepSeek V4.1 Flash decode now runs on fused kernels, and concurrent requests read the weights once per step instead of running one by one. The kernels follow DeepSeek's reference numerics, and with DSpark Lightning MTP on, greedy output now matches decoding without it token for token. By @yohann-bearzi in #4658.

Engram rows on SSD are read exactly instead of faulting whole pages (@williamxie1989 in #4674, e7d4327), and wide prefill chunks prefetch their Engram rows (@rubencu in #4670).

Affine 4-bit and 8-bit KV Cache

Affine4 and Affine8 are new KV cache formats next to TurboQuant. Full-attention KV is packed into 4-bit or 8-bit codes as each prefill chunk is written, and decode runs on a packed attention kernel on any Mac that can build it. Pick it in Model Settings > Memory Saving > KV cache compression. #4659, built on #3582 by @Cianidos.

Qwen3.8-27B-oQ4e-mtp, 32,767-token prompt, 128 greedy tokens. Prefill peak is MLX memory above the weights. Every format returned the passkey.

Several Models at Once Without GPU Hangs

With two or more busy models, the GPU could lock up and macOS reset it, failing requests on every loaded model (#4224). All engines now submit GPU work through one shared command queue. #4637

Distributed Inference for More Models

Distributed inference now runs checkpoints that used to work on one Mac only. #4678

• GLM-5.3 and DeepSeek V4.1 across Macs, with pipeline or tensor parallelism, oQ exports included.
• VLM checkpoints as text models, so Qwen3.6, GLM-5.3 and DeepSeek V4.1 builds with a vision config run and refuse media input up front.
• Plans that match what each Mac loads, so Engram tables, vision towers and draft heads no longer skew the split.
• Staging over Thunderbolt or RDMA, when SSH answers on that link, with the route and throughput logged per Mac.
• Easier pairing, loopback-only nodes are flagged and a missing SSH host key asks for Remote Login (#4647, #3021).

New Features and Improvements

• Claude Desktop: a Claude Desktop toggle maps claude-opus, claude-sonnet and claude-haiku to your tier models, and omlx launch claude_desktop points the app at oMLX (@giuseppebonanno in #3592).
• Realtime transcription: the chat mic and the realtime WebSocket work with every STT model, which transcribes each phrase after a pause (2792f06, from @zviratko's idea in #4655).
• Qwen3.8-Flash-Next expert offload: experts load into shared buffers instead of Python bytes, +5-8% decode on M1 Max with 25% of experts resident (@oddmundleikvoll in #4210, 3166191).
• Lightning MTP on Qwen3.8-Flash-Next: the adaptive controller no longer parks MTP on text where it still wins, and Flash-Next defaults to one draft per cycle for steadier decode (@e-Evolution in #4648).
• Muse Spark Open: checkpoints in the mlx-lm 0.32.2 layout load, including the new mlx-community oQ3e and oQ4e builds, and oQ writes that layout (#4604).
• GLM-5.3-Flash: ModelOpt NVFP4 checkpoints load with their per-expert scales, with or without expert offload (@junmo-kim in #4661, 7a92826).
• Downloads: each row shows the time left from the one-minute average rate (c0d8d6d), and progress is steadier with fewer false stalls on slow links (#4638).
• API: responses carry the engine request id in X-oMLX-Request-ID (#4046).
• Web UI: a number scale in Settings > Appearance for exact digits or compact units with local 万/億/억 units (@LXD-8 in #4633, #4652), light theme surfaces on one neutral gray (@LXD-8 in #4564, #4565), and Chinese for the remaining English strings (@LXD-8 in #4457).

Bug Fixes

• The memory guard no longer throttles prefill into small chunks while memory is free (#4603).
• A lone short prompt keeps one-call admission, cutting time to first text by 26% in a mixed long/short test (@samfenwick in #4595), and the generation-prompt tail runs in the same prefill step (#4503).
• Qwen ANE prefill keeps the loaded model close to its size, 46.75 -> 28.93 GB on the reported model (#4657).
• fp16 SDPA with head dim 256 stays under the M1/M2 thread limit (#4609).
• Qwen3.8-Flash-Next caches grow on the capacity ladder instead of copying the whole bank each step, and extracted rows no longer double (@yoyo930021 in #3963, #4030).
• Tip strip and SpecPrefill draft blocks stay out of the hot cache under memory pressure (7372542).
• DeepSeek V4.1 unload finishes cleanup after a resource error (@rubencu in #4667), and offloaded experts stay on the unfused decode path (cde10a5).
• Streaming qwen3_coder tool calls whose values contain </function> no longer leak into content (@richgoodson in #4654), union-typed XML parameters accept True/False (@yanzhaohui1999 in #4646, ceb1430), and gpt-oss structured output uses the Harmony grammar and stops after one answer (#4656).
• Claude Code's <total_tokens> budget notes are stripped on the chat and Responses paths (@yanzhaohui1999 in #4627).
• Images are downscaled before EXIF orientation and RGB conversion, so a tiny PNG declaring 12000x12000 pixels no longer costs about 1 GiB (@pierre427 in #3885).
• Muse Glimmer keeps its vendored module when another VLM loads first (#4599).
• Web chat keeps the streamed reply when the connection drops (@yanzhaohui1999 in #4645) and exports the chats on screen instead of a stale snapshot (@PatrickSun93 in #4171). Model Settings saves only the fields you edited, so a save no longer overwrites other changes or unloads the model (#3089).
• Embedding models show their context window in settings (#4622), and embedding and rerank usage count tokens after truncation (fdcdc83).
• macOS app: Top K 0 is accepted as off (@HamedShams in #4660), and the menubar All-Time section shows the no-API-key hint (914cf54).
• The Homebrew custom-kernel guard runs the Metal compiler instead of only finding it (@faiyaz7283 in #4643).
• Other: hot-cache flags pass straight through (@l3rett in #4676), and the pi coding agent hint uses its new npm package (@bzz in #4664).

Highlights from the 0.7.1 Release Candidate and Development Releases

• Muse Spark Open. Meta's 816B MoE runs with text, image and tool calls (#4573).
• Models for your Mac. The Recommended tab shows only the oQ models your Mac can run (#4468).
• Splash as an external engine. Qwen3.8 and Qwen3.6 run on Splash with DFlash2 while staying in oMLX's model list, API and dashboard (#4447).
• DFlash in the batched engines. DFlash requests get continuous batching and the prefix cache, and dflash-mlx is no longer a dependency (#4508).
• Decision models and batched prefill. /v1/systemone and /v1/decisions (#4315, #4436), and concurrent prompts prefill in one forward (#4361).
• Reworked settings and chat. Sectioned Model Settings in the web UI and the macOS app, and a chat composer with a reasoning slider (#4415, #4530, #4446).

If you are upgrading from 0.7.0, also read the upgrade notes in 0.7.1rc1: MLX 0.32.4, removed OMLX_* debug variables, removed DFlash settings and new sampling defaults. Benchmarks and every contributor credit are in the 0.7.1rc1, 0.7.1.dev2 and 0.7.1.dev1 release notes.

Upgrade Notes

• Source installs: run make install (or make dev) after updating. Custom kernels moved from per-model packages to attention, sparse_index, quant_matmul and lowbit (#4601), and the old per-model folders under omlx/custom_kernels/ can be deleted.
• API key lockout: clients on other machines are locked out for 30 s after 8 wrong API keys and for 300 s after each further 8, with HTTP 429 and Retry-After. Clients on the same Mac are not limited (#4669, with @billythegod from #4166).
• Splash: Splash 1.3.1 and later no longer load prebuilt incoai/*-Splash packages. The downloader refuses them and names the MLX model with the same weights. NVFP4 and FP8 checkpoints need Splash 1.3.2 (#4679).
• Lightning MTP on Qwen3.8-Flash-Next: without a saved depth setting it now runs one draft per cycle. Set an adaptive max depth to get the adaptive controller back (#4648).

New Contributors

@oddmundleikvoll in #4210, @pierre427 in #3885, @PatrickSun93 in #4171, @bzz in #4664, @rubencu in #4667, @giuseppebonanno in #3592, @faiyaz7283 in #4643, @l3rett in #4676.

Changes since 0.7.1rc1 | Full changelog from 0.7.0