oMLX 0.6.0rc1

This is the first release candidate for oMLX 0.6.0. It rolls up every change since 0.5.7, including the 0.5.8 development releases and 0.6.0.dev1.

This release introduces experimental distributed serving, keeps active generation responsive during concurrent prefills, substantially improves long-context cache and memory behavior, adds several new model families, and expands the admin chat with built-in web search and speech-to-text.

Distributed Serving

• Added experimental distributed serving across Macs. Models can be split with tensor or pipeline parallelism, with capability-aware planning, selective shard staging, per-rank memory guards, liveness supervision, and one-click activation from the Cluster dashboard. Qwen3.6-27B reached 28.6 tok/s across two Macs versus 16.1 tok/s on one Mac with byte-identical output. A 225 GB MiniMax-M3 checkpoint that could not fit either machine alone also loaded across 128 GB and 256 GB Macs and passed its distributed canary. By @ashhart in #2423.

• Added heterogeneous Metal and CUDA model pools. Apple Silicon and NVIDIA CUDA workers can contribute to one logical model-memory pool, with memory-aware contiguous-layer placement, short-lived worker enrollment, ConnectX discovery, and NCCL verification. The current compatibility path keeps physical workers in the outer MLX Ring; the hierarchical Ring-to-NCCL gateway remains future work. By @ashhart in #2591.

• Added SSD-backed prompt reuse for distributed ranks. Each rank stores a process-lifetime chain of cache-boundary snapshots and restores only prefixes available on every rank. Incremental segments keep storage linear: a 12K-token GLM-5.2 chain used 1.17 GB instead of roughly 6.5 GB with cumulative copies. #2620.

Distributed serving is included in the DMG but remains experimental and disabled by default. Enable it from Web Dashboard > Global Settings > Advanced > Distributed Inference, then save the settings and restart oMLX.

Serving Responsiveness and Long-Context Memory

• Active decodes now remain responsive during concurrent prefill. Prefill yields GPU time to running decodes, including streams in another engine, and sizes contended chunks by a target stall time. The feature is enabled by default and can be changed live. Across the tested workloads, overlap decode improved by about 1.6x to 43x; contended prefill took 1.15–2.0x longer while solo prefill remained unchanged. #2633.

• Mixed CacheList prefix storage is now linear instead of quadratic. Sliceable KV members are stored per block while small boundary states remain attached to their checkpoints. On the affected Inkling workload, the legacy path used 282.7 GB for an 84K-token session; the new layout used 15.3 GB for 94K tokens and restored byte-identical output. Incompatible legacy blocks are invalidated automatically on first load. By @Pietroski in #2550.

• GDN recurrent state now uses bounded SSD sidecars by default when the SSD cache is available. The new auto | ssd_sidecar | embedded policy separates recurrent state from ordinary KV storage. RHT-INT16 is the new default sidecar codec, measuring 1.93x less storage than FP32 with 0.002477% mean relative L2 error; restored state still runs in FP32. FP32, BF16, INT8, and RHT-INT8 remain selectable. Existing explicit legacy settings and FP32 deployments are preserved. By @hojin12312 in #2569 and #2644.

• Memory pressure now reclaims the MLX buffer pool before shrinking the hot cache. Reclaimable Metal buffers no longer cause reusable prompt state to be repeatedly discarded. Emergency pressure still enforces immediately, while ordinary hard pressure receives a bounded reclaim grace period. Reported in #2581, fixed by @PhilipJohnBasile in #2635.

• Adaptive prefill admission now accounts for recently reclaimed memory, waits briefly for asynchronous cache cleanup before rejecting long requests, and uses DeepSeek V4’s actual hybrid cache layout instead of the generic KV estimate. Hot-cache promotion failures are also exposed through runtime cache statistics.

Model Support and Performance

• Added Qwen3.8-27B FP8 support and official reasoning levels. Blockwise FP8 checkpoints and embedded MTP tensors load through the Qwen3.5-family compatibility path. reasoning_effort in Chat Completions and reasoning.effort in Responses now reach the model without overriding explicit template arguments. By @scaryrawr in #2653.

• Restored Qwen3.8/Qwen3.5 group-size-128 prefill performance on NAX hardware. Common four-bit checkpoints no longer route through a slower classic Metal fallback when stock MLX can use NAX. On the reported Qwen3.8-27B workload, prompt throughput improved from 513 to 925 tok/s and end-to-end time fell from 10.4 to 6.8 seconds. Pre-NAX Macs retain the existing optimized path. By @True2456 in #2657.

• Added Ling 3.0 Flash support. This includes FP8, official mixed FP4/FP8 checkpoints, oQ/oQe conversion, variable-length batching, and SSD prefix-cache restores. Applying the trained per-layer SwiGLU clamp improved measured HumanEval accuracy from 71.3% to 88.4%. Original implementation and FP4 support by @scaryrawr in #2526, #2534, and #2571.

• Added Meta Muse Glimmer 30B VLM support with DFlash speculative decoding. Text, vision, channel-scoped reasoning, ATEM tool calls, prefix/SSD cache, and oQ quantization are supported. DFlash reached about 1.4x decode on the tested oQ4 pair; image requests continue through the VLM fallback. The vendored implementation was also synchronized with upstream reasoning and FP32 numerical behavior. #2586, #2587, and #2643.

• Added complete Jina Reranker v3.5 support. Projector loading, sliding-window attention, dual matching, block fusion, and reference-score parity are included. By @damascoemi in #2449 and #2494.

• DeepSeek V4 prefill is 23–28% faster across tested context lengths. Native attention routing, 2048-token cache blocks, fused windowed and pooled attention, skipped intermediate lm_head work, in-place pooled-cache appends, and a fused indexer mask improve prompt throughput without changing decode speed. By @DiscoStew6082 and @jonathan308 in #2559, #2562, #2563, and #2568.

• DeepSeek V4 unaligned final prefill tails stay on the native indexer path. A 347,929-token validation reduced full-model MLX peak memory from 142.255 GiB to 105.059 GiB by avoiding a large FP32 fallback workspace. The fix handles partial Metal tiles directly rather than globally splitting scheduler chunks. Reported in #2627, fixed in #2674.

• Added MTPLX sidecar MTP import for compatible Qwen3.6 checkpoints, fixed DeepSeek V4 affine AWQ expert bias/zero-point loading, and removed a per-token host synchronization from grammar-constrained decoding.

Chat, APIs, Reasoning, and Tool Calling

• Added built-in web search to admin chat. Models can use web_search and fetch_url with keyless DDGS metasearch or optional DuckDuckGo, Brave, and SearXNG providers. Results remain visible as source cards across tool rounds. URL fetching rechecks private and loopback destinations after every redirect and caps content before conversion. External API clients are unchanged because only the chat UI advertises these tools. #2596.

• Added file and realtime speech-to-text to chat. Audio STT models switch the composer into transcription mode, while compatible Whisper and Voxtral models expose live microphone input over WebSocket. Whisper file streaming now uses bounded segments with the full anti-hallucination path and per-segment language detection. Qwen3-ASR uses cumulative decoding so multi-token CJK characters remain intact. #2603.

• Fixed streaming Responses reasoning for prompts that already open <think>. DeepSeek V4 reasoning now streams through dedicated reasoning events, creates a separate reasoning item, and reports reasoning token usage instead of leaking thought text through response.output_text.delta. Confirmed with a real checkpoint on Apple Silicon. Reported in #2584, fixed by @leepokai in #2623.

• Hardened tool-call parsing. Tool arguments containing literal close markers are preserved, deeply nested model output now fails closed instead of escaping as RecursionError or SyntaxError, and a malformed call no longer drops a valid neighboring call. #2544, #2593.

• Fixed VLM and Muse Glimmer tool-call paths. VLM requests now carry tool schemas through to protocol parser sessions, preserving JSON-looking values declared as strings. Muse Glimmer also keeps tool calls when a turn opens directly with one instead of returning an empty response. Reported in #2646, fixed in #2645 and #2599.

• Thinking budgets now count only accepted/emitted tokens during both VLM and DSpark MTP speculative decoding. Gemma 4 no longer leaks reasoning into visible content after tool responses, and Ling and Inkling tool protocol variants are parsed without marker leakage.

macOS, UI, and Integrations

• Reworked the macOS self-updater to make bundle replacement crash-safe. A one-shot launchd worker now performs an atomic bundle exchange, so interruption cannot leave /Applications/oMLX.app missing its executable or signature. This addresses the “damaged or incomplete” failures reported on macOS 27 in #2522.

• Important for affected macOS 27 users: install 0.6.0rc1 manually from the DMG once. An older build still runs its old updater while installing this release; the crash-safe updater takes effect for subsequent updates after 0.6.0rc1 is installed.

• Added an Enhanced Readability option for the web admin UI, raising low-contrast helper text and enforcing a 12px minimum without changing KaTeX sizing. By @LXD-8 in #2542.

• Fixed duplicate Dock instances when multiple oMLX app bundles are registered, and changed “All networks” mode to a dual-stack IPv6 wildcard while retaining an explicit IPv4-only option.

• omlx launch codex no longer modifies persistent Codex configuration. omlx launch claude --cross-session now enables Claude Code cross-session messaging explicitly while keeping the privacy-oriented default and unrelated service opt-outs intact. #2519, #2601.

• Hugging Face cache checkpoints now appear as oQ quantization sources under their canonical repository identity, and /v1/models/status exposes favorite and hidden state for external model-management clients.

Upgrade Notes

• This is a release candidate. Please report regressions before the final 0.6.0 release.
• Distributed inference remains experimental and is disabled by default. It is included in the DMG and can be enabled from Web Dashboard > Global Settings > Advanced > Distributed Inference. Save the settings and restart oMLX after enabling it.
• Decode fairness is enabled by default. Set decode_fairness=false to restore the previous concurrent-prefill behavior.
• GDN snapshot storage defaults to auto with RHT-INT16 for configurations without an explicit legacy choice. Existing explicit split/embedded and FP32 settings are preserved.
• Legacy mixed-CacheList SSD blocks are invalidated automatically because their storage layout is incompatible with the new linear format.
• DeepSeek V4 temperature-0 output can differ from earlier builds because the new kernels change floating-point reduction order. The differences occur at near-tied token choices and are not considered a quality regression.
• No manual settings migration is otherwise required.

New Contributors

Thank you to @Maxim-Mazurok, @damascoemi, @jetztmalehrlich, @Pietroski, @studioburnside, @tannerdsilva, @LXD-8, @luziyi, @DiscoStew6082, @fxd0h, @mvdbos, @ashhart, @PhilipJohnBasile, @mkmsyk, @nazerim, @leepokai, and @dawgflymd for their first contributions in this release line.

Full Changelog: v0.5.7...v0.6.0rc1