oMLX is the most convenient way to run MLX models on your Mac, and the fastest way to run them, with custom Metal kernels for GLM-5.2, MiniMax M3, DeepSeek V4, and Qwen3.5/3.6.
oMLX 0.5.2 adds a full statistics layer for the macOS menu bar, Bonsai 1-bit / 2-bit decode kernels, Nemotron-H multi-token prediction, and Lightning MTP recovery under continuous batching, along with a large batch of cache, memory, and stability fixes for long-running servers.
Highlights
- Menu bar statistics. New opt-in status items show live, session-average, and all-time inference throughput (PP and TG in tk/s), plus CPU/GPU/MEM usage bars that merge into a single compact item. Each opens a popover with current values and rolling graphs, and a System Stats submenu adds CPU, GPU, and Memory panels with E/P core usage, load average, uptime, thermal state, GPU utilization, and a memory breakdown. Everything is configured from the new Appearance pane, and host sampling only runs while something displays it. (#2224, #2303)
- Reorganized Models menu. The menubar Models menu now groups Loaded, Favorites, and Library sections with load state and sizes, and the Appearance pane can limit the list to favorites only. By @faisal in #1882
- Bonsai 1-bit / 2-bit decode kernels. Dedicated Metal decode kernels for extreme low-bit Bonsai models, upstreamed from the Prism MLX fork. By @0xClandestine in #2248
- Nemotron-H multi-token prediction. MTP speculative decoding now supports nemotron_h models, covering both the depth-1 cycle and the Lightning MTP chain. By @davidtorcivia in #2277 and #2278
- Lightning MTP recovers under continuous batching. MTP now resumes after a continuous batch shrinks to one request, instead of leaving the surviving request on slower standard decode. By @gilby in #2166
- Faster model downloads. Xet-backed downloads are enabled again. They had been forced off since #1322 because cancellation could not interrupt the transfer; huggingface_hub 1.19.0's session abort now stops them immediately, so cancel, stall recovery, and shutdown all work on the fast path.
- More TTS output formats. Non-streaming
/v1/audio/speechnow returns mp3, opus, flac, and pcm in addition to wav, and unsupported formats get a clean 400 instead of a broken file. Streaming stays wav-only. By @JimStenstrom in #2231, refs #753 and #1013 - M5 MLX kernel bug fix. On M5 machines, some quantized MoE matmuls hit a defective GPU kernel and returned wrong results. oMLX now detects the defect at startup with a quick self-test and reroutes affected calls to known-good kernels, so healthy machines and future fixed mlx builds keep the full fast path. Reported by @richgoodson in #2267
- Torch-free Mistral Small 4 / Devstral loading. The bundled runtime now loads Pixtral and Mistral3 checkpoints without torch, which also fixes Devstral tool calls returning empty messages. Reported by @raucodes in #2263 and @jomei64 in #2291
- Chat history import and export. The chat UI gains a settings modal, chat history download, and import of previously exported chats. By @cobracode in #2279
- oQe imatrix quantization controls in the macOS app. By @jknlsn in #2308
- Faster Qwen3.5/3.6 MoE. The gate and up expert projections are fused into a single gather_qmm across batched and VLM engines. Based on profiling by @builtbycnob in #2238
Changes Since 0.5.2rc2
- Fixed TTS going silent after hours of uptime. mlx-audio sampling ran through a compiled sampler whose RNG state could get stuck, producing identical or empty audio until a full restart; TTS now uses the compile-free omlx samplers. Reported by @raucodes in #2312
- Fixed SpecPrefill on partial-rotary models. The draft scoring path now derives its RoPE width from the actual frequencies instead of head_dim, with zero-padded inv_freq. By @rockminster in #2295
- The engine pool settle barrier now counts in-flight model loads as concurrent activity, and the unresolvable-scheduler memory warning no longer fires for non-streaming engines.
Bug Fixes
Inference and caching
- Fixed decode wedges after request cancellation or batch mutation by keeping lazy batch KV cache operations on the owning engine's Metal stream. Reported by @anicaise-ai in #2235
- Fixed continuous-batching decode failures caused by TurboQuant and dense cache layers diverging after left-padding compaction. Reported by @zviratko in #2237
- Fixed FA-256 prefill slowdowns and GPU preemption failures at long context on M1 through M4 hardware with automatically bounded Metal dispatches. Reported by @zhongyiyang1981 in #2225
- Fixed mixed TurboQuant and dense prefix-cache reconstruction: blocks are restored by their actual payload format, ambiguous or corrupt hits are re-prefilled, and healed chains are cast back to the dense dtype. By @JimStenstrom in #2272
- Fixed DeepSeek V4 Flash ratio-4 overlap compression during decode, preserving previous-window state per batch row and preventing long-generation output drift. By @williamxie1989 in #2271
- Fixed SpecPrefill RoPE wrappers surviving aborted requests or partial cache hits, which could disable sparse prefill or shift later cache positions. By @necoli1822 in #2273
- SpecPrefill parameters now apply on the non-streaming path;
specprefill_keep_pctwas silently ignored. By @Khambampati-Subhash in #2292, reported by @necoli1822 in #2274 - Fixed non-deterministic GLM-5.2 DSA top-k selection when scores tie at the threshold. By @aidiffuser in #2243
- Fixed a failed DFlash arm leaving class-level hooks installed, which corrupted concurrent batched decodes on models sharing attention classes. Reported by @theuseless-ai in #2252
- Fixed the DFlash primary prefill guard double-counting serialized hot-cache memory and rejecting valid long-context requests. By @JimStenstrom in #2229
- Fixed cache-probe results for models with persisted or dynamic chat-template settings such as thinking toggles. By @thornad in #2256
Memory and long-uptime stability
- MoE fusion no longer spikes memory on load. The gate+up fusion freed the original expert weights into the pinned buffer pool, growing the load transient by roughly 2/3 of the expert bytes (~40 GB on a 122B model). The pool is now drained after each fused layer. Reported by @raucodes in #2304
- Prefix index no longer leaks on long uptimes. Prefix index entries are dropped together with their cache blocks, index hits are re-validated block by block before reuse, and aborted requests trigger the same deferred Metal pool clear as normal completions.
- The prefill guard self-recovers from a pooled-buffer wedge, instead of latching the server into a state where every request queues forever. By @JimStenstrom in #2306, fixes #2179
- Fixed model re-discovery orphaning in-flight loads, which left the loaded engine unreachable with its memory unreclaimable until restart. Reported by @guruswami-ai in #2307
- Fixed model swapping with the memory guard disabled: loading a second model now evicts instead of wedging the server. Reported by @raucodes in #2290
- Fixed engine leases accumulating after client disconnects until the server wedged. Reported by @jzupnick in #2260
- Fixed resident KV tensors accumulating across multi-turn SSD cache traffic by removing closure reference cycles from nstate serialization. By @bitfieldz in #2258
Model loading and quantization
- Fixed custom kernels silently falling back to slow paths on Homebrew installs due to an unresolved @rpath/libmlx.dylib dependency. Reported by @zhongyiyang1981 in #2233
- Made oQ/oQe calibration subsampling deterministic, so repeated runs of the same recipe produce identical artifacts. Reported by @zviratko in #2293
- Fixed the VLM engine silently ignoring OptiQ vision sidecars on gemma4_unified. Reported by @donkuk in #2298
- Fixed oQe calibration on models that need the compatibility loader. By @seeker-cyber-maker in #2299
API and admin
/v1/responsesnow passes chat template kwargs through. By @cloud-of-equality in #2287python -m omlx.servernow resolves a working admin settings path and no longer passes stale arguments to server initialization. By @richgoodson in #2265 and @Khambampati-Subhash in #2242- Fixed prefill throughput metrics for non-streaming OpenAI, Anthropic, Responses API, DFlash, and multi-prompt completion paths. By @Khambampati-Subhash in #2240
- External benchmarks now support provider-specific extra body parameters, distinguish endpoint failures from wrong answers, and no longer fail on thinking models due to a too-small preflight token budget. By @earthking11 in #2296 and #2297, reported by @zviratko in #2309
- Fixed accuracy benchmark queue cancellation allowing stale queue chains to interrupt a newly started benchmark. By @JimStenstrom in #2230
- Fixed the desktop UI ignoring a manual "embedding" model type override. Reported by @mveroone in #2270
macOS app and web chat
- Cleared profile fields (context window, max tokens, kwargs) now actually reset on apply instead of silently keeping their old values.
- Fixed a chat theme flash on load and a missing translation on the download button.
- Fixed highlighted text visibility in the web chat dark theme. By @HaloFour in #2222
Downloads
- Fixed Hugging Face download progress when the dry-run size probe fails, with safetensors-based estimation during download and measured size on completion. By @JimStenstrom in #2266
Upgrade Notes
- pip installs now require huggingface-hub >= 1.19.0 for the xet session abort API. The bundled macOS app already ships it.
- If you had "Show Live Activity in Menu Bar" enabled, it migrates automatically to the new LIV status item; the old inline text next to the menu bar icon is gone. All menu bar items are configured in Settings under Appearance.
New Contributors
Thank you to @HaloFour, @Khambampati-Subhash, @necoli1822, @williamxie1989, @bitfieldz, @faisal, @earthking11, @seeker-cyber-maker, @0xClandestine, @davidtorcivia, @cobracode, @cloud-of-equality, @jknlsn, and @rockminster for their first contributions to oMLX in this release.