jundot/omlx v0.5.2rc2

0.5.2rc2

on GitHub

This release candidate adds Bonsai 1-bit / 2-bit decode kernels, Nemotron-H multi-token prediction, faster Hugging Face downloads with working cancellation, and more TTS output formats, plus an MLX kernel bug fix for M5 machines and a batch of memory and stability fixes for long-running servers.

Highlights

• Bonsai 1-bit / 2-bit decode kernels. Dedicated Metal decode kernels for extreme low-bit Bonsai models, upstreamed from the Prism MLX fork. By @0xClandestine in #2248
• Nemotron-H multi-token prediction. MTP speculative decoding now supports nemotron_h models, covering both the depth-1 cycle and the Lightning MTP chain, along with a fix for models whose hidden output is already post-norm. By @davidtorcivia in #2277 and #2278
• Faster model downloads. Xet-backed downloads are enabled again. They had been forced off since #1322 because cancellation could not interrupt the transfer; huggingface_hub 1.19.0's session abort now stops them immediately, so cancel, stall recovery, and shutdown all work on the fast path.
• More TTS output formats. Non-streaming /v1/audio/speech now returns mp3, opus, flac, and pcm in addition to wav, and unsupported formats get a clean 400 instead of a broken file. Streaming stays wav-only. By @JimStenstrom in #2231, refs #753 and #1013
• Chat history import and export. The chat UI gains a settings modal, chat history download, and import of previously exported chats. By @cobracode in #2279
• M5 MLX kernel bug fix. On M5 machines, some quantized MoE matmuls hit a defective GPU kernel and returned wrong results. oMLX now detects the defect at startup with a quick self-test and reroutes affected calls to known-good kernels, so healthy machines and future fixed mlx builds keep the full fast path. Reported by @richgoodson in #2267
• oQe imatrix quantization controls in the macOS app. By @jknlsn in #2308

Bug Fixes

• MoE fusion no longer spikes memory on load. The gate+up fusion freed the original expert weights into the pinned buffer pool, growing the load transient by roughly 2/3 of the expert bytes (~40 GB on a 122B model) and pushing idle weights into the compressor. The pool is now drained after each fused layer. Reported by @raucodes in #2304
• Prefix index no longer leaks on long uptimes. Prefix index entries are now dropped together with their cache blocks, index hits are re-validated block by block before reuse, and aborted requests trigger the same deferred Metal pool clear as normal completions.
• The prefill guard self-recovers from a pooled-buffer wedge, instead of latching the server into a state where every request queues forever. By @JimStenstrom in #2306, fixes #2179
• Model re-discovery no longer orphans in-flight loads. A discovery pass racing an in-flight model load could leave the loaded engine unreachable with its memory unreclaimable until restart. Reported by @guruswami-ai in #2307
• SpecPrefill parameters now apply on the non-streaming path; specprefill_keep_pct was silently ignored. By @Khambampati-Subhash in #2292, reported by @necoli1822 in #2274
• /v1/responses now passes chat template kwargs through. By @cloud-of-equality in #2287
• python -m omlx.server now resolves a working admin settings path. By @richgoodson in #2265, asked by @Khambampati-Subhash in #2282
• External API intelligence benchmarks no longer fail on thinking models; the preflight token budget was too small for reasoning output. Reported by @zviratko in #2309
• Cleared profile fields (context window, max tokens, kwargs) now actually reset on apply instead of silently keeping their old values.
• Fixed a chat theme flash on load and a missing translation on the download button.

Upgrade Notes

• pip installs now require huggingface-hub >= 1.19.0 for the xet session abort API. The bundled macOS app already ships it.

New Contributors

Thank you to @0xClandestine, @davidtorcivia, @cobracode, @cloud-of-equality, and @jknlsn for their first contributions to oMLX in this release.