jundot/omlx v0.5.2.dev1

0.5.2.dev1

on GitHub

I have limited availability this week due to scheduled leave, so responses may be delayed. Thank you for your patience. I will follow up on recent issue reports early next week.

This development release restores Lightning MTP acceleration after continuous batches shrink back to one request, adds live inference activity to the macOS menu bar, and improves TurboQuant KV cache correctness, prefix-cache recovery, and long-context kernel reliability, with targeted metrics, downloader, and admin fixes after 0.5.1.

Highlights

• Lightning MTP performance recovery. MTP now resumes after a continuous batch shrinks to one request, instead of leaving the surviving request on slower standard decode. By @gilby in #2166
• TurboQuant KV cache correctness. TurboQuant and dense caches now stay aligned when batch rows leave, preventing decode shape mismatches. Reported by @zviratko in #2237
• Safer prefix-cache recovery. Mixed TurboQuant and dense blocks are restored by their actual payload format, while ambiguous or corrupt hits are re-prefilled. By @JimStenstrom in #2272
• Long-context kernel improvements. FA-256 prefill uses bounded Metal dispatches, Qwen3.5/3.6 MoE fuses gate/up projections, and GLM-5.2 top-k selection is deterministic. (#2225, #2238, #2243)
• Live inference activity in the macOS menu bar. The opt-in display shows prefill, generation speed, token count, and queued requests. By @aosama in #2224

New Features

• Added opt-in live inference activity to the macOS menu bar. By @aosama in #2224
• Added fused Qwen3.5/3.6 MoE gate/up expert projections across batched and VLM engines. Based on profiling by @builtbycnob in #2238
• Added Lightning MTP recovery after continuous batches shrink to one request. By @gilby in #2166

Bug Fixes

• Fixed Lightning MTP remaining on standard decode after concurrent requests completed, allowing the surviving request to recover MTP acceleration. By @gilby in #2166
• Fixed decode wedges after request cancellation or batch mutation by keeping lazy batch KV cache operations on the owning engine's Metal stream. Reported by @anicaise-ai in #2235
• Fixed continuous-batching decode failures caused by TurboQuant and dense cache layers diverging after left-padding compaction. Reported by @zviratko in #2237
• Fixed FA-256 prefill slowdowns and GPU preemption failures at long context on M1 through M4 hardware with automatically bounded Metal dispatches. Reported by @zhongyiyang1981 in #2225
• Fixed mixed TurboQuant and dense prefix-cache reconstruction, wrong-width codec guesses, corrupt metadata handling, rotating-cache geometry ambiguity, and dtype promotion during chain healing. By @JimStenstrom in #2272
• Fixed resident KV tensors accumulating across multi-turn SSD cache traffic by removing recursive closure reference cycles from nstate serialization. By @Spirotot in #2258
• Fixed DeepSeek V4 Flash ratio-4 overlap compression during decode, preserving previous-window state per batch row and preventing long-generation output drift. By @williamxie1989 in #2271
• Fixed SpecPrefill RoPE wrappers surviving aborted requests or partial cache hits, which could disable sparse prefill or shift later cache positions. By @necoli1822 in #2273
• Fixed non-deterministic GLM-5.2 DSA top-k selection when scores tie at the threshold. By @aidiffuser in #2243
• Fixed the DFlash primary prefill guard double-counting serialized hot-cache memory and rejecting valid long-context requests. By @JimStenstrom in #2229
• Fixed cache-probe results for models with persisted or dynamic chat-template settings such as thinking toggles. By @technosophics in #2256
• Fixed Hugging Face download progress when the dry-run size probe fails, with safetensors-based estimation during download and measured size on completion. By @JimStenstrom in #2266
• Fixed accuracy benchmark queue cancellation allowing stale queue chains to interrupt a newly started benchmark. By @JimStenstrom in #2230
• Fixed prefill throughput metrics for non-streaming OpenAI, Anthropic, Responses API, DFlash, and multi-prompt completion paths. By @Khambampati-Subhash in #2240
• Fixed the python -m omlx.server entry point passing stale or mismatched arguments to server initialization. By @Khambampati-Subhash in #2242
• Fixed highlighted text visibility in the web chat dark theme. By @HaloFour in #2222

New Contributors

Thank you to @williamxie1989, @necoli1822, @Khambampati-Subhash, @Spirotot, and @HaloFour for their first contributions to oMLX in this release.

Full Changelog: v0.5.1...v0.5.2.dev1