jundot/omlx v0.5.4.dev1

0.5.4.dev1

on GitHub

This development release adds a context benchmark that measures the usable context window on the current Mac, makes long-context memory admission more predictable, adds MiMo V2.5 text support and Laguna S-2.1 DFlash acceleration, and overhauls the web chat experience. It also makes throughput benchmarks more representative and includes a broad set of inference, quantization, integration, and stability fixes since 0.5.3.

Highlights

• Context benchmark. Measures the largest context the current Mac can prefill, verifies it with a real request, and applies the result to model settings. Available through the admin API, web dashboard, and macOS app. (#2390, #2391)
• Prefill priority controls. Choose max context to shrink prefill chunks for larger prompts or speed to keep full-size chunks for maximum throughput. The setting applies live across the web dashboard and macOS app.
• Safer long-context memory admission. Admission accounts for KV layout and transient memory before work starts. Oversized prompts fail immediately with a typed 400, while reclaimable MLX buffers are drained before aborting an active request. (#2390)
• Laguna S-2.1 DFlash support. Poolside Laguna targets and official drafts now support adaptive speculative decoding, strict compatibility checks, rollback, mixed caches, and prefix-cache round trips. The contributed benchmark measured 2.03x speedup with exact greedy parity. By @FabioMalpezzi in #2392
• MiMo V2.5 text support. Xiaomi MiMo V2.5 checkpoints now load through the native text engine, with model discovery and oQ integration. Closes #1486
• Overhauled web chat. Adds model selection, context tracking, chat search, configurable image limits, better scrolling, keyboard shortcut help, and expanded chat settings. By @PowerSpy in #2379
• More representative throughput benchmarks. Benchmarks now use natural text and enforce exact prompt lengths. Speculative-decoding results may be lower than older runs but better reflect real workloads. By @JimStenstrom in #2343, reported by @shengpeng7 in #2254
• Faster heterogeneous embedding batches. Inputs are grouped by length and restored to caller order, avoiding excessive padding. Measured throughput improved by roughly 4x to 4.6x across text and VLM embedding models. By @1qh in #2370

New Features

• Added a global prefill priority setting. max context dynamically shrinks prefill chunks to maximize usable context, while speed requires full-size chunks for maximum throughput. The setting is available in the web dashboard and macOS app.
• Added --memory-guard off. Selecting a tier or custom ceiling now explicitly enables the guard instead of silently doing nothing when it was disabled in settings.
• Added MXFP8 source-model support to oQ quantization.
• VLM checkpoints can now be admitted to the text engine using the language-model weight size instead of charging the unused vision tower against memory admission.
• Added generate, regenerate, and copy controls to macOS sub-key creation. By @tobomobo in #2348
• Added first-use diagnostics for the SDPA-256 tiled route and large prefix-cache divergence points.
• Claude Code launches now disable the LSP tool to keep agentic prompt prefixes stable for prefix-cache reuse.

Performance

• Faster GLM-5.2 long-context decode for single-stream workloads. With the native custom kernel active, non-MTP decode now uses a fused DSA indexer scan after 4k cached tokens. On an M3 Ultra, end-to-end GLM-5.2 decode improved by 1.7% at 30k context and 3.1% at 125k. MTP verification and continuous batches keep the existing path. By @aidiffuser in #2245
• Throttled prefill chunks now align to reusable minimum-chunk multiples, improving measured prefill throughput by 6.8% on Gemma 4 and 8.4% on Qwen3.6 under tight memory ceilings while remaining neutral on roomy systems. (#2390)

Bug Fixes

Inference and caching

• Fixed hard-memory-pressure recovery livelocking on continuously busy servers by draining reclaimable MLX buffers at the scheduler step boundary. By @cebertowicz in #2342
• Fixed external VLM MTP being skipped when a request completed through chunked prefill. By @ethannortharc in #2361
• Fixed mRoPE positions remaining constant across multi-token speculative verification windows, which could commit incorrectly rotated keys into the VLM cache. By @gilbert-barajas in #2363
• Fixed MiniCPM-o text-only requests failing when the mRoPE fallback path had no initialized position state. Reported by @goldyard2022 in #2387
• Fixed prefix-cache reconstruction running outside the owning engine's Metal stream, which could wedge concurrent fully cached prefills. Reported by @anicaise-ai in #2330
• Fixed SDPA patches capturing an old attention function before TurboQuant was installed, causing TurboQuant prefills to fail or hang.
• Fixed the SDPA-256 patch not rebinding mlx-vlm submodules and restored the unfused fast path when the memory guard is disabled.
• Fixed multi-token stop-sequence prefixes leaking into completion output when a stop sequence spanned multiple generated tokens. Reported by @Xcoder1990 in #2386
• Fixed SpecPrefill request overrides being ignored by the VLM non-streaming path. By @JimStenstrom in #2338
• Fixed frequency_penalty being ignored by VLM-hosted autoregressive models. By @JimStenstrom in #2339
• Fixed failed context benchmark probes leaving a stale transient-memory estimate that could make every retry fail admission immediately.
• Fixed empty thinking blocks appearing in web chat responses.

Model loading and quantization

• Fixed quantized Laguna checkpoints with legacy router names failing to load because router scales, biases, and per-layer quantization overrides were not remapped. By @layer4down in #2337
• Fixed current DFlash draft checkpoints storing RoPE and DFlash configuration under Transformers 5.x nested keys. By @JimStenstrom in #2340
• Fixed mistral-common models selecting a tokenizer backend that could drop control tokens during prompt construction. By @JimStenstrom in #2341
• Fixed MiniMax M3 oQ conversion dropping unpacked shared experts. Reported by @avlp12 in #2356
• Fixed oQ calibration admission using process memory without accounting for the active Metal ceiling.
• Fixed GLM-5.2 DSA indexers being quantized below Q8, producing checkpoints that could not load. Reported by @noonkho in #2347
• Fixed Qwen3-VL embedding and rerank models failing on image inputs after the torch-free image processor replacement.
• Fixed empty vision configuration stubs being misclassified as usable vision towers during model discovery.
• Fixed the Homebrew custom-kernel build verifying only three of the four bundled kernels; Bonsai is now checked as well. Reported by @goldyard2022 in #2388

API, benchmarks, and integrations

• Fixed XML tool-call fallbacks coercing array and object parameters without consulting the tool schema, and fixed JSON-quoted string values retaining their surrounding quotes. Reported by @bobacus in #2332
• Fixed external benchmark streams recognizing only reasoning_content; reasoning, analysis, and responses without observable content timing now produce correct metrics or N/A instead of zero or multi-million token-per-second results. By @fparrav in #2375
• Fixed the dashboard showing a different memory reserve than the server on 16 GB to 23 GB Macs.
• Memory rejection messages now identify the actual binding ceiling and the setting that can raise it.
• Fixed the Codex integration reporting the renamed ChatGPT desktop app as missing and added fallback discovery of its bundled CLI. Reported by @xiamubobby in #2353 and @BeHappy0o0o0o0 in #2157

macOS app and localization

• Added an Appearance action that restores a hidden menu bar icon and reinstated menubar.log diagnostics for visibility and recovery failures.
• Added and refined Russian localization across the macOS app and admin dashboard. By @DrMaks22 in #2329, #2373, and #2376

Upgrade Notes

• The minimum supported Transformers version is now 5.12.1 and remains capped below 5.13.
• Throughput benchmark results for MTP and DFlash models are not directly comparable with results from 0.5.3 or earlier because benchmark prompts now use natural text instead of highly repetitive filler.
• --memory-guard off explicitly disables the guard. Selecting safe, balanced, aggressive, or --memory-guard-gb enables it. off cannot be combined with a custom ceiling.

New Contributors

Thank you to @layer4down, @cebertowicz, @tobomobo, @gilbert-barajas, @1qh, @PowerSpy, and @FabioMalpezzi for their first contributions to oMLX in this release.

Full Changelog: v0.5.3...v0.5.4.dev1