jundot/omlx v0.5.4rc1

0.5.4rc1

on GitHub

This release candidate promotes the 0.5.4 development line with a context benchmark that measures the usable context window on the current Mac, broader and faster Lightning MTP support, improved DFlash acceleration and observability, MiMo V2.5 text and oQ support, and a major web chat refresh. It also tightens Claude Code context handling, model conversion, Metal stream safety, configuration persistence, and inference stability ahead of the final 0.5.4 release.

Highlights

• Context benchmark. Measures the largest context the current Mac can prefill, verifies it with a real request, and applies the result to model settings. Available through the admin API, web dashboard, and macOS app. (#2390, #2391)

• Lightning MTP prompt priming. The MTP head now learns from prompt hidden states during prefill instead of starting decode with an empty cache. This improves acceptance for prompt-specific identifiers and content across Qwen3.5/3.6, DeepSeek V4, and GLM-5.2. (#2402)

  On an M3 Ultra with Qwen3.6-27B-oQ4e-mtp, TurboQuant KV 4-bit enabled, temperature=0.6, and max_tokens=512, prompt priming raised acceptance from 70–75% to 78–84% and improved decode throughput by approximately 5–18%.

  Context	Before → after, decode tok/s
  Code 1K	46.0 → 54.4 (+18%)
  Code 16K	42.1 → 47.7 (+13%)

  Results are averages of two repetitions.

• Lightning MTP for merged Gemma 4 checkpoints. Gemma 4 assistant heads can now be merged into a single checkpoint and use the standard batched decode path with paged, prefix, and SSD caches. oQ can create these merged checkpoints directly, while concurrent or unproductive speculative workloads fall back safely to standard decode. (#2404)

  gemma-4-31B-oQ4e — decode tok/s
  Parentheses show speedup over MTP off and draft acceptance.

  Context	Off	DFlash	VLM MTP	Lightning MTP
  Code 4K	28.7	34.1 (1.19×, 56%)	54.2 (1.89×, 70%)	56.4 (1.97×, 88%)
  Code 16K	25.7	36.9 (1.44×, 59%)	51.9 (2.02×, 82%)	50.7 (1.97×, 89%)
  Story 4K	28.7	39.0 (1.36×, 62%)	37.6 (1.31×, 37%)	35.2 (1.23×, 59%)
  Story 16K	25.7	37.4 (1.46×, 60%)	32.9 (1.28×, 38%)	42.2 (1.64×, 82%)

• Faster, more resilient Gemma 4 MTP. Added fused multi-row verify attention, a default depth ceiling of 8, device-specific threadgroup probing, and an escape hatch that hands unproductive speculation back to standard decode.

• Laguna S-2.1 and Gemma 4 DFlash improvements. Poolside Laguna targets and official drafts support adaptive speculative decoding, strict compatibility checks, rollback, mixed caches, and prefix-cache round trips. Gemma 4 unified checkpoints are now accepted, and the customized DFlash runtime reduces synchronization overhead to a single device-to-host transfer per speculative cycle. (#2392, #2403, #2175)

  4K-context DFlash throughput — decode tok/s
  Each cell shows the two measured repetitions.

  Model pair	Before	After	Improvement
  Qwen3-4B-bf16 + DFlash-b16	45.7 / 46.4	47.7 / 48.4	+4.3%
  Gemma-4-26B-A4B-IT-4bit + 26B-A4B-DFlash	95.1 / 142.9	113.0 / 162.9	+14–19%

• MiMo V2.5 text and oQ support. Xiaomi MiMo V2.5 checkpoints load through the native text engine and can now be quantized from native FP8 weights, including fused and tensor-parallel QKV layouts.

• Correct Claude Code context handling. oMLX now reports real token usage instead of inflating it toward an artificial target. Claude Code launches receive the model's actual context window for deterministic auto-compaction, and models configured below 48K context are rejected before launch. (#2400)

• Overhauled web chat. Adds model selection, context tracking, chat search, configurable image limits, better scrolling, keyboard shortcut help, expanded chat settings, and stable message order when switching conversations. By @PowerSpy in #2379

• More representative throughput benchmarks. Benchmarks now use natural text and enforce exact prompt lengths. Speculative-decoding results may be lower than older runs but better reflect real workloads. By @JimStenstrom in #2343, reported by @shengpeng7 in #2254

• Faster heterogeneous embedding batches. Inputs are grouped by length and restored to caller order, avoiding excessive padding. Measured throughput improved by roughly 4x to 4.6x across text and VLM embedding models. By @1qh in #2370

• Merge external MTP heads during oQ quantization. oQ can now combine compatible donor MTP heads directly into quantized Gemma 4 and Qwen3.5/3.6 outputs. The resulting single-folder -mtp checkpoints can be enabled through the standard mtp_enabled setting without managing a separate assistant model. (#2404)

New Features

• Added a global prefill priority setting. max context dynamically shrinks prefill chunks to maximize usable context, while speed keeps full-size chunks for maximum throughput.
• Added --memory-guard off. Selecting a tier or custom ceiling explicitly enables the guard.
• Added MXFP8 source-model support to oQ quantization.
• VLM checkpoints can be admitted to the text engine using only the language-model weight size instead of charging the unused vision tower against memory admission.
• The macOS Performance screen now shows the effective memory ceiling and warns when the Metal limit is the binding cap. (#1463)
• The admin dashboard now exposes active DFlash engines, runtime cache statistics, and DFlash phase timings.
• Added generate, regenerate, and copy controls to macOS sub-key creation. By @tobomobo in #2348
• Added first-use diagnostics for the SDPA-256 tiled route and large prefix-cache divergence points.
• Claude Code launches disable the LSP tool by default to keep agentic prompt prefixes stable for prefix-cache reuse.

Performance

• MTP prompt priming improves draft acceptance for code identifiers and prompt-specific content with approximately 1–2% additional prefill cost. (#2402)
• Gemma 4 Lightning MTP uses fused verification attention for global layers and automatically falls back to standard decode when speculation stops paying off.
• DFlash decode now performs a single device-to-host transfer per cycle instead of four synchronizations plus a hard evaluation. Measured gains ranged from 3.5–4.8% on Qwen3 and 15–20% on Gemma 4. (#2403)
• GLM-5.2 long-context decode uses a fused DSA indexer scan after 4K cached tokens for eligible single-stream workloads. M3 Ultra measurements improved by 1.7% at 30K context and 3.1% at 125K. By @aidiffuser in #2245
• Throttled prefill chunks align to reusable minimum-chunk multiples, improving measured prefill throughput by 6.8% on Gemma 4 and 8.4% on Qwen3.6 under tight memory ceilings. (#2390)

Bug Fixes

Inference and caching

• Fixed VLM MTP silently dropping per-request grammar constraints, thinking budgets, and repetition or presence penalties. Requests requiring logits processors now use the standard batch path, and incompatible model-level settings are rejected or migrated safely.
• Fixed Metal cache clears racing with in-flight work during prefill cleanup, GLM decode, and VLM MTP decode by synchronizing the actual engine or generation stream first. By @JimStenstrom in #2412 and #2413
• Fixed hard-memory-pressure recovery livelocking on continuously busy servers by draining reclaimable MLX buffers at scheduler step boundaries. By @cebertowicz in #2342
• Fixed external VLM MTP being skipped when a request completed through chunked prefill. By @ethannortharc in #2361
• Fixed mRoPE positions remaining constant across multi-token speculative verification windows. By @gilbert-barajas in #2363
• Fixed MiniCPM-o text-only requests failing when the mRoPE fallback had no initialized position state. Reported by @goldyard2022 in #2387
• Fixed prefix-cache reconstruction running outside the owning engine's Metal stream. Reported by @anicaise-ai in #2330
• Fixed DFlash prefix-cache round trips when sidecar entries were trimmed.
• Fixed fallback engines losing the originally requested runtime signature, which could cause subsequent requests to select inconsistent engine configurations.
• Fixed SDPA patches capturing an old attention function before TurboQuant was installed.
• Fixed the SDPA-256 patch not rebinding mlx-vlm submodules and restored the unfused fast path when the memory guard is disabled.
• Fixed multi-token stop-sequence prefixes leaking into completion output. Reported by @Xcoder1990 in #2386
• Fixed SpecPrefill request overrides being ignored by the VLM non-streaming path. By @JimStenstrom in #2338
• Fixed frequency_penalty being ignored by VLM-hosted autoregressive models. By @JimStenstrom in #2339
• Fixed failed context benchmark probes leaving stale transient-memory estimates.
• Fixed empty thinking blocks appearing in web chat responses.

Tool calling and streaming

• Fixed unfinished paired tool-call envelopes disappearing when generation ended before the closing marker. OpenAI chat completions, Anthropic Messages, and the Responses API now recover the buffered content when no structured tool call can be parsed. (#2420)
• Fixed XML tool-call fallbacks coercing arrays and objects without consulting the tool schema.
• Fixed JSON-quoted string arguments retaining their surrounding quotes. Reported by @bobacus in #2332

Model loading and quantization

• Fixed oQ routing text-only multimodal checkpoint families such as MiMo V2.5 through mlx-vlm instead of mlx-lm. By @blightbow in #2416
• Fixed native FP8 calibration admission using the compressed source footprint even though calibration materializes BF16 weights.
• Fixed MiMo V2.5 fused, tensor-parallel QKV tensors being dequantized or combined incorrectly. By @blightbow in #2417
• Fixed text-only oQ outputs retaining multimodal configuration and processor sidecars after vision or audio weights were removed. By @blightbow in #2421
• Fixed Gemma 4 shared-KV oQe calibration.
• Fixed merged Gemma 4 MTP weights reaching the plain mlx-lm text path when MTP was disabled.
• Fixed known misconverted Unlimited-OCR tokenizer metadata so affected checkpoints can load with the correct tokenizer behavior. (#2424)
• Fixed quantized Laguna checkpoints with legacy router names failing to load. By @layer4down in #2337
• Fixed current DFlash draft checkpoints storing RoPE and DFlash configuration under Transformers 5.x nested keys. By @JimStenstrom in #2340
• Fixed mistral-common models selecting a tokenizer backend that could drop control tokens. By @JimStenstrom in #2341
• Fixed MiniMax M3 oQ conversion dropping unpacked shared experts. Reported by @avlp12 in #2356
• Fixed oQ calibration admission ignoring the active Metal ceiling.
• Fixed GLM-5.2 DSA indexers being quantized below Q8. Reported by @noonkho in #2347
• Fixed Qwen3-VL embedding and rerank models failing on image inputs.
• Fixed empty vision configuration stubs being misclassified as usable vision towers.
• Fixed the Homebrew custom-kernel build omitting Bonsai verification. Reported by @goldyard2022 in #2388

API, configuration, and integrations

• Fixed Claude Code token usage and auto-compaction using different context scales. Reported usage now always matches the real tokens processed by the model. By @fparrav in #2400
• omlx serve now persists all explicit non-secret CLI configuration safely without capturing API keys or unrelated environment overrides. Settings files are restricted to the owner on POSIX systems. By @ShiroKSH in #2239
• Per-request INFO summaries now include the actual serving model across completions, chat, embeddings, reranking, Anthropic Messages, and the Responses API. By @JimStenstrom in #2411
• Fixed external benchmark streams recognizing only reasoning_content; reasoning, analysis, and streams without observable timing now produce correct metrics or N/A. By @fparrav in #2375
• Fixed the dashboard showing a different memory reserve than the server on 16–23 GB Macs.
• Memory rejection messages now identify the binding ceiling and the setting that can raise it.
• Fixed the Codex integration reporting the renamed ChatGPT desktop app as missing and added bundled CLI fallback discovery. Reported by @xiamubobby in #2353 and @BeHappy0o0o0o0 in #2157

macOS app, web UI, and localization

• Fixed web chat messages appearing out of order after switching conversations.
• Added an Appearance action that restores a hidden menu bar icon and reinstated menubar.log diagnostics.
• Completed Simplified and Traditional Chinese translations across the admin dashboard. By @NirvanaCh7 in #2415
• Added and refined Russian localization across the macOS app and admin dashboard. By @DrMaks22 in #2329, #2373, and #2376

Upgrade Notes

• The minimum supported Transformers version is now 5.12.1 and remains capped below 5.13.
• Throughput benchmark results for MTP and DFlash models are not directly comparable with results from 0.5.3 or earlier because benchmark prompts now use natural text instead of repetitive filler.
• --memory-guard off explicitly disables the guard. Selecting safe, balanced, aggressive, or --memory-guard-gb enables it. off cannot be combined with a custom ceiling.
• Claude Code context scaling has been removed. claude_code.context_scaling_enabled and claude_code.target_context_size are no longer used; token usage and auto-compaction now follow the model's configured max_context_window.
• Claude Code integration requires a configured context window of at least 48K tokens.
• VLM MTP cannot be combined with model-level guided grammar, thinking budget, repetition penalty, or presence penalty settings. Existing conflicting configurations preserve the output-shaping settings and disable VLM MTP during migration.

New Contributors

Thank you to @layer4down, @cebertowicz, @tobomobo, @gilbert-barajas, @1qh, @PowerSpy, @FabioMalpezzi, @ShiroKSH, and @NirvanaCh7 for their first contributions to oMLX in the 0.5.4 release line.

Full Changelog: v0.5.3...v0.5.4rc1