jundot/omlx v0.5.4 on GitHub

oMLX 0.5.4 adds native support and acceleration for DeepSeek V4 Flash 0731, Inkling Small, Step-3.7-Flash, MiMo V2.5, and Laguna S-2.1. It also expands Lightning MTP, DFlash, SpecPrefill, context benchmarking, and long-context cache reliability across the server, web dashboard, and macOS app.

Highlights

DeepSeek V4 Flash 0731 gains DSpark Lightning MTP with up to 85.6% faster code decoding. Embedded DSpark weights, oQ/oQe, and Metal kernels are supported with matching greedy output. (#2460)
Inkling Small is supported and accelerated with 1.18–1.23x MTP speedups. Text and vision serving, oQ/oQe, composite caches, and built-in eight-depth Lightning MTP are included. Community layouts also load; audio is not yet supported. (#2438, #2463; reported by @studioburnside in #2451)
The context benchmark measures usable context on the current Mac. It verifies the largest successful prefill and can apply the result through the API, web dashboard, or macOS app. (#2390, #2391)
Accelerated benchmark runs reach the leaderboard. MTP, TurboQuant, DFlash, SpecPrefill, and VLM MTP results now upload with feature flags, exact model names, and per-run host metrics.
MTP benchmarks can choose code or novel contexts. Selectable lengths and bundled English, Japanese, and Korean corpora make accelerated results easier to compare across workloads.
Laguna S-2.1 and Gemma 4 DFlash are up to 20% faster and more robust. Compatibility checks, rollback, mixed caches, prefix-cache round trips, unified Gemma checkpoints, and fewer synchronizations improve decode. By @FabioMalpezzi in #2392 and @richgoodson in #2175, fixes #2153; runtime work in #2403, refs #1737.
MiMo V2.5 text and oQ are supported. Native text loading and FP8 quantization cover fused and tensor-parallel QKV layouts. Closes #1486.
Step-3.7-Flash gains native Lightning MTP with 1.22–1.30x faster decode. The StepFun nextn path also enables previously undetected VLM-shaped heads. By @True2456 in #2450.
Lightning MTP prompt priming improves Qwen3.6 decode by up to 18% for about 1–2% extra prefill cost. Prompt hidden states improve acceptance across Qwen3.5/3.6, DeepSeek V4, and GLM-5.2. (#2402)
Merged Gemma 4 checkpoints reach up to 1.97x faster decode with Lightning MTP. oQ can merge Gemma 4 and Qwen3.5/3.6 donor heads into single-folder checkpoints, with fused verification and safe fallback. (#2404)
The web chat has been overhauled. Model selection, context tracking, search, image limits, scrolling, shortcuts, settings, and stable conversation ordering are included. By @PowerSpy in #2379.
Throughput benchmarks better represent real workloads. Natural text and exact prompt lengths replace repetitive filler. By @JimStenstrom in #2343, reported by @shengpeng7 in #2254.

New Features

Prefill priority is configurable. max context shrinks chunks for larger prompts, while speed keeps full-size chunks for throughput. (#2390)
The memory guard can be disabled explicitly. --memory-guard off now has clear semantics, while choosing a tier or custom ceiling enables the guard.
Text-only VLM loads use language-model size for admission. Unused vision weights no longer reduce the memory budget for text-engine serving.
DFlash observability is expanded. The dashboard shows active engines, cache state, phase timings, acceptance, and fallback reasons; precision-mismatched draft pairs warn clearly. (#2396; by @FabioMalpezzi in #2445)
oQ accepts MXFP8 source models. Native MXFP8 checkpoints can enter the standard quantization workflow.
Maximum reasoning effort is selectable. Model settings now expose the highest supported reasoning-effort level.
Claude Code launches keep agent prefixes stable. The LSP tool is disabled by default so repeated agent prompts can reuse the prefix cache.
Memory limits are visible in the macOS app. The Performance screen shows the effective ceiling and warns when the Metal cap is binding. (#1463)

Performance

SpecPrefill reuses persistent target prefixes. Stable system and tool prefixes survive requests and restarts, reducing repeated target prefill. By @aosama in #2440, closes #2177.
Heterogeneous embedding batches are roughly 4–4.6x faster. Length grouping avoids excess padding while preserving caller order. By @1qh in #2370.
GLM-5.2 long-context decode is faster. Fused DSA indexer scanning and 32-head sparse-MLA tuning improve eligible single-stream workloads. By @aidiffuser in #2244 and #2245.
Throttled prefills are 6.8% faster on Gemma 4 and 8.4% faster on Qwen3.6. Minimum-chunk alignment improves reuse under tight memory ceilings. (#2390)
Deep Inkling MTP blocks stay warm. A lag-triggered keepalive refold prevents later depth probes from paying the full gap refold or drafting from stale rows.
DeepSeek affine MoE avoids accidental FP32 work. Short BF16 routes now preserve the intended FP16 affine kernel path. Reported by @yohann-bearzi in #2467.

Bug Fixes

Inference and caching

DeepSeek V4 prefix snapshots stay compact. Pooling-cache blocks now store append-only deltas, preventing post-response memory growth on very long SSD-cache writes. Reported by @neutrospec in #2465.
SpecPrefill preserves a mandatory trailing context window. Recent chat structure can no longer be dropped by sparse selection. By @JimStenstrom in #2442, fixes #2439 and addresses reports in #419, #548, #1401, and #334.
Metal cache clears wait for the owning stream. Prefill cleanup, GLM decode, and VLM MTP no longer race in-flight GPU work. By @JimStenstrom in #2412 and #2413.
Busy servers recover from hard memory pressure. Reclaimable MLX buffers drain at scheduler step boundaries instead of livelocking. By @cebertowicz in #2342, completes #2306.
Prefix-cache reconstruction stays on the engine stream. Fully cached concurrent prefills no longer wedge during restore. Reported by @anicaise-ai in #2330.
VLM MTP preserves output-shaping settings. Grammar constraints, thinking budgets, and repetition or presence penalties no longer disappear silently; incompatible configurations migrate or use standard decode safely. (#2399)
VLM speculative windows advance mRoPE positions. Multi-token verification no longer commits incorrectly rotated cache keys. By @gilbert-barajas in #2363.
DFlash cache round trips preserve runtime identity. Trimmed sidecars and fallback engines retain valid prefix state and the originally requested runtime signature. (#2406)
SDPA patches compose with TurboQuant and mlx-vlm. Attention is resolved at call time, VLM submodules are rebound, and the memory-guard-off fast path remains available. (#2283)
MiniCPM-o text-only requests initialize mRoPE fallback state. Text requests no longer fail when no vision position state exists. Reported by @goldyard2022 in #2387.
Multi-token stop prefixes stay out of output. Partial stop sequences no longer leak before the full sequence matches. Reported by @Xcoder1990 in #2386.
Prefill memory estimates reject only real overages. Short speed-priority tails and isolated noisy samples no longer inflate admission costs. Reported by @funnygeeker in #2434; by @fparrav in #2414.
Chunked prefill can enter external VLM MTP. Requests no longer skip speculation when prefill completes in chunks. By @ethannortharc in #2361, fixes #2219.
VLM request overrides propagate consistently. SpecPrefill settings and frequency_penalty now reach non-streaming VLM-hosted models. By @JimStenstrom in #2338 and #2339.
Immediate 4K repeats can reuse an in-flight cache store. The scheduler now defers highly related requests briefly instead of re-prefilling the full prompt. By @wolfyy970 in #2472, fixes #2471.

Model loading and quantization

Homebrew grammar installs no longer risk an xgrammar import crash. The compatible native dependency pair is pinned. Reported by @zviratko in #2428.
Step-3.7 checkpoint handling is hardened. oQ and model loading now share one nextn prefix resolver.
MiMo V2.5 conversion follows the text path. Native FP8 calibration budgets BF16 materialization correctly, fused QKV shards dequantize independently, and text-only outputs drop multimodal sidecars. By @blightbow in #2416, #2417, and #2421.
Gemma 4 conversion handles shared KV and merged MTP safely. oQe calibration works, and disabled merged heads no longer reach the plain text engine.
Community Inkling MLX layouts load correctly. pipenetwork and mlx-community naming variants are mapped to the supported architecture. Reported by @studioburnside in #2451.
Inkling MTP heads are detected in oQ. The settings UI now recognizes the checkpoint's built-in MTP layout.
MiniMax M3 oQ preserves unpacked shared experts. Conversion no longer drops the shared-expert weights. Reported by @avlp12 in #2356.
GLM-5.2 DSA indexers stay at Q8. oQ no longer creates lower-bit indexers that cannot load. Reported by @noonkho in #2347.
Qwen3-VL embedding and rerank models accept images again. The torch-free processor follows the expected Transformers image contract.
Legacy Laguna quantization metadata is remapped. Router scales, biases, and per-layer overrides now load. By @layer4down in #2337.
DFlash reads Transformers 5.x nested configuration. Current draft checkpoints expose their RoPE and DFlash settings correctly. By @JimStenstrom in #2340.
mistral-common models use the HF-native tokenizer backend. Control tokens are preserved during prompt construction. By @JimStenstrom in #2341.
Unlimited-OCR tokenizers self-repair known bad metadata. Affected converted checkpoints load with the correct tokenizer behavior. (#2424)
Embedding compile state is detected accurately. The load-time smoke test now traces attention_mask, so mask-branching models no longer claim compilation while serving eagerly. By @JimStenstrom in #2461, reported by @alantmiller in #2447.
Empty vision stubs are treated as text-only. Model discovery no longer invents a usable vision tower from an empty configuration.
Homebrew verifies the Bonsai kernel. All four bundled custom kernels are checked after build. Reported by @goldyard2022 in #2388.

API, tools, benchmarks, and integrations

MCP SDK 2.0 works again. Snake-case and legacy result fields are handled together, restoring server initialization, schemas, errors, and structured content. By @aaramos in #2464.
Serve configuration persists safely. Explicit non-secret CLI settings are stored without API keys or unrelated environment overrides, with owner-only permissions on POSIX. By @ShiroKSH in #2239.
Claude Code uses the real model context and requires at least 48K. Reported token usage and auto-compaction now share the configured window. By @fparrav in #2400.
Unclosed tool-call streams preserve buffered content. OpenAI chat, Anthropic Messages, and Responses recover unfinished paired envelopes when no structured call can be parsed. (#2420)
Streamed completions keep one response ID. Strict OpenAI-compatible accumulators no longer discard later chunks. By @ShiroKSH in #2462.
XML tool-call fallback follows the schema. Arrays and objects are coerced correctly, and JSON-quoted strings lose their extra quotes. Reported by @bobacus in #2332.
partial: true reaches VLM chat templates. The option now works for text conversations served through VLM-class checkpoints. By @JimStenstrom in #2435, reported by @dougy86 in #2357.
Codex integration finds the renamed ChatGPT app. Bundled CLI fallback discovery replaces the stale application-name check. Reported by @xiamubobby in #2353 and @BeHappy0o0o0o0 in #2157.
Skipped benchmark uploads terminate cleanly. Subscribers now receive a terminal SSE event instead of waiting forever.
External benchmark reasoning metrics are honest. reasoning_content, reasoning, and analysis variants produce valid timing or N/A. By @fparrav in #2375.

macOS app, web UI, and localization

Chinese and Russian localization coverage is expanded. Simplified and Traditional Chinese are complete in the dashboard, and Russian strings are added or refined across the app and dashboard. By @NirvanaCh7 in #2415 and @DrMaks22 in #2329, #2373, and #2376.

Upgrade Notes

Transformers 5.12.1 or newer is required. The dependency remains capped below 5.13.
Claude Code context scaling has been removed. claude_code.context_scaling_enabled and claude_code.target_context_size are ignored in favor of max_context_window.
Claude Code requires at least a 48K context window. Smaller configured windows are rejected before launch.
VLM MTP has model-setting compatibility limits. Guided grammar, thinking budget, repetition penalty, and presence penalty cannot be combined with model-level VLM MTP; migration preserves output shaping and disables MTP.
Memory-guard flags have explicit enablement rules. off disables the guard, while a tier or --memory-guard-gb enables it; off cannot be combined with a custom ceiling.
Benchmark results are not directly comparable with 0.5.3. MTP and DFlash prompts now use natural text instead of repetitive filler.

New Contributors

Thank you to @layer4down, @cebertowicz, @tobomobo, @gilbert-barajas, @1qh, @PowerSpy, @FabioMalpezzi, @ShiroKSH, @NirvanaCh7, @True2456, @wolfyy970, and @aaramos for their first contributions to oMLX in the 0.5.4 release line.

Full Changelog: v0.5.3...v0.5.4

jundot/omlx v0.5.4 0.5.4 on GitHub