Hotfix release: Addresses DeepSeek V4 prompt-tail visibility (#2490), tiered-cache compatibility (#2487), and sparse-prefill fallback cache safety (#2484).
Highlights from 0.5.4
- DeepSeek V4 Flash 0731 gains DSpark Lightning MTP with up to 85.6% faster code decoding. Embedded DSpark weights, oQ/oQe, and Metal kernels are supported with matching greedy output. (#2460)
- Inkling Small is supported and accelerated with 1.18–1.23x MTP speedups. Text and vision serving, oQ/oQe, composite caches, and built-in eight-depth Lightning MTP are included. Community layouts also load; audio is not yet supported. (#2438, #2463; reported by @studioburnside in #2451)
- The context benchmark measures usable context on the current Mac. It verifies the largest successful prefill and can apply the result through the API, web dashboard, or macOS app. (#2390, #2391)
- Accelerated benchmark runs reach the leaderboard. MTP, TurboQuant, DFlash, SpecPrefill, and VLM MTP results now upload with feature flags, exact model names, and per-run host metrics.
- MTP benchmarks can choose code or novel contexts. Selectable lengths and bundled English, Japanese, and Korean corpora make accelerated results easier to compare across workloads.
- Laguna S-2.1 and Gemma 4 DFlash are up to 20% faster and more robust. Compatibility checks, rollback, mixed caches, prefix-cache round trips, unified Gemma checkpoints, and fewer synchronizations improve decode. By @FabioMalpezzi in #2392 and @richgoodson in #2175, fixes #2153; runtime work in #2403, refs #1737.
- MiMo V2.5 text and oQ are supported. Native text loading and FP8 quantization cover fused and tensor-parallel QKV layouts. Closes #1486.
- Step-3.7-Flash gains native Lightning MTP with 1.22–1.30x faster decode. The StepFun nextn path also enables previously undetected VLM-shaped heads. By @True2456 in #2450.
- Lightning MTP prompt priming improves Qwen3.6 decode by up to 18% for about 1–2% extra prefill cost. Prompt hidden states improve acceptance across Qwen3.5/3.6, DeepSeek V4, and GLM-5.2. (#2402)
- Merged Gemma 4 checkpoints reach up to 1.97x faster decode with Lightning MTP. oQ can merge Gemma 4 and Qwen3.5/3.6 donor heads into single-folder checkpoints, with fused verification and safe fallback. (#2404)
- The web chat has been overhauled. Model selection, context tracking, search, image limits, scrolling, shortcuts, settings, and stable conversation ordering are included. By @PowerSpy in #2379.
- Throughput benchmarks better represent real workloads. Natural text and exact prompt lengths replace repetitive filler. By @JimStenstrom in #2343, reported by @shengpeng7 in #2254.
Full Changelog: v0.5.4...v0.5.5