incoai/splash 1.4.0

Splash 1.4.0

on GitHub

Splash 1.4.0 adds Qwen3.8-Flash-Next, SSD streaming for experts and PLE, active KV spill, and up to 1M context. Adaptive draft verification and mixed structured-output batching improve decode throughput.

Qwen3.8-Flash-Next

Flash-Next GGUFs now load with QSA sparse attention and an automatically selected DFlash2 draft (#467–#469):

splash serve --model ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF:Q2_0

Unsloth's Flash-Next GGUFs also load. Flash-Next MLX checkpoints are not supported yet.

On an M5 Pro (20-core GPU, 64 GB), Q2_0 measured 95 tok/s decode, 1,312 tok/s prefill at 32K, 171 ms cached TTFT, and 171 tok/s aggregate decode with four concurrent requests. These use selected SPEED-Bench coding prompts over HTTP with 1,024-token outputs; prefill rate uses time to the first token, and all 16 replayed prompts hit the cache (#521).

SSD streaming

The planner shares one memory budget between weights, context and streaming caches (#463, #468, #474–#481, #495):

• MoE experts: GGUF and MLX targets keep experts resident when they fit, otherwise stream them from the SSD. --expert-memory SIZE sets their RAM store; off loads every expert.
• Flash-Next PLE: the n-gram table streams when it cannot fit, with a default 512 MiB row cache. --ple-memory SIZE changes that cache; off loads the whole table.
• Active KV: Flash-Next can spill a running request's KV to the SSD, keeping its sparse index and hot blocks in RAM. --sparse-kv-memory SIZE controls the hot cache. The 27B and 35B keep active KV resident.

Streaming is automatic and trades SSD reads for RAM. KV spill requires a writable SSD cache with available quota. --max-memory leaves room for other applications and shrinks the automatic stores. Streamed prefill runs in passes so other requests can decode and cancellations take effect between passes.

Up to 1M context

All three families support static YaRN beyond their native 262,144-token window. Add --max-context 1M to enable it; 1M means 1,000,000 tokens, with an upper limit of 1,010,000 (#431, #469, #518).

The requested context must fit available RAM and SSD resources. YaRN stays off within the native window and changes outputs even on short prompts when enabled, so request the longer window when needed.

Faster decode

MoE targets adapt how many draft tokens they verify, avoiding work on unlikely proposals (#487). Against verifying every row, Flash-Next Q2_0 with a 12G expert store gained 13–38% decode throughput at one to four concurrent requests on an M5 Max 36 GB. The 35B UD-Q2_K_XL with a 4G store gained 6.5–42% on an M6 24 GB.

JSON Schema and ordinary requests now share decode batches (#520). With two of each on an M5 Max 36 GB, aggregate throughput rose 349 → 457 tok/s for the 35B MLX 4-bit and 138 → 210 tok/s for the 27B MLX 4-bit. Single-request throughput was unchanged within noise.

These comparisons measure individual changes. Metal optimizations also improve MoE projections, gathering, low-bit decode and sparse attention.

Other changes

• Preserve conversation replay states during routine memory reclamation, and durable restart points when the SSD cache fills. Fix spill-resume accounting, pinned-cache lifetime errors and engine relaunch with spilled KV (#484, #512, #513, #524).
• A single unexpected exit no longer clears persistent cache; repeated failures or engine-reported execution faults during startup probation still can (#498).
• Honor explicit --max-cache-disk SIZE quotas subject to the volume's free-space floor. Fix a host-step race that could cause GPU timeouts, and avoid watchdog restarts while streamed commands make progress (#479, #494, #515).
• Load split GGUFs, Q2_0 and additional GGUF norm/router/gate tensor types. Add network-free install.local.prepare_local_model() for local checkpoints (#451, #452, #500).
• Normalize tool-schema type aliases and preserve untyped arguments as text. Bound schema/prompt caches by bytes; PDF preparation no longer holds up unrelated requests while waiting (#514).
• Remove stray </think> tags after reasoning ends; report streaming, KV spill and verification mode in /status (#486, #481, #487).

Upgrade notes

• Default --verify-rows auto can change MoE greedy and seeded outputs as timings and server history change. Use --verify-rows all for repeatability with isolated requests and identical cached prefixes.
• Affected weight layouts, including the 27B and 35B MLX checkpoints, start their persistent cache cold once; it refills during use.
• Direct python -m server.server integrations should drop --tokenizer and use prepare_local_model() (#500).

Thanks to @janhilgard and @Vardaan97.

Thanks also to Strata for inspiring some of the design choices in this release.

Full changelog: 1.3.2...1.4.0