Splash 1.6.0

It's been longer than usual since 1.5.0. Sorry for the wait.

This release has few kernel changes. Most of the work went into the engine: how it uses memory, schedules requests, and recovers when something goes wrong. In real serving, with several requests, long contexts, and a memory limit, these decide whether Splash stays responsive. We think they matter more there than another few percent of decode speed.

Serving under a memory limit

Splash 1.6.0 fixes long stalls seen in 1.5.0 under a memory limit. On an M5 Max with 36 GB serving Qwen3.8-Flash-Next Q2_0 at --max-memory 18G, the longest pause in a stream dropped from 300–406 s to 0.92 s. On the same load, prefill throughput rose 1.27x, decode throughput rose 1.25x, and TTFT fell 26%. On a 24 GB Mac serving Qwen3.8-27B UD-IQ3_XXS, TTFT fell 41%. These are concurrent serving results; single-request decode stays within 2% of 1.5.0.

• One memory ledger. Every allocation goes through one ledger, with one order for what to free when memory runs short. Cached data the model can rebuild goes first; memory another request is using goes last. /status reports it under memory_governor.
• Idle KV no longer slows every command. 1.5.0 made all cached KV resident for every GPU command, so idle cache slowed decode by up to 5.8x near the GPU's memory limit. Each command now makes resident only the KV it reads. (#644)
• Elastic expert store. With --expert-memory auto (the default), streamed experts grow as memory allows and give memory back when requests or other apps need it. --expert-memory SIZE still fixes the size.
• SSD cache keeps making room. A full cache replaces old copies instead of refusing new ones.
• Fairer scheduling. A request that has to wait for memory no longer holds up the requests behind it. A long prompt gets its turn after three commands have passed it, as in 1.5.0.
• More context, not less. Qwen3.6-35B-A3B no longer serves less context when given a larger memory budget.
• Smaller prefill chunks use a 64-row tile on M5/M6 and smaller M3/M4 GPUs. (#621)

Reliability

• Engine restarts. A failed engine restarts with a backoff. Restarts stop at once on model files that cannot be served, and after six quick failures in a row (1.5.0 stopped after three). A start that waits for memory keeps retrying.
• Quarantine. A request that crashes the engine twice is refused with 400 request_quarantined instead of crashing it again.
• Sandboxed PDFs. PDFs render in a sandbox with no network access and no environment variables.
• A request whose KV spills to the SSD no longer fails with capacity_exhausted while memory is about to free up. (#622, fixes #613)

Models

• Split GGUFs at a repository's root install as one model.
• Qwen3.8-Flash-Next GGUFs with BF16 or F16 PLE tables load.
• Flint GGUFs from lmstudio-community, ggml-org, bartowski, mradermacher, and prithivMLmods install.

Splash for Mac

• Drawer: press ⌘J for terminals and live logs from the server and downloads.
• Smoother app: replies stream without redrawing, long chats scroll steadily, and the composer and controls move together.
• Clearer launch options: short names, an explanation beside each option, and only the options the selected model can use.
• Start failures say why. The server the app started also stops if the app crashes.
• Context ring: a ring beside Send shows how much of the context the prompt uses. (#642)

APIs

• POST /v1/chat/completions/count_tokens counts a Chat request's tokens without running it. (#642)
• /status breaks each request's time to first token down by cause. (#620)
• A tool call without a function name is returned as text instead of being dropped. (#632)
• Stop sequences are matched as output arrives. Long stop sequences no longer slow generation.

Fixes

• Thinking budgets shrink with the remaining context to leave room for the answer. A budget cutoff now follows the chat template, so follow-up turns reuse more cached KV.
• With http_proxy set, the launcher and coding agents now reach the local server directly. 1.5.0 sent the server's API key to the proxy.
• splash pi sends the API key to every Pi version.
• A Finder .DS_Store file no longer breaks local models.
• A pipelined request no longer resets the response.
• The app lists only models whose files are still on this Mac.

Upgrade

• App: choose Splash → Check for Updates…, or download the latest DMG.
• CLI: brew update && brew upgrade splash.
• App installed with Homebrew: brew update && brew upgrade --cask splash-app.

If you use both the CLI and app, keep them on the same release.

• /status is schema 8. Two fields were removed. Requests ended by a stop sequence count as completed.
• Stop sequences: each is limited to 16,384 characters. Messages stop_sequences must be an array of nonblank strings; a single string now gets a 400 error.
• GGUF screening: a GGUF with tensors the model would not use, such as an attention bias, is refused when it loads.
• PDF input needs /usr/bin/sandbox-exec. Inside another sandbox, PDF requests get 503 document_unavailable.
• Persistent cache: a GGUF model's cache starts fresh once after the upgrade. The weights are not downloaded again.
• Downloads are refused only when they don't fit on the disk. The SSD cache shrinks or turns off on its own when space runs short.

Full details are in the 1.6.0 release notes.

Thanks to @linson007, @pcvelz, and @coco-alen for contributions, and to @q-p, @sunnybluesea, and @oinani0721 for detailed reports in #613 and #644.

Full changelog: 1.5.0...1.6.0