v1.7.1 — Sushi Cooler, made right: memory, refusals, long conversations
- Every request stays inside the memory it was admitted with: Qwen3.8's speculative requests are now billed for what
their draft rounds hold (a short one needs up to about 560 MB more free; the advertised context does not move),
DeepSeek counts its memory margin once, and a raised--wired-margin-gibapplies once again, under the wired limit,
as on 1.6.0. Leaks after a failed speculative request, after joined requests with a logit bias, and after every reload
of an EXL3 or STQ pack are closed; concurrent and sampled requests decode together on Sushi Cooler again, and a
failure fails only the requests it touched. Where free RAM rather than the GPU limit binds, the advertised context is
now the one a request at that length is admitted at, so it can come out smaller. - Long Qwen3.8 conversations keep going: when a turn shares the last turn's cached entry and nothing else frees
enough room, the turn takes that entry over instead of being refused, whether it adds to the conversation, is shorter
or regenerates the last reply (on a 32 GB Mac, a 131k-token turn that was refused is now served); turns that share
part of an entry keep their restore points and prefill less. - DeepSeek-V4.1 is steadier: a long prompt no longer re-reads decoder layers from the SSD while other requests
decode or after a failed memory borrow, regenerating a reply reuses the conversation, a cancelled turn no longer
costs its retry the cache (or hands it another conversation's), and an image request is no longer refused for memory
a prompt of the same size gets. A greedy or seeded DSpark request on the release checkpoint repeats its text: on an
M5 Ultra 256 GB with its experts streamed at--ssd-budget-gb 200, 26.6 tok/s decode cold and 38.6 tok/s repeated. - Refusals say what to do: a request or load refused for memory names the
max_tokens, context and SSD streaming
memory that fit, offers streaming when a pack only fits streamed, names the Settings
label beside each flag, prints one unit with the need rounded up and what is available rounded down, and never
suggests a wired limit past the Mac's RAM. Sushibar shows the whole reason, the web page keeps it on screen until
dismissed, every request to a refused model gets it, and a load the GPU limit refuses answers at once. - APIs and server: errors on
/v1/messagescome in Anthropic's shape; a Responses WebSocket turn refused before it
runs ends with an error frame, andstore: falseis echoed as sent; a client that stops reading no longer blocks an
unload (a busy unload answers 409 within 30 s); a large SSD prompt cache
no longer stops new connections; the web page's Thinking panel stays where you scroll it. The HTTP request-size limit
is a setting,server.max_request_mb(Settings > Server, 1-1,024 MiB, 128 by default): thanks @Cianidos. - On the MLX route (
--qwen-mlx-engine, GLM-5.3-Flash, MiMo): a kv8, kv6 or kv4 cache grows on MLX's page grid, so
its appends are never copied past the bill; a bill moves by a few MB, and on a tight Mac the advertised context can
drop by a few hundred tokens. Sushi Cooler is unchanged by this. - Known issue: DeepSeek-V4.1-Flash's FP8 release checkpoint does not load at its default SSD streaming memory on a
256 GB Mac; set--ssd-budget-gb 215(the refusal names it). Thanks @nikolay-bratanov, @younesehb, @NikolaKrstic94
and @cyrilmhansen for the reports behind this release.