github beamivalice/sushi v1.7.2
sushi v1.7.2

one hour ago

v1.7.2 — Memory you can count on, faster concurrent Qwen

Bugs fixed in this release, by severity and area:

Severity Memory & admission Server & API Models & loads Engine & tests Total
High 21 10 2 2 35
Medium 21 16 8 4 49
Low 17 21 7 6 51
Total 59 47 17 12 135
  • The context sushi names is the context it admits: requests are checked against the GPU limit macOS declares
    (iogpu.wired_limit_mb) less one safety margin, as in 1.6.0, not against free RAM that other apps move. A 64 GB Mac
    at the recommended limit serves Qwen3.8-Flash-Next STQ 2.75bpw at a 262K context with kv8 and images. A request that
    fits once the running ones finish now waits its turn instead of being refused, and the margin is a setting
    (--wired-margin-gib, Settings › GPU memory).
  • Concurrent Qwen requests are faster on Sushi Cooler: sampled requests draft with MTP again (up to 28% more
    throughput at two rows), four joined requests read their hyper-connections in one pass (11% faster), and prefill no
    longer waits between chunks. DeepSeek-V4.1 prefills 8,192 tokens a chunk by default.
  • A thinking model never ends on an empty turn: when a model stops inside its thought with no answer or tool call,
    sushi closes the thought and lets it continue once, on chat completions, /v1/messages and Responses, and says so in
    finish_details. On by default; turn it off in Settings › Replies or with --no-continue-after-thought. A client can
    also continue a reply that holds only its thought.
  • Exact memory bills: every request reserves its cache at its last row before anything writes, the prefill's last
    forward is billed at its full width, and a DeepSeek-V4.1 prompt of a million tokens is admitted and runs. Cached
    conversations restored on the MLX engine no longer copy their cache every token.
  • Stricter, clearer APIs: logprobs list exactly the text a reply returns; a field of the wrong type is refused by
    name; a reply cut inside a tool call never returns half a call; HTTP heads, chunked bodies and WebSocket frames follow
    the protocol to the byte; every request after a shutdown gets the shutdown error.
  • Model management you can trust: one folder is one model however its path is spelled, --max-resident-models holds
    while two models load, a failed load names its own failure, and sushi kld without --ssd-budget-gb takes serve's
    memory default (fully resident on a 192 or 256 GB Mac).
  • Thanks to @balr0g and @NikolaKrstic94 for the logs that found both memory causes, and to @yanun0323 for the
    thinking-turn report and the Strata numbers behind it.

Don't miss a new sushi release

NewReleases is sending notifications on new releases.