github beamivalice/sushi v1.7.0
sushi v1.7.0

one hour ago

v1.7.0 — Sushi Cooler by default, faster DeepSeek, a lighter prompt cache

  • Qwen3.8 and DeepSeek-V4.1 run on Sushi Cooler, sushi's own Metal engine, by default: every Qwen3.8 STQ pack and
    its BF16 checkpoint, and every DeepSeek-V4.1-Flash pack including its own release checkpoint, prefill, decode, batch
    concurrent requests and read images on it, on M1 through M5 Macs, with the tokens MLX gives. A repeat start skips the
    engine's warm-up, and nothing compiles mid-request. --qwen-mlx-engine, --ds41-mlx-engine or the model's Cooler
    checkbox keeps a model on MLX.
  • DeepSeek-V4.1-Flash is faster and steadier: about 3% faster decode, faster long-prompt prefill, about a third less
    CPU while generating, cached turns that answer about 120 ms sooner, and agent turns that no longer re-read their
    previous reply. Sampled replies no longer slip in a random token (1.6.1-dev2 and 1.7.0-dev1), and greedy answers no
    longer depend on drafting or chunk size. On a Mac that streams its experts from the SSD, a new opt-in setting, "Faster
    SSD streaming (not exact)" in the web UI or --expert-cache-bias, trades a little answer quality for speed: the
    router favours an expert already in memory over a slightly better one it would read from the SSD (about 2x faster
    decode at 0.2 on a 64 GB Mac, about 30% at 0.5 on a 32 GB Mac whose cache holds far fewer experts; off by default, so
    answers stay exact). Thanks @odokododok for it.
  • A lighter prompt cache: a Qwen3.8 conversation writes about three quarters less to the SSD (six turns on the STQ
    4bpw pack: 370 MB instead of 1,409 MB, same replies), holding its newest restore points in up to 250 MB of RAM so a
    regenerated answer restores at once, even after a restart. The SSD cache is back to full speed when saving and
    restoring, and stays within its budget on disk.
  • Less CPU, sooner answers: about 20% less CPU over a full benchmark on Qwen3.8-Flash-Next with decode about 5%
    faster, and a 400-turn tool-calling conversation builds its prompt in 2 to 4 ms instead of up to 0.6 s.
  • Every request runs inside its memory budget: a wide prefill no longer runs a 32 GB Mac out of memory, a 1536²
    image on Qwen3.8 takes 1.9 GB instead of 22.8 GB, and what does not fit is refused by name instead of crashing.
  • Exact prompts and tool calls on every API: chat, Anthropic Messages and Responses render one prompt and say one
    reply; tool arguments arrive spelled as the model wrote them; GLM's forced tool calls are enforced as Qwen's are; a
    strict JSON schema is enforced as written or refused with a reason; min_p sampling joins every API and the chat
    page.
  • Removed: --mtp-typical, --mtp-tokenv3 and --mtp-greedy-tail, with their settings: speculative decoding
    accepts drafts by the exact rule only, so a seeded request repeats its text. A settings file that still names them
    loads with a warning.

Don't miss a new sushi release

NewReleases is sending notifications on new releases.