github beamivalice/sushi v1.1.0
sushi v1.1.0

latest releases: v1.7.0-dev3, v1.7.0-dev2, v1.7.0-dev1...
one month ago
  • MiMo-V2.6-Flash: sushi's second model. MiMo-V2.6-Flash-Sushi-2.3bpw serves text and image input from one
    resident pack with native MTP, up to its full 1M-token context on a 128 GB Mac.
  • Sushi packs stream from SSD: --ssd-budget-gb N keeps the trunk resident and streams the routed experts from
    SSD, so a Mac with less memory than the pack can serve it; replies are identical to a resident load. Every Sushi
    Qwen pack and MiMo-V2.6-Flash-Sushi-2.3bpw stream, and Sushi-2bpw serves on a 32 GB M1 Max at a 20 GB budget.
  • Faster Flash-Next: on an M5 Max, Sushi-4bpw decodes 8% faster (83 -> 89 tok/s) and prefills a 10k-token prompt
    19% faster (1,796 -> 2,139 tok/s). oMLX's tensor-unit sparse attention now serves prefill from the first sparse
    chunk, GDN prefill runs a software-pipelined recurrence, batched decode overlaps GPU work with graph building, and
    --prefill-decode-share keeps decoders moving while a long prompt prefills. Thanks @STRML and @cowboycoderhq.
  • Better and smaller packs: new expert weights for Sushi-4bpw (KLD 0.0632 -> 0.0592) and Sushi-3bpw
    (0.1047 -> 0.1036), and Sushi-2bpw for 48 GB Macs.
  • Live sessions on /metrics.json: every in-flight request and every cached conversation, with its phase,
    context against the model's limit and the GPU memory its KV holds. Thanks @yoyo930021 and @ddalcu.
  • Updates itself: sushi update installs the newest release after checking its SHA-256 and signature and keeps
    the old one for --rollback; a daily check, one-click update from the chat page, or brew install beamivalice/tap/sushi.
  • Fixes: MiMo long-context memory returns to the OS, SSD prompt-cache restores keep one copy, JSON-constrained
    logprobs pair with their tokens, and a failed model load can be retried after a rescan. Thanks @brandondyal and
    @jasontitus; the EXL3 engine is now a module mlx-serve builds against, thanks @ddalcu.

Don't miss a new sushi release

NewReleases is sending notifications on new releases.