beamivalice/sushi v1.0.5

sushi v1.0.5

on GitHub

v1.0.5 — Sushi-2.6bpw at full speed

• Sushi-2.6bpw for 64 GB Macs, as fast as Sushi-3bpw: the new Qwen3.8-Flash-Next pack carries 43.95 GiB of
  weights, 5.4 GiB less than Sushi-3bpw, so a 64 GB Mac serves 250k tokens of context at 8-bit KV. Its experts now run
  on the same fast kernels as Sushi-3bpw's: 18% faster decode and 12% faster prompts on an M5 Max, output unchanged.
• Faster Flash-Next decoding: when a reply copies text already in the conversation (a file returned with an edit,
  a tool call carrying a file), speculative decoding drafts from that text, 16-21% faster file edits and 9-11% faster
  file writes; linear attention, hyper-connections and the sparse-attention indexer also take fewer GPU dispatches.
  Output is unchanged; several of these are ported from mlx-serve, thanks @STRML.
• Long conversations stay cached: by default the RAM prompt cache holds a whole conversation where memory allows,
  a session longer than the cache keeps the longest prefix that fits, and sessions past about 250k tokens restore
  from the SSD cache with half the memory. Ported in part from mlx-serve, thanks @STRML, @brandondyal and
  @celestial-rose.
• Forced tool calls that work: tool_choice required (Anthropic any) or a named function now makes
  Qwen3.8-Flash-Next call one of the declared tools on every API, after its thinking; naming an undeclared function
  is a 400. Streamed and non-streamed thinking are now the same text, and a continued reply resumes after its
  closed think block.
• --preserve-thinking on|off: keep every turn's thinking in the prompt (the default) or only the latest turn's,
  per model in model-settings.json or per request with chat_template_kwargs.preserve_thinking.
• sushi run qwen3.8-flash-next: sushi run and sushi pull name the Sushi packs (:2.6bpw, :4bpw, 3bpw by
  default) instead of models sushi does not serve.
• Thanks, @jasontitus: for this release's faster MTP verification on Flash-Next (hyper-connection weights read
  once per group of draft tokens), the fix that stops long prompts being refused while the GPU finishes earlier work,
  and the build-from-source docs, and, belatedly, for v1.0.4's 4x faster prompts on M1–M4 Macs and parallel n-gram
  reads on 64 GB Macs.