beamivalice/sushi v1.4.1

sushi v1.4.1

on GitHub

v1.4.1 — Faster DeepSeek prefill, DeepSeek with no flags

• DeepSeek-V4.1 prefills up to 47% faster and decodes up to 13% faster: on an M5 Max with the 1.75bpw pack at 128K
  tokens of code, prefill goes from 829 to 1,217 tok/s and decode from 63.7 to 72.0. The attention core runs on the
  matrix units, the shared expert rides the routed experts' launches, a prompt chunk another chunk follows skips the
  decoder tail, and the hyper-connection mix runs on a better tile; only the attention change moves output, at rounding
  level with KLD unchanged within noise.
• DeepSeek-V4.1 runs with no flags: a 1,000,000-token context, a 256,000-token answer budget, and an SSD budget from
  the Mac's RAM: 80 GB on a 96 GB Mac, 96 GB on a 128 GB Mac (108 GB with iogpu.wired_limit_mb at 120000 or more),
  fully resident on a 192 or 256 GB Mac. --ctx-size, --max-tokens and --ssd-budget-gb still override them.
• DeepSeek-V4.1 needs less memory and runs on more Macs: prefill takes about 8 GB less, the token embedding is read
  from SSD, the routers stay as stored and the image tower is half the size, all returned to the expert cache; M1 and M2
  Macs no longer abort, seeded requests repeat exactly, and a JSON schema keeps DSpark on.
• Qwen and GLM do less work per token with byte-identical output: fewer GPU launches per MoE layer, agent turns no
  longer prefill the previous thought again, SSD restore points every 8192 tokens up to 1M, and a reasoning budget keeps
  MTP and DSpark speculating through the answer.
• Streams match the non-streamed answer in more cases: <think> tags quoted in code and thoughts re-opened
  mid-answer stream as the non-streamed reply holds them, five more malformed tool-call shapes parse, /v1/responses
  keeps the model's reasoning across a tool loop, and a chat template that fails is refused by name.
• Server additions: /v1/messages/count_tokens for Claude Code, POST /metrics/enable and SSD-cache write metrics
  on the dashboard, a queued request says so, GIF and BMP images, idle models unload before a long prompt is refused,
  and a running stream keeps a quarter of the time while another request prefills.
• Code for models sushi does not serve is gone: /v1/embeddings, the nvfp4 quantization mode, GGUF discovery and
  unused tool-call dialects; a GGUF path is still refused by name.
• Thanks to @stooit for the batched-decode ! fix (#74), @cosmic-quokka for count_tokens, @cyrilmhansen for the
  cache-write metrics request, and @STRML for the plugin attention work this release ports.