MakazhanAlpamys/Soup v0.71.33

v0.71.33 — soup draft: know whether speculative decoding is actually worth it

on GitHub

Everyone tells you to bolt a draft model onto your server for a free speedup. Nobody tells you to measure it first. Now you can — and on the pair we validated, the honest answer was don't.

What's New

soup draft measure — reports a draft's acceptance rate (the fraction of your target's own greedy tokens the draft would have proposed correctly — the metric the Medusa/EAGLE papers report) plus real plain-vs-assisted tok/s. Exit 0 / 2 (below --min-acceptance) / 1, so CI can gate on it.

soup draft measure --target ./my-tuned-model \
  --draft HuggingFaceTB/SmolLM2-135M-Instruct \
  --prompts prod-prompts.jsonl

soup draft distill — distils your tuned target into a tiny draft base (logit KD through the existing task: distill trainer) and emits a dense model, loadable straight as an assistant_model.

Auto-wired into serving — drafts land in a local registry that soup serve --auto-spec consults before the built-in pairing table, so a draft you trained yourself just gets used. Plus soup draft list.

Draft and target must share a tokenizer; a mismatch is refused up front (speculative decoding proposes draft token ids into the target's vocabulary — a mismatched pair silently produces garbage instead of failing).

The measured result — we withdrew our own speedup claim

This release was planned as "make your fine-tune ~1.5–2× faster to serve." The live run says that's false at this scale, so we didn't ship the claim.

On SmolLM2-360M-Instruct ← SmolLM2-135M-Instruct (RTX 3050):

5× more training moved acceptance by 0.0 points, and assisted decoding was a net slowdown. A small same-family draft is already near its capacity ceiling for agreeing with the target, and logit KD can't buy capacity it doesn't have.

That negative result is the feature working. soup draft measure is the number you want before you enable speculative decoding, not after. Run it on your pair rather than trusting anyone's speedup claim — including ours.

Install / Upgrade

pip install --upgrade soup-cli

Fixed

• The hardware-fit gate no longer refuses to train small models. model_size_from_name didn't understand an M (millions) suffix, so SmolLM2-135M fell through to the 7B default, was predicted to need ~14 GB of weights, and was blocked — every draft-sized model hit this. 1.7B was also being read as 7B, while Qwen2.5-7B-Instruct-1M correctly stays 7B (1M is the context).
• Community fixes riding along: batched PRM reward forward (#301, thanks @Ekaanksh-dev), serve --backend vllm no longer force-enables trust_remote_code, multi-adapter serving actually switches adapters, vision datasets reject out-of-directory image paths, KD term aligns with the CE term, and more — see CHANGELOG.

Security

• soup draft distill refuses -o . and won't overwrite a pre-existing non-draft directory without --force (the merge replaces -o wholesale, so this would have deleted a working tree after a long run).
• Path containment now refuses Windows junctions / reparse points, which S_ISLNK misses — shutil.rmtree would otherwise follow one outside cwd.
• A locally-registered draft is strict-safetensors-checked before serve loads it, so a poisoned registry entry can't become a pickle-deserialization RCE.
• Generated training configs escape every user-supplied string (a newline in --target could otherwise inject sibling YAML keys).

Known Limitations

1. Distillation did not improve acceptance on the validated pair. Whether it pays off for a genuinely diverged fine-tune, or a larger target/draft pair, is unproven on a 4 GB box.
2. Speculative decoding was a net slowdown (0.55–0.64×) on a small same-family pair — the draft's forward pass costs more than the tokens it saves.
3. Acceptance is teacher-forced greedy agreement — exact, deterministic, and the right number for ranking drafts, but not the accepted-token count of a sampling run.
4. Same-tokenizer only. Cross-tokenizer drafts (ULD-aligned + universal assisted decoding) are deferred.