MakazhanAlpamys/Soup v0.71.27

v0.71.27 — Fine-tune Doctor 🩺

on GitHub

What's New — Fine-tune Doctor 🩺

Two new pure-CPU pre-flight commands that catch the top silent fine-tune failures before you burn a single GPU-hour. No competitor (Unsloth / Axolotl / LlamaFactory) ships any of these.

soup data doctor <data> --model <id|path> — chat-template compatibility report over 8 checks:

• eos_in_labels — the #1 "my model never stops generating" bug: every assistant turn's trained span must actually contain an EOS/EOT token. Checks every turn, not just the last.
• bos_duplication — template + tokenizer both prepending BOS
• system_role — Mistral-style templates that reject a leading system turn
• plus chat_template, template_render, generation_markers, unknown_roles, and truncation_risk (p95 rendered length vs max_length)
• Same OK / MINOR / MAJOR taxonomy as soup diagnose; exit 0 = OK/MINOR, exit 2 = MAJOR.

soup data doctor ... --show-mask N — renders N sample rows with per-token trained / masked colouring through the real collator path (answer-only, per-message-train-field, or RAFT span-mask). Not a reimplementation — the same masking dispatch the report itself uses, so an assistant-mask bug is visible instantly.

soup data lint <data> — preference-data linter for dpo/orpo/simpo/ipo/bco/kto:

• length_bias — chosen systematically longer than rejected (the #1 silent DPO degradation), reported as a Cohen's d effect size
• label_imbalance (KTO), near_duplicates (MinHash), identical_pairs (chosen == rejected), prompt_leak

Validated live against the real HuggingFaceTB/SmolLM2-135M-Instruct tokenizer — that smoke pass caught two genuine bugs synthetic fixtures missed (EOS span-search scope; a real jinja2.exceptions.TemplateError from a no-system-role guard).

Also in this release

• soup diagnose --evidence hardened against a TOCTOU symlink swap (O_NOFOLLOW + fstat-on-open-fd), closing a v0.71.25 known limitation.
• Judge-URL SSRF fix — soup eval gate / soup train --gate no longer accept a hostname prefix bypass like http://localhost.attacker.com. Thanks @CODING-DARSH (#288, closes #283).
• Vocabulary expansion actually applied — data.add_new_tokens / data.new_special_tokens are now honoured in the text SFT path (#287, closes #289) and the vision/audio SFT paths (#291, closes #290); previously accepted by the schema and silently ignored. Thanks @CODING-DARSH.

Install / Upgrade

pip install --upgrade soup-cli            # core CLI (light, no PyTorch)
pip install --upgrade 'soup-cli[train]'   # + training stack

Try it:

soup data doctor train.jsonl --model HuggingFaceTB/SmolLM2-135M-Instruct
soup data doctor train.jsonl --model <id> --show-mask 3
soup data lint prefs.jsonl                # dpo/orpo/kto preference data

Security

• soup data doctor strips C0 control characters (keeping tab/newline/CR) from dataset-derived content before it reaches the terminal — rich.markup.escape() only neutralises [...] tag syntax, not raw ESC bytes, so an untrusted training row could otherwise carry a terminal-injection payload (title-bar / OSC-8 spoofing). --output JSON is unaffected.
• soup diagnose --evidence TOCTOU symlink hardening (see above).
• Judge-URL SSRF hostname-prefix-bypass fix (see above).

Known Limitations

• --show-mask is CPU-rendered, not a live forward pass — it reconstructs the exact labels tensor soup train would produce, but does not run the model, so it can't catch a collator wiring bug that only diverges at train time.
• soup data lint's near-duplicate check is advisory-only without pip install 'soup-cli[data]' — datasketch is optional; the check reports OK with a skip message rather than failing closed (same behaviour as soup data dedup).
• Chat-template checks are heuristic, not exhaustive — the 8 checks cover the failure modes seen across the bundled recipe catalog's tokenizers; an unusual custom template could still slip through. soup data doctor is a pre-flight net, not a formal proof.

Full changelog: https://github.com/MakazhanAlpamys/Soup/blob/main/CHANGELOG.md