MakazhanAlpamys/Soup v0.71.38

v0.71.38 — The gate grows teeth

on GitHub

What's New

soup ship answers the one question that matters after a fine-tune — did my task improve WITHOUT silently breaking everything else? This release makes the "without breaking everything else" half real.

Leg 2 (the catastrophic-forgetting / regression gate that carries the whole SHIP / DON'T-SHIP claim) used to be 15 trivia prompts scored by raw substring — it credited "B" for "Berlin", "3" for "13", and had zero items for tool-calling, safety, or JSON validity.

• A gate that actually catches regressions. A fixed, extraction-based scorer over seven bundled, offline suites: MCQ · arithmetic · tool-calling · JSON validity · safety/refusal. A tune that WINS your task but quietly breaks tool-calling now gets a DON'T SHIP — the old gate waved it through.
• Zero new dependencies, no network. Every suite ships in the wheel and is scored by the pure scorers Soup already has — no lm-eval, no download. Each suite is >20 items so a single-item flip (1/N < 0.05) trips the default threshold instead of being rounded away.
• Distinct exit codes for CI. Usage errors moved off exit 2 to 3, so 2 now means only DON'T-SHIP — a typo'd flag is no longer indistinguishable from a caught regression.
• Riders. Removed the dead --task-mode pairwise "reserved for a later release" gate (it shipped in v0.71.31); soup diagnose now documents seven probes and re-exports all seven score_* functions.

soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
soup ship --evidence scores.json    # offline verdict, no model load

Install / Upgrade

pip install --upgrade soup-cli

Security

• Per-item scorer exceptions are isolated: a RecursionError from json.loads on a pathologically-nested model output scores that one item as a failure rather than aborting the run.
• Model output fed to the scorers is length-capped (64 KiB) before any regex / JSON parse; the bundled-fixture loader is symlink-rejected and size-capped (mirrors the existing behaviour-battery loader).
• The bundled mini_safety suite ships only generic, non-operational harmful-request prompts (refusal probes), consistent with the existing behaviour-battery fixtures.

Known Limitations

1. The behavioural suites are proof-of-mechanism on tiny models. A 135M base scores 0.0 on mini_tool_call (can't tool-call) and 0.0 on mini_safety (no refusal phrasing), so both are the "both-zero → no signal" case; leg 2 only detects a DROP. The caught-tool-call-regression headline is proven deterministically (injected generators over the real bundled fixtures), not with a real tool-capable model (none fits the 4 GB dev box).
2. The fixed scorer is a breaking behaviour change — existing soup ship runs and any committed --baseline captured before v0.71.38 can flip verdicts; recompute baselines.
3. mini_safety scores refusal-rate as an absolute — it catches under-refusal on harmful prompts, not over-refusal on benign prompts.
4. The MCQ extractor rewards letter-answering — a model that answers with the option value ("Au") instead of the letter ("C") scores wrong (consistent across base+tuned, so no false regression).
5. Suites are ~24–40 items — enough to make the 0.05 threshold bite, not a full lm-eval benchmark; --general-suite <lm-eval-names> remains the heavy-tier override.