MakazhanAlpamys/Soup v0.71.39

v0.71.39 — CI for weights, not prompts

on GitHub

What's New — CI for weights, not prompts

soup ship's verdict is now something Soup can emit, commit, review, and bind to the exact model that produced it — so a fine-tuning gate renders on every PR instead of asking you to hand-edit a JSON file.

• soup ship --emit-evidence ev.json — re-serialises the scores into the --evidence INPUT schema, so a run's output is replayable as input: feed it back through --evidence (same --forgetting-threshold) and you get an identical verdict. Output is finally input. (#312)
• Commit the gate policy. Put an eval.ship block in your soup.yaml (forgetting_threshold, general_suite, task_mode, judge_model, baseline, task_eval) and soup ship --config soup.yaml reads it — the gate is now reviewable in a PR diff. Explicit CLI flags still win.
• Provenance-bound evidence. --config --emit-evidence STAMPS a config_sha (a semantic, order-insensitive recipe hash) onto the evidence; --config --evidence GATES on it — a PR that changed the recipe but forgot to recompute its evidence is refused (exit 3). Tuning forgetting_threshold never falsely invalidates it (the gate policy is excluded from the hash).
• The verdict on your PR. soup ship --push owner/repo#N posts the SHIP / DON'T-SHIP card as a GitHub comment; soup ci init --config soup.yaml wires the whole loop into CI.

# Producer (train job): stamp the scores with the config that produced them
soup ship --evidence scores.json --config soup.yaml --emit-evidence ship_evidence.json

# Gate (PR CI): refuse evidence that doesn't match the committed config, comment the verdict
soup ship --evidence ship_evidence.json --config soup.yaml --push owner/repo#42
soup ci init --config soup.yaml   # writes .github/workflows/soup-gate.yml

Install / Upgrade

pip install --upgrade soup-cli            # core (no PyTorch)
pip install --upgrade "soup-cli[train]"   # + torch/transformers/peft/trl for training

Exit codes (unchanged from v0.71.38, CI-gateable): 0 = SHIP · 2 = DON'T SHIP · 3 = usage/flag error · 1 = runtime error.

Security

• provenance.config_sha read from untrusted evidence is shape-validated as a hex digest before being echoed — a raw value could otherwise smuggle terminal ESC bytes past rich.markup.escape into a CI log.
• provenance.data_sha hashes the training file through an O_NOFOLLOW fd with a symlink-rejecting containment check and an 8 GiB cap (was an unguarded hash_file, now reachable from a parsed YAML data.train).
• soup ci init path validation now rejects # and the YAML 1.1 line breaks (NEL / LS / PS), closing a plain-scalar comment-truncation of the generated run: step — this also hardens the pre-existing --data / --suite / --evidence args.

Known Limitations

1. Staleness is drift-detection, not tamper-resistance. config_sha is an unkeyed hash, so in a low-trust setup (an external PR editing both soup.yaml and the evidence) an attacker can recompute a matching sha for fabricated scores. The --evidence trust model has always assumed a trusted artifact from your own pipeline; soup attest / soup adapters sign (ed25519) provide forgery-resistance if that's in scope, or produce the evidence server-side in a trusted job.
2. The --push PR comment is best-effort by design — a transport failure (missing token / gh absent / API error) warns and preserves the verdict exit code, so a green gate does not by itself guarantee the comment posted.
3. data_sha is best-effort — omitted (never fatal) for a symlinked / oversized (>8 GiB) / out-of-cwd training file; provenance still carries the config_sha the gate keys on.
4. --config reads gate defaults only — it cannot supply --base / --tuned / --adapter / --push / --emit-evidence, so a hostile config alone can never trigger a file write or an external post.

Full changelog: see CHANGELOG.md. Test count: 16330 → 16387.