MakazhanAlpamys/Soup v0.71.40

v0.71.40 — soup reward synth: generate a reward verifier from your data

on GitHub

What's New

soup reward synth — point it at a JSONL of reference (gold) outputs and it infers a deterministic reward verifier, writes a readable / committable .py reward function, and — the part nobody else does — refuses to emit one that can't tell your references from auto-generated bad answers. Nothing in TRL / Unsloth / Axolotl / OpenRLHF synthesizes a reward.

• Four families, auto-detected (or --kind): numeric (\boxed{} / #### / last-number, exact or --tolerance), json_schema (induced keys + types + required), regex (positional char-classes), tool_call (per-tool required/allowed argument binding — a call can't borrow another tool's keys).
• A calibration report is the moat — the verifier is loaded back and run against its own references (must accept ≥ 90%) and auto-perturbed negatives (must reject). A degenerate always-accept verifier is refused (exit 2), never silently shipped. --plan-only previews; --output-report saves the JSON.
• No new exec surface — the emitted file rides the existing reward_fn: reward.py path; you read, edit, commit, and diff it.
• Reward ensembles now train — reward_fn: "accuracy,format" loads as multiple rewards (and unlocks the rm_ensemble reward-hack detector). A recipe that shipped exactly this used to crash with Unknown reward function. (#311)

soup reward synth references.jsonl -o reward.py --output-report calib.json
# then: training: { reward_fn: reward.py }   (or an ensemble: reward_fn: "accuracy,format")

Install / Upgrade

pip install -U soup-cli

Security

• The emitted verifier bakes the induced spec via repr() (untrusted gold strings can't break out); the one value formatted raw into the emitted file — the -o path shown in its header — is sanitized so a crafted path can't inject code into the later-exec'd file.
• References / output / report paths are cwd-contained, O_NOFOLLOW, and size/row-capped; the emitted regexes are linear (no ReDoS).
• training.reward_fn gains input validation (null-byte / blank / oversize / empty-comma-segment).

Known Limitations

• v1 is deterministic verifiers only (numeric / json_schema / regex / tool_call); judge-based is a future tier and LLM-codegen is out of scope.
• The numeric verifier prefers answer markers (\boxed{} / ####) then falls back to "last number" — mark your answers (standard RLVR practice); a correct answer followed by an incidental trailing number in un-marked free text can misscore.
• Calibration proves discrimination, not full correctness — it proves the verifier separates references from perturbed negatives; the emitted .py is readable and editable precisely so a human reviews it.
• The GRPO comma-ensemble was validated proof-of-mechanism on SmolLM2-135M; a full multi-reward run on a real reward model / 7B+ is tracked in the existing scale ask (#286).