MakazhanAlpamys/Soup v0.70.0

v0.70.0 — Loop Hardening

on GitHub

What's New

Six surfaces that protect the training loop from the failure modes that cost a real GPU-hour. Schema-only release — live trainer-callback / math kernel wiring deferred to v0.70.1 per the project's established stub-then-live cadence (matches v0.50.0 / v0.62.0 / v0.69.0).

• soup train --reward-hack-detector info_rm|rm_ensemble — early-warning when the policy starts gaming the reward model. info_rm tracks the InfoRM Cluster-Separation Index across training (Wang et al. 2024, arXiv 2402.09345); rm_ensemble tracks pairwise variance across an RM ensemble. --reward-hack-halt auto-stops on HACK verdict (≥30% relative drop). Composes with v0.34 soup why so the anomaly explainer can name reward-hacking specifically. Math kernels live now (compute_cluster_separation, compute_rm_ensemble_divergence, classify_hack_signal with OK/WARN/HACK at 0.10 / 0.30); live HF Trainer callback in v0.70.1.

• soup train --uld-strategy wasserstein|topk_align — cross-tokenizer distillation (Boizard et al. 2024, arXiv 2402.12030). The v0.53.2 distillation path assumes student and teacher share a vocabulary; the moment vocabs differ, column-wise logit alignment breaks. wasserstein computes 1D Wasserstein distance between sorted teacher/student logit distributions — no alignment required. topk_align picks top-K teacher logits and maps via BPE overlap. Bounded vocab sizes [1, 262144] cover multilingual SentencePiece + GPT-OSS 200K. Live projection module in v0.70.1.

• soup train --minillm-enabled — MiniLLM-style reverse-KL on-policy distillation (Gu et al. 2024, arXiv 2306.08543). Bundles the three stability tricks scattered across the paper: teacher-mixed sampling (--minillm-teacher-mix-ratio 0.3), length normalisation on rollouts, and a small pretrain-loss anchor (--minillm-pretrain-anchor-weight 0.1 requires --minillm-pretrain-anchor-path pre.jsonl). Cross-validators reject silent no-op combos. Live callback in v0.70.1.

• soup train --rl-checkpoint-save-every-steps N — mid-epoch checkpoint for PPO/GRPO. TorchTune explicitly punts this; Soup ships the real save_state / load_state surface here. Captures optimizer state, optional ref-model, optional rollout/replay buffer; --rl-checkpoint-keep-last 3 retains the last N. Composes with v0.32 spike recovery + v0.40.0 ref-model regen. Live save_state / load_state in v0.70.1.

• soup iterative-dpo --rounds N --pairs-per-round 500 [--plan-only] — sample → RM-score → re-pair → retrain over N rounds. Frozen IterativeDPOPlan with consecutive-round_index invariant + per-round artifact paths. --plan-only renders the canonical plan + exits 0; without it the deferred-live runner exits 3 with explicit v0.70.1 marker. Recipe glue around existing TRL primitives; v0.70.1 wires the live soup train --task dpo subprocess loop.

• soup train --echo-trap-enabled --echo-trap-threshold 0.6 --echo-trap-halt — RAGEN-style detection of trajectory degeneration during multi-turn agent RL (Zhu et al. 2025, arXiv 2504.14437). Pure-Python n-gram repetition rate per trajectory + batch mean. OK / WARN / TRAP taxonomy at 0.30 / 0.60. Composes with v0.53.11 #127 GRPOStabilityCallback — both detectors fire in the same training step without duplicating trajectory collection. Live HF Trainer callback in v0.70.1.

Install / Upgrade

pip install --upgrade soup-cli

Verify:

soup version          # soup v0.70.0
soup iterative-dpo --help
soup --help | grep iterative-dpo

Security

• Validator hardening: every numeric input rejects bool explicitly (bool is subclass of int in Python — matches the project bool-as-int policy across v0.30 / v0.34 / v0.41 / v0.50 / v0.65 / v0.67 / v0.69).
• math.isfinite rejection of NaN / Inf on every float (mirrors v0.32 / v0.41 / v0.50 / v0.62 policy).
• Null-byte rejection + per-field length caps on every string (32 chars for enum names, 4096 chars for paths, 512 chars for model ids).
• Closed allowlists (frozenset) on every enum: SUPPORTED_HACK_DETECTORS, SUPPORTED_ULD_STRATEGIES, _RL_TASKS, VERDICTS (×2).
• MappingProxyType-wrapped metadata registries so callers cannot mutate at runtime.
• DoS caps on every math kernel: RM ensemble capped at 32 RMs; echo-trap _MAX_NGRAM_N=32, _MAX_TRAJECTORY_TOKENS=1_000_000, _MAX_BATCH_TRAJECTORIES=100_000; iterative-DPO _MAX_ROUNDS=100, _MAX_PAIRS_PER_ROUND=1_000_000.
• Frozen dataclasses (post-init validation, FrozenInstanceError on mutation) on every public record.
• Tuples (not Lists) on every frozen collection field.
• No top-level import torch / from torch in any v0.70.0 utility module (4 source-grep regression tests).
• Cross-validators reject silent no-ops: reward_hack_halt=True without detector rejected; uld_top_k with wasserstein rejected; topk_align without uld_top_k rejected; MiniLLM anchor weight ↔ path mutual-requirement; echo_trap_halt=True without echo_trap_enabled rejected; all minillm_* tunables without minillm_enabled=True rejected.
• Task-gates: reward_hack / echo_trap require task in {grpo, ppo}; uld + minillm require task='distill'; rl_checkpoint requires task in {grpo, ppo}. MLX rejected on every Part with distinct error message.
• Deferred-live stubs validate inputs FIRST then raise NotImplementedError("v0.70.1") so misconfigured callers fail fast even though the live callback is not yet wired.
• CLI exit-code policy: typer.Exit(2) for validation rejection, typer.Exit(3) for deferred-live.

Known Limitations

1. Every live wiring is deferred to v0.70.1 — build_reward_hack_callback, build_uld_projection, build_minillm_callback, build_rl_checkpoint_callback, run_iterative_dpo, build_echo_trap_callback all raise NotImplementedError with explicit v0.70.1 markers after validating inputs. Schema accepts every new flag; trainer dispatch / CLI invocations are gated by validators so misconfigured runs fail loudly at config load. Same stub-then-live cadence as v0.50.0 / v0.52.0 / v0.61.0 / v0.62.0 / v0.67.0 / v0.68.0 / v0.69.0.
2. InfoRM detector requires operator-supplied (good, bad) splits — the v0.70.1 callback will inject this from the GRPO advantage signal; the v0.70.0 math kernel accepts the two lists directly.
3. RM-ensemble detector needs ≥2 RM evaluators wired up via existing v0.19.0 / v0.46.0 judge / RM surfaces — the v0.70.1 callback will fan out the prompt to every configured RM.
4. ULD topk_align is the simpler strategy — defers BPE-overlap heuristics to v0.70.1; today the schema gate guarantees top_k is present when chosen.
5. MiniLLM pretrain_anchor_path is shape-only at schema time — cwd-containment + symlink rejection deferred to v0.70.1 disk-read site (matches v0.69.0 build_dag source policy).
6. RL checkpoint state only manifests, not weights — RLCheckpointState.to_dict serialises the manifest; v0.70.1 wires torch.save(state_dict) + tar of optimizer/ref-model/rollout state.
7. Iterative DPO orchestrator is plan-only — soup iterative-dpo --plan-only renders the canonical per-round paths; the v0.70.1 runner will subprocess-invoke soup train --task dpo --resume between rounds (matches v0.53.5 #116 proxy_run_for_weights policy).
8. Echo-trap detector is whitespace-tokenised — n-grams over decoded strings, not tokeniser ids. Sub-word echo trapping is invisible until v0.70.1 adds a tokeniser-aware variant.
9. No multi-detector composition yet — operators can set only one reward_hack_detector at a time; v0.70.1 will allow concurrent InfoRM + RM-ensemble probes.

Test count: 11487 → 11824 (+337 net). 12-invariant self-review against the full project checklist all green across all 6 Parts. Manual CPU smokes for every CLI happy + failure path.

🤖 Generated with Claude Code