MakazhanAlpamys/Soup v0.71.11

v0.71.11 — GRPO / RL callbacks (live)

on GitHub

The reward-safety, distillation, and RL-loop surfaces that shipped schema-only in v0.70.0 are now live, validated end-to-end on SmolLM2-135M (RTX 3050, 4 GB).

What's New

• soup train --reward-hack-detector info_rm|rm_ensemble (#235) — a live GRPO TrainerCallback watches per-step rewards, computes an InfoRM cluster-separation drop (info_rm) or RM-ensemble divergence (rm_ensemble), classifies OK/WARN/HACK, logs the verdict, and halts on HACK with --reward-hack-halt. rm_ensemble needs ≥2 reward functions. Rewards are captured by wrapping your reward functions through a shared thread-safe buffer — no TRL monkeypatching.
• soup train --echo-trap-enabled (#240) — catches RAGEN-style degenerate repetition in multi-turn agent RL: per-trajectory n-gram repetition → OK/WARN/TRAP against --echo-trap-threshold, halts with --echo-trap-halt.
• soup train --rl-checkpoint-save-every-steps N (#238) — mid-epoch RL checkpoints (adapter + optimizer state + manifest) every N steps so a long GRPO/PPO run survives a crash; pruned to --rl-checkpoint-keep-last.
• task: distill with --uld-strategy wasserstein|topk_align (#236) — live cross-tokenizer Universal Logit Distillation loss, handling student/teacher vocab-size mismatch.
• task: distill with --minillm-enabled (#237) — MiniLLM teacher-mixed length-normalised reverse-KL with an optional pretrain anchor (--minillm-pretrain-anchor-path / --minillm-pretrain-anchor-weight).
• soup iterative-dpo (#239) — runs the full sample → reward-score → build-pairs → DPO-train loop across rounds; each round samples from the previous adapter and trains a fresh LoRA on the harvested pairs. --plan-only renders the plan without running.
• GRPO variant fallback warning (#159) — a --grpo-variant custom loss that falls back to the base trainer (installed TRL renamed the loss inputs) now logs a one-shot WARNING instead of silently degrading.
• In-place reference-model EMA (#160) — --ref-model-ema-alpha updates the reference model in place (no full state_dict round-trip), cheap at 70B+ scale; warns once if 0 parameters overlap.

Install / Upgrade

pip install -U soup-cli
pip install -U 'soup-cli[train]'   # with the training stack

Security

• Iterative-DPO config serialisation uses yaml.safe_dump; per-round paths reject control characters (\n\r\t).
• The MiniLLM pretrain-anchor reader and iterative-DPO prompt reader are cwd-contained with TOCTOU symlink rejection and per-line / per-row byte caps.
• The default iterative-DPO trainer spawns soup train via an argv list (no shell).

Known Limitations

• MiniLLM teacher-mix is an offline distribution blend, not on-policy rollout sampling — the paper's true teacher-mixed sampling needs an autoregressive rollout loop on top of the distill trainer (tracked follow-up).
• Cross-tokenizer ULD handles vocab-size mismatch, not fully-disjoint tokenizers — a genuinely different BPE needs a token-sequence alignment step (tracked follow-up).
• rm_ensemble requires ≥2 reward functions — with one RM the detector is a no-op (divergence is undefined for one RM).
• The #159 fallback WARNING fires only when the installed TRL renames the compute_loss inputs (covered by a unit test; can't be forced from a tiny-model smoke).

Full changelog: https://github.com/MakazhanAlpamys/Soup/blob/main/CHANGELOG.md