MakazhanAlpamys/Soup v0.48.0

v0.48.0 — Adaptive Training (BETA)

on GitHub

What's New

v0.48.0 — Adaptive Training (BETA): Two research-grade techniques that close the "10% of data beats 100%" gap — dynamic curriculum re-weighting and a Bayesian data-mixing optimiser. Both ship BETA:-flagged until reference-benchmark validation lands; the schema, math kernels, and CLI surface are stable, the live HF Trainer callback + scikit-optimize backend wire-in lands in v0.48.1.

• training.curriculum_dynamic: true layers on the static curriculum bucketer. Every N steps the trainer aggregates per-sample loss + grad-norm into a per-bucket uncertainty signal, runs it through a softmax with floor (water-filling so no bucket ever drops below curriculum_dynamic_floor), and re-weights the sampler. Multi-rank launches must wire an all_reduce hook on per-bucket stats — validate_distributed_curriculum rejects un-coordinated multi-rank runs upfront so the well-known DDP-divergence footgun fails fast.
• soup runs curriculum-curve <run_id>. ASCII visualiser of bucket-weight evolution over training. Containment + TOCTOU symlink rejection + 50 MB / 100k-line caps.
• soup data mix --optimize --budget 1h --datasets a.jsonl,b.jsonl,.... Runs N short proxy runs over candidate mixture weights and writes a mix_recipe.yaml you can splice into soup.yaml under data.interleave. Per-candidate proxy failures are isolated (logged at DEBUG, sentinel high loss recorded) so a single OOM combo does not kill the whole search. partial=True surfaced when the budget cap trips mid-loop.
• soup data mix --apply <recipe.yaml>. Re-loads a recipe and prints the canonical data.interleave block. Cwd containment + symlink rejection + 256 KB file cap; yaml.safe_load only.
• +168 net new tests (6242 → 6410). Floor-strict invariant, water-fill correctness, frozen-dataclass mutations, simplex constraint on MixCandidate, isolated-proxy-crash regression, 50 MB and 100k-row caps, POSIX symlink rejection, KeyboardInterrupt propagation, every cross-validator branch.

Install / Upgrade

pip install --upgrade soup-cli==0.48.0

Security

• compute_bucket_weights water-fill design fix (code-review HIGH): removed harmful trailing renorm that could push elements below floor when accumulated float error left sum slightly > 1.0.
• os.lstat on original path before realpath (security HIGH, post-CI hotfix): the v0.48.0 first-cut lstat'd the realpath, which had already followed the symlink. Fixed in all 4 path-input sites (validate_datasets, write_mix_recipe, load_mix_recipe, soup runs curriculum-curve). Matches v0.46.0 _reject_symlink_target pattern.
• DoS caps: _MAX_HISTORY_ROWS=100_000 on render_curve + parse_history_jsonl; 50 MB file cap + 100k-line streaming cap on curriculum-curve history reader; _MAX_DATASETS=32 on soup data mix.
• YAML key injection defended at render_mix_recipe_yaml (rejects newlines / null bytes / oversize dataset paths).
• Proxy crash isolation: run_mix_optimizer per-candidate exceptions logged at DEBUG + sentinel _MAX_LOSS + continue; KeyboardInterrupt/SystemExit re-raised. Matches v0.33.0 #47 CrossDocCollator / v0.40.3 judge_filter_pairs policy.

Known Limitations

1. Live HF Trainer callback for dynamic curriculum deferred to v0.48.1 — math kernel, schema gates, and visualiser ship; the callback wiring with torch.distributed.all_reduce of per-bucket stats lands in v0.48.1. Same stub-then-live pattern as v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro.
2. Curriculum-Aware multi-trainer restricted to sft/pretrain — non-SFT trainers rejected at SoupConfig level because per-sample loss semantics differ. Multi-trainer expansion deferred to v0.48.1.
3. soup data mix live proxy training deferred to v0.48.1 — the CLI ships with a synthetic offline proxy (quadratic penalty around uniform mixture) so the budget tracker, optimiser surface, and recipe writer can be exercised without GPUs. Live integration with a short soup train proxy run is the v0.48.1 deliverable.
4. scikit-optimize backend opt-in — when scikit-optimize is not installed, the optimiser falls back to a deterministic Dirichlet sampler. Wrapping skopt.Optimizer behind the OptimizerProtocol ducktype is also in v0.48.1.
5. MixOptimizationReport.elapsed_seconds includes failed-candidate time — acceptable for v0.48.0; the budget tracker still trips correctly.
6. Both features ship BETA:-prefixed in CLI help and field descriptions until reference-benchmark validation lands in v0.48.1.

🤖 Generated with Claude Code