What's New
When training goes wrong, Soup now tells you why instead of dumping a stack trace.
soup why— heuristic explainer over the most recent (or named) run. Detects NaN/Inf loss, plateaus (loss flat for ≥30 steps), divergence (loss > 3× initial), persistent high gradient norm, and out-of-band learning rates. Plain-English diagnoses with concrete next steps. Pure rule-based — no model calls.soup tui— full-screen Textual dashboard for runs and live metrics. Two-pane layout (run list + detail), keyboard-driven (rrefresh,qquit). New[tui]extra:pip install 'soup-cli[tui]'..crashbundles — auto-written on training exception to./.soup-crashes/crash_<utc>_<hex>.crash. Self-contained JSON: classified failure kind (oom/nan/cuda/dataloader/nccl/other), redacted error trace, GPU state at crash time, env summary, last-50 metric rows, recursively redacted config. Ready to attach to a GitHub issue without leakinghf_*/sk-*/Bearer …tokens or\$HOME.soup runs replay <id>— re-renders summary panel + downsampled loss curve (≤2000 points) from SQLite history.soup train --profile— torch.profiler Chrome-trace JSON to<output>/profiles/<run_id>.trace.json. Open inchrome://tracingor Perfetto.- Per-run cost —
soup runs showprints estimated\$from captured GPU device × duration. Renders—for CPU / MPS / unknown GPUs (no fabricated zeros). --log-level quiet|normal|verbose|debug— global flag wires a Rich-formatted logger on thesoupnamespace. DEBUG enables timestamps + module paths.
Net +110 tests (3818 → 3928); all five review-agent waves (python / code / security / tdd / smoke) clean before tag.
Install / Upgrade
```bash
pip install -U soup-cli
pip install 'soup-cli[tui]' # for soup tui
```
Security
.crashbundle generator recursively redacts token-shaped strings (hf_*/sk-*/Bearer …) in any captured config and metric tail before serialisation, so a.crashfile shared on a public issue cannot leak credentials.output_dirreduced toos.path.basename. Containment viaos.path.realpath + commonpath(Windows-safe).ValueError(notPermissionError) on outside-cwd so callers can't silently swallow withexcept OSError. Filename appendssecrets.token_hex(4)to avoid same-second collisions. Bundle truncated toMAX_BUNDLE_BYTES=1_000_000.tracker.get_runLIKE-prefix match escapes%/_/\and usesESCAPE '\\'so a craftedrun_idcannot widen the match (mirrors v0.26.0 registry policy).- Lazy schema migration tolerates
duplicate columnrace when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). runs.pyshow/replay/clean:markup_escapeon user-suppliedrun_idbefore printing;cleanswitched to project-standardos.path.realpath + is_under_cwd(wasPath.resolve() + relative_to(), broken on Windows 8.3 short names).tui_app.pymarkup_escapes every DB-sourced string (run_id / experiment_name / base_model / task / status) before rendering into Textual widgets.profiling.resolve_trace_pathrejects empty /./..///\\/ null-byterun_id(closes theoutput_dir/profiles/../trace.jsonescape) and usesos.path.realpath + is_under_cwd. Profiles dir is created only on successful torch import.run_cost.estimate_run_cost_usdrejectsboolinnum_gpus(bool is a subclass of int — same defence as v0.30.0Candidate); duration clamped to[0, 1 year].log_level.parse_log_levelrejects non-string and null-byte input;setup_loggingreplaces the handler on tier change so DEBUG actually togglesshow_time/show_path.
Known Limitations
Carry-over from prior releases (infrastructure-blocked, not v0.34.0 follow-ups):
- #23 — MLX e2e test still gated on a paid
macos-14ARM runner. - #39 / #40 — pipeline-parallel + multi-node still gated on
torch.pipeliningmaturity / 2-machine validation hardware. - #41 — multi-GPU recipe smoke-train CI still gated on a 4×A100 runner budget.
- #45 — kernel auto-compose benchmark loop still gated on a CUDA-equipped CI runner.
- License — Soup is Apache-2.0 (since 2026-04-21). Downstream redistributors must retain
NOTICEper §4(d).
PyPI auto-publishes via Trusted Publisher OIDC on tag push. GHCR Docker image builds on release: published.