github MakazhanAlpamys/Soup v0.34.0
v0.34.0 — Observability & Dev UX

latest releases: v0.77.2, v0.77.1, v0.77.0...
6 months ago

What's New

When training goes wrong, Soup now tells you why instead of dumping a stack trace.

  • soup why — heuristic explainer over the most recent (or named) run. Detects NaN/Inf loss, plateaus (loss flat for ≥30 steps), divergence (loss > 3× initial), persistent high gradient norm, and out-of-band learning rates. Plain-English diagnoses with concrete next steps. Pure rule-based — no model calls.
  • soup tui — full-screen Textual dashboard for runs and live metrics. Two-pane layout (run list + detail), keyboard-driven (r refresh, q quit). New [tui] extra: pip install 'soup-cli[tui]'.
  • .crash bundles — auto-written on training exception to ./.soup-crashes/crash_<utc>_<hex>.crash. Self-contained JSON: classified failure kind (oom/nan/cuda/dataloader/nccl/other), redacted error trace, GPU state at crash time, env summary, last-50 metric rows, recursively redacted config. Ready to attach to a GitHub issue without leaking hf_* / sk-* / Bearer … tokens or \$HOME.
  • soup runs replay <id> — re-renders summary panel + downsampled loss curve (≤2000 points) from SQLite history.
  • soup train --profile — torch.profiler Chrome-trace JSON to <output>/profiles/<run_id>.trace.json. Open in chrome://tracing or Perfetto.
  • Per-run cost — soup runs show prints estimated \$ from captured GPU device × duration. Renders — for CPU / MPS / unknown GPUs (no fabricated zeros).
  • --log-level quiet|normal|verbose|debug — global flag wires a Rich-formatted logger on the soup namespace. DEBUG enables timestamps + module paths.

Net +110 tests (3818 → 3928); all five review-agent waves (python / code / security / tdd / smoke) clean before tag.

Install / Upgrade

```bash
pip install -U soup-cli
pip install 'soup-cli[tui]' # for soup tui
```

Security

  • .crash bundle generator recursively redacts token-shaped strings (hf_* / sk-* / Bearer …) in any captured config and metric tail before serialisation, so a .crash file shared on a public issue cannot leak credentials. output_dir reduced to os.path.basename. Containment via os.path.realpath + commonpath (Windows-safe). ValueError (not PermissionError) on outside-cwd so callers can't silently swallow with except OSError. Filename appends secrets.token_hex(4) to avoid same-second collisions. Bundle truncated to MAX_BUNDLE_BYTES=1_000_000.
  • tracker.get_run LIKE-prefix match escapes % / _ / \ and uses ESCAPE '\\' so a crafted run_id cannot widen the match (mirrors v0.26.0 registry policy).
  • Lazy schema migration tolerates duplicate column race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh).
  • runs.py show/replay/clean: markup_escape on user-supplied run_id before printing; clean switched to project-standard os.path.realpath + is_under_cwd (was Path.resolve() + relative_to(), broken on Windows 8.3 short names).
  • tui_app.py markup_escapes every DB-sourced string (run_id / experiment_name / base_model / task / status) before rendering into Textual widgets.
  • profiling.resolve_trace_path rejects empty / . / .. / / / \\ / null-byte run_id (closes the output_dir/profiles/../trace.json escape) and uses os.path.realpath + is_under_cwd. Profiles dir is created only on successful torch import.
  • run_cost.estimate_run_cost_usd rejects bool in num_gpus (bool is a subclass of int — same defence as v0.30.0 Candidate); duration clamped to [0, 1 year].
  • log_level.parse_log_level rejects non-string and null-byte input; setup_logging replaces the handler on tier change so DEBUG actually toggles show_time / show_path.

Known Limitations

Carry-over from prior releases (infrastructure-blocked, not v0.34.0 follow-ups):

  • #23 — MLX e2e test still gated on a paid macos-14 ARM runner.
  • #39 / #40 — pipeline-parallel + multi-node still gated on torch.pipelining maturity / 2-machine validation hardware.
  • #41 — multi-GPU recipe smoke-train CI still gated on a 4×A100 runner budget.
  • #45 — kernel auto-compose benchmark loop still gated on a CUDA-equipped CI runner.
  • License — Soup is Apache-2.0 (since 2026-04-21). Downstream redistributors must retain NOTICE per §4(d).

PyPI auto-publishes via Trusted Publisher OIDC on tag push. GHCR Docker image builds on release: published.

Don't miss a new Soup release

NewReleases is sending notifications on new releases.