MakazhanAlpamys/Soup v0.58.0

v0.58.0 — soup loop: CLI-first data flywheel

on GitHub

What's New

soup loop — the production data flywheel, all from the CLI. Every competitor stops at training. Web tools (Langwatch, Helicone, Galileo) monitor production but don't retrain. Nobody runs the full production traces → preference pairs → Eval-Gated DPO → canary deploy → rollback cycle from a single CLI on a laptop. v0.58.0 connects 8 of Soup's existing uniques (v0.26 Trace-to-Preference + Eval-Gated Training + Registry lineage + Quant-Lobotomy verdicts + Soup Cans + v0.25 Autopilot + v0.54 Advise + v0.55 Eval Design + v0.56 Diagnose) into one workflow.

• soup loop init <served-model> --eval <suite> --baseline registry://<id> — one-time setup writing .soup/loop.yaml (atomic write, cwd-contained, lstat-based symlink-rejected — no lexists race).
• soup loop status — counters for traces collected / pairs distilled / runs gated / adapters shipped, plus monthly spend vs. budget and runs-today vs. daily cap.
• soup loop watch [--detach] [--max-iterations N] — foreground or background daemon. --detach spawns python -m soup_cli.cli loop watch --foreground via argv-list subprocess.Popen (no shell). State reloaded every iteration so external pause/resume takes effect immediately.
• soup loop pause / soup loop resume — atomic status flip via immutable LoopState.with_status API.
• soup loop canary <new-adapter> --traffic 5% --autoroll-on-regress — deterministic SHA-256 hash routing (_HASH_MOD=10000 buckets, ±0.01% split granularity). Sticky-on-rollback: a flaky verdict can't ping-pong traffic.
• soup loop replay [<iteration-id>] — list or pretty-print iteration manifests under .soup-loops/<iter-id>/iteration.json.
• Budget guardrails. --monthly-budget 50usd composes with v0.34 per-run cost; daemon refuses next iteration when projected spend would exceed cap. --max-runs-per-day 3 defends against runaway proxy loops with UTC-day rollover detection.

Install / Upgrade

```bash
pip install -U soup-cli
```

Security

Three review waves fixed 1 CRITICAL + 7 HIGH + 9 MEDIUM + 2 LOW total before tag:

• python-reviewer wave 1 — BucketStats verdict comparison moved inside the threading lock; TOCTOU lstat-before-write (no lexists guard) on `_check_path` + `init_state`; NUL-byte rejection on the SHA-256 hash input.
• code-review wave 2 — watch preserves paused status across daemon restart (was unconditionally flipping to running on entry and stopped on exit); budget-skipped iterations no longer produce manifests under .soup-loops/; canary_autoroll_on_regress flag now persisted to LoopState (was dropped); route() uses math.ceil not round for sub-bucket fraction predictability; parse_budget_string(\"usd\") raises friendly "must include a numeric value" error; list_iterations swallows OSError from os.listdir.
• security + tdd wave 3 — _check_dir TOCTOU mirrors _check_path pattern; exact-boundary tests at _MAX_STR_FIELD=512 and _MAX_FILE_BYTES=1 MiB; bool-rejection on 4 numeric counters; empty-string rejection on 3 optional-str fields.
• subprocess.Popen uses argv list (no shell) for loop watch --detach. All paths are cwd-contained + symlink-rejected. Atomic state-file writes via tempfile.mkstemp + os.replace with POSIX 0o600 perms. 1 MiB cap on state file. Sticky-on-rollback prevents canary flap.

Known Limitations

1. Stage callbacks ship as no-op stubs — default_harvest / default_train / default_gate / default_deploy / default_cost are deterministic placeholders so the daemon runs end-to-end on CPU without a model. Wiring v0.26 trace-to-pref + eval-gate + v0.30 multi-adapter deploy is operator-driven via WatchConfig; pre-wired versions land in v0.58.1.
2. Soup Can packaging deferred to v0.58.1 — iteration manifests are laid out so a v0.26 Soup Can wrapper hook can ship without re-shaping files, but Registry-DAG append + automatic soup history lineage is the v0.58.1 deliverable.
3. `--detach` is a single-process subprocess — no setsid / nohup-style daemonization. On Linux pair with systemd or tmux; on Windows the subprocess survives the parent CLI exit.
4. No automatic budget refill on UTC month rollover — spent_this_month_usd is reset by the operator (or by writing a fresh loop.yaml); the daemon does not auto-detect month boundaries.
5. Direct `code-reviewer` / `security-reviewer` agents thrash on the project's 800+ KB release-notes history; the `general-purpose` agent with focused "read these 7 files only" prompts produced equivalent findings (3 review waves total).

Test count: 8998 → 9193 (+195 net in tests/test_v0580.py). 188 pass + 1 POSIX-only symlink test skipped on Windows.