github MakazhanAlpamys/Soup v0.26.0
v0.26.0 — Red and Blue Ocean (closing the flywheel)

latest releases: v0.78.0, v0.77.2, v0.77.1...
7 months ago

Red and Blue Ocean — closing the fine-tuning flywheel.

Every artifact now has lineage, every training run can be eval-gated,
every quant-check has a verdict, every recipe is shareable. Five Parts
(A-E) ship together as v0.26.0.

What's New

Part A — Local Model Registry

  • soup registry push/list/show/diff/search/promote/delete — SQLite-backed at ~/.soup/registry.db
  • soup history <name> — lineage DAG tree
  • Config diff + per-benchmark eval delta between any two entries

Part B — Eval-Gated Training

  • training.eval_gate + soup train --gate <suite.yaml> — halt on regression vs baseline
  • soup eval gate — standalone post-hoc verdict
  • Baselines via registry://<id> or JSON file; on_regression: stop|warn|continue

Part C — Trace-to-Preference

  • soup data from-traces — LangChain + OpenAI-style logs → DPO/KTO pairs
  • Signals: thumbs_up, regenerations, user_edit
  • soup data review — preview pairs before training

Part D — Quant-Lobotomy Checker

  • soup eval quant-check --before X --after Y --tasks t.jsonl — OK / MINOR / MAJOR verdict
  • table | json | markdown output formats
  • registry:// refs with artifact-kind filter

Part E — Soup Cans

  • .can = tar.gz of manifest + config + data_ref
  • soup can pack/inspect/verify/fork — shareable + reproducible recipes
  • Safe extraction: filter="data" on py3.12+, symlink rejection + commonpath check
  • Format version pinned to 1

Shared

  • New soup_cli/utils/paths.py — single is_under_cwd helper (Windows 8.3 short-name safe)
  • 43 ready-made recipes now (was 45 counted as "9 new" in v0.25.0; stable counter)

Install / Upgrade

pip install --upgrade soup-cli
# or
docker pull ghcr.io/makazhanalpamys/soup:v0.26.0

Security

  • Registry: name/tag validation, LIKE-wildcard escaping, 600 perms on POSIX, cycle detection in lineage walks, resolve() raises on ambiguous prefix
  • Eval gate: fail-safe error handling (structured errors treated as regressions under on_regression="stop"); SSRF allowlist on judge URLs
  • Trace harvester: 100k-line cap; PII warning panel; path containment
  • Quant check: path containment; artifact-kind filter avoids picking wrong model
  • Soup Cans: HTTPS-only DataRef; 100MB size cap on pack + fork; dunder-key + null-byte rejection in fork modifications (prototype pollution prevention); manifest author length cap + ISO-8601 created_at validation; safe tar extraction

Full per-Part security notes: SECURITY.md.

Known Limitations

These ship as stubs in v0.26.0; live implementations planned for v0.26.1. Each becomes a tracked issue.

  1. soup eval gate --model + soup eval quant-check use a stub generator (empty completions); live model loading via transformers/GGUF/AWQ backends comes in v0.26.1.
  2. soup data from-traces — judge validation (quality.py) and passive collection via soup serve --trace-log <dir> not yet implemented.
  3. soup can run + soup can publish — one-shot fetch → train → eval → deploy orchestrator and HF Hub publish not yet implemented. pack/inspect/verify/fork fully work.
  4. soup eval --attach-to-registry <id> flag and soup export auto-artifact registration — deferred to v0.26.1 (manually attach via registry push for now).

Credits

Review-driven hardening across 4 agents (python / code / security / tdd) on every Part. All CRITICAL/HIGH/MEDIUM/LOW findings fixed before release.

Full changelog: v0.25.1...v0.26.0

Don't miss a new Soup release

NewReleases is sending notifications on new releases.