Red and Blue Ocean — closing the fine-tuning flywheel.
Every artifact now has lineage, every training run can be eval-gated,
every quant-check has a verdict, every recipe is shareable. Five Parts
(A-E) ship together as v0.26.0.
What's New
Part A — Local Model Registry
soup registry push/list/show/diff/search/promote/delete— SQLite-backed at~/.soup/registry.dbsoup history <name>— lineage DAG tree- Config diff + per-benchmark eval delta between any two entries
Part B — Eval-Gated Training
training.eval_gate+soup train --gate <suite.yaml>— halt on regression vs baselinesoup eval gate— standalone post-hoc verdict- Baselines via
registry://<id>or JSON file;on_regression: stop|warn|continue
Part C — Trace-to-Preference
soup data from-traces— LangChain + OpenAI-style logs → DPO/KTO pairs- Signals:
thumbs_up,regenerations,user_edit soup data review— preview pairs before training
Part D — Quant-Lobotomy Checker
soup eval quant-check --before X --after Y --tasks t.jsonl— OK / MINOR / MAJOR verdicttable | json | markdownoutput formatsregistry://refs with artifact-kind filter
Part E — Soup Cans
.can= tar.gz of manifest + config + data_refsoup can pack/inspect/verify/fork— shareable + reproducible recipes- Safe extraction:
filter="data"on py3.12+, symlink rejection + commonpath check - Format version pinned to 1
Shared
- New
soup_cli/utils/paths.py— singleis_under_cwdhelper (Windows 8.3 short-name safe) - 43 ready-made recipes now (was 45 counted as "9 new" in v0.25.0; stable counter)
Install / Upgrade
pip install --upgrade soup-cli
# or
docker pull ghcr.io/makazhanalpamys/soup:v0.26.0Security
- Registry: name/tag validation, LIKE-wildcard escaping, 600 perms on POSIX, cycle detection in lineage walks,
resolve()raises on ambiguous prefix - Eval gate: fail-safe error handling (structured errors treated as regressions under
on_regression="stop"); SSRF allowlist on judge URLs - Trace harvester: 100k-line cap; PII warning panel; path containment
- Quant check: path containment; artifact-kind filter avoids picking wrong model
- Soup Cans: HTTPS-only DataRef; 100MB size cap on pack + fork; dunder-key + null-byte rejection in fork modifications (prototype pollution prevention); manifest author length cap + ISO-8601 created_at validation; safe tar extraction
Full per-Part security notes: SECURITY.md.
Known Limitations
These ship as stubs in v0.26.0; live implementations planned for v0.26.1. Each becomes a tracked issue.
soup eval gate --model+soup eval quant-checkuse a stub generator (empty completions); live model loading via transformers/GGUF/AWQ backends comes in v0.26.1.soup data from-traces— judge validation (quality.py) and passive collection viasoup serve --trace-log <dir>not yet implemented.soup can run+soup can publish— one-shotfetch → train → eval → deployorchestrator and HF Hub publish not yet implemented.pack/inspect/verify/forkfully work.soup eval --attach-to-registry <id>flag andsoup exportauto-artifact registration — deferred to v0.26.1 (manually attach viaregistry pushfor now).
Credits
Review-driven hardening across 4 agents (python / code / security / tdd) on every Part. All CRITICAL/HIGH/MEDIUM/LOW findings fixed before release.
Full changelog: v0.25.1...v0.26.0