What's New

v0.69.0 — Data Engineering Pro. Five surfaces that turn dataset prep from "throw a JSONL at the trainer" into a first-class engineering workflow. soup build is dbt-for-SFT — a DAG of ref()-connected dataset transforms with incremental materialization and content-hash row-diff so re-runs re-tokenize only changed rows. soup expect is Great Expectations for chat data with exit code 3 on suite failure (CI gate). soup data gen-magpie plans the Magpie synthetic generator. soup data persona-mix samples a prompt × persona × style matrix with a bundled diversity set + topic-entropy metric. soup data brain-rot implements the arXiv 2510.13928 detector with OK/MINOR/MAJOR verdicts and --strict mode for refusing slop datasets.

• soup build <manifest.yaml> [--dry-run] — dbt-for-SFT DAG parser + topological sort + plan rendering. Each model declares kind: incremental|table|view, a transform, and either a source: (seed) or refs: [...] (derived). Cross-validators reject ambiguous shapes. compute_row_hash + incremental_diff(prev, new) -> IncrementalDiffReport(added, changed, removed, unchanged) are the kernel that lets the live runner (v0.69.1) re-tokenize only changed rows. Live runner deferred to v0.69.1.
• soup expect <data.jsonl> <suite.yaml> — Expectations suite over chat data. Four built-ins: expect_no_pii (reuses v0.47.0 detect_pii — Presidio backend when [data-pro] installed), expect_token_length_between, expect_no_refusal_pattern (reuses v0.56.0 looks_like_refusal), expect_chosen_preferred_over_rejected_by_judge (operator-injected judge callable). _dispatch_expectation passes args raw so per-expectation validators fire authentically — no silent int()/float() coercion bypass. Exit 0 = pass, 2 = validation rejection, 3 = suite failure.
• soup data gen-magpie --base <m> --provider ollama|anthropic|vllm --target N [--plan-only] — Magpie technique (Xu et al. 2024) schema + plan. Live generation deferred to v0.69.1.
• soup data persona-mix --prompts <jsonl> --n N --output <jsonl> — Persona-Hub-style diversity sampler. Bundled 12-persona × 5-style matrix (operator can swap in --personas/--styles JSONLs). Atomic JSONL write via tempfile.mkstemp + os.replace. New compute_topic_diversity Shannon-entropy kernel.
• soup data brain-rot <data.jsonl> [--strict] [--max-major-fraction 0.25] — arXiv 2510.13928 brain-rot detector. Two orthogonal slop scorers (triviality + popularity-signal); worst-signal composite. Same OK/MINOR/MAJOR taxonomy as v0.26 / v0.56 / v0.65 (≥0.85 OK, ≥0.60 MINOR, else MAJOR). --strict exits 3 when MAJOR-row fraction exceeds the threshold so training pipelines can refuse to materialise slop datasets.
• +262 tests (11225 → 11487) across 5 new files. Review-fix coverage: 1 CRITICAL + 4 HIGH + 5 MEDIUM + 4 LOW across 4 review waves (security + code + python + TDD). Lint clean.

Install / Upgrade

```bash
pip install --upgrade soup-cli

or

docker pull ghcr.io/makazhanalpamys/soup:v0.69.0
```

Security

• CRITICAL — centralised TOCTOU helper. Replaces 3 duplicate os.lstat + S_ISLNK + realpath + is_under_cwd blocks in build_dag.load_build_yaml / expectations.load_suite_yaml / expect._load_jsonl_rows with the shared paths.enforce_under_cwd_and_no_symlink helper (v0.59.0 single-source-of-truth policy — mirrors v0.40.6 / v0.53.5 centralisation enforcement).
• HIGH — DoS caps on every new JSONL loader. Brain-rot data path capped at 1 GiB + 1M rows; persona-mix --prompts/--personas/--styles capped at 100 MiB + 100k entries; soup expect data path capped at 1 GiB + 1M rows.
• HIGH — TOCTOU symlink rejection on every write target. persona-mix --output now passes through enforce_under_cwd_and_no_symlink so a pre-placed out.jsonl -> /etc/cron.d/x cannot redirect atomic writes.
• HIGH — Validator-bypass fixed. magpie.build_magpie_config now explicit isinstance(quality_filter, bool); expectations._dispatch_expectation passes raw args to per-expectation validators (was int()/float()-coercing inputs which silently bypassed _check_token_bound bool-rejection).
• MEDIUM — BuildModel cross-validator rejects ambiguous shapes at schema load (refs=() + source=None degenerate; refs + source mutually exclusive).
• MEDIUM — --max-major-fraction CLI-boundary validation: bool / NaN / Inf / out-of-range rejected before any scoring pass.
• All operator-controlled strings pass through rich.markup.escape before any Rich console embedding.
• Closed-allowlist validators across SUPPORTED_MODEL_KINDS / SUPPORTED_EXPECTATIONS / SUPPORTED_MAGPIE_PROVIDERS / BRAIN_ROT_VERDICTS, all bool-first / null-byte / oversize / case-insensitive normalisation.

Known Limitations

1. Live soup build runner deferred to v0.69.1 — run_build raises NotImplementedError; --dry-run produces a fully-validated topological plan today. Live transform-resolver registry, incremental-diff tokenize loop, and DuckDB/SQLite materialisation backend land in v0.69.1.
2. Live Magpie generator deferred to v0.69.1 — run_magpie raises NotImplementedError; --plan-only renders the resolved MagpieConfig Panel today. Live wiring of v0.20.0 Ollama / Anthropic / vLLM providers + chat-template-prefix harvest loop + v0.47 quality-filter chain lands in v0.69.1.
3. BuildModel.source schema-time containment NOT enforced — only shape (non-empty, null-byte-free, length-capped). Live v0.69.1 runner must apply enforce_under_cwd_and_no_symlink before reading source paths. Schema permits relative paths so a build can be planned offline before the data lands on disk.
4. expect_chosen_preferred_over_rejected_by_judge advisory-pass when judge_fn=None — every row gets default score 1.0 (assume chosen wins). Production callers MUST supply a real judge.
5. Brain-rot detector is English-keyword-only — _LOW_EFFORT_TOKENS and _CLICKBAIT_PHRASES are English-language tokens. Operators training on non-English corpora will need to extend or pre-translate.
6. Persona-Hub uses bundled 12-persona × 5-style set by default — full Tencent Persona-Hub corpus (200k personas) NOT bundled. Operators pass --personas <jsonl> + --styles <jsonl> to swap in the full set.

Full changelog

`v0.68.0...v0.69.0` — v0.68.0...v0.69.0