MakazhanAlpamys/Soup v0.71.36

v0.71.36 — Data Moat II: semantic dedup, topic map, canaries, replay

on GitHub

A semantic layer over your training data, plus two tools for what a fine-tune forgets and what it leaks.

What's New

soup data dedup --semantic — dedup by meaning, not shared tokens. Catches reworded duplicates that MinHash's shingling scores as distinct (measured 0.88–0.91 cosine on rewordings it missed). Zero new dependencies — reuses transformers from the [train] extra.

soup data topics <data> — cluster your data, label each cluster with c-TF-IDF terms, and get a coverage table plus a warning for thin topics:

│ function / python / code │ 3444 │    82.0% │
│ theorem / proof / math   │  252 │     6.0% │
│ refuse / harmful / safe  │   42 │     1.0% │
topic 'refuse / harmful / safe' is thin: 1.0% of rows (42/4200)

soup data canary insert|check — prove whether a model memorized your data. Insert K high-entropy secrets, then rank each one's loss against never-inserted controls drawn from the same space and sharing the same carrier prompt. Exit 2 on a leak, so CI can gate. Measured on SmolLM2-135M: a memorized set lands at percentile 0.0 (loss 1.7–2.5) against a clean model's 4.1–6.2.

soup train --replay old.jsonl --replay-ratio 0.1 — continual-learning rehearsal. Interleaves a seeded sample of your old data so the new task doesn't erase it. r is a share of the final mixed set, rows are interleaved rather than appended, and an undersized pool reports a shortfall instead of repeating rows.

The honest result: the paraphrase headline was withdrawn

Semantic dedup was planned as a paraphrase catcher. Measured with all-MiniLM-L6-v2, it isn't one — and the numbers say so plainly:

A real paraphrase scores lower than two genuinely different instructions. Paraphrases span 0.49–0.76; distinct rows span 0.54–0.76. They overlap — no threshold separates them, so lowering --threshold to chase paraphrase recall deletes real training rows. The default (0.8) is deliberately conservative, and the shipped claim is the narrower one the data supports: it catches rewordings MinHash misses.

We'd rather ship the measured claim than the planned one.

Fixed

• The hardware-fit gate refused to train any model you merged yourself. A merged checkpoint (soup merge -o ./mymodel) has no size marker in its name, so the size guesser returned its 7B default, predicted ~16 GB of VRAM and refused — blocking soup merge → train-from-merged entirely. Local checkpoints are now measured from their safetensors header (0.135B actual vs 7.0B guessed).
• pip install 'soup-cli[extra]' hints printed without the extra. Rich ate the bracket, so 17 "install the missing dependency" sites — including Typer help — told you to run pip install 'soup-cli', which succeeds and leaves the feature just as broken. Affected [eval], [data], [serve], [ui], [tui], [compile], [mcp], [carbon] and others.
• The canary verdict cried wolf. "MAJOR if any canary looks memorized" ignores that with 16 canaries, chance alone dips one below the line ~15% of the time — and MAJOR exits 2. The verdict now tests the count against a binomial tail (α=0.05): false alarms on a clean model drop from 15% to 1.2%, while a partial leak (2 of 10) still fires.
• Replay rows bypassed the image/audio path-traversal validation the primary dataset receives.

Install / Upgrade

pip install --upgrade soup-cli
pip install --upgrade 'soup-cli[train]'   # needed for --semantic and data topics

Security

• The canary manifest is the sensitive artifact, not the dataset: anyone holding it can reproduce the secrets. It is written 0600 on POSIX, cwd-contained, symlink-refused, and entry-capped. check --output embeds the same secrets and gets the same treatment plus a warning.
• Manifest-derived strings are stripped of control bytes before reaching the terminal — rich.markup.escape() neutralises [...] but passes a raw ESC through, and the manifest is designed to be shared.
• canary insert writes the manifest before the poisoned dataset, so a failure can't leave canaried data on disk with nothing to identify it.
• Replay file paths are cwd-contained and get the same media-traversal validation as the primary dataset.

Known Limitations

• Semantic dedup is not a paraphrase detector (see above). Whether a stronger embedding model separates the distributions is unmeasured.
• Replay is proof-of-mechanism at 135M + LoRA. Direction correct — 7% better old-task retention than a no-replay control — but forgetting without it was only +4%, i.e. mild. The effect size at full fine-tuning or 7B+ is unproven on a 4 GB box.
• Canary exposure is the sampled-control approximation, not full-space rank enumeration. "No exposure" is not proof of no memorization.
• data topics / dedup --semantic require [train] and download an embedding model. Plain MinHash dedup stays on the light core.
• Replay v1 is sft/pretrain only and rejects packing/multipack — both concatenate rows into fixed blocks, so the ratio stops being meaningful.
• The O(n²) semantic dedup and k-means carry row caps that refuse loudly rather than subsampling silently.

Tests: 16001 → 16283. Full suite green (16154 passed, 127 skipped, 0 failed); CI green across 11 jobs.