MakazhanAlpamys/Soup v0.61.0

v0.61.0 — Unlearning & Knowledge Edit

on GitHub

What's New

Two of the most under-served axes in fine-tuning land as first-class trainers: unlearning (NPO / SimNPO / RMU — the legal-liability axis upstream TRL avoids) and knowledge editing (ROME / MEMIT / AlphaEdit — research-coded everywhere, productized nowhere). Hospital data teams handling a GDPR deletion request, labs patching factual errors without retraining, and security teams responding to a CSAM/PII leak all get a CLI workflow that composes with the existing Registry + eval-gate + diagnose surface.

Schema-only release: live trainer + kernel wiring lands in v0.61.1 (matches the established v0.50.0 / v0.52.0 / v0.53.0 stub-then-live cadence).

• task='unlearn' + NPO / SimNPO / RMU. training.unlearn_method selects the backend; data.forget_set (rows to unlearn) + data.retain_set (capability anchors) validated at schema-load. UnlearnTrainerWrapper stub accepts forward-compat kwargs so v0.61.1 wiring is purely additive.
• soup eval unlearning <run-id> --benchmark tofu|muse|wmdp — three orthogonal scores on the project's OK / MINOR / MAJOR taxonomy: Forget Quality (pre/post forget-loss ramp), Model Utility (retain accuracy preserved), PrivLeak (MIA AUC distance from 0.5). Bundled TOFU mini-fixture; MUSE + WMDP loaders v0.61.1.
• soup edit set --base <m> --method rome|memit|alphaedit --subject "..." --target "..." [--plan-only] — surgical locate-and-edit. Per-method default MLP layer. --plan-only validates + prints the resolved EditPlan and exits 0 today; without it the CLI surfaces the deferred-live NotImplementedError at exit code 3 (distinct from validation rejection exit 2).
• Sequential edit governor + norm-blowup detection. Auto-switches ROME → AlphaEdit at edit-count threshold (default 10) AND on detected Frobenius blowup. Refuses further edits past the max cap or after BLOWUP via GovernedEditError so callers can distinguish governance refusals from other failures.
• soup edit diff <before-run> <after-run> --probes p.jsonl — knowledge-injection diff visualizer. Frozen DiffReport; probe-file loader is cwd-contained, symlink-rejected, size-capped (16 MiB / 1000 rows / 4096-char per prompt with WARNING-log truncation). Shape + table renderer ship now; live before/after generation v0.61.1.

Install / Upgrade

pip install --upgrade soup-cli
soup version  # should print v0.61.0

Security

• TOCTOU lstat-before-write on every operator-supplied path via shared paths.atomic_write_text helper.
• load_evidence_file + load_probes cwd-contained + os.lstat + S_ISLNK rejection (CRITICAL: rejected BEFORE realpath, which would silently resolve symlinks — CI caught this on POSIX).
• Frozen dataclasses for every report/plan surface; EditGovernor counters declared as real fields (slots-safe, asdict / replace compatible).
• run_unlearn_eval distinguishes "missing key → neutral OK" from "present-but-invalid → ValueError" so operator typos surface instead of silently scoring OK.
• registry_id validated BEFORE the deferred attach so a crafted null-byte / oversize id can't crash the v0.61.1 hook.
• Exit-code policy: eval unlearning exits 2 on MAJOR (CI gate); edit set exits 3 on deferred-live NotImplementedError (distinct from exit 2 = validation rejection).
• importlib.resources for TOFU fixture path (zipapp / namespace-package safe).
• Test count: 9446 → 9571 (+125 net).

Known Limitations

1. Every live wiring (NPO/SimNPO/RMU loss kernels, ROME/MEMIT/AlphaEdit closed-form updates, UnlearnTrainerWrapper.setup) deferred to v0.61.1 — schema accepts every flag today so soup.yaml written now works the moment v0.61.1 ships.
2. MUSE + WMDP fixtures NOT bundled (only TOFU mini-set); operators wanting MUSE / WMDP today supply pre-computed evidence JSON via --evidence.
3. soup edit diff is shape-only — live model loading + before/after generation lands in v0.61.1.
4. Registry-attach for edited models is plan-only — v0.61.1 wires the parent_adapter lineage write.
5. EditGovernor is single-process (no SQLite persistence).
6. Norm-blowup detection depends on operator-supplied norm_delta — v0.61.1's live apply_edit will compute + thread it through.
7. Unlearn task gated to non-mlx (mlx-lm doesn't expose per-token logp scoring needed for NPO / SimNPO).
8. apply_edit does not consult EditGovernor automatically — v0.61.1 wires the optional governor: EditGovernor kwarg.

Full release notes: .claude/CLAUDE.md v0.61.0 entry.

🤖 Generated with Claude Code