MakazhanAlpamys/Soup v0.66.0

v0.66.0 — Post-train X-rays

on GitHub

What's New

v0.66.0 — Post-train X-rays: 5 surfaces extend soup diagnose from 6 failure modes to 10. Mechanistic interpretability shipped CLI-first because hosted vendors structurally can't (SaaS unit economics break on O(N) influence functions). Closes v0.57 #171 — the NotImplementedError blame stub is lifted with a DataInf-style influence-function approximation.

• soup probe sae-diff <sae> <pre.json> <post.json> — Sparse-Autoencoder feature attribution. Pure-numpy: ReLU(activations @ W_enc + b_enc) → per-feature mean post-pre delta → top-K + L2 drift. Closed HF_HUB_ALLOWLIST over 8 known SAE families (Gemma Scope / Pythia / SAE-Lens Llama / OpenAI GPT-2-small). Containment + O_NOFOLLOW open + 64-tensor cap.
• soup adapters blame --top-k 50 — Live DataInf-style runner closes #171. Per-row score = cos(grad_row, grad_probe) × |grad_row|. Operator supplies probe_fn or falls back to a deterministic synthetic probe so the surface always returns a real BlameResult (never raises).
• soup probe sleeper <base> [--evidence ev.json] — Calibrated linear probe per base. 6 bundled bases (Llama-2/3 8B, Mistral-7B, Qwen2-7B, Gemma-2 2B/9B). OK / MINOR / MAJOR at 1% / 5% defection rate. Exit 2 on MAJOR for CI gating.
• soup probe interference <losses.json> — Pairwise N×N adapter interference matrix. OK / MINOR / MAJOR at 5% / 20% bands. Surfaces which adapter pairs you cannot deploy together via soup serve --adapters. MAJOR worst-pair exits 2.
• soup probe pack <base> — Manifest assembler. Per-base probe bundle (sleeper / sae / truth / harm; last two reserved for v0.66.x).

Install / Upgrade

pip install --upgrade soup-cli

Security

• TOCTOU defence via O_NOFOLLOW — load_sae_weights and _count_dataset_rows now probe-open with os.O_NOFOLLOW so a symlink swap between the containment check and the read raises immediately rather than silently following.
• Process-independent deterministic seeds — _probe_weights (sleeper) and _default_synthetic_probe (blame) replace process-salted hash() with hashlib.sha256(key).hexdigest()[:16] (64-bit seed). CI reproducibility now works regardless of PYTHONHASHSEED.
• Rich-markup escape on operator-controlled adapter / verdict / description / layer strings before embedding into markdown render. A crafted adapter name like [link=evil]click[/] cannot inject Rich markup.
• TypeError on bool/non-str verdict — SleeperProbeResult and InterferenceCell type-check verdict before the membership check, so verdict=True raises TypeError (was: misleading ValueError).
• Non-numeric loss rejection in soup probe interference CLI — JSON true/false deserialise to Python bool (subclass of int), explicit guard fires first.
• DoS caps: 10M-row hard reject on dataset row count (no silent truncation); _DEFAULT_SYNTH_PROBE_CAP=100_000 so a 10M-row default-probe never allocates 1.28 GB; _MAX_ADAPTERS=16 (256-cell matrix ceiling); _MAX_DESCRIPTION_LEN=4096 on ProbeEntry; safetensors len(keys) > 64 pre-check before tensor materialization.
• Frozen dataclasses everywhere + __post_init__ validation; tests assert pytest.raises(FrozenInstanceError) (replaces generic (AttributeError, Exception) which would silently pass on dropped frozen flag).

Known Limitations

• Bundled sleeper-probe weights are deterministic but SYNTHETIC — _probe_weights derives from a SHA-256 seed so CI is reproducible, but the weights are NOT real Anthropic-style calibrated probes. Real weights ship in v0.66.x.
• No live HF Hub auto-download for SAE / probe weights — operators must already have the safetensors file locally; the allowlist is the recorded-provenance closed set, not a fetcher.
• truth + harm probe kinds reserved — PROBE_KINDS accepts them today as a forward-compat surface, but no runtime utility consumes them yet.
• run_blame synthetic-probe default is offline-only — scores from probe_fn=None are not meaningful (seeded RNG). Real workflows pass a real probe_fn returning gradients from a held-out probe set.
• Interference matrix uses operator-supplied losses — Soup does not measure them automatically; operators run soup serve --adapters A,B (v0.22), capture per-prompt losses, and feed the JSON.
• SAE feature diff uses operator-supplied activations — operators capture hidden states externally via model.forward hooks.

Test count

10577 → 10836 (+259 net across 7 new test files).