MakazhanAlpamys/Soup v0.65.0

v0.65.0 — Eval Depth

on GitHub

What's New

Eval Depth — calibrated judges + behaviour battery + capability auto-suite + CheckList DSL + IRT subset selector. Closes axis 4 of the roadmap. Evals as a first-class surface, not an afterthought. SaaS evals (Galileo, Braintrust) ship judges but not calibration; lm-eval-harness is the closest to capability auto-suite but wiring is painful; CheckList for LLMs hasn't been revived. Soup ships all five locally.

• Judge calibration with conformal abstention — eval/calibrate.py extended with SCOPE/CJE-style bidirectional pairwise judging. PairwiseJudgement carries first/second/oracle verdicts; fit_position_bias returns slot-flip coefficient ∈ [-1, 1]; conformal_threshold(scores, *, alpha) emits the α-quantile abstention threshold; ensure_judge_calibrated(report) refuses to score with an uncalibrated judge (RuntimeError on missing report / low agreement / extreme bias). v0.43.0 KL-divergence surface preserved.

• soup eval behavior <run-id> --battery xstest|harmbench|jailbreakbench|elephant|syceval — bundled behaviour battery with pre/post diff. 5 tiny probe sets ship under soup_cli/data/_fixtures/behavior/ (harmful prompts REDACTED; operators pull real sets from upstream papers). Word-boundary regex agreement rejects "safe" in "unsafe" false positives. OK/MINOR/MAJOR thresholds match v0.26 / v0.56 taxonomy.

• soup eval capability <run-id> --suite full|fast|math|code — pre-bundled MMLU-Pro / GPQA / BBEH / AIME / MATH-500 / HumanEval+ / SWE-bench-Verified profile selector with sane lm-eval-harness task ids. Emits the (benchmark, lm-eval task) manifest for downstream soup eval benchmark chaining.

• soup eval checklist <spec.yaml> — Ribeiro et al. 2020 CheckList behavioural DSL. Three test kinds — mft (Minimum Functionality), inv (Invariance), dir (Directional Expectation). Word-boundary matching prevents "and" matching "sand". Per-test OK/MINOR/MAJOR verdict.

• soup eval irt-subset <responses.jsonl> --size full|small|tiny — Item Response Theory eval-cost optimizer. 1PL Rasch closed-form fit + high-info subset selection. 5-10x cut in eval bills without losing ranking power. Streams via os.fdopen(fd) so 256 MiB JSONL doesn't materialise in RAM.

• +271 new tests (10306 → 10577). Review-fix coverage from 2 review waves: 0 CRITICAL + 6 HIGH (TOCTOU O_NOFOLLOW + os.fstat on SAME fd across every new read path; importlib.resources.files / Traversable / op replaces fragile os.path.join(str(pkg_root)) for namespace-package safety; word-boundary regex agreement in behavior_battery + checklist_dsl) + 9 MEDIUM (CLI _validate_run_id gate; 16 MiB evidence cap with O_NOFOLLOW; _MAX_ROWS cap counts skipped lines toward total; INV all-whitespace responses no longer pass; _write_json_output dedup; named-test error in parse_checklist_spec) + 7 LOW. Manual CPU smokes for every command incl. failure modes (unknown battery / size / kind / empty run_id / outside-cwd evidence / MAJOR diff all exit 2).

Install / Upgrade

pip install --upgrade soup-cli

PyPI auto-publishes via Trusted Publisher OIDC on tag push (usually 1-2 min). GHCR Docker image builds on release publish (~10-15 min).

Security

v0.65.0 — Eval Depth ships 5 LIVE parts with TOCTOU O_NOFOLLOW + os.fstat on SAME fd across every new read path (load_checklist_spec, load_response_rows, _read_evidence_json) — earlier double-lstat-on-path was a race the attacker could win by swapping the file between calls. load_battery_probes namespace-package safety uses importlib.resources.files Traversable / operator + as_file context manager + os.lstat + S_ISLNK rejection + 4 MiB cap. Word-boundary regex agreement in behavior_battery._agreement_rate and checklist_dsl._mft_pass. CLI _validate_run_id gate rejects null bytes / oversize / empty before any value flows into report payloads. 16 MiB evidence file cap via os.fstat on the O_NOFOLLOW-opened descriptor. Closed allowlists wrapped in MappingProxyType / frozenset everywhere; bool-first / null-byte / oversize / case-insensitive normalisation on every validator. Test count: 10306 → 10577 (+271 net). Review-fix coverage: 0 CRITICAL + 6 HIGH + 9 MEDIUM + 7 LOW across 2 review waves.

See SECURITY.md for full hardening notes.

Known Limitations

1. Live lm-eval-harness invocation deferred to v0.65.1 — soup eval capability emits the planned (benchmark, lm_eval_task) manifest for the chosen profile but does NOT itself run lm-eval; operators chain into the existing soup eval benchmark surface. Same stub-then-live cadence as v0.46.0 / v0.44.0 soup quantize design (Typer commands aren't safe to re-enter).
2. Live model-driven soup eval behavior deferred to v0.65.1 — without --evidence, the command emits a neutral OK report; with --evidence it scores against operator-supplied pre/post response arrays. Live wiring that loads a base + adapter, generates responses on the bundled probe set, and emits the diff is the v0.65.1 deliverable.
3. Behaviour battery probe sets are tiny placeholders — XSTest / HarmBench / JailbreakBench / ELEPHANT / SycEval ship as 4-8 rows each. The harmful-prompt batteries (HarmBench / JailbreakBench) ship REDACTED placeholders (e.g. "[harmful-prompt-redacted-A]") — Soup will not ship verbatim harmful prompts. Operators wanting the real sets pull them from the upstream papers.
4. CheckList DSL inv test responses must match prompt count — when evidence has fewer responses than prompts, the test verdicts MAJOR (defensive); operators should ensure response arrays are aligned with the prompts tuple.
5. IRT model is 1PL Rasch only — 2PL (item discrimination) and 3PL (guessing) deferred to v0.65.x. The 1PL closed-form fit is exact when ability is centred at θ=0; for multi-respondent matrices, operators should run a real Rasch package.
6. conformal_threshold requires confidence scores in [0, 1] — operators using log-prob judge scores must normalise via sigmoid first.
7. Judge calibration agreement metric is asymmetric — agreement_rate is first_winner == oracle (treating the first-arrangement verdict as canonical). Operators wanting symmetric agreement (both orderings agree with oracle) should compute it themselves from the PairwiseJudgement list.
8. ensure_judge_calibrated is in-process only — no persistent calibration registry; operators must keep the JudgeCalibrationReport and pass it to every scoring call. v0.65.x may add a registry artifact kind judge_calibration.