MakazhanAlpamys/Soup v0.47.0

v0.47.0 — Data Forge

on GitHub

What's New

v0.47.0 — Data Forge: Synthetic-data pipeline with full provenance + a lighter-weight data-quality scorecard. Two CLIs, one philosophy: every synthetic row carries its audit trail, and every quality check is pure-Python so it runs anywhere — no GPU, no 200 MB Presidio model.

Synthetic Data Forge

• soup data forge --docs <dir> --task sft|preference|tool --target-rows N — multi-stage pipeline: chunk documents → call a judge → active-learning prune via Jaccard distance → JSONL output plus a separate provenance manifest linking every synthetic row back to source doc + judge label + filter score. 10 000-doc cap; closed task allowlist; atomic staged-tempfile writes; os.lstat + S_ISLNK rejection on every write target. Live Ollama / Anthropic / vLLM judge providers wire through --judge-provider in v0.47.1; the in-tree default is a deterministic offline stub.

Data Quality Scorecard

• soup data score --input rows.jsonl — composite scorecard: PII flagged, toxic flagged, language distribution, educational-value mean, decontamination removed. Pure Python, no heavy deps.
• soup data decontaminate --benchmarks mmlu,gsm8k,humaneval — drops rows that overlap public benchmarks via n-gram containment. Closed benchmark-name allowlist (mmlu / gsm8k / humaneval / truthfulqa / arc / hellaswag).
• soup data toxicity / langdetect / pii / educational — standalone subcommands, JSONL-in, enriched JSONL-out. Compose freely.

ReDoS hardening

• Phone + credit-card PII regexes rewritten to eliminate nested optional quantifiers; input pre-capped to 50 KB before finditer. Pathological near-miss inputs (100 KB of "1 " * N + "x") complete promptly instead of hanging.

Stats

• +116 net new tests (6126 → 6242).
• All five review agents (python / code / security / TDD / verification-loop) green after CRITICAL→LOW fixes applied.

Install / Upgrade

pip install --upgrade soup-cli==0.47.0

Security

• ReDoS-hardened PII regex set; 50 KB pre-cap on detect_pii.
• math.isfinite guard on every unit-float validator (NaN AND ±Inf rejected before bounds check).
• is_under_cwd containment on every input + output path; os.lstat + S_ISLNK rejection at every write target.
• Atomic writes via tempfile.mkstemp + os.replace — mid-stream failure never leaves a partial file.
• _require_str rejects null bytes consistently across data_forge and data_score.
• compute_scorecard per-row exceptions logged at DEBUG (no silent swallow).
• All user-controlled strings in CLI output pass through rich.markup.escape.

Known Limitations

1. Live judge providers deferred to v0.47.1 — soup data forge ships with a deterministic offline echo stub only; live Ollama / Anthropic / vLLM via --judge-provider lands next. Stub-then-live pattern matches v0.27.0 MII / v0.37.0 multipack / v0.46.0 Part A.
2. Decontamination benchmark corpora not bundled — --benchmarks validates the flag but operates on an empty corpus; operator-supplied --benchmark-file lands in v0.47.1.
3. [data-pro] extras deferred — Llama-Guard-3-1B variant for toxicity, FineWeb-Edu classifier for educational value, full Presidio for PII, fastText / langdetect for language detection all ship behind opt-in extras in v0.47.1.
4. Provenance manifest stores absolute realpath in source_doc — operators sharing manifests publicly should redact paths before publishing; kept as-is for audit-trail completeness.

🤖 Generated with Claude Code