MakazhanAlpamys/Soup v0.63.0

v0.63.0 — Production Trace Ecosystem

on GitHub

What's New

5 LIVE top-level commands that close axis 7 of the roadmap. Pull traces from any SaaS dashboard, mine your prod logs, ship gates. No deferred stubs — every Part works on day one.

• soup ingest --source <name> --logs <jsonl> — Universal trace importer with 6 adapters: Langfuse / LangSmith / Helicone / OpenPipe / OTel / OpenAI Stored Completions. Output is a normalised JSONL trace stream that soup data from-traces (v0.26) consumes. SaaS pulls happen out-of-band (zero credential-handling threat surface — Soup parses the local export, never makes the network call). PII reminder panel on every ingest.
• soup prune-prompt --input <jsonl> --output <jsonl> --min-frequency 0.95 — Detect + strip the longest shared system-prompt prefix across training rows so the FT model internalises it. OpenPipe's signature trick, OSS. Binary-search over up to 32 candidate templates finds the longest threshold-meeting prefix (no early-exit on the 100% match).
• soup data active-sample --input <jsonl> --budget N — Surface the most uncertain prod traces for human review. Max-entropy on single rm_score (peak at 0.5) OR pairwise disagreement on dual rm_scores. Composes with soup eval human (v0.19).
• soup ab --input <jsonl> --metric latency|judge_score|retry_rate — Wald's classic SPRT for the point alternative — proper sequential A/B with early-stop guarantees. LLR is a martingale under H0 so Type-I error is controlled at every stopping time per the optional stopping theorem. Composes with soup loop canary (v0.58).
• soup drift-alarm --reference <jsonl> --live <jsonl> --threshold 0.2 — Rolling KL divergence on whitespace-tokenised output distribution. Optional --slack-url / --discord-url webhooks SSRF-validated (loopback HTTP only, RFC1918 / 169.254.x rejected). Exit code 3 on drift for cron-friendly automation.

Install / Upgrade

pip install --upgrade soup-cli

Compatible with Python 3.9+. CI green on ubuntu / macOS / windows × 3.9 / 3.11 / 3.12.

Security

5 new commands ship hardened from day one:

• SSRF parity with v0.51.0 on --slack-url / --discord-url webhooks: scheme allowlist {http, https}, null-byte + control-char rejection (CRLF defence), plain HTTP only for loopback hosts, RFC1918 / 169.254.x / reserved / multicast IPs rejected via ipaddress.ip_address.
• Path containment via shared is_under_cwd (os.path.realpath + commonpath) + null-byte rejection BEFORE any file open on every command.
• DoS caps on every input surface: _MAX_INGEST_LINES=1_000_000, _MAX_SCAN_ROWS=100_000 + _MAX_ROW_CHARS=1_000_000, _MAX_BUDGET=100_000, _MAX_SAMPLES_PER_ARM=1_000_000, _MAX_WEBHOOK_URL_LEN=4096, _MAX_TEXT_LEN=1_000_000.
• TraceRecord.metadata wrapped in MappingProxyType post-init so callers cannot mutate a returned record.
• post_webhook never crashes the drift check — broad except Exception so telemetry / alerting failure cannot mask a real drift verdict.
• Auth env-var resolution is read-only — Soup surfaces which env var is set (LANGFUSE_KEY / LANGSMITH_API_KEY / HELICONE_API_KEY / OPENPIPE_API_KEY / OPENAI_API_KEY / OTEL_EXPORTER_OTLP_HEADERS) but never makes the network call.

Review-fix coverage across 2 reviewer waves: 1 CRITICAL (mSPRT log-likelihood-ratio sign error → replaced with Wald point-alternative SPRT, fixes unbounded Type-I error) + 2 HIGH (detect_common_prefix early-exit on 100% match returned shortest not longest; _MAX_SCAN_ROWS DoS cap used pass instead of break) + 3 MEDIUM (TraceRecord MappingProxyType immutability; _AUTH_ENV deduplicated; drift_alarm precedence parens) + 2 LOW (pooled_se dead-branch refactor; mean_uncertainty NaN guard) + 8 follow-up boundary / regression tests. See full per-version security log in SECURITY.md.

Known Limitations

• No live SaaS API pull — soup ingest parses the JSONL export the operator pulls from their dashboard or vendor SDK. Soup itself never makes the network call, keeping the credential-handling threat surface zero. Auto-pull adapters per source are tracked for v0.63.1 once we have a designed-for-pull auth flow.
• prune-prompt is whitespace-tokenised — detect_common_prefix operates on raw characters, not tokens. Operators whose system prompts contain UTF-8 multi-byte sequences mid-prefix may see the prefix end in the middle of a code point.
• Active-learning K>2 RMs deferred to v0.63.1 — score_uncertainty accepts up to 2 RM scores; 3+ score rows fall back to a range-based diversity score (max - min) until proper variance-based diversity ships.
• mSPRT single-metric per pass — multi-metric correction (Bonferroni / Holm) is operator-controlled. Operators monitoring (latency, judge_score, retry_rate) jointly should run three passes with alpha adjusted.
• mSPRT assumes Gaussian-like data — for binary metrics like retry_rate as a boolean, the operator must pre-aggregate per-prompt rates so the per-prompt averages are approximately Gaussian.
• drift-alarm uses whitespace tokeniser — sub-word drift (token leakage at the BPE level) is invisible. A tokenizer-aware variant via --tokenizer flag is tracked for v0.63.1.
• No automatic webhook on soup ingest / soup prune-prompt / soup ab — only soup drift-alarm ships webhook delivery. Operators wanting alerts on the other commands should wrap them in their CI.
• post_webhook swallows every exception — broad except Exception because telemetry / alerting must NEVER crash the analysis run. Operators monitoring whether webhook delivery actually succeeded should check the CLI's stdout printout.
• Zero-variance arms in mSPRT → continue forever — both arms with identical constant values can never reject H0 (proper SPRT behaviour: without observed noise the test cannot bound Type-I error honestly). Operators expecting "huge effect" to reject must include realistic measurement variability.

Test count: 9816 → 10035 (+219 net).