MakazhanAlpamys/Soup v0.43.0

v0.43.0 — Tracker & Eval Pro

on GitHub

What's New

18 features across 4 areas — closes the observability gap with all three competitors in one release.

Tracker integrations

• New --tracker flag on soup train accepts mlflow / swanlab / trackio (mutually exclusive with --wandb / --tensorboard). Closed allowlist via MappingProxyType.
• PostHog telemetry payload schema lands as opt-IN (SOUP_TELEMETRY=1) with hardware-info-only fields — no model names, dataset paths, or config contents.

Eval metrics

• Pure-Python BLEU + ROUGE-1 / ROUGE-2 / ROUGE-L for soup eval custom.
• effective_tokens_per_second metric — unmasked_tokens / wall_clock_seconds (returns None when wall_clock ≤ 0).
• KL-divergence calibration framework (soup_cli.eval.calibrate.run_calibration) with OK / MINOR / MAJOR thresholds at 0.05 / 0.20.
• Model Arena Elo tournament (soup_cli.eval.arena.Tournament, K=32, 256-model cap).
• ceval / cmmlu / aider_polyglot benchmark name allowlist.

Profiling extras

• memory_snapshot_context wraps torch.cuda.memory._record_memory_history with cwd-confined snapshot path.
• nccl_bandwidth_check reference table (h100 / a100 / v100 / rtx-series, NVLink + PCIe).
• write_vscode_launch writes a .vscode/launch.json with soup train + pytest configs, symlink-rejected at the target path.

Demo datasets

• New soup data demo lists 4 ready-to-use JSONL fixtures (alpaca / sharegpt / dpo / grpo). soup data demo alpaca_demo --output ./mine.jsonl copies the bundle for instant soup train warm-up.

Install / Upgrade

```bash
pip install -U soup-cli
```

Security

• New --tracker allowlist + validate_tracker_name (null-byte / >32-char / unknown rejection).
• _REPORT_TO_BACKENDS registry wrapped in MappingProxyType.
• Tournament model names reject Rich markup metacharacters `[` and `]` so leaderboard rows can't be markup-injected.
• resolve_snapshot_path rejects null-byte / . / .. / path separators in run_id; base_dir rejects null-byte / absolute / .. segments before realpath join.
• write_vscode_launch uses os.lstat + S_ISLNK symlink rejection at the target path regardless of force (TOCTOU defence).
• copy_bundle_to stages writes to a sibling temp file with symlink rejection, then os.replace-atomic rename — mid-stream rejection never leaves a partial file at the user-visible target.
• Telemetry payload is closed-key (soup_version / command / python major.minor / os / arch / duration); no leak of model names, dataset paths, config contents.
• memory_snapshot_context narrows the RuntimeError catch to record() entry only — defends against a wide-catch double-yield generator bug.

Known Limitations

1. MLflow / SwanLab / Trackio live integration deferred to v0.43.1 — schema accepts the names and HF Trainer threads them through report_to, but Soup does not bundle the upstream packages; HF raises a generic ImportError when the package is absent.
2. PostHog telemetry network code deferred to v0.43.1 — payload schema + opt-in env var ship; no network code yet.
3. Aider Polyglot live runner deferred to v0.43.1 — name in the allowlist; live wiring requires upstream aider-chat + Docker.
4. soup doctor --nccl measurement CLI deferred — nccl_bandwidth_check accepts a measured value and classifies it; live measurement via torch.distributed.all_reduce is a follow-up.
5. memory_snapshot_context real CUDA path only via mocks in unit tests — graceful fallback when torch / CUDA missing; real GPU snapshot requires a GPU runner.
6. examples/data/ fixtures shipped at the repo root (not as package data inside soup_cli/); _bundle_source_path derives repo_root via str(importlib.resources.files("soup_cli")) which is editable-install + wheel friendly but may break in zipapp / namespace-package installs.