MakazhanAlpamys/Soup v0.53.10

v0.53.10 — Quick wins + packaging + UX wiring

on GitHub

v0.53.10 closes 7 deferred packaging + UX issues from the v0.40 → v0.53 long tail. Pure no-GPU work — every feature is a pyproject.toml edit, lazy-import helper, frontend tab, or trainer callback that builds on machinery already exposed by v0.53.6 / v0.53.9.

What's New

• [mix] + [data-pro] extras (#150, #113). pip install soup-cli[mix] bundles scikit-optimize so soup data mix --optimize runs the real Bayesian loop. pip install soup-cli[data-pro] bundles langdetect + presidio-analyzer for broader language / PII coverage on top of the v0.47.0 regex baseline. Both helpers fall through silently when the optional package is missing.
• PostHog env override (#154). New SOUP_POSTHOG_KEY + SOUP_POSTHOG_ENDPOINT env vars let operators point opt-in telemetry at their own PostHog project. Sentinel-based default cleanly distinguishes "caller omitted endpoint" from "caller pinned the default" so an explicit pin is NOT silently overridden by env. Endpoint goes through the v0.51.0 HTTPS + private-IP rejection policy.
• --hub on chat / serve / infer / merge / export / push (#152). Closes the v0.53.8 known-limitation that only train and data download could route through ModelScope / Modelers. New shared utils/hubs.prefetch_model_from_hub + apply_hub_to_cli_model helpers; push uses utils.hubs.upload_repo (skips HF-specific Collections + model-card auto-render).
• soup data download --hub live SDK (#153). Lifts the v0.53.8 advisory-only path. ModelScope uses MsDataset.load(...), Modelers uses openmind_hub.snapshot_download(repo_type='dataset', ...). Friendly pip install <sdk> advisory when missing.
• Web UI Tool Outputs panel (#155). New tab polls /api/tool-outputs?limit=100 every 3 s; XSS-safe via document.createElement + td.textContent (no innerHTML for user-controlled fields). Bearer token threaded via the v0.53.9 SPA bootstrap.
• SFT callback record_call wiring (#156). SoupTrainerCallback.on_step_end records tool_calls counts from kwargs['inputs'] into the global tool buffer so the Web UI panel populates from real training batches. Best-effort — training never crashes on observation failure.

Install / Upgrade

pip install -U soup-cli
# or with the new extras:
pip install -U 'soup-cli[mix,data-pro]'

Security

13 review-fixes (4 HIGH / 5 MEDIUM / 4 LOW) landed in-release:

• HIGH PostHog endpoint precedence — sentinel-based default prevents env from silently overriding an explicit pin
• HIGH Absolute path leak in local_path advisory reduced to os.path.relpath
• HIGH Rich markup escape on base / local_path / cache_dir (defends against [bold red]evil[/] injection from a crafted base)
• HIGH Callback # noqa: BLE001 per project policy
• MEDIUM import time moved out of try block; oversize-key + explicit-empty-key rejection tests; source-grep regression guards
• MEDIUM prefetch_model_from_hub outside-cwd cache_root rejection
• LOW Empty-list + bool-True tool_calls no-op tests; push uses upload_repo + validate_hub_name regression guard

Slug regex [^A-Za-z0-9._-]+ → __ neutralises path-separator + .. escapes in base before the cache-dir join; final is_under_cwd realpath check on the resulting cache dir. See SECURITY.md for the detailed fix-note.

Known Limitations

1. langdetect is not thread-safe — _langdetect_fast rebinds langdetect.DetectorFactory.seed = 0 on every call to keep determinism; concurrent calls from multiple threads can interleave (inherent to the upstream library). The scoring pipeline is single-threaded today; tracked as a follow-up if multi-threaded scoring lands.
2. Llama-Guard-3-1B toxicity classifier NOT bundled in [data-pro] — license + ~600 MB weight blob keep it out of an extras install. Operators wanting it use pip install transformers + pipeline('text-classification', model='meta-llama/Llama-Guard-3-1B') manually.
3. apply_hub_to_cli_model argument routing heuristic — the helper prefers rewriting base_model when set, else falls back to model. For a command where --model is a remote repo id AND --base is unset, the model arg is rewritten.
4. push --hub <non-hf> skips HF-specific features — HF Collections + model-card auto-render are HF-only; non-HF uploads use the raw upload_repo path.
5. PostHog endpoint precedence design — to lock the default URL against SOUP_POSTHOG_ENDPOINT env, callers must pass endpoint=_POSTHOG_ENDPOINT (the module constant) explicitly. Omitting endpoint honors the env override (intended).
6. --hub covers model commands — data download --hub is also live, but data push --hub, data forge --hub, etc. still route through HF. Multi-data-command rollout tracked for v0.53.11+.

Test count: 8285 → 8330 (+45 in tests/test_v05310.py).

🤖 Generated with Claude Code