MakazhanAlpamys/Soup v0.53.8

v0.53.8 — Remote data + Hubs + Trackers (wave 2)

on GitHub

What's New

Six features wired live across cloud storage, alternative model hubs, experiment trackers, and HF Spaces.

• fsspec live remote loaders. data.train: s3://bucket/data.jsonl (plus gs:// / gcs:// / az:// / abfs:// / abfss:// / oci://) now loads through fsspec with the v0.42.0 SSRF-hardened URI validator (bucket regex + userinfo/query/fragment rejection). Threads data.streaming + data.buffer_size. Friendly Rich panel names the pip install <driver> advisory when the backend SDK is missing. Row count capped at 1M to defend against pathological remote objects.

• Live ModelScope + Modelers hub dispatcher. New utils/hubs.download_repo() / upload_repo() lazy-import per backend (huggingface_hub, modelscope, openmind-hub). commands/train.py pre-fetches non-HF base into .soup_hub_cache/<sanitized-slug>/ and reuses cached snapshots on resume. soup data download --hub <name> flag plumbed (live for HF; non-HF dataset SDKs tracked for v0.53.9).

• [trackers] extra + missing-dep advisory. pip install soup-cli[trackers] bundles MLflow + SwanLab + Trackio. When you pass --tracker mlflow without the package installed, soup train now surfaces a clear pip install advisory before construction instead of HF Trainer's generic ImportError. Uses importlib.util.find_spec (non-executing probe) so swanlab can't initialise network threads during the absence check.

• PostHog telemetry network (opt-IN, silent-fail). utils/trackers.send_telemetry_payload() lazy-imports httpx with a 1-second hard timeout. HTTPS-only with SSRF re-validation (private-IP / link-local / RFC1918 rejected). Disabled unless SOUP_TELEMETRY=1; every exception swallowed so telemetry can never crash training.

• Bundled demo fixtures as package data. soup data demo fixtures (4 JSONLs) migrated to soup_cli/data/_fixtures/ so they ship inside the wheel — zipapp / namespace-package safe. _bundle_source_path falls back to legacy examples/data/ for editable installs.

• HF Space SDK auto-pick from requirements.txt. soup deploy hf-space --template-dir <path> now reads the rendered requirements.txt and picks space_sdk="streamlit" when the file lists streamlit, "gradio" otherwise. Closes the v0.40.2 known limitation.

Install / Upgrade

pip install --upgrade soup-cli           # core
pip install --upgrade soup-cli[trackers] # + mlflow / swanlab / trackio
pip install --upgrade soup-cli[remote]   # + fsspec / s3fs / gcsfs / adlfs

Security

Three review-agent waves (python / code / security) ran in parallel; 16 findings fixed:

• HIGH: cwd-containment on download_repo / upload_repo local_dir / folder_path; Windows ..\ traversal defence on .soup_hub_cache slug; cfg.base = local_path replaced with cfg.model_copy(update=...) so Pydantic field validators rerun (matches v0.33.0 #47 immutability policy).
• MEDIUM: SSRF re-validation on telemetry endpoint override (private-IP / link-local / RFC1918 rejected); 1M-row cap on remote dataset materialisation; 256 KB cap on detect_space_sdk input; rich.markup.escape on ImportError print; modelscope.push_model commit_message kwarg removed (would TypeError at runtime); importlib.util.find_spec instead of __import__ to avoid swanlab side-effects.
• LOW: bool rejection on local_dir; SDK detection rules documented; idempotent pre-fetch via cache probe.

Test count: 8162 → 8257 (+66 new tests + 29 net coverage adjustments).

Known Limitations

1. Hub dispatch is single-command live. commands/train.py (pre-fetch) and soup data download --hub (advisory only). Wider rollout for chat / serve / infer / merge / export / push tracked for v0.53.9.
2. No hf-id → modelscope-id translation table. A base: meta-llama/Llama-3.1-8B with hub: modelscope passes the literal id to ModelScope; if the repo isn't published there under that id, you'll see a clean HF Hub-style 404.
3. PostHog API key is a placeholder. Live deployments will swap via env var once the privacy policy ships. Telemetry remains opt-IN + silent-fail until then.
4. soup data download --hub modelscope is advisory-only. Non-HF dataset download needs the modelscope dataset SDK (different API from snapshot_download); v0.53.9.
5. Package-data migration keeps legacy examples/data/ duplicates for repo-relative scripts. Future patch may remove the duplicates.
6. [trackers] extra pin-floor mlflow>=2.0.0. Older mlflow setups can override via direct install.
7. fsspec streaming path requires HF datasets for the data.streaming=true codepath; without it, falls back to direct fsspec.open line-by-line.

v0.53.7 Carry-over

The /v1/tools/bash endpoint still returns HTTP 501 — RLVR sandbox child-process isolation insufficient for /bin/sh -c. The marker has been bumped from v0.53.8 to v0.53.9 (container / namespace work scheduled there).

🤖 Generated with Claude Code