MakazhanAlpamys/Soup v0.54.0

v0.54.0 — soup advise: the pre-flight decision

on GitHub

What's New

soup advise <data.jsonl> --goal "..." — the layer above autopilot. Before you spend 8 hours on a GPU, spend 60 seconds in the CLI:

• Verdict engine. Classifies your task (factual_lookup / style_shaping / format_conversion / reasoning / tool_use / summarization / classification), profiles your dataset (row count, type/token diversity, label variance, chosen/rejected detection, reasoning-trace detection), and renders a Verdict with one of PROMPT_ENG / RAG / SFT / DPO / GRPO. Includes a confidence score, a reason, the criterion that would reverse the decision, and the literal next command (soup autopilot --data … --task sft).

• Probe runner. soup advise --probe adds a 10-minute ROI estimate: zero-shot + few-shot baselines, an RAG baseline, and a 100-step LoRA probe. v0.54.0 ships heuristic stubs (no GPU required); real model loading lands in v0.54.1 with forward-compatible model / device / lr / timeout_seconds kwargs already on the signature.

• Explain. soup advise explain prints the rubric — which rule fired, evidence, the next command, and what would flip the verdict.

• Cross-project learning. soup advise compare reads ~/.soup/advise_history.jsonl (atomic cross-process writes via fcntl.flock on POSIX / sidecar <path>.lock on Windows) and shows your prior verdicts across projects so the next decision is informed by the previous N.

• Why blue-ocean. No trainer library has an incentive to tell users not to train — Unsloth's funnel, Axolotl's hosted business, LLaMA-Factory's Alibaba alignment all monetise the training event. The advisor role is structurally orthogonal to every incumbent's revenue model.

Install / Upgrade

pip install -U soup-cli

Docker:

docker pull ghcr.io/makazhanalpamys/soup:0.54.0

Security

5 review-agent waves (python / code / security / tdd / architect) ran on this release; 16 findings fixed (0 CRITICAL + 5 HIGH + 7 MEDIUM + 4 LOW) before tag:

• argv rewriter scoped to argv[1] == "advise" (defends against false-positive rewrites)
• atomic write + symlink reject on ~/.soup/advise_last.json scratch file
• symlink rejection on _read_last_verdict (defends against planted-symlink leaking /etc/passwd through json.load)
• cross-process file locking on history append (fcntl POSIX, sidecar <path>.lock Windows)
• per-line 64 KB cap on load_history (DoS defence)
• cwd-containment + os.lstat + S_ISLNK rejection on every path input (TOCTOU policy)
• SOUP_ADVISE_HISTORY_PATH env override containment-checked to $HOME / $CWD / tempdir
• bool / finite / NUL / oversize guards on every public input
• Rich markup escape on all user-controlled output

Full details in SECURITY.md.

Known Limitations

1. Live probe runner deferred to v0.54.1 — synth_probe_baselines and synth_probe_lora_delta ship as heuristic stubs that derive deltas from dataset shape; the real zero/few-shot + RAG + 100-step LoRA pipeline lands in v0.54.1 (same stub-then-live cadence as v0.27.0 MII / v0.37.0 multipack / v0.50.0 GRPO Plus). Forward-compat kwargs (model / device / lr / timeout_seconds) are already on the public signature so v0.54.1 is additive only.
2. RAG verdict is advisory-only — Soup does not ship a RAG runtime; when build_verdict returns RAG, the next_command_for helper prints a note pointing the user at pgvector / weaviate / faiss rather than a soup rag invocation.
3. base_model_proximity not measured — the DatasetProfile field exists with full validation, but neither compute_dataset_profile nor the v0.54.0 probe stubs populate it; v0.54.1 will set it from a held-out logit-agreement probe.
4. Cross-project learning is read-only on compare — advise_history.jsonl records verdicts, but build_verdict does NOT yet bias its rubric based on prior outcomes. v0.54.1 deliverable.
5. record_verdict(path=...) bypasses containment — explicit path kwarg is documented as trusted-internal-only; operator-supplied paths must arrive via SOUP_ADVISE_HISTORY_PATH.
6. Windows file locking uses a sidecar .lock file — POSIX uses fcntl.flock on the data file directly; Windows uses a separate <target>.lock because msvcrt.locking is byte-range-positional. The sidecar persists across runs by design.

Test count

8400 → 8571 (+136 in tests/test_v0540.py covering all three Parts; 5 Windows-skipped symlink tests; CI green across ubuntu-latest / windows-latest / macos-latest × Python 3.9 / 3.11 / 3.12).

🤖 Generated with Claude Code