MakazhanAlpamys/Soup v0.27.0

v0.27.0 — Multi-GPU Mastery

on GitHub

What's New

• Multi-GPU Mastery — soup train --gpus auto auto-detects GPU count + NVLink/PCIe interconnect and prints the exact accelerate launch command with topology-aware NCCL env hints (applied via os.environ.setdefault so user / launcher overrides always win).
• ZeRO++ — --deepspeed zero++ (alias zero_pp): Stage-3 with hierarchical partitioning + quantized weights + quantized gradients. Cuts inter-node traffic 4-8x on 8+ GPUs.
• FSDP2 + torch.compile — training.use_fsdp2_compile: true wires torch_compile=True into TrainingArguments via extracted apply_fsdp_training_kwargs helper. Validator enforces FSDP + CUDA + backend=transformers + torch>=2.2/accelerate>=0.27; rejects DeepSpeed+compile coexistence (prevents cryptic runtime crash).
• DeepSpeed-MII backend — soup serve --backend mii registered with dependency check. Live pipeline wiring ships in v0.27.1; v0.27.0 exits non-zero with a clear stub warning (no silent mis-start).
• Pipeline parallelism — training.parallelism: pipeline + pipeline_stages [1, 16] with validator (requires stages >= 2 + CUDA + gpu_count >= stages). Execution wiring ships in v0.27.1; config validates today with a Rich Panel notice.
• New multi-GPU recipes — llama3-70b-fsdp2, qwen3-32b-zeropp, deepseek-v3-pipeline. Recipe count 43 → 46.

Install / Upgrade

pip install -U soup-cli

Verify with soup version (should show v0.27.0) and soup train --help (should list --gpus).

Security

• --gpus bounds: rejects bool, non-digit, zero, negative, values above MAX_GPU_COUNT=128
• --gpus auto on 0-GPU host prints explicit yellow warning (no silent no-op)
• Rich markup escaped on --config path before embedding in the multi-GPU advice Panel
• accelerate launch argv assembled via shlex.quote per element (copy-paste safe)
• build_accelerate_argv validates num_processes >= 1, mixed_precision Literal (no/fp16/bf16/fp8), num_machines bounded [1, 256]
• ZeRO++ integer literals (int(1e9) not float) so DeepSpeed strict JSON accepts
• validate_fsdp2_compile_config rejects DeepSpeed + torch.compile combination
• pipeline_stages Pydantic bounds [1, 16]; parallelism Literal data|pipeline
• NCCL env (NCCL_P2P_DISABLE / NCCL_IB_DISABLE / NCCL_NVLS_ENABLE) applied via os.environ.setdefault only

Known Limitations

• Auto-reexec of accelerate launch — currently advisory (prints command). Auto-re-exec deferred to v0.27.1.
• Live DeepSpeed-MII server pipeline — backend is registered + dependency-checked but exits with a stub warning. Live inference pipeline ships in v0.27.1.
• Live pipeline-parallel execution — config + validation ship today; actual pipelined forward/backward via torch.distributed.pipelining ships in v0.27.1.
• Multi-node launch — single-node only in v0.27.0. Multi-node num_machines > 1 argv is generated but not fully tested end-to-end.
• Recipe CI smoke-train validation — the 3 new multi-GPU recipes load cleanly but are not exercised by a 100-step CI smoke-train (deferred to v0.31.0).

Full commit: b7abd2a
Tests: 2511 → 2585 (+74). All 4 review agents + re-review wave applied; no remaining findings.