MakazhanAlpamys/Soup v0.50.0

v0.50.0 — GRPO Plus

on GitHub

v0.50.0 closes the GRPO/RL gap with Unsloth and Axolotl. 22 features across objective variants, long-context + memory-efficient RL, multi-turn agent rollouts, stability/efficiency knobs, and Process Reward Models + Vision RL.

This is a schema-only release: every new flag parses, every cross-validator fires, every error message names the actual problem. Live loss kernels, vLLM sleep-mode plumbing, ART/RULER/NeMo Gym/OpenEnv launchers, and the PRM trainer wrapper land in v0.50.1 — schema accepts the values now so configs are stable.

What's New

7 GRPO objective variants

New training.grpo_variant:

• gspo — Group Stabilized Policy Optimization (Unsloth)
• dapo — Decoupled Advantage Policy Optimization (Unsloth + Axolotl)
• dr_grpo — Doubly Robust GRPO (Unsloth + Axolotl)
• bnpo — Batch Normalized Policy Optimization (Unsloth)
• two_sided — symmetric clipping (requires grpo_delta)
• rft — Reinforced Fine-Tuning (Unsloth)
• standard — default GRPO (DeepSeek-R1 style)

Plus grpo_fp16: true (explicit FP16 RL, complementing existing FP8 RL).

Long-context + memory-efficient RL

• training.long_context_grpo: true — schema gate for Unsloth-grade long-context GRPO (380K B200 / 110K H100); wires Tiled MLP from v0.56.0 Part A once it lands.
• training.vllm_sleep_mode: true — between-rollouts vLLM standby (Unsloth + Axolotl).

Multi-turn agent rollout backends

New training.rollout_backend:

• art — OpenPipe ART multi-turn agent rollouts (Unsloth)
• ruler — RULER agent eval framework (Unsloth)
• nemo_gym — NVIDIA NeMo Gym single/multi-turn rollouts (Unsloth + Axolotl)
• openenv — generic rollout_func protocol (Unsloth + Axolotl)

7 GRPO stability/efficiency knobs (Axolotl + Unsloth parity)

• ref_model_ema_alpha — exponential moving average for policy → reference sync
• replay_buffer_size — bounded replay buffer for rollouts
• async_grpo_prefetch — overlap rollout + train
• tis_threshold + mask_truncated_completions — truncated importance sampling
• defer_rerolling — defer re-rolling identical prompts
• skip_zero_advantage — skip backward pass on zero-advantage samples
• off_policy_mask_threshold — token/sequence off-policy mask gate

Process Reward Models + Vision RL

• New top-level task='prm' — Process Reward Model / stepwise-supervised training (paired with data.format='prm' from v0.42.0).
• New training.vision_grpo: true — Vision RL / VLM RL on Qwen2-VL / Pixtral / InternVL (requires modality='vision' and task ∈ {grpo, ppo}).

Install / Upgrade

pip install --upgrade soup-cli

Or via Docker:

docker pull ghcr.io/makazhanalpamys/soup:v0.50.0

Security

5 sequential review agents (python-review, code-review, security-review, tdd-guide, verification-loop) flagged HIGH/MEDIUM/LOW findings; every finding was fixed before commit:

• Explicit field_validator(mode='after') for grpo_delta NaN/Inf rejection (Pydantic le=1.0 only incidentally rejects NaN; made explicit so a future Pydantic change cannot regress the guard).
• Null-byte rejection on task AND backend strings in long-context-GRPO and vLLM-sleep compat helpers; matches validate_grpo_variant / validate_rollout_backend policy.
• Bool guard on use_ring_attention parameter (parity with the bool-before-int policy enforced on every numeric field this release).
• grpo_fp16 added to the GRPO-only task-gate — silently no-opping on task='sft' is no longer possible.
• vllm_sleep_mode requires task='grpo' — sleep mode is a between-rollouts feature, meaningless on SFT.
• _reject_bool_on_grpo_numerics field_validator rejects bool on every Part D numeric field + grpo_delta before Pydantic's True→1 coercion (matches v0.30.0 Candidate / v0.41.0 Part B lr_groups / v0.43.0 Part B policy).

All new closed-allowlist validators (validate_grpo_variant, validate_rollout_backend) reject non-string / bool / empty / null-byte / oversize inputs with actionable case-insensitive normalisation. Registries wrapped in MappingProxyType (matches v0.36.0 _REGISTRY policy); specs are frozen dataclasses (matches v0.32.0 SpikeRecoveryStrategy policy).

Test count: 6490 → 6729 (+239 across 5 new test files).

Known Limitations

• Every live loss kernel and launcher is deferred to v0.50.1. apply_variant_loss (GRPO variants), apply_vllm_sleep_mode, launch_rollout (ART/RULER/NeMo Gym/OpenEnv), and build_prm_trainer all raise NotImplementedError with explicit v0.50.1 markers. Same stub-then-live pattern as v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro / v0.45.0 plugins / v0.46.0 ExecuTorch / v0.48.0 curriculum / v0.49.0 LongLoRA.
• Long-context GRPO requires Tiled MLP (scheduled to land in v0.56.0). Until then, long_context_grpo: true parses but the runtime fallback prints an advisory.
• Vision RL allowlist does not verify that the base model is actually a VLM (Qwen2-VL / Pixtral / InternVL); the upstream trainer surfaces the error loudly when it fails to load the vision tower.
• PRM trainer routing not registered — task='prm' schema-validates but soup train will print "PRM trainer not yet wired (v0.50.1)" until the live wrapper lands.
• Stability knobs do not interact at v0.50.0 — the 7 new Part D fields all schema-validate but none are wired into a live callback in this release; replay_buffer_size, defer_rerolling, and skip_zero_advantage are pure schema lock-ins.
• vllm_sleep_mode is a vLLM ≥0.7 feature — Soup does not auto-upgrade vLLM when the flag is set; users must pip install -U vllm themselves.
• grpo_fp16 is captured but not yet routed — the flag rejects on non-grpo tasks per code-review HIGH fix, but the live mixed-precision picker still uses the v0.32.0 auto_mixed_precision path; full FP16-RL routing lands in v0.50.1.

🤖 Generated with Claude Code