MakazhanAlpamys/Soup v0.41.0

v0.41.0 — Optimizer & PEFT Zoo

on GitHub

What's New

Closes the optimizer-breadth gap with LlamaFactory + Axolotl in a single release.

• 14 new optimizers — BAdam, APOLLO (apollo_adamw), Adam-mini, lomo / adalomo, grokadamw, schedule_free_adamw / schedule_free_sgd, muon, dion, came_pytorch, plus TorchAO ao_adamw_{fp8,4bit,8bit}. Closed allowlist rejects typos at config-load time with an actionable error.
• Per-module LR groups — training.lr_groups: {q_proj: 1e-4, mlp: 5e-5} (or list-of-pairs / list-of-dicts). First-match-wins routing; remaining params fall through to the base lr. Capped at 32 entries with regex / null-byte / NaN-Inf hardening + best-effort ReDoS probe. PyYAML scientific notation (1e-4 without dot) is auto-coerced.
• LoftQ quantization-aware LoRA init — training.lora.init_strategy: loftq (with loftq_iter and loftq_bits ∈ {2, 4, 8}) jointly initialises A/B with low-bit base weights. Composes with QLoRA for stronger adapter quality on aggressive quantization.
• LLaMA Pro block expansion + Mixture-of-Depths schemas — training.expand_layers (1-64) + training.freeze_trainable_layers (signed, |x| ≤ 1000) + training.use_mod. Schema fields ship in v0.41.0 to lock the YAML surface; full live wiring lands in v0.41.1 (mirrors v0.27.0 / v0.37.0 stub-then-live releases).
• Friendly aliases — training.load_in_8bit: true / load_in_16bit: true remap quantization for users coming from LF / Axolotl conventions. Mutually exclusive; rejected when combined with explicit Quant Menu formats.
• +120 net new tests — 5122 → 5242. Covers optimizer allowlist, lr_groups parsing/runtime/schema-roundtrip, LoftQ + LLaMA Pro validators, alias remap rules, frozen LrGroup dataclass, ReDoS / bool-as-int / null-byte / YAML-string-coercion hardening.

Install / Upgrade

pip install --upgrade soup-cli
# or pin
pip install soup-cli==0.41.0

Security

• optimizer field uses a closed frozenset allowlist (HF-native + bnb-backed + 14 v0.41.0 additions). Non-string / empty / null-byte / >64-char rejected at schema-load. _OPTIMIZER_PACKAGES wrapped in types.MappingProxyType so the registry cannot be mutated at runtime (matches v0.36.0 _REGISTRY policy).
• lr_groups patterns: ≤256 chars, no null bytes, must re.compile, plus a best-effort ReDoS probe (compiled.search("a"*128)) catches catastrophic-backtracking patterns at schema-load. LR values: (0.0, 1.0] bounds + math.isfinite (NaN AND ±inf rejected) + bool rejection. Duplicates rejected. LrGroup is @dataclass(frozen=True).
• expand_layers / freeze_trainable_layers use field_validator(mode="before") so bool is rejected BEFORE Pydantic's ge/le silently coerces True to 1 (matches the project bool-as-int policy applied since v0.30.0).
• load_in_8bit / load_in_16bit aliases use the is True policy: explicit False is treated as "no preference" (never silently rewrites the field). Mutually-exclusive both-True rejected. Combining alias=True with an explicit Quant Menu format raises rather than silently overriding the user's pick. Alias-driven rewrite uses direct self.quantization = ... (Pydantic v2 BaseModel non-frozen path), NOT object.__setattr__ — code review caught that the latter would silently bypass any future field validator on quantization.
• _count_layers uses hasattr(layers, "__len__") instead of try/except TypeError so legitimate __len__ bugs surface loudly.

Known Limitations

1. LLaMA Pro live wiring deferred to v0.41.1 — expand_model_blocks raises NotImplementedError on non-zero block counts; the schema fields lock the YAML surface so v0.41.1 can land the trainer-side wiring (zero-init weights, freeze original layers, route through HF Trainer) without breaking config compat. Same stub-then-live pattern as v0.27.0 MII / v0.37.0 multipack.
2. Mixture-of-Depths live wiring deferred to v0.41.1 — use_mod=True parses but the actual selective-token routing patch lands in v0.41.1.
3. Optimizer dependency check is advisory — validate_optimizer_name accepts every entry in the allowlist regardless of whether the underlying package is installed; a friendly "pip install <required_package(name)>" advisory fires at trainer-construction time. Schema-load happens before deps can be probed and CI environments often lack optional optimizer packages.
4. load_in_8bit / load_in_16bit rewrite quantization only when set to True — explicit False is intentionally a no-op (treated as "no preference") so a YAML with load_in_8bit: false and quantization: 4bit still trains in 4-bit.

🤖 Generated with Claude Code