What's New
Continued Pre-training (task: pretrain)
- New training task for domain adaptation on raw text data
plaintextdata format:{"text": "..."}in JSONL, or plain.txtfiles (one document per line)- Template:
soup init --template pretrain
Mixture of Experts (MoE) Support
- Auto-detection of MoE architectures: Mixtral, Qwen3 MoE (30B-A3B), DeepSeek V3, DBRX, OLMoE, JetMoE
- ScatterMoE LoRA (
moe_lora: true): applies LoRA to both attention layers and expert FFN layers moe_aux_loss_coeff: configurable router load-balancing loss (default: 0.01)- MoE support works across all training tasks (SFT, DPO, KTO, ORPO, etc.)
- Template:
soup init --template moe
Test & Quality
- 85 new tests across
test_pretrain.pyandtest_moe.py - 1002 total tests, 47 test files, 57.6% coverage
- All review gates passed (python-review, code-review, security-review, tdd)
Install / Upgrade
```bash
pip install --upgrade soup-cli
```