MakazhanAlpamys/Soup v0.6.0

v0.6.0 — Quantization-Aware Training

on GitHub

What's New

Quantization-Aware Training (QAT) — Train with simulated quantization for significantly better post-quantization quality.

• quantization_aware: true config option — enables QAT during training
• Uses torchao Int8WeightOnlyConfig to insert fake quantization ops
• Works with all training tasks: SFT, DPO, GRPO, and vision modality
• Train command shows + QAT in setup panel and validates config
• Not compatible with unsloth backend (use backend: transformers)
• After QAT training, export to GGUF normally with soup export

When to use QAT vs post-training quantization

• QAT (quantization_aware: true): Better quality when deploying with aggressive quantization (int8/int4). ~5-10% slower training, but the model learns to compensate for quantization noise.
• Post-training quantization (default): Faster training, good enough for most use cases.

Install / Upgrade

# Upgrade
pip install --upgrade soup-cli

# Install QAT support
pip install 'soup-cli[qat]'

Stats

• 490 tests across 37 files
• 35 new tests for QAT