MakazhanAlpamys/Soup v0.72.0

v0.72.0 — Layer Streaming (BETA)

on GitHub

Layer streaming (BETA) — fine-tune models that don't fit in your card.

The frozen base model lives in CPU RAM and is streamed into a small pool of
pre-allocated VRAM buffers one decoder layer at a time, so peak VRAM is
bounded by the size of one layer instead of the whole model. Only the LoRA
adapters, their gradients and optimizer state stay resident.

Slower than resident training — but these models did not run on the card at all.

What's New

training.stream_layers: true

# soup.yaml — then just: soup train --config soup.yaml
training:
  stream_layers: true      # base streams from RAM; only the adapter trains
  stream_source: ram       # 'auto' resolves to this; 'disk' is v0.72.2
  stream_buffers: 2        # double buffering (2–8)
  batch_size: 1
  quantization: none       # NF4 streaming lands in v0.72.1

It is a config key, not a CLI flag.

Measured numbers

All on the development box — RTX 3050 Laptop 4 GB, Windows 11, 16.9 GB RAM,
batch 1, gradient checkpointing on, 50 steps after 10 warm-up:

Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run OOMs.

The honest cost: 1.43× slower than resident, measured at 0.5B — the only
apples-to-apples comparison available on this box, because 1.5B and above cannot
run resident there at all.

Correctness was gated before any of it was written

The mechanism was proven before a line of src/ existed:

• streamed vs resident logits: bit-exact (max abs diff 0.0)
• layer-0 LoRA gradient non-zero, on all 30 layers — a severed graph still lowers
  the loss, so this is the only check that catches it
• 100-step loss curve identical to resident
• same seed twice → identical loss (catches stream races)

Those properties now run in CI on every platform.

Under the hood

• soup_cli/utils/layer_stream.py — tier choice, pinned-vs-pageable decision,
  architecture allowlist, VRAM/throughput arithmetic (no torch import)
• soup_cli/utils/layer_shard.py — rewrites a checkpoint into one safetensors
  shard per decoder layer, one tensor at a time, so sharding a model that does
  not fit never needs it to fit
• soup_cli/utils/layer_stream_runtime.py — buffer pool, CPU-RAM weight source,
  prefetch scheduler on a dedicated CUDA stream, streamed layer wrapper

Shards are cached under ~/.soup/layer-stream/ (override with
SOUP_LAYER_STREAM_CACHE_DIR).

Install / Upgrade

pip install -U "soup-cli[train]"

Layer streaming needs the [train] extra (torch / transformers / peft / trl).
Requires an NVIDIA GPU and enough system RAM to hold the base model.

Security

• The shard cache is keyed to a fingerprint of the source checkpoint (per-shard
  name + size + mtime), not just the model name. A base retrained in place
  re-shards instead of silently training against stale weights — a slug-keyed
  cache with only a dtype check would not have caught that.
• shard_checkpoint bounds its own writes to $HOME / $CWD / $TMPDIR
  via realpath rather than trusting its caller, resolves symlinked ancestors,
  and skips symlinked source shards.
• SOUP_LAYER_STREAM_CACHE_DIR is ignored if it contains control characters or
  escapes those roots.
• DoS caps on layer count, tensor element count and total tensor count; atomic
  writes throughout; an oversized base is refused before minutes of disk I/O.

Known Limitations

• BETA, and proof-of-mechanism at 3B. Nothing above 3B was measured on this
  hardware. No 8B / 14B claim is supported by any number here.
• The 3B figure is a lower bound. This box tops out at 7.65 GB of page-locked
  host memory, so a 5.55 GB base plus a CUDA context did not fit and that run used
  a pageable store, which makes the host-to-device copy synchronous. The cost is
  visible as GPU utilisation 96.8% (pinned, 1.5B) → 79.3% (pageable, 3B). The
  fallback is automatic and prints the cost rather than absorbing it.
• Scope is frozen — RAM tier, bf16, task: sft, Llama/Qwen, batch size 1, no
  gradient accumulation, no --resume. Every refusal names the release that lifts
  it: 4-bit (NF4) streaming is v0.72.1; the disk tier, larger batches,
  accumulation, checkpoint/resume and more architectures are v0.72.2;
  preference losses are v0.72.3.
• expandable_segments:True is silently ignored on Windows — Soup probes for
  it and never claims it is active.
• Numbers are Windows/WDDM and therefore systematically pessimistic vs Linux.
• The shard cache costs disk space roughly equal to the base model.
• The pre-flight prints the plan (tier, store size, buffers) but no tok/s
  forecast — a fabricated per-card estimate would break the measured-numbers-only
  rule, and the only calibration point is this one 4 GB laptop.
• MPS (Apple Silicon) is untested; the end-to-end training-step test is skipped
  there rather than asserting support that was never run.
• The 14B-on-8 GB reference benchmark is hardware-blocked — it needs an 8 GB
  card and 32 GB of RAM.

Tests

16576 → 16735 (+159 in tests/test_v07200.py), 324 test files.