MakazhanAlpamys/Soup v0.71.22

v0.71.22 — Perf & measure polish

on GitHub

What's New

Performance & measurement polish — four hot-path lifts validated live on a real RTX 3050.

• MiniLLM on-policy KV-cache (#263) — the on-policy reverse-KL rollout now threads past_key_values so each step forwards only the new token. The cache-support probe unwraps a PEFT/LoRA student via get_base_model(), so LoRA distillation students actually get the cache. Cached output is bit-for-bit identical to the no-cache path; resolves the v0.71.18 O(L²) rollout limitation.
• soup serve --mole KV-cache (#262) — each of the N task adapters gets its own KV cache in lockstep, created fresh per generate() call (no cross-request leak). Byte-identical to the legacy path on a real MoLE; top-k zero-weight adapters still skipped.
• deploy-autopilot live generator factories (#143) — a first-party transformers loader factory replaces the inject-only hooks; run_measure scores the baseline once and pre-validates the whole candidate list up front, so a typo no longer burns N live model loads or doubles peak VRAM. The _DEPLOY_MEASURE_* test seams are retained.
• Live-codec TTS, partial (#265) — load_audio_mono probes soundfile.info for duration + byte caps before decoding and reads via an O_NOFOLLOW fd (TOCTOU-safe). A real SNAC-backed Orpheus encode of a 24 kHz wav produced 42 codec tokens. The other four TTS families keep their per-family codec dependency gate.

Install / Upgrade

pip install --upgrade soup-cli

Security

• TOCTOU-safe audio reads (O_NOFOLLOW fd + soundfile.info metadata pre-probe + byte cap before any multi-GB decode-into-RAM).
• deploy-measure candidate echo truncated to 64 chars; live-measure failures surface a friendly message with no traceback leak.

Known Limitations

• KV-cache correctness is validated (cache == no-cache equality) on tiny artifacts; large-model throughput gain is not measured on the 4 GB dev box.
• #265 stays open — the live-codec data.format='audio' SNAC encode path is validated for Orpheus only; the other four TTS families keep their per-family codec dependency gate.
• deploy-measure's real quantized (bnb 4bit) load is CUDA + bitsandbytes-gated; on Windows / no-bnb the injected _DEPLOY_MEASURE_* seams are the validated path.
• MoLE serve KV-cache assumes single-sequence (B == 1) decode.

Full suite: 14067 passed / 117 skipped, exit 0 (14184 tests total; 293 files).