MakazhanAlpamys/Soup v0.49.0

v0.49.0 — Long Context & Architecture

on GitHub

What's New

Four RoPE-scaling additions that close the long-context gap with Unsloth and LlamaFactory — YaRN, Dynamic NTK hardening, LongLoRA S² shifted-sparse attention (schema-only), and full Llama 3.1 NTK-aware scaling.

• YaRN RoPE scaling. New training.rope_scaling_type: yarn with explicit tunables yarn_factor / yarn_attn_factor / yarn_beta_fast / yarn_beta_slow. Pure-Python kernels (yarn_find_correction_dim, yarn_find_correction_range, yarn_linear_ramp_mask, yarn_get_mscale) mirror the YaRN paper §3.4/§3.5 with bool / NaN / Inf rejection on every numeric input. The range helper auto-disambiguates high <= low so the ramp-mask denominator is non-zero on adversarial inputs.
• Llama 3.1 NTK-aware (full impl). New training.rope_scaling_type: llama3 emits the canonical {factor, original_max_position_embeddings, low_freq_factor=1.0, high_freq_factor=4.0} block. scale_inv_freq_llama3 classifies each frequency by its wavelength relative to old_context_len=8192. detect_llama3_rope_in_config reads HF model configs and accepts both rope_scaling.type and rope_scaling.rope_type keys.
• LongLoRA S² shifted-sparse attention (schema gate). New training.use_longlora: true requires task=sft, backend=transformers, a Llama-family base, and use_ring_attention=false. Schema gate ships now so misconfigured runs fail fast; live forward override deferred to v0.49.1.
• Dynamic NTK hardened. Existing path verified; get_rope_scaling_config('dynamic', ...) accepts both absolute target lengths and scaling factors.
• Public-API hardening. get_rope_scaling_config validates target_length / original_length / yarn_factor for bool / NaN / Inf / non-positive at the public boundary — prevents direct callers from emitting {factor: NaN} into HF model configs even when bypassing the Pydantic schema.

Install / Upgrade

pip install --upgrade soup-cli
# or
pip install soup-cli==0.49.0

Security

• Public-boundary input validation on soup_cli/utils/long_context.py::get_rope_scaling_config — target_length / original_length / yarn_factor reject bool / NaN / Inf / non-positive (matches v0.30.0 Candidate / v0.34.0 estimate_run_cost_usd / v0.41.0 Part B lr_groups policy).
• scale_inv_freq_llama3 rejects bool on every numeric parameter (inv_freq, scale_factor, low_freq_factor, high_freq_factor, old_context_len).
• yarn_get_mscale raises on non-finite input rather than silently clamping to identity.
• detect_llama3_rope_in_config uses explicit is None instead of or idiom — falsy-but-set values cannot silently fall through (matches v0.40.6 review-fix policy).
• is_llama_model uses a word-boundary regex with null-byte rejection + 512-char cap — substring "llama" inside an unrelated identifier does NOT match (mirrors v0.39.0 is_gemma4_model / v0.44.0 is_llama4_model policy). ReDoS-bounded by the size cap and the absence of nested quantifiers.
• validate_longlora_compat emits distinct error messages per failure mode (mlx vs other non-transformers backends) per v0.34.0 review-fix policy.

Known Limitations

1. LongLoRA live LlamaAttention.forward override deferred to v0.49.1 — apply_longlora_forward_override raises NotImplementedError with the v0.49.1 marker; the schema gate ships now so misconfigured runs cannot reach trainer construction. Mirrors v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro stub-then-live pattern.
2. LongLoRA architecture allowlist is Llama 1/2/3.x + CodeLlama only — Mistral / Qwen / Phi expansion tracked for v0.49.1+.
3. Llama 3.1 NTK auto-detect helper ships but is not yet wired into apply_long_context_config — detect_llama3_rope_in_config is exposed so trainer wiring can pick it up when needed.
4. YaRN math kernels are reference / config-emit — the actual RoPE rotation runs inside HF Transformers' _compute_yarn_parameters. The pure-Python kernels are exposed for testing and direct config emission.

+80 net new tests (6410 → 6490). Full suite green on ubuntu / windows / macOS × Python 3.9 / 3.11 / 3.12.