MakazhanAlpamys/Soup v0.42.0

v0.42.0 — Data Pipeline Pro

on GitHub

What's New

18 data-tooling features in one release — closes the gap with Axolotl + LlamaFactory + Unsloth.

• 5 new data formats — prm (PRM stepwise-supervised), pre_tokenized (skip auto-tokenize), input_output (template-free segments+labels), video, multimodal (axolotl typed content-parts: text / image / audio / video). Each is loud-fail validated.
• Remote dataset loading (schema gate live; fsspec wiring deferred to v0.42.1) — data.train: s3://... / gs:// / gcs:// / az:// / abfs:// / abfss:// / oci://. Bucket regex per S3/GCS spec; userinfo / fragment / query strings rejected (the query-string gate is SSRF-adjacent — fsspec backends interpret ?endpoint_url=... as config overrides). Plus streaming: bool, buffer_size: int [1, 1_000_000], shards: int [1, 1024].
• AOT preprocessing — new soup data preprocess <config> emits a deterministic 16-char SHA-256 cache key from (dataset, tokenizer, max_length, format). Pair with the new data.tokenized_path schema field.
• Multi-dataset interleave + advanced masking — data.interleave: concat / under / over / {strategy: probs, probs: [...]} (sum-to-1 ±1e-6, max 32 datasets). Plus mask_history, train_on_prompt (mutually exclusive with train_on_responses_only), eval_on_each_dataset, split_thinking for Qwen3 <think> masking, image_min_pixels / image_max_pixels / image_resize_algorithm, video_fps / video_maxlen / video_dir.
• Vocab expansion + custom prompt strategies — add_new_tokens / new_special_tokens (cap 10_000, no duplicates, null-byte rejected) + resize_vocab cross-validator. New prompt_strategy: my_module.path:my_fn field (regex-validated; runtime invocation in v0.42.1). Plus skip_prepare_dataset / remove_unused_columns escape hatches.
• Document ingestion — new soup data ingest <doc.pdf> (PDF / DOCX / MD / TXT) → JSONL. Lazy-imports pypdf / python-docx so missing optional deps never break soup data --help. Symlink-rejected, cwd-confined.

Install / Upgrade

pip install -U soup-cli

Or via Docker: docker pull ghcr.io/makazhanalpamys/soup:v0.42.0

Existing v0.41.x configs continue to load unchanged — every v0.42.0 field is optional and defaults to its v0.41.x behaviour.

Security

• _REMOTE_SCHEMES wrapped in MappingProxyType so the allowlist cannot be mutated at runtime (mirrors v0.36.0 / v0.41.0 policy).
• validate_remote_uri rejects userinfo / fragment / query string. Query-string forwarding to fsspec backends would be SSRF-adjacent (config overrides like ?endpoint_url=...).
• Bucket regex ^[a-zA-Z0-9][a-zA-Z0-9._\-]{0,62}$ matches RFC-3986 + S3/GCS naming (1-63 chars).
• Bool-rejected-before-int on every numeric input (buffer_size / shards / image_pixels / video_maxlen / num_datasets / max_length / PRM labels[i] — bool is a subclass of int).
• Null-byte + length caps on every string-shaped input (paths, URIs, tokens, regex patterns, video field, prompts).
• is_under_cwd containment on video_dir / tokenized_path schema fields and on soup data preprocess --config / --output and soup data ingest <file> / --output (uses os.path.realpath + commonpath, Windows 8.3-short-name safe — NOT Path.resolve() + relative_to()).
• os.lstat + S_ISLNK symlink rejection on the ingest input file (TOCTOU defence — mirrors v0.33.0 #22 prune_checkpoints policy).
• PRM converter type-checks every completions[i] (string) and every labels[i] (bool, NOT int — matches v0.30.0 Candidate policy); caps at 10_000 steps.
• InterleaveSpec is a frozen dataclass; mutation raises FrozenInstanceError (matches v0.32.0 / v0.39.0 / v0.41.0 policy).
• Image-pixel validator threads info.field_name so error messages name the actual field that tripped — fixes a misleading-error UX gap caught in code review.

Known Limitations

• fsspec live loaders deferred to v0.42.1 — schema accepts s3:// / gs:// / etc. on data.train, but the live loader path raises a friendly ImportError advisory; the schema gate prevents misconfiguration. Same stub-then-live pattern as v0.27.0 MII / v0.37.0 multipack / v0.41.0 LLaMA Pro.
• AOT preprocess live tokenize deferred to v0.42.1 — soup data preprocess emits the planned cache key + target path so users can wire data.tokenized_path into their YAML; the actual tokenize loop lands in v0.42.1.
• Custom prompt-strategy runtime deferred to v0.42.1 — data.prompt_strategy: my_module:my_fn parses at schema load but is not invoked yet; runtime importer + transform call lands in v0.42.1.
• PRM live trainer wiring deferred to v0.50 — format='prm' parses + validates via _convert_prm but the upstream PRM trainer integration ships in v0.50 per the original roadmap.
• Markdown ingest does not split on headings — current soup data ingest writes the whole MD file as a single row. Heading-aware splitting tracked for v0.42.1.
• Video / multimodal converters validate path/url null-byte and length but do NOT verify the underlying file exists or is readable — file existence is the trainer's responsibility (matches v0.40.5 reward_model policy).

Tests

5242 → 5389 (+147 net new tests). Lint clean across soup_cli/ + tests/.