github MakazhanAlpamys/Soup v0.78.0
v0.78.0 — a continual-learning loop, soup ship per-row audit, native MLX chat and merge

one hour ago

v0.78.0, 2026-11-18

218 of the 271 pull requests in this release came from 48 people outside the maintainer.
Thank you: the Contributors section below names each of these contributors and what was found or built.

v0.78.0 is a cut of main after v0.77.2, not a backport: it carries every pull request merged since that tag. By the CHANGELOG count that is 60 additions, 20 changes, 2 deprecations, 169 fixes and 3 security entries. This is not a fixes-only release, and it refuses some input v0.77.2 accepted: a boolean in a numeric training field, an enabled eval gate on a path that never ran one, nan on six numeric options, an --output that is one of soup ship's own inputs, and more. It also changes some numbers: soup adapters merge weights, the tool_call_args_subset score (which can move a soup ship verdict and a soup eval retention result), the VRAM pre-flight, and the weights of classifier, reranker, cross_encoder and asr runs. Read the upgrade notes under Install / Upgrade before you upgrade a config or a script. Coming from v0.76.x or older, read the v0.77.0 notes first and run:

soup doctor --config soup.yaml

Every change, with its pull request, is in CHANGELOG.md, section [0.78.0]. Compare with the previous release: v0.77.2...v0.78.0.

What's New

Added

  • A continual-learning loop (#2897, which folded 17 stacked pull requests into one). soup serve gives every /v1/chat/completions and /v1/messages answer a response_id (also the X-Response-Id header), --trace-log writes one v1 conversation record per answer, and --feedback-log turns on POST /v1/feedback (thumb, rating, edit or comment, by id) (#2532); soup serve --adapter-dir with POST /v1/adapters/load loads a LoRA adapter into a running server without a restart (#2516). soup loop learns from thumbs, ratings and edits with sft (the new default), dpo or kto (#2538, #2560), paraphrases the prompts people corrected (#2573), judges a candidate with soup ship --gate loop-v2 by the corrections it learned (#2570, #2588), starts a canary at 5% of keyed traffic and withdraws it when people rate it worse (#2518, #2539), and writes iteration, version and progress logs (#2554). New: soup loop set and soup loop rollback, soup loop watch --detach with a PID file and a log (#2540), soup ship --base-adapter to compare two adapters (#2511), and a soup loop canary that refuses an adapter whose last verdict was DON'T SHIP, named or given as the path of its directory under .soup-loops/adapters (--allow-dont-ship overrides it and the override is recorded; #3002). The six JSONL formats are defined in docs/loop-formats.md with JSON Schemas (#2505).
  • soup ship shows what its number is made of (#2870, #2894, #2831, #2933). Per task it counts the rows that got better, worse and stayed the same, the answers cut off at the token limit and the failed generations; --output verdict.json also writes a per-row audit file verdict.task-rows.jsonl; it prints how much of the base the adapter's update changes; verdict and evidence files carry schema_version: 2. --task-mode reward --reward-fn <accuracy|format|verifiable:math|verifiable:json_schema|path.py> scores each answer with a function from 0 to 1; a .py file is executed with your permissions, only when named, and its hash is recorded.
  • Layer streaming does more. A forward-only inference mode built from an existing shard cache, with adapter reload (soup_cli.utils.stream_inference, #2509, #2510, #2636); a pre-quantised bitsandbytes NF4 checkpoint imports byte for byte and trains under stream_layers: true (#2628, #2653); BCO and IPO train on a streamed base (#2896, #2915), and so does task: preference with a single dpo, orpo or simpo loss (#2843); a checkpointed NF4 layer switches to a packed-state backward when the pre-flight predicts the dense one would not fit (#2655); on Windows the disk tier's page-locked staging checks system commit headroom first (#2658); soup prune-layer-cache removes stale cache slots, and the disk pre-flight lists them (#2647).
  • More on Apple Silicon. Native MLX chat (soup chat --backend mlx, #2123), fusion into a standalone dense checkpoint (soup merge --backend mlx, #2124), selective per-message loss masks (#2128), consumption of existing preprocessing caches (#2131), vocabulary expansion with trainable new tokens (#2684), linear RoPE extension (#2710) and dense Llama SFT (#2818).
  • New commands and flags. soup eval retention measures forgetting across a series of model versions against a stored record of pinned control tasks, the file named by --ledger (#2552); soup data leakage reports exact input-text overlap between training and evaluation rows (#2833); soup docs search searches the bundled guides offline (#2716); soup eval quant-check --performance adds latency, tokens per second and peak memory (#2689); soup export --format onnx --validate-onnx compares an ONNX graph's logits with its source (#2695); soup diagnose --evidence --require-all-modes (#2857); soup deploy hf-space can pick a hardware tier (#2815).
  • Training and tracking. The experiment tracker records a GRPO run's per-step score and its spread beside the loss (#2891), the live panel shows zero-spread groups (#2819) and the estimated finish time (#2707), runs record the project's Git commit and clean/dirty state (#2692), and soup runs compare shows full config differences and the latest stored evaluation scores (#2706). task: classifier reports accuracy and F1 (#2789), and reranker and cross_encoder report ranking quality when validation rows carry query_id (#2637).
  • Eight recipes (181 in all): qwen3.5-4b-grpo, qwen3.5-27b-grpo, qwen3.5-122b-a10b-dpo, qwen3.5-122b-a10b-grpo, qwen3.5-397b-a17b-grpo, minimax-m3-dpo, qwen3.5-9b-pretrain and qwen3.5-2b-pretrain (#2522, #2605, #2606, #2660, #2718, #2719, #2720, #2721).
  • A library of Triton kernels for the Qwen2 QLoRA SFT step, soup_cli.kernels.packed_lora (#2623). It has no config key and nothing in soup train reads it yet. Also: a guarded resident route for Kimi-K2-Base and -Instruct on transformers 5.x (#2759), soup doctor prints an NF4 rounding note under an RTX 50-series card (#2587).

Changed

  • soup loop defaults and rules (#2560, #2539, #2532): soup loop init makes an sft loop; a canary is judged by the ratings on its answers with a significance test; soup serve refuses --trace-log, --record-thumbs and --feedback-log on the vLLM, SGLang and MII backends instead of ignoring them; a --trace-log line has a new shape.
  • The VRAM pre-flight prices differently. Padded SFT runs are priced by their longest row, so runs it refused because of data.max_length now start (#2934); 8-bit runs with checkpointing are charged for what bitsandbytes keeps (#2925); layer streaming is charged its cuBLAS workspaces (#2749); a task: distill run is priced with its teacher (#2950); soup profile --model, soup profile and the MCP profile tool take seq_len and shape (#2871).
  • Resident 4-bit and 8-bit runs on a bf16 card keep embeddings, norms and an untied head in bf16 instead of fp32 (#2683): about 25% faster and 3.8 GiB lower peak on Qwen2.5-7B NF4 on an RTX 4090, loss unchanged within a tolerance.
  • soup train --push-as creates the repository private; --push-public keeps it public (#2802). soup train --cloud refuses a config whose local files cannot be shipped (#2627). soup serve --max-tokens above 16384 exits 2 (#2898). training.vllm_sleep_mode: true is refused on task: grpo (#2773). The mlx extra needs mlx>=0.31.2 (#2779).
  • Numbers that moved. soup adapters merge scales each input's term by its weight instead of its square (#2927); scoring: "tool_call_args_subset" counts Correct only at full credit (#2939); DPO, ORPO and MLX preference training cut a prompt only when the row overflows (#2555, #2659); soup runs compare shows more (#2706); judge task rows are validated before any generation (#2881).

Deprecated

  • training.warmup_auto: true and a training.forgetting_threshold away from 0.1 load with a warning and are refused in v0.79.0 (#2814). soup deploy hf-space with the Streamlit template prints a deprecation warning (#2834).

Fixed: the ones that matter most

  • Config and options are checked where they are read. Booleans in numeric fields (#2462), wrong-typed values in 38 training fields (#2382), integers past the float range (#2912, #2748, #2838), nan on six ranged options (#2917), and an unknown key with no close match now names a next step (#1823). The eval gate refuses a suite that cannot load, a repeated key, a boolean threshold, baseline: previous and an unsupported path instead of training without it (#2264, #2873, #2697, #2640, #2564).
  • Training. soup train --deepspeed no longer stops before the first step under fp16 (#2276); task: classifier, reranker, cross_encoder and asr full fine-tune on fp32 master weights, where 16-bit weights barely moved at the default learning rate (#2949); training.curriculum: true trains in sorted order (#2924); fp16 auto_mixed_precision loads the frozen base in fp16 (#2921); the BOS token survives a prompt cut in KTO, BCO, ORPO and SimPO (#2544, #2561, #2572); Ctrl+C and .checkpoint_now save at an optimizer-step boundary (#2414); soup train --resume on task: ppo is refused instead of restarting from step 0 (#2772); the GRPO watchdog names the parameters whose gradient was not finite (#2645); soup sweep applies the quantization guard of soup train and binds each arm to its base revision (#2665, #2799, #2671).
  • Eval, ship and serve. soup serve --backend transformers honours stop and stop_sequences and streams while it generates (#2630, #2514); --max-tokens is bounded on soup serve, soup chat and soup bench infer (#2855, #2898, #2787, #2788); soup ship evaluates in native bf16 or fp16 for the requested card (#2669); judge and calibration inputs are checked: finite scores and weights, no duplicate criteria, no empty calibration (#2693, #2701, #2699, #2513); soup drift-alarm refuses an input with no tokens instead of saying OK (#2955).
  • No silent overwrites, empty results or crashes. soup airgap-bundle checks size and paths before reading (#2568, #2690); soup adapters merge refuses adapters it cannot merge (#2926); soup data sample, soup data mix --apply, soup iterative-dpo, soup tokenizer train and soup expect report bad input instead of crashing or writing nothing useful (#2931, #2905, #2946, #2958, #2947, #2945); soup ship refuses an --output that is one of its inputs (#2894); files starting with a UTF-8 byte-order mark keep their first record in soup drift-alarm, soup prune-prompt, soup distill-prompt, soup iterative-dpo and soup data active-sample (#2901, #2902, #2959, #2943, #2879).
  • Hub, cloud and export. RunPod jobs use REST API v2, because v1 was retired on 2026-11-15 (#2877); the built-in Gradio and Streamlit Space templates no longer raise TypeError and publish in one commit (#2657, #2641, #2649); Qwen3.5 GGUF export works on a fresh checkout after the llama.cpp pin moved to b12423 (#2910); soup export --format torchao merges the adapter first (#2920); soup merge --low-memory fetches the files and shards a Hub base needs (#2267, #2900).
  • Text shown as typed. More than 30 pull requests make the commands print paths, names and messages as typed: brackets and :name: text are kept and terminal control characters are removed from the display only. What is left is under Known Limitations.

The measurement records behind this release are in benchmarks/ (42 new records; the index is benchmarks/README.md). The pre-flight memory estimate changed in v0.78.0 (#2749, #2871, #2934), so the preprint's validation describes the estimator of v0.77.x.

Install / Upgrade

pip install -U soup-cli               # or: pipx upgrade soup-cli
soup --version                        # soup v0.78.0
soup doctor --config soup.yaml        # before the first run of an existing config
  • From v0.77.2: a valid soup.yaml loads unless it sets a boolean in a numeric training field or one of the other values listed first in the upgrade notes; the notes list the rest.
  • From v0.76.x or older: read the v0.77.0 notes first. Python 3.10 stays on v0.76.x (pip install "soup-cli[train]<0.77"). This release supports Python 3.11 to 3.13. v0.76.x gets no further security fixes after this release.
  • Docker: docker pull ghcr.io/makazhanalpamys/soup:0.78.0 (also latest); the host driver must support CUDA 12.8.

Upgrade notes

A soup.yaml that loads on 0.77.2 loads on 0.78.0, except for the values listed in the first item (refused now). Coming from 0.76.x or older, read the 0.77.0 and 0.77.2 upgrade notes below first, and run soup doctor --config soup.yaml before the first run of an existing config.

  • Config values that loaded on 0.77.2 and are refused now. Each message names the field and what to change.
    • A boolean in a numeric field of training or training.lora (lr: true, batch_size: true, lora.r: false): 0.77.2 coerced false to 0 and true to 1; 0.78.0 refuses it with <field> must be a number, not a bool (or an int). Write the number you meant (#2462).
    • training.vllm_sleep_mode: true on task: grpo: it never reached a vLLM engine; delete the line (#2773).
    • training.eval_gate.enabled: true (or --gate) on backend: mlx and on task: prm, unlearn, moe_lora_routing: these paths never attached a gate and trained without one; set enabled: false or remove the block, and run soup eval or soup ship after training instead (#2564). A training.eval_gate.suite that cannot load now stops soup train with exit 1 instead of training without a gate: fix the suite file the message names, or disable the gate (#2264). training.eval_gate.baseline: previous is refused; use a path to a baseline results file (#2640).
    • Eval suites and rubrics: a boolean threshold, a repeated key in a task mapping, duplicate rubric criterion names and a non-finite rubric weight are refused (#2697, #2873, #2699, #2701); --write-baseline onto the suite, custom-task or judge-prompts file exits 3 (#2694); a calibration file with calibrated: true and num_pairs: 0 is refused (#2513).
    • Data: reranker / cross_encoder query_id of mixed or non-string/non-integer types (#2637), task: embedding rows where only some have a negative (#2563), a task: unlearn retain_set with no usable row (#2577).
  • Deprecated, refused in v0.79.0: training.warmup_auto: true and training.forgetting_threshold set away from 0.1 load with a warning that they do nothing; delete them (#2814). soup deploy hf-space with the Streamlit template prints a deprecation warning (#2834).
  • Commands that stop where they used to continue.
    • soup train --push-as <repo> now creates the repository private; add --push-public to keep creating a public one (#2802). soup train --cloud refuses a config with local input files at plan time, and --push-as / --gate with --cloud (#2627).
    • soup serve --max-tokens above 16384 exits 2 before any model loads (#2898); soup chat --max-tokens and soup bench infer --max-tokens below 1 exit 2 (#2787, #2788). soup serve --trace-log, --record-thumbs and --feedback-log are refused with --backend vllm, sglang and mii (exit 2; they were silently ignored there): drop the flag, or serve with --backend transformers to keep recording (#2532).
    • soup train --resume / --hf-resume on task: ppo is refused (#2772). One Ctrl+C no longer stops a Hugging Face Trainer run: the first saves and continues, the second saves and stops, the third interrupts (#2414).
    • soup ci init --python with a leading zero (03.11) exits 1 (#2632); soup airgap-bundle with a missing folder exits 1 (#2568); soup adapters merge of adapters it cannot merge exits 2 (#2926); soup merge --low-memory of an adapter whose init rewrites the base is refused (#2732); soup data generate asks again until --count and exits 1 after three empty requests (#2418).
    • nan is refused like inf on soup data validate --min-valid-fraction, soup draft measure --min-acceptance, soup data toxicity --threshold, soup data decontaminate --threshold, soup data score --threshold and soup data forge --uncertainty-threshold, with a usage error that names the option (exit 3 for data validate, 2 for the others) before any input is read; pass a number (#2917). soup data sample refuses -n below 1 and a --pct that is not in (0, 100] with exit 1 (#2931). soup drift-alarm exits 1 on an input with no tokens instead of reporting OK; fix the producer of that input (#2955). soup iterative-dpo with no usable prompt row exits 1 before it loads a model or writes pairs (#2958). soup tokenizer train --output naming an existing file is refused before training (#2947).
    • Exit codes: soup data mix --apply with invalid YAML or invalid recipe weights 1 -> 2, so a script that tests for 1 must test for 2 (#2905, #2946); soup expect with an unreadable suite or dataset 1 -> 3, and exit 1 now means the judge was unreachable (#2945).
    • soup ship: when every task generation fails on one side, --task-mode metric exits 1 (#2870); --output names that cannot hold <stem>.task-rows.jsonl exit 3 (#2894); an --output or --emit-evidence that is the same file as one of the run's inputs, or as the other output, exits 3 before anything is read, loaded or written, where it used to replace the input with the verdict (#2894); an adapter directory whose weights are a pickle (an adapter_model.bin, .pt, .pth, .pkl, .pickle or .ckpt file, or a pickle posing as .safetensors) is refused on --adapter and --base-adapter with exit 3 (unsafe weight file (pickle / PyTorch-classic): adapter_model.bin): re-save the adapter as safetensors, for example model.save_pretrained(path, safe_serialization=True), and pass that directory (#2897); the verdict, the evidence and --output gain keys, none removed, including a top-level base_adapter that is null unless --base-adapter is used.
    • soup can run refuses a can that holds <redacted> values and names their key paths: supply the values in the can's config.yaml and run soup train on it, and keep credentials out of a config before packing it (#2883).
  • soup ship --emit-evidence writes schema_version: 2. soup-cli 0.77.x refuses such a file (evidence has unsupported field(s): 'judged', 'schema_version', 'task_rows', exit 3); upgrade whatever reads the evidence (the CI gate, the MCP ship_evidence tool). 0.78.0 reads versions 1 and 2 (#2870).
  • soup adapters merge weights. Each input's own term is now scaled by its weight, where 0.77.2 used the weight squared: the numbers of every linear, dare and svd merge of two or more inputs change (an equal-weight pair doubles each input's own term, 0.25 to 0.5), and the weights an earlier --strategy cmaes run printed reproduce a different adapter. Re-run merges whose output you compare or ship (#2927).
  • A scoring rule got stricter. scoring: "tool_call_args_subset" counts a row as Correct only at full credit, so soup eval custom, soup ship metric mode and soup eval retention report a lower Accuracy for suites that use it, and the exit status can change for an unchanged model (a soup ship that ended 0 can end 2). Record base and tuned scores again with this version before comparing (#2939).
  • The VRAM pre-flight prices differently. An 8-bit run with gradient checkpointing is priced higher (#2860 in #2925), layer streaming charges the cuBLAS workspaces (#2749), on Windows it checks system commit headroom before page-locking (#2658), and a task: distill run is priced with its teacher and divergence (#2950): a run that launched close to the card's limit can be refused; --allow-oom-attempt, a lower batch_size or data.max_length launch it. A padded text SFT run the old pre-flight refused because of data.max_length now starts when its longest row fits (#2934). soup profile / the MCP profile tool take seq_len and shape (#2871), and their JSON gains teacher_memory_gb, distill_memory_gb and teacher_generation_memory_gb (0 outside task: distill).
  • Runs are not bit-comparable with 0.77.2 for: resident 4-bit / 8-bit on a bf16 card (embeddings, norms and an untied head stay bf16; faster, lower peak, #2683), fp16 auto_mixed_precision on a bf16 card (#2921), curriculum: true (rows now in sorted order, #2924), the BOS token kept after a keep_end prompt cut (#2544, #2561, #2572), the DPO / ORPO / MLX prompt cut (#2555, #2659), soup data split --stratify and --stratify-semantic, which now give the requested validation and test counts, so the same seed can place different rows in train, validation and test (#2954), Fast-LoRA 4-bit dtype boundary (#2383, #2546), soup ship on a CUDA card without bf16, which now evaluates in float16 (#2669), and task: classifier, reranker, cross_encoder and asr, which now full fine-tune on fp32 master weights: twice the memory over a 16-bit checkpoint, so a run that fit the card on 0.77.x can run out of memory (train an adapter with classifier_lora: true or asr_lora: true, or lower training.batch_size; #2949).
  • soup loop (already in 0.77.2) behaves differently. soup loop init without --train-method now creates an sft loop that trains on corrections (existing loop.yaml files keep their method; pass --train-method dpo to keep preference pairs, #2560); the pre-wired harvest reads the feedback log and trains only when new feedback arrived (#2538, #2531); a canary is judged by the ratings on its answers with a significance test, needs 30 rated answers on each side unless --canary-max-wait ends the wait, and a canary that got no answers is withdrawn (#2539); the iteration, version and progress logs are written (#2554); soup loop watch --detach leaves .soup/loop.pid and .soup/loop.log and refuses a second --detach (#2540); a --trace-log line has ts as an ISO 8601 UTC string and tokens as an object, and a line whose schema_version is not 1 is skipped by soup loop and soup data from-traces (#2532). A hand-edited loop.yaml with a monthly_budget_usd or spent_this_month_usd that is not finite or is above 1e15 was read before and is refused now (exit 2, naming the field): write a number from 0 to 1e15, or remove the line (#2897).
  • Databases, servers and files. ~/.soup/experiments.db gains nullable columns in place on first use; loss, lr and speed may be NULL where 0.77.2 stored 0.0 (#2589, #2810, #2891); a database written by 0.78.0 stays readable by 0.77.2. soup ui and the MCP run tools return output_dir relative to the working directory, or a bare folder name; read the absolute path from your own records or from soup runs show on the host (#2807). soup serve --backend transformers honours stop / stop_sequences and streams while it generates (#2514, #2630).
  • Crash bundles and .can files are redacted before they are written; read a bundle before sharing it (#2883). RL-checkpoint optimizer state loads with weights_only=True (#2916).
  • GGUF export: the llama.cpp pin moved from b5270 to b12423; delete ~/.soup/llama.cpp to have the next export clone the pinned tag (#2910). The mlx extra now needs mlx>=0.31.2 (#2779).
  • Python API: JudgeResults.compute() and the calibration helpers raise ValueError for non-finite or oversized inputs (#2701, #2838); decide_performance_flags treats an explicit vram_headroom_gb=0.0 as zero headroom (#2885); soup_cli.utils.profiler.profile_from_config returns a truthy soup_cli.utils.hardware_fit.FitUnavailable (with .field and .reason) instead of None when a run cannot be priced, so test isinstance(result, dict) (#2871).

Security

This release carries 3 entries under Security in the CHANGELOG, from changes merged on 2026-11-16 (2) and 2026-11-17 (1). No advisory is published with it. A security review of the code merged for this release, run on 2026-11-17, recorded 9 findings (0 critical, 1 high, 1 medium, 7 low): 5 are fixed in this release and 4 are listed under Known Limitations. See SECURITY.md for how to report a problem and for the supported versions: 0.78.x is supported, 0.77.x gets security fixes until v0.79.0 is released, and the 0.76.x line ends with this release.

Known Limitations

New in this release:

  • soup_cli.kernels.packed_lora is a library: it has no config key and soup train does not read it. Its GPU tests run on Linux CUDA only (pytest -m gpu --no-cov) and its Triton modules are left out of the coverage measurement (#2623). By design; no issue.
  • Layer-streaming inference is forward only: no KV cache, no generation, no command (#2509 stays open).
  • The VRAM estimate is fitted on small dense models. The 8-bit checkpointing term is calibrated on three (Qwen2.5-0.5B, Qwen2.5-1.5B, SmolLM2-360M) and gradient_checkpointing: medium is outside the calibrated range and only warns (#2925); the padded-SFT lift reads at most 20,000 rows or 90 seconds and otherwise prices at data.max_length (#2934); a distillation step counts as outside the calibrated range, and uld_strategy and minillm_enabled are priced at the largest measured figure (#2950); a micro-batch that fills 512 tokens can still need more than the estimate (#2871). Tracked in #3021 (calibration beyond 1,024 tokens per micro-batch).
  • The Apple Silicon features (chat, merge, caches, vocabulary expansion, RoPE extension, dense Llama SFT, preference and GRPO) are covered only by the hosted macOS MLX test job and fixtures. By design; no issue.
  • The RunPod controller uses REST API v2 from the published reference and has not been run against a real account; the remote script still depends on the Pod image's system Python (#2877, #2098, #2192).
  • soup ship --task-mode reward with a .py file executes that file with your permissions: the recorded hash says which file scored a verdict, it is not a defence against a hostile file. verifiable:code and judge_rank stay refused. By design; no issue.
  • Kimi K2: the resident route for Base and Instruct is guarded and rests on tiny-checkpoint evidence only, the kimi-k2-sft recipe stays unavailable, and Thinking and Instruct-0905 have no supported route (#2759, #2104).
  • GGUF export clones llama.cpp b12423 by tag, not by commit, and an existing ~/.soup/llama.cpp checkout is reused as it is; a file written by the new converter can differ from one written with b5270 (#2910). By design; no issue.
  • The loop: disk. .soup-loops/ grows with every training run and nothing is pruned or rotated: each run keeps its adapter, recipe, logs and verdict files, pairs/<hash>.jsonl is rewritten for each new set of rows, and each append to iterations.jsonl or versions.jsonl reads the file once. Archive old .soup-loops/adapters/* by hand once they can no longer be a rollback target. Tracked in #3015 (soup loop prune).
  • The loop: running and stopping. soup loop init --force does not check for a running daemon; there is no soup loop stop (soup loop pause takes effect at the next cycle boundary and a running soup train child keeps going); two foreground soup loop watch processes are not guarded against each other; a stop between an adapter load and the end of its cycle, and between a rollback or canary log line and the loop.yaml update, is not reconciled. Running the same command again is safe. Tracked in #3016 and #3017.
  • The loop: deploy. A refused deploy is retried once per poll, three times, with no back-off: a soup serve restart longer than a few minutes ends the retries, and the adapter must then be deployed by hand with soup loop canary <adapter directory>. Tracked in #3018.
  • The loop: feedback is training data. On a loopback server started without --tool-auth-token any local user or process can post to /v1/feedback; start soup serve with a token on a shared machine. The loop does not check an edit for intent, and the loop-v2 gate checks a candidate against the corrections it was trained on, so a poisoned edit that the model learns passes that check; the canary's ratings and soup loop rollback are the other defences. soup loop reads --trace-dir files whole: keep --trace-log-cap-mb at its default on a server others can reach. By design; no issue.
  • The loop: processes and files. The loop's own children start without the current directory on sys.path (-P); a Python process that soup train or soup ship starts in turn does not, so run the loop in a directory you control. A stage that raises is retried after 5 minutes, then 30, and held after the third failure in the same stage until soup loop resume; failures in alternating stages restart the count. soup ship refuses an --output that is one of its inputs but does not compare the training file a --config names: do not name your training data as an output. Tracked in #3019 and #3020.
  • The loop: quality. The default recipe (all linear modules, r 64 / alpha 128, lr 2e-4, 8 epochs) regressed small evaluation suites in a study on one project; milder defaults and anchors are untested. A 40-row suite's base side read 0.900 once and 0.925 six times on the same model between soup ship calls, so a 0.05 threshold is half noise; and eval.custom.score_semantic is word overlap, so a short correct answer against a long reference can score 0.0. Tracked in #3014, #3012 and #3013.
  • Some commands still print an exception's text through Rich without escaping it, so a name taken from an untrusted file can render as markup in an error line; the tracker is #2489. Two patterns in the paraphrase step and the loop-v2 gate are slow on a pathological reply, which comes from the paraphrase service you configured and is limited to 64 KiB.
  • Three guards in the loop's reconcile code and in the per-row file writer have no test that fails when they are removed, and a native-RoPE test compares float tables with exact equality and failed once on a hosted runner (#2948). The guards are tracked in #3022.
  • A live soup ship verdict is slow on a laptop card: a full verdict for a 0.5B model took 35 minutes for 20 task rows plus the eight bundled mini suites on an RTX 5070 Laptop (8 GB), with other work on the machine, at about 10% GPU utilisation and one busy CPU core, and it prints nothing between loading the models and the result panel unless you pass --progress jsonl. Expect tens of minutes on a card of that class. On a quiet box with CPU only, a 135M model and 5 task rows, a live verdict took 324 s with a .py scoring function and one general suite, 544 s with --noise-floor 2, and 1252 s for the verdict child of a soup loop iteration, which runs all eight bundled mini suites. Tracked in #3011.
  • The gpu-marked test test_configured_workspace_is_visible_to_allocator_in_fresh_process fails on a CUDA card because its child script has no graph for backward (#2991); the fix is #2994, in the release after this one. Hosted CI does not run gpu-marked tests.
  • The loop's quantization: auto picks 4-bit when the Hub is offline, because the base's parameter count cannot be read: an offline soup loop run trains in 4-bit unless --sft-quantization none is given. The reason is written to the iteration record (sft quantization auto: 4bit (parameter count unknown)) but not printed. Tracked in #3033.
  • Partial fixes whose issues stay open: the settings a backend or task never applies are refused only on the paths fixed so far (#1811); the base-revision binding covers the transformers backend, soup merge, soup sweep and native MLX text training, not Unsloth loads or local and cached-path hints (#1902); soup data leakage finds exact input-text overlap only (#2672); task: preference and BCO and IPO stream, but preference_loss_weights blends are refused (#2766); Qwen3.5 and MiniMax recipes use inherited settings and are not tuned or measured (#275); the text-shown-as-typed tracker (#2489); fields that reach no consumer outside the staged table (#1814) and soup doctor --config coverage (#1815).
  • Not exercised by hand before this release. The pre-release smoke ran in two passes, the first on an 8 GB laptop card and the second on CPU with SmolLM2-135M and 5 task rows. The second pass did run a live soup ship with a .py scoring function (the file's sha256 is in the verdict and a replay gives the same verdict), a live soup ship with --noise-floor 2, one full soup loop iteration to a recorded end state (DON'T SHIP), soup loop canary with and without --allow-dont-ship, and the --detach guards. Not run: the deploy and the canary after a SHIP verdict (the one iteration ended DON'T SHIP) and a canary decided by 30 real ratings; a paraphrase endpoint other than the served model; soup docs search from an installed wheel; the reranker arm of the classifier sweep and task: asr (the model is not in the offline cache); soup diagnose --require-all-modes; soup eval quant-check --performance, soup eval retention with live model loads and soup runs compare; the refusal of a merge whose tokenizer is missing; the benchmark:reward refusal as an eval metric; the Ctrl+C three-signal behaviour (it needs a real console); the interactive "Start training?" prompt; Triton, MLX, vLLM, SGLang, MII, ONNX, torchao, cloud submission and Hub writes (not available, or never part of a smoke); and 18 of the 19 gpu-marked test files.
  • The loop: the manual canary check. soup loop canary refuses an adapter whose last verdict was DON'T SHIP unless --allow-dont-ship is passed, by run name or by the path to its directory (new in this release: v0.77.2 had no such check). The check applies to the adapter's own directory under .soup-loops/adapters, not to a checkpoint directory inside it, and on a case-insensitive file system it applies to a run name written in the letter case the run was recorded with.

Carried over from v0.77.2, still open:

  • training.fast_lora has no speedup claim and no end-to-end claim; its correctness half is open (#792).
  • training.stream_recompute is SFT only; auto works on the RAM tier only, gains nothing on a read-bound disk tier, and at a high VRAM fill it is slower.
  • stream_expert_layout: per_expert covers the DeepSeek-V3 family only and costs step time (about 2x to 2.7x on Moonlight-16B); two_read_experts has no hardware speed record yet (#2170).
  • lora.rank_pattern and alpha_pattern are refused on MLX and Unsloth until those trainers apply them (MLX: #1996).
  • soup merge --low-memory refuses adapters it cannot reproduce exactly (DoRA, modules_to_save, rank or alpha patterns, embedding adapters, and PiSSA, OLoRA, CorDA, LoftQ or LoRA-GA initialisations) and names the regular merge instead.
  • soup serve's /v1/tools/python is unavailable (501) on Windows and on Linux with Python 3.11, as /v1/tools/bash already was.
  • Layer streaming remains BETA and opt-in; --cuda-graphs remains experimental.

Fixed since v0.77.2 and no longer limits: the stale layer-stream cache slots (#2196, soup prune-layer-cache), freeze_layers with a string target_modules (#2201), unknown-key and cross-field config errors without a next step (#1816), GGUF export of Qwen3.5 (#1911), native MLX loading and generation for soup chat (#1986), KTO and BCO losing the BOS token and the KTO length cap reading whole columns (#2487, #2486), soup adapters bisect --eval-command quoting on Windows (#2488), emoji aliases and lost backslashes in path output (#2533), soup ci init --python accepting 03.11 (#2534), judge rows checked only during generation (#2535), and the soup draft distill --plan-only preview wrapping (#2536).

Contributors

Grouped by person, in order of merged pull requests; each line says what was found or built. 48 people outside the maintainer, 218 merged pull requests. Thank you to all 48.

  • Nariman Abdikarim (@Nari-Ab), 45: 45 pull requests across env, lock, ingest, export, prune-prompt, data-mix, compile-tools, distill-prompt, ra-dit, advise, agent, steer, quantize, data, tunability, ui, hubs, cli, ship, autopilot, can, rewind, diagnose, edit, fetch, recipes, eval, cache, expect, tokenizer, train, cost (#2525, #2526, #2527, #2528, #2529, #2530, #2550, #2557, #2558, #2559, #2565, #2566, #2580, #2581, #2582, #2583, #2584, #2594, #2595, #2596, #2597, #2598, #2612, #2613, #2614, #2615, #2616, #2874, #2875, #2879, #2880, #2901, #2902, #2903, #2904, #2905, #2943, #2944, #2945, #2946, #2947, #2958, #2959, #2960, #2962)
  • AbayAbdumutalip (@abdumutalipabay0), 18: 18 pull requests across recipes, merge, export, tracker, ui, sweep, mcp, migrate, sft, vram, pretrain, trainers (#2718, #2719, #2720, #2721, #2732, #2739, #2741, #2780, #2799, #2800, #2817, #2920, #2921, #2924, #2925, #2936, #2949, #2950)
  • Arman (@AubakirovArman), 17: 17 pull requests across ship, distill, eval, runs, export, monitoring, data, plan, config, bench, stream, infer (#2669, #2686, #2689, #2691, #2692, #2695, #2819, #2833, #2838, #2839, #2844, #2852, #2886, #2912, #2919, #2952, #2953)
  • shubham-dev-stack (@Shubham-dev-stack), 12: 12 pull requests across airgap, runs, adapters, ci, draft, cli, diff, eval, autopilot, serve, data, recipes (#2568, #2602, #2603, #2632, #2633, #2685, #2744, #2881, #2885, #2928, #2931, #2932)
  • Harshit Sharma (@harshitthek), 9: 9 pull requests across registry, bench, deploy, serve (#2592, #2600, #2604, #2611, #2641, #2642, #2649, #2670, #2978)
  • Pernav Jain (@here-2007), 9: 9 pull requests across mlx, docs, preference, training (#2635, #2684, #2710, #2716, #2818, #2821, #2843, #2896, #2915)
  • Abdulwaarith Zakariyya (@abdulwaarith0), 8: name the field that stops the VRAM estimate, and say before the Hub is read (#2444); say why the pre-flight fit is not checked (#2673); run the Q/K/V harness's bf16 cells under aten_half_matmuls (#2682); runs clean keeps every run's best checkpoint in a shared output directory (#2812); refuse a --max-tokens count below 1 before any backend loads (#2855); refuse a --max-tokens count above 16384 before any backend loads (#2898); refuse a merge of adapters that cannot be merged instead of writing an empty adapter (#2926); adapters merge scales each input's delta by its weight, not its square (#2927)
  • CapyRiver (@capyriver), 8: Add reproducible T4 CUDA measurements for Fast-LoRA MLP (#2621); Fix doubled braces in built-in chat Space templates (#2657); record RTX 3050 QuEST capacity run (#2715); account for cuBLAS workspaces in layer streaming (#2749); align supported tasks and refusals (#2786); add Hugging Face Space hardware selection (#2815); explain rejected Streamlit Spaces (#2834); reject duplicate task keys in eval-gate suites (#2873)
  • Yang Bai (@ybai08), 8: read the KL lender from its map batch in the prompt cap (#2543); keep the BOS when the cap cuts a prompt from the end (#2544); match modules_to_save by plain suffix, as PEFT does (#2553); keep the BOS when the cap cuts a prompt from the end (#2561); refuse a set where only some rows have a negative (#2563); refuse an unusable retain_set before any model loads (#2577); name the gradients that were non-finite before the clip (#2645); record the failed MLX SimNPO qualification run (#2809)
  • Faisal Fayaz (@Faisal-Fayaz), 7: keep LoRA input dtype at the 4-bit boundary (#2383); keep the SwiGLU LoRA dX add in fp32 (#2546); resolve the run id for eval benchmark and custom (#2743); an int past the double range is not a number (#2748); refuse a --max-tokens count below 1 before loading (#2787); refuse a count below 1 before the model loads (#2788); the #792 cause harness, one arm per candidate operation (#2922)
  • David (@1DMB1), 6: native resident route for Kimi-K2-Base/Instruct on transformers 5.x (#2759); name the suite file and field in schema-load errors (#2869); move the llama.cpp pin to b12423 so Qwen3.5 converts by default (#2910); refuse an evidence file that records a fully failed task side (#2911); refuse tokenless inputs instead of reporting OK (#2955); reject oversized evidence fractions without overflow (#2979)
  • Nurkhan Esenbek (@kok-o), 6: wire manual checkpoints and graceful SIGINT saves (#2414); add captured-token student objective (#2443); skip identical latest thumbs pairs (#2521); add Qwen3.5 4B text GRPO variant (#2522); add Qwen3.5 27B text GRPO variant (#2605); add Qwen3.5 122B-A10B text DPO variant (#2606)
  • Tal (@taljeon), 6: keep leaderboard averages finite on overflow (#2693); protect gate input files from baseline writes (#2694); reject boolean task thresholds (#2697); reject duplicate judge rubric criteria (#2699); keep judge weighted and aggregate scores finite (#2701); use resolved run IDs for comparison and Aider results (#2705)
  • Alibek Berik (@AlibekBerik), 5: ranking metrics for reranker and cross_encoder (#2637); accuracy and F1 for task: classifier (#2789); record the step speed the panel shows, not 0.0 (#2810); the validation row records no speed, not 0.0 (#2842); tool_call_args_subset is matched only at full credit (#2939)
  • Aditya Pratap Singh (@akkupratap323), 4: skip JSON lines that aren't records in read_audit_tail (#2591); say when a run id prefix matches several runs (#2617); refuse a malformed curriculum history with exit 2 (#2796); fetch the shards a Hub base's index names for --low-memory (#2900)
  • AmixDigital (@Amix29), 4: native interactive chat using the shared loader (#2123); fuse native LoRA into standalone dense checkpoints (#2124); Honor selective message loss masks in native MLX SFT (#2128); consume existing pretokenized SFT caches (#2131)
  • Rymbek Nazerke (@nazek52), 4: preserve preference prompts when complete rows fit (#2659); add Qwen3.5-122B-A10B text MoE GRPO variant (#2660); Show full local run comparisons (#2706); Estimate training finish in the live panel (#2707)
  • nurik (@noncalan), 4: show paths, registry ids and errors literally (#2841); show paths, model names and typed values literally (#2847); show bases, paths, adapter names and errors literally (#2887); show lock fields, paths, bounds and errors literally (#2890)
  • vaputa (@vaputa), 4: refuse enabled gates on unsupported training paths (#2564); share the Windows colon filename skip (#2625); bind each arm to one base revision (#2671); bind native training loads to the recorded base revision (#2884)
  • Ahmet (@ANO3457), 2: retry generation until target count is reached (#2418); keep leading BOS when keep_end cuts an overlong prompt in ORPO and SimPO (#2572)
  • artikulus1301 (@artikulus1301), 2: load the adapter tokenizer before writing the merged weights (#2761); measure and print how much the adapter changes the base (#2831)
  • DEVEN KALATHIYA (@deven1003), 2: refuse a training.eval_gate.suite that cannot load before training starts (#2264); fetch a Hub base's support files and skip consolidated weights in --low-memory (#2267)
  • Samarth Mukhija (@Samearth17), 2: reject booleans in numeric fields (#2462); preserve exact stratified split counts (#2954)
  • Srinivasan R (@Srinivasan8888), 2: DPO and ORPO cut the prompt only when the row overflows (#2555); fake every torchao module the NVFP4 resolver reaches (#2688)
  • Aaron Probha (@AaronProbha18), 1: feat(stream)[#2196] : list stale layer-cache slots in the disk pre-flight; add soup cache prune (#2647)
  • Batyrkhan (@Batyrkhantzy111), 1: Two-read comparator for the four-run rule, with --step-dir (#2269)
  • Dinmukhamed Bekeshev (@BDaaac), 1: refuse baseline previous and unresolvable baselines (#2640)
  • Alexis (@bernalalexis-try), 1: name a location for cross-field errors and a next step for unmatched keys (#1823)
  • zgy (@bfs2021), 1: give each merge its own scratch directory (#2310)
  • Can Akyuz (@canakyuz), 1: fix preference_loss_weights bco blend timing (#2252)
  • Salil Mhatre (@Deadpool2000), 1: sanitize output_dir for machine-readable surfaces (#2807)
  • Erizoned (@Erizoned), 1: add --require-all-modes for evidence runs (#2857)
  • George Tonmoy Roy (@george07-t), 1: refuse eval-gate suites with a non-string top-level key (#2021)
  • Izumx (@Izumx), 1: warn about --token only when it is typed, not when HF_TOKEN fills it (#2918)
  • Emil Jafarov (@jafarovvvvvvv), 1: start the pinned-arena hardware test from an empty host cache (#2681)
  • Keshemir (@Keshemir), 1: refuse nan like inf on the six ranged float options (#2917)
  • nurpeisovnurbolofficial (@nurpeisovnurbolofficial), 1: resolve run prefixes before reading metrics (#2794)
  • PANDIARAJ (@Pandiaraj-1), 1: validate kto row shape before loading any model (#2435)
  • Sajjad P. Savoji (@SajjadPSavoji), 1: reject calibrated judge reports with zero pairs (#2513)
  • Shivansh Shukla (@ShivanshShukla), 1: close the streaming runtime when setup() fails (#2378)
  • Shu Chen (@shu-chen-dot-ca), 1: report wrong-typed training values as validation errors (#2382)
  • Shutaru (@Shutaru), 1: externalize bulk #2162 evidence and retain numerical audits (#2644)
  • Siddhardha Nanda (@SID-6921), 1: stat-check a file against the size cap before hashing it (#2690)
  • Sofia5050 (@Sofia5050), 1: declare the mlx floor that mlx-lm needs (#2779)
  • Vansh Sharma (@VanshSharmaPES), 1: match the config's precision to the trainer's (#2276)
  • Kratos (@wanjinhao1), 1: Fix #2220: drop RST backticks from the neftune refusal and use singular 'requires' in soup export's gguf-on... (#2259)
  • xiemocun (@xiemocun), 1: resolve a string lora.target_modules inside a freeze plan (#2289)
  • Ziyadinkhan Kudaibergenuly (@Ziyadin23), 1: preserve line endings in quoted CSV fields (#2687)

Reports that led to fixes in this release: @abdumutalipabay0 (#2787, #2788, #2794, #2796, #2812, #2855, #2885, #2898, #2928), @Batyrkhantzy111 (#2414), @baurvtm-stack (#2918), @duysengali-kuanysh (#2689, #2695, #2819, #2833), @Faisal-Fayaz (#2781), @noncalan (#2877), @Samearth17 (#2418), @taljeon (#2513, #2838, #2900, #2911, #2955, #2979).

Don't miss a new Soup release

NewReleases is sending notifications on new releases.