v0.78.0, 2026-11-18
218 of the 271 pull requests in this release came from 48 people outside the maintainer.
Thank you: the Contributors section below names each of these contributors and what was found or built.
v0.78.0 is a cut of main after v0.77.2, not a backport: it carries every pull request merged since that tag. By the CHANGELOG count that is 60 additions, 20 changes, 2 deprecations, 169 fixes and 3 security entries. This is not a fixes-only release, and it refuses some input v0.77.2 accepted: a boolean in a numeric training field, an enabled eval gate on a path that never ran one, nan on six numeric options, an --output that is one of soup ship's own inputs, and more. It also changes some numbers: soup adapters merge weights, the tool_call_args_subset score (which can move a soup ship verdict and a soup eval retention result), the VRAM pre-flight, and the weights of classifier, reranker, cross_encoder and asr runs. Read the upgrade notes under Install / Upgrade before you upgrade a config or a script. Coming from v0.76.x or older, read the v0.77.0 notes first and run:
soup doctor --config soup.yaml
Every change, with its pull request, is in CHANGELOG.md, section [0.78.0]. Compare with the previous release: v0.77.2...v0.78.0.
What's New
Added
- A continual-learning loop (#2897, which folded 17 stacked pull requests into one).
soup servegives every/v1/chat/completionsand/v1/messagesanswer aresponse_id(also theX-Response-Idheader),--trace-logwrites one v1 conversation record per answer, and--feedback-logturns onPOST /v1/feedback(thumb, rating, edit or comment, by id) (#2532);soup serve --adapter-dirwithPOST /v1/adapters/loadloads a LoRA adapter into a running server without a restart (#2516).soup looplearns from thumbs, ratings and edits with sft (the new default), dpo or kto (#2538, #2560), paraphrases the prompts people corrected (#2573), judges a candidate withsoup ship --gate loop-v2by the corrections it learned (#2570, #2588), starts a canary at 5% of keyed traffic and withdraws it when people rate it worse (#2518, #2539), and writes iteration, version and progress logs (#2554). New:soup loop setandsoup loop rollback,soup loop watch --detachwith a PID file and a log (#2540),soup ship --base-adapterto compare two adapters (#2511), and asoup loop canarythat refuses an adapter whose last verdict was DON'T SHIP, named or given as the path of its directory under.soup-loops/adapters(--allow-dont-shipoverrides it and the override is recorded; #3002). The six JSONL formats are defined indocs/loop-formats.mdwith JSON Schemas (#2505). soup shipshows what its number is made of (#2870, #2894, #2831, #2933). Per task it counts the rows that got better, worse and stayed the same, the answers cut off at the token limit and the failed generations;--output verdict.jsonalso writes a per-row audit fileverdict.task-rows.jsonl; it prints how much of the base the adapter's update changes; verdict and evidence files carryschema_version: 2.--task-mode reward --reward-fn <accuracy|format|verifiable:math|verifiable:json_schema|path.py>scores each answer with a function from 0 to 1; a.pyfile is executed with your permissions, only when named, and its hash is recorded.- Layer streaming does more. A forward-only inference mode built from an existing shard cache, with adapter reload (
soup_cli.utils.stream_inference, #2509, #2510, #2636); a pre-quantised bitsandbytes NF4 checkpoint imports byte for byte and trains understream_layers: true(#2628, #2653); BCO and IPO train on a streamed base (#2896, #2915), and so doestask: preferencewith a single dpo, orpo or simpo loss (#2843); a checkpointed NF4 layer switches to a packed-state backward when the pre-flight predicts the dense one would not fit (#2655); on Windows the disk tier's page-locked staging checks system commit headroom first (#2658);soup prune-layer-cacheremoves stale cache slots, and the disk pre-flight lists them (#2647). - More on Apple Silicon. Native MLX chat (
soup chat --backend mlx, #2123), fusion into a standalone dense checkpoint (soup merge --backend mlx, #2124), selective per-message loss masks (#2128), consumption of existing preprocessing caches (#2131), vocabulary expansion with trainable new tokens (#2684), linear RoPE extension (#2710) and dense Llama SFT (#2818). - New commands and flags.
soup eval retentionmeasures forgetting across a series of model versions against a stored record of pinned control tasks, the file named by--ledger(#2552);soup data leakagereports exact input-text overlap between training and evaluation rows (#2833);soup docs searchsearches the bundled guides offline (#2716);soup eval quant-check --performanceadds latency, tokens per second and peak memory (#2689);soup export --format onnx --validate-onnxcompares an ONNX graph's logits with its source (#2695);soup diagnose --evidence --require-all-modes(#2857);soup deploy hf-spacecan pick a hardware tier (#2815). - Training and tracking. The experiment tracker records a GRPO run's per-step score and its spread beside the loss (#2891), the live panel shows zero-spread groups (#2819) and the estimated finish time (#2707), runs record the project's Git commit and clean/dirty state (#2692), and
soup runs compareshows full config differences and the latest stored evaluation scores (#2706).task: classifierreports accuracy and F1 (#2789), andrerankerandcross_encoderreport ranking quality when validation rows carryquery_id(#2637). - Eight recipes (181 in all):
qwen3.5-4b-grpo,qwen3.5-27b-grpo,qwen3.5-122b-a10b-dpo,qwen3.5-122b-a10b-grpo,qwen3.5-397b-a17b-grpo,minimax-m3-dpo,qwen3.5-9b-pretrainandqwen3.5-2b-pretrain(#2522, #2605, #2606, #2660, #2718, #2719, #2720, #2721). - A library of Triton kernels for the Qwen2 QLoRA SFT step,
soup_cli.kernels.packed_lora(#2623). It has no config key and nothing insoup trainreads it yet. Also: a guarded resident route for Kimi-K2-Base and -Instruct on transformers 5.x (#2759),soup doctorprints an NF4 rounding note under an RTX 50-series card (#2587).
Changed
soup loopdefaults and rules (#2560, #2539, #2532):soup loop initmakes an sft loop; a canary is judged by the ratings on its answers with a significance test;soup serverefuses--trace-log,--record-thumbsand--feedback-logon the vLLM, SGLang and MII backends instead of ignoring them; a--trace-logline has a new shape.- The VRAM pre-flight prices differently. Padded SFT runs are priced by their longest row, so runs it refused because of
data.max_lengthnow start (#2934); 8-bit runs with checkpointing are charged for what bitsandbytes keeps (#2925); layer streaming is charged its cuBLAS workspaces (#2749); atask: distillrun is priced with its teacher (#2950);soup profile --model,soup profileand the MCPprofiletool takeseq_lenandshape(#2871). - Resident 4-bit and 8-bit runs on a bf16 card keep embeddings, norms and an untied head in bf16 instead of fp32 (#2683): about 25% faster and 3.8 GiB lower peak on Qwen2.5-7B NF4 on an RTX 4090, loss unchanged within a tolerance.
soup train --push-ascreates the repository private;--push-publickeeps it public (#2802).soup train --cloudrefuses a config whose local files cannot be shipped (#2627).soup serve --max-tokensabove 16384 exits 2 (#2898).training.vllm_sleep_mode: trueis refused ontask: grpo(#2773). Themlxextra needsmlx>=0.31.2(#2779).- Numbers that moved.
soup adapters mergescales each input's term by its weight instead of its square (#2927);scoring: "tool_call_args_subset"counts Correct only at full credit (#2939); DPO, ORPO and MLX preference training cut a prompt only when the row overflows (#2555, #2659);soup runs compareshows more (#2706); judge task rows are validated before any generation (#2881).
Deprecated
training.warmup_auto: trueand atraining.forgetting_thresholdaway from0.1load with a warning and are refused in v0.79.0 (#2814).soup deploy hf-spacewith the Streamlit template prints a deprecation warning (#2834).
Fixed: the ones that matter most
- Config and options are checked where they are read. Booleans in numeric fields (#2462), wrong-typed values in 38
trainingfields (#2382), integers past the float range (#2912, #2748, #2838),nanon six ranged options (#2917), and an unknown key with no close match now names a next step (#1823). The eval gate refuses a suite that cannot load, a repeated key, a boolean threshold,baseline: previousand an unsupported path instead of training without it (#2264, #2873, #2697, #2640, #2564). - Training.
soup train --deepspeedno longer stops before the first step under fp16 (#2276);task: classifier,reranker,cross_encoderandasrfull fine-tune on fp32 master weights, where 16-bit weights barely moved at the default learning rate (#2949);training.curriculum: truetrains in sorted order (#2924); fp16auto_mixed_precisionloads the frozen base in fp16 (#2921); the BOS token survives a prompt cut in KTO, BCO, ORPO and SimPO (#2544, #2561, #2572); Ctrl+C and.checkpoint_nowsave at an optimizer-step boundary (#2414);soup train --resumeontask: ppois refused instead of restarting from step 0 (#2772); the GRPO watchdog names the parameters whose gradient was not finite (#2645);soup sweepapplies the quantization guard ofsoup trainand binds each arm to its base revision (#2665, #2799, #2671). - Eval, ship and serve.
soup serve --backend transformershonoursstopandstop_sequencesand streams while it generates (#2630, #2514);--max-tokensis bounded onsoup serve,soup chatandsoup bench infer(#2855, #2898, #2787, #2788);soup shipevaluates in native bf16 or fp16 for the requested card (#2669); judge and calibration inputs are checked: finite scores and weights, no duplicate criteria, no empty calibration (#2693, #2701, #2699, #2513);soup drift-alarmrefuses an input with no tokens instead of saying OK (#2955). - No silent overwrites, empty results or crashes.
soup airgap-bundlechecks size and paths before reading (#2568, #2690);soup adapters mergerefuses adapters it cannot merge (#2926);soup data sample,soup data mix --apply,soup iterative-dpo,soup tokenizer trainandsoup expectreport bad input instead of crashing or writing nothing useful (#2931, #2905, #2946, #2958, #2947, #2945);soup shiprefuses an--outputthat is one of its inputs (#2894); files starting with a UTF-8 byte-order mark keep their first record insoup drift-alarm,soup prune-prompt,soup distill-prompt,soup iterative-dpoandsoup data active-sample(#2901, #2902, #2959, #2943, #2879). - Hub, cloud and export. RunPod jobs use REST API v2, because v1 was retired on 2026-11-15 (#2877); the built-in Gradio and Streamlit Space templates no longer raise
TypeErrorand publish in one commit (#2657, #2641, #2649); Qwen3.5 GGUF export works on a fresh checkout after the llama.cpp pin moved tob12423(#2910);soup export --format torchaomerges the adapter first (#2920);soup merge --low-memoryfetches the files and shards a Hub base needs (#2267, #2900). - Text shown as typed. More than 30 pull requests make the commands print paths, names and messages as typed: brackets and
:name:text are kept and terminal control characters are removed from the display only. What is left is under Known Limitations.
The measurement records behind this release are in benchmarks/ (42 new records; the index is benchmarks/README.md). The pre-flight memory estimate changed in v0.78.0 (#2749, #2871, #2934), so the preprint's validation describes the estimator of v0.77.x.
Install / Upgrade
pip install -U soup-cli # or: pipx upgrade soup-cli
soup --version # soup v0.78.0
soup doctor --config soup.yaml # before the first run of an existing config
- From v0.77.2: a valid
soup.yamlloads unless it sets a boolean in a numerictrainingfield or one of the other values listed first in the upgrade notes; the notes list the rest. - From v0.76.x or older: read the v0.77.0 notes first. Python 3.10 stays on v0.76.x (
pip install "soup-cli[train]<0.77"). This release supports Python 3.11 to 3.13. v0.76.x gets no further security fixes after this release. - Docker:
docker pull ghcr.io/makazhanalpamys/soup:0.78.0(alsolatest); the host driver must support CUDA 12.8.
Upgrade notes
A soup.yaml that loads on 0.77.2 loads on 0.78.0, except for the values listed in the first item (refused now). Coming from 0.76.x or older, read the 0.77.0 and 0.77.2 upgrade notes below first, and run soup doctor --config soup.yaml before the first run of an existing config.
- Config values that loaded on 0.77.2 and are refused now. Each message names the field and what to change.
- A boolean in a numeric field of
trainingortraining.lora(lr: true,batch_size: true,lora.r: false): 0.77.2 coercedfalseto0andtrueto1; 0.78.0 refuses it with<field> must be a number, not a bool(oran int). Write the number you meant (#2462). training.vllm_sleep_mode: trueontask: grpo: it never reached a vLLM engine; delete the line (#2773).training.eval_gate.enabled: true(or--gate) onbackend: mlxand ontask: prm,unlearn,moe_lora_routing: these paths never attached a gate and trained without one; setenabled: falseor remove the block, and runsoup evalorsoup shipafter training instead (#2564). Atraining.eval_gate.suitethat cannot load now stopssoup trainwith exit 1 instead of training without a gate: fix the suite file the message names, or disable the gate (#2264).training.eval_gate.baseline: previousis refused; use a path to a baseline results file (#2640).- Eval suites and rubrics: a boolean
threshold, a repeated key in a task mapping, duplicate rubric criterion names and a non-finite rubric weight are refused (#2697, #2873, #2699, #2701);--write-baselineonto the suite, custom-task or judge-prompts file exits 3 (#2694); a calibration file withcalibrated: trueandnum_pairs: 0is refused (#2513). - Data: reranker / cross_encoder
query_idof mixed or non-string/non-integer types (#2637),task: embeddingrows where only some have anegative(#2563), atask: unlearnretain_setwith no usable row (#2577).
- A boolean in a numeric field of
- Deprecated, refused in v0.79.0:
training.warmup_auto: trueandtraining.forgetting_thresholdset away from0.1load with a warning that they do nothing; delete them (#2814).soup deploy hf-spacewith the Streamlit template prints a deprecation warning (#2834). - Commands that stop where they used to continue.
soup train --push-as <repo>now creates the repository private; add--push-publicto keep creating a public one (#2802).soup train --cloudrefuses a config with local input files at plan time, and--push-as/--gatewith--cloud(#2627).soup serve --max-tokensabove 16384 exits 2 before any model loads (#2898);soup chat --max-tokensandsoup bench infer --max-tokensbelow 1 exit 2 (#2787, #2788).soup serve --trace-log,--record-thumbsand--feedback-logare refused with--backend vllm,sglangandmii(exit 2; they were silently ignored there): drop the flag, or serve with--backend transformersto keep recording (#2532).soup train --resume/--hf-resumeontask: ppois refused (#2772). One Ctrl+C no longer stops a Hugging Face Trainer run: the first saves and continues, the second saves and stops, the third interrupts (#2414).soup ci init --pythonwith a leading zero (03.11) exits 1 (#2632);soup airgap-bundlewith a missing folder exits 1 (#2568);soup adapters mergeof adapters it cannot merge exits 2 (#2926);soup merge --low-memoryof an adapter whose init rewrites the base is refused (#2732);soup data generateasks again until--countand exits 1 after three empty requests (#2418).nanis refused likeinfonsoup data validate --min-valid-fraction,soup draft measure --min-acceptance,soup data toxicity --threshold,soup data decontaminate --threshold,soup data score --thresholdandsoup data forge --uncertainty-threshold, with a usage error that names the option (exit 3 fordata validate, 2 for the others) before any input is read; pass a number (#2917).soup data samplerefuses-nbelow 1 and a--pctthat is not in (0, 100] with exit 1 (#2931).soup drift-alarmexits 1 on an input with no tokens instead of reportingOK; fix the producer of that input (#2955).soup iterative-dpowith no usable prompt row exits 1 before it loads a model or writes pairs (#2958).soup tokenizer train --outputnaming an existing file is refused before training (#2947).- Exit codes:
soup data mix --applywith invalid YAML or invalid recipe weights 1 -> 2, so a script that tests for 1 must test for 2 (#2905, #2946);soup expectwith an unreadable suite or dataset 1 -> 3, and exit 1 now means the judge was unreachable (#2945). soup ship: when every task generation fails on one side,--task-mode metricexits 1 (#2870);--outputnames that cannot hold<stem>.task-rows.jsonlexit 3 (#2894); an--outputor--emit-evidencethat is the same file as one of the run's inputs, or as the other output, exits 3 before anything is read, loaded or written, where it used to replace the input with the verdict (#2894); an adapter directory whose weights are a pickle (anadapter_model.bin,.pt,.pth,.pkl,.pickleor.ckptfile, or a pickle posing as.safetensors) is refused on--adapterand--base-adapterwith exit 3 (unsafe weight file (pickle / PyTorch-classic): adapter_model.bin): re-save the adapter as safetensors, for examplemodel.save_pretrained(path, safe_serialization=True), and pass that directory (#2897); the verdict, the evidence and--outputgain keys, none removed, including a top-levelbase_adapterthat isnullunless--base-adapteris used.soup can runrefuses a can that holds<redacted>values and names their key paths: supply the values in the can'sconfig.yamland runsoup trainon it, and keep credentials out of a config before packing it (#2883).
soup ship --emit-evidencewritesschema_version: 2. soup-cli 0.77.x refuses such a file (evidence has unsupported field(s): 'judged', 'schema_version', 'task_rows', exit 3); upgrade whatever reads the evidence (the CI gate, the MCPship_evidencetool). 0.78.0 reads versions 1 and 2 (#2870).soup adapters mergeweights. Each input's own term is now scaled by its weight, where 0.77.2 used the weight squared: the numbers of everylinear,dareandsvdmerge of two or more inputs change (an equal-weight pair doubles each input's own term, 0.25 to 0.5), and the weights an earlier--strategy cmaesrun printed reproduce a different adapter. Re-run merges whose output you compare or ship (#2927).- A scoring rule got stricter.
scoring: "tool_call_args_subset"counts a row as Correct only at full credit, sosoup eval custom,soup shipmetric mode andsoup eval retentionreport a lower Accuracy for suites that use it, and the exit status can change for an unchanged model (asoup shipthat ended 0 can end 2). Record base and tuned scores again with this version before comparing (#2939). - The VRAM pre-flight prices differently. An 8-bit run with gradient checkpointing is priced higher (#2860 in #2925), layer streaming charges the cuBLAS workspaces (#2749), on Windows it checks system commit headroom before page-locking (#2658), and a
task: distillrun is priced with its teacher and divergence (#2950): a run that launched close to the card's limit can be refused;--allow-oom-attempt, a lowerbatch_sizeordata.max_lengthlaunch it. A padded text SFT run the old pre-flight refused because ofdata.max_lengthnow starts when its longest row fits (#2934).soup profile/ the MCPprofiletool takeseq_lenandshape(#2871), and their JSON gainsteacher_memory_gb,distill_memory_gbandteacher_generation_memory_gb(0 outsidetask: distill). - Runs are not bit-comparable with 0.77.2 for: resident 4-bit / 8-bit on a bf16 card (embeddings, norms and an untied head stay bf16; faster, lower peak, #2683), fp16
auto_mixed_precisionon a bf16 card (#2921),curriculum: true(rows now in sorted order, #2924), the BOS token kept after akeep_endprompt cut (#2544, #2561, #2572), the DPO / ORPO / MLX prompt cut (#2555, #2659),soup data split --stratifyand--stratify-semantic, which now give the requested validation and test counts, so the same seed can place different rows in train, validation and test (#2954), Fast-LoRA 4-bit dtype boundary (#2383, #2546),soup shipon a CUDA card without bf16, which now evaluates in float16 (#2669), andtask: classifier,reranker,cross_encoderandasr, which now full fine-tune on fp32 master weights: twice the memory over a 16-bit checkpoint, so a run that fit the card on 0.77.x can run out of memory (train an adapter withclassifier_lora: trueorasr_lora: true, or lowertraining.batch_size; #2949). soup loop(already in 0.77.2) behaves differently.soup loop initwithout--train-methodnow creates an sft loop that trains on corrections (existingloop.yamlfiles keep their method; pass--train-method dpoto keep preference pairs, #2560); the pre-wired harvest reads the feedback log and trains only when new feedback arrived (#2538, #2531); a canary is judged by the ratings on its answers with a significance test, needs 30 rated answers on each side unless--canary-max-waitends the wait, and a canary that got no answers is withdrawn (#2539); the iteration, version and progress logs are written (#2554);soup loop watch --detachleaves.soup/loop.pidand.soup/loop.logand refuses a second--detach(#2540); a--trace-logline hastsas an ISO 8601 UTC string andtokensas an object, and a line whoseschema_versionis not 1 is skipped bysoup loopandsoup data from-traces(#2532). A hand-editedloop.yamlwith amonthly_budget_usdorspent_this_month_usdthat is not finite or is above 1e15 was read before and is refused now (exit 2, naming the field): write a number from 0 to 1e15, or remove the line (#2897).- Databases, servers and files.
~/.soup/experiments.dbgains nullable columns in place on first use; loss, lr and speed may be NULL where 0.77.2 stored0.0(#2589, #2810, #2891); a database written by 0.78.0 stays readable by 0.77.2.soup uiand the MCP run tools returnoutput_dirrelative to the working directory, or a bare folder name; read the absolute path from your own records or fromsoup runs showon the host (#2807).soup serve --backend transformershonoursstop/stop_sequencesand streams while it generates (#2514, #2630). - Crash bundles and
.canfiles are redacted before they are written; read a bundle before sharing it (#2883). RL-checkpoint optimizer state loads withweights_only=True(#2916). - GGUF export: the llama.cpp pin moved from
b5270tob12423; delete~/.soup/llama.cppto have the next export clone the pinned tag (#2910). Themlxextra now needsmlx>=0.31.2(#2779). - Python API:
JudgeResults.compute()and the calibration helpers raiseValueErrorfor non-finite or oversized inputs (#2701, #2838);decide_performance_flagstreats an explicitvram_headroom_gb=0.0as zero headroom (#2885);soup_cli.utils.profiler.profile_from_configreturns a truthysoup_cli.utils.hardware_fit.FitUnavailable(with.fieldand.reason) instead ofNonewhen a run cannot be priced, so testisinstance(result, dict)(#2871).
Security
This release carries 3 entries under Security in the CHANGELOG, from changes merged on 2026-11-16 (2) and 2026-11-17 (1). No advisory is published with it. A security review of the code merged for this release, run on 2026-11-17, recorded 9 findings (0 critical, 1 high, 1 medium, 7 low): 5 are fixed in this release and 4 are listed under Known Limitations. See SECURITY.md for how to report a problem and for the supported versions: 0.78.x is supported, 0.77.x gets security fixes until v0.79.0 is released, and the 0.76.x line ends with this release.
Known Limitations
New in this release:
soup_cli.kernels.packed_lorais a library: it has no config key andsoup traindoes not read it. Its GPU tests run on Linux CUDA only (pytest -m gpu --no-cov) and its Triton modules are left out of the coverage measurement (#2623). By design; no issue.- Layer-streaming inference is forward only: no KV cache, no generation, no command (#2509 stays open).
- The VRAM estimate is fitted on small dense models. The 8-bit checkpointing term is calibrated on three (Qwen2.5-0.5B, Qwen2.5-1.5B, SmolLM2-360M) and
gradient_checkpointing: mediumis outside the calibrated range and only warns (#2925); the padded-SFT lift reads at most 20,000 rows or 90 seconds and otherwise prices atdata.max_length(#2934); a distillation step counts as outside the calibrated range, anduld_strategyandminillm_enabledare priced at the largest measured figure (#2950); a micro-batch that fills 512 tokens can still need more than the estimate (#2871). Tracked in #3021 (calibration beyond 1,024 tokens per micro-batch). - The Apple Silicon features (chat, merge, caches, vocabulary expansion, RoPE extension, dense Llama SFT, preference and GRPO) are covered only by the hosted macOS MLX test job and fixtures. By design; no issue.
- The RunPod controller uses REST API v2 from the published reference and has not been run against a real account; the remote script still depends on the Pod image's system Python (#2877, #2098, #2192).
soup ship --task-mode rewardwith a.pyfile executes that file with your permissions: the recorded hash says which file scored a verdict, it is not a defence against a hostile file.verifiable:codeandjudge_rankstay refused. By design; no issue.- Kimi K2: the resident route for Base and Instruct is guarded and rests on tiny-checkpoint evidence only, the
kimi-k2-sftrecipe stays unavailable, and Thinking and Instruct-0905 have no supported route (#2759, #2104). - GGUF export clones llama.cpp
b12423by tag, not by commit, and an existing~/.soup/llama.cppcheckout is reused as it is; a file written by the new converter can differ from one written withb5270(#2910). By design; no issue. - The loop: disk.
.soup-loops/grows with every training run and nothing is pruned or rotated: each run keeps its adapter, recipe, logs and verdict files,pairs/<hash>.jsonlis rewritten for each new set of rows, and each append toiterations.jsonlorversions.jsonlreads the file once. Archive old.soup-loops/adapters/*by hand once they can no longer be a rollback target. Tracked in #3015 (soup loop prune). - The loop: running and stopping.
soup loop init --forcedoes not check for a running daemon; there is nosoup loop stop(soup loop pausetakes effect at the next cycle boundary and a runningsoup trainchild keeps going); two foregroundsoup loop watchprocesses are not guarded against each other; a stop between an adapter load and the end of its cycle, and between arollbackorcanarylog line and theloop.yamlupdate, is not reconciled. Running the same command again is safe. Tracked in #3016 and #3017. - The loop: deploy. A refused deploy is retried once per poll, three times, with no back-off: a
soup serverestart longer than a few minutes ends the retries, and the adapter must then be deployed by hand withsoup loop canary <adapter directory>. Tracked in #3018. - The loop: feedback is training data. On a loopback server started without
--tool-auth-tokenany local user or process can post to/v1/feedback; startsoup servewith a token on a shared machine. The loop does not check an edit for intent, and the loop-v2 gate checks a candidate against the corrections it was trained on, so a poisoned edit that the model learns passes that check; the canary's ratings andsoup loop rollbackare the other defences.soup loopreads--trace-dirfiles whole: keep--trace-log-cap-mbat its default on a server others can reach. By design; no issue. - The loop: processes and files. The loop's own children start without the current directory on
sys.path(-P); a Python process thatsoup trainorsoup shipstarts in turn does not, so run the loop in a directory you control. A stage that raises is retried after 5 minutes, then 30, and held after the third failure in the same stage untilsoup loop resume; failures in alternating stages restart the count.soup shiprefuses an--outputthat is one of its inputs but does not compare the training file a--confignames: do not name your training data as an output. Tracked in #3019 and #3020. - The loop: quality. The default recipe (all linear modules, r 64 / alpha 128, lr 2e-4, 8 epochs) regressed small evaluation suites in a study on one project; milder defaults and anchors are untested. A 40-row suite's base side read 0.900 once and 0.925 six times on the same model between
soup shipcalls, so a 0.05 threshold is half noise; andeval.custom.score_semanticis word overlap, so a short correct answer against a long reference can score 0.0. Tracked in #3014, #3012 and #3013. - Some commands still print an exception's text through Rich without escaping it, so a name taken from an untrusted file can render as markup in an error line; the tracker is #2489. Two patterns in the paraphrase step and the loop-v2 gate are slow on a pathological reply, which comes from the paraphrase service you configured and is limited to 64 KiB.
- Three guards in the loop's reconcile code and in the per-row file writer have no test that fails when they are removed, and a native-RoPE test compares float tables with exact equality and failed once on a hosted runner (#2948). The guards are tracked in #3022.
- A live
soup shipverdict is slow on a laptop card: a full verdict for a 0.5B model took 35 minutes for 20 task rows plus the eight bundled mini suites on an RTX 5070 Laptop (8 GB), with other work on the machine, at about 10% GPU utilisation and one busy CPU core, and it prints nothing between loading the models and the result panel unless you pass--progress jsonl. Expect tens of minutes on a card of that class. On a quiet box with CPU only, a 135M model and 5 task rows, a live verdict took 324 s with a.pyscoring function and one general suite, 544 s with--noise-floor 2, and 1252 s for the verdict child of asoup loopiteration, which runs all eight bundled mini suites. Tracked in #3011. - The gpu-marked test
test_configured_workspace_is_visible_to_allocator_in_fresh_processfails on a CUDA card because its child script has no graph for backward (#2991); the fix is #2994, in the release after this one. Hosted CI does not run gpu-marked tests. - The loop's
quantization: autopicks 4-bit when the Hub is offline, because the base's parameter count cannot be read: an offlinesoup looprun trains in 4-bit unless--sft-quantization noneis given. The reason is written to the iteration record (sft quantization auto: 4bit (parameter count unknown)) but not printed. Tracked in #3033. - Partial fixes whose issues stay open: the settings a backend or task never applies are refused only on the paths fixed so far (#1811); the base-revision binding covers the transformers backend,
soup merge,soup sweepand native MLX text training, not Unsloth loads or local and cached-path hints (#1902);soup data leakagefinds exact input-text overlap only (#2672);task: preferenceand BCO and IPO stream, butpreference_loss_weightsblends are refused (#2766); Qwen3.5 and MiniMax recipes use inherited settings and are not tuned or measured (#275); the text-shown-as-typed tracker (#2489); fields that reach no consumer outside the staged table (#1814) andsoup doctor --configcoverage (#1815). - Not exercised by hand before this release. The pre-release smoke ran in two passes, the first on an 8 GB laptop card and the second on CPU with SmolLM2-135M and 5 task rows. The second pass did run a live
soup shipwith a.pyscoring function (the file's sha256 is in the verdict and a replay gives the same verdict), a livesoup shipwith--noise-floor 2, one fullsoup loopiteration to a recorded end state (DON'T SHIP),soup loop canarywith and without--allow-dont-ship, and the--detachguards. Not run: the deploy and the canary after a SHIP verdict (the one iteration ended DON'T SHIP) and a canary decided by 30 real ratings; a paraphrase endpoint other than the served model;soup docs searchfrom an installed wheel; the reranker arm of the classifier sweep andtask: asr(the model is not in the offline cache);soup diagnose --require-all-modes;soup eval quant-check --performance,soup eval retentionwith live model loads andsoup runs compare; the refusal of a merge whose tokenizer is missing; thebenchmark:rewardrefusal as an eval metric; the Ctrl+C three-signal behaviour (it needs a real console); the interactive "Start training?" prompt; Triton, MLX, vLLM, SGLang, MII, ONNX, torchao, cloud submission and Hub writes (not available, or never part of a smoke); and 18 of the 19 gpu-marked test files. - The loop: the manual canary check.
soup loop canaryrefuses an adapter whose last verdict was DON'T SHIP unless--allow-dont-shipis passed, by run name or by the path to its directory (new in this release: v0.77.2 had no such check). The check applies to the adapter's own directory under.soup-loops/adapters, not to a checkpoint directory inside it, and on a case-insensitive file system it applies to a run name written in the letter case the run was recorded with.
Carried over from v0.77.2, still open:
training.fast_lorahas no speedup claim and no end-to-end claim; its correctness half is open (#792).training.stream_recomputeis SFT only;autoworks on the RAM tier only, gains nothing on a read-bound disk tier, and at a high VRAM fill it is slower.stream_expert_layout: per_expertcovers the DeepSeek-V3 family only and costs step time (about 2x to 2.7x on Moonlight-16B);two_read_expertshas no hardware speed record yet (#2170).lora.rank_patternandalpha_patternare refused on MLX and Unsloth until those trainers apply them (MLX: #1996).soup merge --low-memoryrefuses adapters it cannot reproduce exactly (DoRA,modules_to_save, rank or alpha patterns, embedding adapters, and PiSSA, OLoRA, CorDA, LoftQ or LoRA-GA initialisations) and names the regular merge instead.soup serve's/v1/tools/pythonis unavailable (501) on Windows and on Linux with Python 3.11, as/v1/tools/bashalready was.- Layer streaming remains BETA and opt-in;
--cuda-graphsremains experimental.
Fixed since v0.77.2 and no longer limits: the stale layer-stream cache slots (#2196, soup prune-layer-cache), freeze_layers with a string target_modules (#2201), unknown-key and cross-field config errors without a next step (#1816), GGUF export of Qwen3.5 (#1911), native MLX loading and generation for soup chat (#1986), KTO and BCO losing the BOS token and the KTO length cap reading whole columns (#2487, #2486), soup adapters bisect --eval-command quoting on Windows (#2488), emoji aliases and lost backslashes in path output (#2533), soup ci init --python accepting 03.11 (#2534), judge rows checked only during generation (#2535), and the soup draft distill --plan-only preview wrapping (#2536).
Contributors
Grouped by person, in order of merged pull requests; each line says what was found or built. 48 people outside the maintainer, 218 merged pull requests. Thank you to all 48.
- Nariman Abdikarim (@Nari-Ab), 45: 45 pull requests across
env,lock,ingest,export,prune-prompt,data-mix,compile-tools,distill-prompt,ra-dit,advise,agent,steer,quantize,data,tunability,ui,hubs,cli,ship,autopilot,can,rewind,diagnose,edit,fetch,recipes,eval,cache,expect,tokenizer,train,cost(#2525, #2526, #2527, #2528, #2529, #2530, #2550, #2557, #2558, #2559, #2565, #2566, #2580, #2581, #2582, #2583, #2584, #2594, #2595, #2596, #2597, #2598, #2612, #2613, #2614, #2615, #2616, #2874, #2875, #2879, #2880, #2901, #2902, #2903, #2904, #2905, #2943, #2944, #2945, #2946, #2947, #2958, #2959, #2960, #2962) - AbayAbdumutalip (@abdumutalipabay0), 18: 18 pull requests across
recipes,merge,export,tracker,ui,sweep,mcp,migrate,sft,vram,pretrain,trainers(#2718, #2719, #2720, #2721, #2732, #2739, #2741, #2780, #2799, #2800, #2817, #2920, #2921, #2924, #2925, #2936, #2949, #2950) - Arman (@AubakirovArman), 17: 17 pull requests across
ship,distill,eval,runs,export,monitoring,data,plan,config,bench,stream,infer(#2669, #2686, #2689, #2691, #2692, #2695, #2819, #2833, #2838, #2839, #2844, #2852, #2886, #2912, #2919, #2952, #2953) - shubham-dev-stack (@Shubham-dev-stack), 12: 12 pull requests across
airgap,runs,adapters,ci,draft,cli,diff,eval,autopilot,serve,data,recipes(#2568, #2602, #2603, #2632, #2633, #2685, #2744, #2881, #2885, #2928, #2931, #2932) - Harshit Sharma (@harshitthek), 9: 9 pull requests across
registry,bench,deploy,serve(#2592, #2600, #2604, #2611, #2641, #2642, #2649, #2670, #2978) - Pernav Jain (@here-2007), 9: 9 pull requests across
mlx,docs,preference,training(#2635, #2684, #2710, #2716, #2818, #2821, #2843, #2896, #2915) - Abdulwaarith Zakariyya (@abdulwaarith0), 8: name the field that stops the VRAM estimate, and say before the Hub is read (#2444); say why the pre-flight fit is not checked (#2673); run the Q/K/V harness's bf16 cells under aten_half_matmuls (#2682); runs clean keeps every run's best checkpoint in a shared output directory (#2812); refuse a --max-tokens count below 1 before any backend loads (#2855); refuse a --max-tokens count above 16384 before any backend loads (#2898); refuse a merge of adapters that cannot be merged instead of writing an empty adapter (#2926); adapters merge scales each input's delta by its weight, not its square (#2927)
- CapyRiver (@capyriver), 8: Add reproducible T4 CUDA measurements for Fast-LoRA MLP (#2621); Fix doubled braces in built-in chat Space templates (#2657); record RTX 3050 QuEST capacity run (#2715); account for cuBLAS workspaces in layer streaming (#2749); align supported tasks and refusals (#2786); add Hugging Face Space hardware selection (#2815); explain rejected Streamlit Spaces (#2834); reject duplicate task keys in eval-gate suites (#2873)
- Yang Bai (@ybai08), 8: read the KL lender from its map batch in the prompt cap (#2543); keep the BOS when the cap cuts a prompt from the end (#2544); match modules_to_save by plain suffix, as PEFT does (#2553); keep the BOS when the cap cuts a prompt from the end (#2561); refuse a set where only some rows have a negative (#2563); refuse an unusable retain_set before any model loads (#2577); name the gradients that were non-finite before the clip (#2645); record the failed MLX SimNPO qualification run (#2809)
- Faisal Fayaz (@Faisal-Fayaz), 7: keep LoRA input dtype at the 4-bit boundary (#2383); keep the SwiGLU LoRA dX add in fp32 (#2546); resolve the run id for eval benchmark and custom (#2743); an int past the double range is not a number (#2748); refuse a --max-tokens count below 1 before loading (#2787); refuse a count below 1 before the model loads (#2788); the #792 cause harness, one arm per candidate operation (#2922)
- David (@1DMB1), 6: native resident route for Kimi-K2-Base/Instruct on transformers 5.x (#2759); name the suite file and field in schema-load errors (#2869); move the llama.cpp pin to b12423 so Qwen3.5 converts by default (#2910); refuse an evidence file that records a fully failed task side (#2911); refuse tokenless inputs instead of reporting OK (#2955); reject oversized evidence fractions without overflow (#2979)
- Nurkhan Esenbek (@kok-o), 6: wire manual checkpoints and graceful SIGINT saves (#2414); add captured-token student objective (#2443); skip identical latest thumbs pairs (#2521); add Qwen3.5 4B text GRPO variant (#2522); add Qwen3.5 27B text GRPO variant (#2605); add Qwen3.5 122B-A10B text DPO variant (#2606)
- Tal (@taljeon), 6: keep leaderboard averages finite on overflow (#2693); protect gate input files from baseline writes (#2694); reject boolean task thresholds (#2697); reject duplicate judge rubric criteria (#2699); keep judge weighted and aggregate scores finite (#2701); use resolved run IDs for comparison and Aider results (#2705)
- Alibek Berik (@AlibekBerik), 5: ranking metrics for reranker and cross_encoder (#2637); accuracy and F1 for task: classifier (#2789); record the step speed the panel shows, not 0.0 (#2810); the validation row records no speed, not 0.0 (#2842); tool_call_args_subset is matched only at full credit (#2939)
- Aditya Pratap Singh (@akkupratap323), 4: skip JSON lines that aren't records in read_audit_tail (#2591); say when a run id prefix matches several runs (#2617); refuse a malformed curriculum history with exit 2 (#2796); fetch the shards a Hub base's index names for --low-memory (#2900)
- AmixDigital (@Amix29), 4: native interactive chat using the shared loader (#2123); fuse native LoRA into standalone dense checkpoints (#2124); Honor selective message loss masks in native MLX SFT (#2128); consume existing pretokenized SFT caches (#2131)
- Rymbek Nazerke (@nazek52), 4: preserve preference prompts when complete rows fit (#2659); add Qwen3.5-122B-A10B text MoE GRPO variant (#2660); Show full local run comparisons (#2706); Estimate training finish in the live panel (#2707)
- nurik (@noncalan), 4: show paths, registry ids and errors literally (#2841); show paths, model names and typed values literally (#2847); show bases, paths, adapter names and errors literally (#2887); show lock fields, paths, bounds and errors literally (#2890)
- vaputa (@vaputa), 4: refuse enabled gates on unsupported training paths (#2564); share the Windows colon filename skip (#2625); bind each arm to one base revision (#2671); bind native training loads to the recorded base revision (#2884)
- Ahmet (@ANO3457), 2: retry generation until target count is reached (#2418); keep leading BOS when keep_end cuts an overlong prompt in ORPO and SimPO (#2572)
- artikulus1301 (@artikulus1301), 2: load the adapter tokenizer before writing the merged weights (#2761); measure and print how much the adapter changes the base (#2831)
- DEVEN KALATHIYA (@deven1003), 2: refuse a training.eval_gate.suite that cannot load before training starts (#2264); fetch a Hub base's support files and skip consolidated weights in --low-memory (#2267)
- Samarth Mukhija (@Samearth17), 2: reject booleans in numeric fields (#2462); preserve exact stratified split counts (#2954)
- Srinivasan R (@Srinivasan8888), 2: DPO and ORPO cut the prompt only when the row overflows (#2555); fake every torchao module the NVFP4 resolver reaches (#2688)
- Aaron Probha (@AaronProbha18), 1: feat(stream)[#2196] : list stale layer-cache slots in the disk pre-flight; add soup cache prune (#2647)
- Batyrkhan (@Batyrkhantzy111), 1: Two-read comparator for the four-run rule, with --step-dir (#2269)
- Dinmukhamed Bekeshev (@BDaaac), 1: refuse baseline previous and unresolvable baselines (#2640)
- Alexis (@bernalalexis-try), 1: name a location for cross-field errors and a next step for unmatched keys (#1823)
- zgy (@bfs2021), 1: give each merge its own scratch directory (#2310)
- Can Akyuz (@canakyuz), 1: fix preference_loss_weights bco blend timing (#2252)
- Salil Mhatre (@Deadpool2000), 1: sanitize output_dir for machine-readable surfaces (#2807)
- Erizoned (@Erizoned), 1: add --require-all-modes for evidence runs (#2857)
- George Tonmoy Roy (@george07-t), 1: refuse eval-gate suites with a non-string top-level key (#2021)
- Izumx (@Izumx), 1: warn about --token only when it is typed, not when HF_TOKEN fills it (#2918)
- Emil Jafarov (@jafarovvvvvvv), 1: start the pinned-arena hardware test from an empty host cache (#2681)
- Keshemir (@Keshemir), 1: refuse nan like inf on the six ranged float options (#2917)
- nurpeisovnurbolofficial (@nurpeisovnurbolofficial), 1: resolve run prefixes before reading metrics (#2794)
- PANDIARAJ (@Pandiaraj-1), 1: validate kto row shape before loading any model (#2435)
- Sajjad P. Savoji (@SajjadPSavoji), 1: reject calibrated judge reports with zero pairs (#2513)
- Shivansh Shukla (@ShivanshShukla), 1: close the streaming runtime when setup() fails (#2378)
- Shu Chen (@shu-chen-dot-ca), 1: report wrong-typed training values as validation errors (#2382)
- Shutaru (@Shutaru), 1: externalize bulk #2162 evidence and retain numerical audits (#2644)
- Siddhardha Nanda (@SID-6921), 1: stat-check a file against the size cap before hashing it (#2690)
- Sofia5050 (@Sofia5050), 1: declare the mlx floor that mlx-lm needs (#2779)
- Vansh Sharma (@VanshSharmaPES), 1: match the config's precision to the trainer's (#2276)
- Kratos (@wanjinhao1), 1: Fix #2220: drop RST backticks from the neftune refusal and use singular 'requires' in soup export's gguf-on... (#2259)
- xiemocun (@xiemocun), 1: resolve a string lora.target_modules inside a freeze plan (#2289)
- Ziyadinkhan Kudaibergenuly (@Ziyadin23), 1: preserve line endings in quoted CSV fields (#2687)
Reports that led to fixes in this release: @abdumutalipabay0 (#2787, #2788, #2794, #2796, #2812, #2855, #2885, #2898, #2928), @Batyrkhantzy111 (#2414), @baurvtm-stack (#2918), @duysengali-kuanysh (#2689, #2695, #2819, #2833), @Faisal-Fayaz (#2781), @noncalan (#2877), @Samearth17 (#2418), @taljeon (#2513, #2838, #2900, #2911, #2955, #2979).