open-clip-torch 4.2.1

v4.2.1

on Python PyPI

The first 4.x release on PyPI, and it adds official pretrained NaFlex CLAP audio-text models. It also moves training to
a new stack and adds several model families. Earlier 4.x version numbers were development versions only.

Pretrained image-text inference is intended to remain compatible, but training scripts and downstream integrations need review. See the migration guide. To stay on 3.x, run pip install 'open_clip_torch<4'.

Highlights

• NaFlex CLAP: five Apache 2.0 audio-text models (27M to 340M parameters) that take variable-length audio, trained on LAION-Audio-XL: XXS, XS, S, M, L
• NaFlex CLIP: variable-resolution and variable-aspect image towers with token-budget batching
• NaFlex GenLIP / GenLAP: generative image and audio captioning with prefix-LM attention (training configs, no pretrained weights yet)
• Modern text towers: RoPE, SwiGLU/ReLU², and RMSNorm options, per-batch variable-length text, and Hugging Face ModernBERT towers
• MaMMUT and CoCa v2: MaMMUT with the released LAION weights, plus corrected coca2_* configs
• Training stack: TrainingTask wrappers, dict-based batches, FSDP2 (--fsdp), torch.compile strategies, and length bucketing
• Safer checkpoint loading: all checkpoint loads use weights_only=True, and OpenAI weights now load from the Hugging Face Hub
  instead of JIT archives

Upgrade notes

• PyPI now requires torch>=2.6 (was >=2.0)
• The default --precision is now amp_bf16 (was amp), and SigLIP's --loss-dist-impl now defaults to gather
• --horovod, --torchscript, and --trace are removed
• load_openai_model, list_openai_models, build_model_from_openai_state_dict, and trace_model are removed; use
  create_model_from_pretrained(..., pretrained='openai') for OpenAI weights
• Dataloaders now emit dict batches, and training code should pass a TrainingTask instead of a separate model and loss
• python -m open_clip_train.legacy_main remains available as a compatibility path for older image-text training scripts

See the migration guide for the full list of changes.

What's Changed

• Refactor checkpoint deletion based on save frequency by @verena-hallitschke in #1140
• Change output format from 'NCHW' to 'NLC' in forward_intermediates of TextTransformer by @vladan-stojnic in #1143
• Hoist DDP unwrap to top of evaluate() so zero_shot_eval runs on bare module by @shaun0927 in #1155
• Fix #1146, ability to set trust_remote_code from hub config is disabled by @rwightman in #1156
• Tinkering with a minimalist task wrapper. Remove Horovod, add FSDP2. WIP by @rwightman in #1136
• Add chunked logit computation to SigLipLoss for memory efficiency by @avocardio in #1145
• Refactor NaFlex WebDataset pipeline around dict batches. by @rwightman in #1158
• Re-do CLAP integration on top of Task changes. Replaces PR #1137 by @rwightman in #1159
• Task compile refactor & TrainState by @rwightman in #1160
• Fix an issue with dataloader worker processes and audio preprocessing… by @rwightman in #1161
• Add optimizer kwarg passthrough by @rwightman in #1164
• Python314 and compile tweaks by @rwightman in #1165
• More memory efficient chunked retrieval eval by @rwightman in #1169
• NaFlexGenLIP experiment by @rwightman in #1168
• NaFlex + audio by @rwightman in #1170
• Add per-modality pre_norm option to GenLIP/GenLAP by @rwightman in #1171
• Change contract of Task to return loss + report dicts by @rwightman in #1172
• Add seq len probs option by @rwightman in #1173
• Update loss logging by @rwightman in #1174
• Add bias control (on/off) for moderntext, by @rwightman in #1175
• Initial trackio logging support by @rwightman in #1177
• Add bias config for GenLIP / GenLAP models and RMSNorm tweaks by @rwightman in #1178
• Misc NaFlex stuff by @rwightman in #1179
• Setup default audio_seq_len for naflex audio models by @rwightman in #1180
• Backward compat change in mel patchify to transpose spatial f vs t by @rwightman in #1182
• Rejigging some data key uses, add support for multiple caption keys by @rwightman in #1183
• Fix NaFlex worker-padded sample count after schedule shuffle by @chenghuichen in #1189
• Allow persistent dataloader workers to be toggled via params by @rwightman in #1185
• MaMMUT and CoCa2 (fixed) by @rwightman in #1188
• Use compact key-only masks for GenLIP/GenLAP encoders by @chenghuichen in #1194
• Add z-loss for next-token CE losses, add dtype support for the CE by @rwightman in #1192
• Set the default eval length for NaFlex ViT-B-16 by @chenghuichen in #1197
• Fix GenerationMixin compat with transformers>=5.14.0 by @joan8627 in #1200
• Fix int8 quantization (Linear8bitLt) and document save/load - fixes #1012, #921 by @joan8627 in #1198
• Add support for pushing audio and TikToken based models to HF hub by @rwightman in #1202
• ci: update action versions in ci.yml to current majors by @katsugtgz in #1203
• Mammut2 cleanup, NaFlexViT CoCa2 by @rwightman in #1204
• Tokenizer cleanup by @rwightman in #1196
• Naflex patch interp by @rwightman in #1207
• Replace model-name sniffing with ModelTraits and other cleanup. by @rwightman in #1209
• Further cleanup by @rwightman in #1210
• Prevent mixed-precision overflow in SigLIP loss reduction by @zjn20030811 in #1206
• An improved NaFlex loss scale impl that covers grad accumulation by @rwightman in #1215
• Siglip chunk + transform interp fixes by @rwightman in #1216
• Respect interpolation for rectangular evaluation crops by @trueoneplusone in #1213
• Bound SigLIP chunked training memory with activation recomputation by @trueoneplusone in #1214
• Add some experimental linear probe / multi-label zero-shot experiments by @rwightman in #1217
• Fix Hub uploads after removal of legacy repository parser by @kocean9-freedom in #1228
• Release prep fixes by @rwightman in #1231
• Stabilize chunked SigLIP loss with diagonal sign flips by @rwightman in #1234
• Accept list image sizes in timm and NaFlex transform setup by @MohammadHijjawi97 in #1235
• Drop collator-only batch keys from accum and distill model forwards. Alternative to #1237 by @rwightman in #1238
• Remove torchaudio dep by @rwightman in #1239
• Add pretrained NaFlex CLAP checkpoints by @GLJS in #1230
• Caption val token weighting by @rwightman in #1241
• Prepare for 4.2.1 release by @rwightman in #1242

New Contributors

• @verena-hallitschke made their first contribution in #1140
• @vladan-stojnic made their first contribution in #1143
• @shaun0927 made their first contribution in #1155
• @avocardio made their first contribution in #1145
• @chenghuichen made their first contribution in #1189
• @joan8627 made their first contribution in #1200
• @katsugtgz made their first contribution in #1203
• @zjn20030811 made their first contribution in #1206
• @trueoneplusone made their first contribution in #1213
• @kocean9-freedom made their first contribution in #1228
• @MohammadHijjawi97 made their first contribution in #1235
• @GLJS made their first contribution in #1230

Full Changelog: v3.3.0...v4.2.1