pypi transformers 5.20.0
Release v5.20.0
on Python PyPI

4 hours ago

Release v5.20.0

New Model additions

Muse Spark Open

image

Muse Spark Open is a multimodal mixture-of-experts model from Meta that takes text, images, audio, and video as input and generates text. It has 816B total parameters (42B active per token) and a context window of up to 1,048,576 tokens shared across modalities. It is trained for agentic coding tasks such as multi-file editing, debugging, and tool use. Checkpoints are available in BF16 and NVFP4 formats.

Links: Documentation | Blog Post

Ling 3.0

Ling 3.0 Flash is a hybrid Mixture-of-Experts language model from InclusionAI. Its decoder groups five Kimi Delta Attention (KDA) layers with one Multi-head Latent Attention (MLA) layer. The MLA blocks use interleaved rotary embeddings and a head-wise output gate, and the feed-forward blocks combine routed and shared experts. It is natively supported in Transformers as BailingHybridForCausalLM, so it loads without trust_remote_code=True.

Links: Documentation

Kolibri1

Kolibri 1 is Aleph Alpha's mixture-of-experts reasoning model, focused on German and English. Each of its 50 layers routes every token to 6 of 384 experts using sigmoid routing, and adds one shared expert. Four sliding-window attention layers alternate with one full-attention layer, and RoPE is applied only in the sliding-window layers.

Links: Documentation

MiniCPM-V 4.7

MiniCPM-V is a series of efficient multimodal large language models developed by OpenBMB. Like MiniCPM-V 4.6, the MiniCPM-V 4.7 architecture pairs a SigLIP vision encoder that has a window-attention merger with a Qwen3.5 language model backbone, and supports both 4x and 16x visual downsampling modes.

The main addition over 4.6 is canvas M-RoPE: instead of numbering visual tokens along a single 1-D sequence, the model lays every image out on a 2-D canvas and assigns each visual token a (temporal, height, width) position, so slices of the same image keep their spatial relationship and video frames keep their temporal order.

UnlimitedOcr

UnlimitedOcr is a single 3B-parameter OCR model from Baidu with a standard 32,768-token context length. It builds on DeepSeek OCR and replaces all decoder attention layers with Reference Sliding Window Attention (R-SWA). This keeps the KV cache constant during decoding, so the model can transcribe dozens of pages in a single forward pass. It supports "gundam" mode (640x640 tiles with dynamic cropping) and "base" mode (a single 1024x1024 view), and it can output region detections with bounding boxes.

Links: Documentation | Paper

UltraBERT

UltraBERT is a family of bidirectional text encoders trained with a combination of masking ratios, pruning, and distillation for classification and retrieval tasks. The released encoders support sequences of up to 32,768 tokens. They use rotary position embeddings, RMSNorm, SwiGLU feed-forward layers, and alternating local and global attention (every third layer is global). Checkpoints include UltraBERT-150M and UltraBERT-380M for masked language modeling and fine-tuning, and UltraBERT-Embed-150M for text embeddings.

Links: Documentation | Paper

MOSS-Transcribe-Diarize

MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model for long-form multi-speaker transcription, speaker diarization, and timestamps. It combines a Whisper-style encoder, a 4× frame merge step, a multi-modal projector, and a Qwen3 language model. It produces joint transcription and diarization in a single pass, with segments formatted as [start][S01]text[end]. It supports long-form audio through Whisper-window chunking in the processor, promptable transcription with hotword biasing, and 50+ languages.

Links: Documentation

Breaking changes

🚨 General breaking notice that inference with EP has regressions in speed; we are working on fixing this ASAP! 🚨

Deprecated checks and methods have been removed from cache_utils.py (affecting models such as minimax_m3_vl and pi0), so any code still calling those deprecated cache methods must switch to the current cache API.

Accidental imports from other models were removed from several model files (e.g. embedding_gemma2, ijepa, lightglue), and the modular converter now avoids adding such cross-model imports. If you imported these names through those model modules, import them from their original modules instead.

The minimum kernels version is now 0.17.2, which allows use_fallback=KernelizeFallback.ALL so that a failed kernel load falls back to the original implementation instead of raising an error. Upgrade kernels to 0.17.2 or later.

  • 🚨 Bump min kernels to 0.17.2 for better fallback (#49235) by @SunMarc

A new SubConfigSpec centralizes the initialization of sub-configs, so default sub-configs are collected from dataclass fields instead of by instantiating the class, and this touches the configuration utilities and many model configs. Custom or existing nested configs may need to be updated to declare their sub-configs through this mechanism.

The LUKE and mLUKE tokenizers now treat entity special tokens such as <ent> as special tokens (not split, honored by skip_special_tokens, normalized) while keeping the same spacing as regular vocabulary tokens when decoding. Tokenization and decoding output involving these tokens may therefore differ from before, so check any code or saved outputs that depend on the old behavior.

Exporters

Transformers exporters now support quantization directly during export across Dynamo, ONNX, OpenVINO, and ExecuTorch. The new unified quantization API integrates PT2E, ONNX Runtime, and NNCF quantizers, enabling static and dynamic quantization as well as weight-only compression (INT8/INT4), with configurable calibration workflows. This makes it easier to export optimized models for efficient inference across different backends.

For more information see the docs: https://huggingface.co/docs/transformers/exporters_quantization

Parallelization

This release improves distributed training and inference. FSDP × TP model and optimizer checkpoints can now be saved and loaded across different parallel configurations, and expert-parallel dispatch, tensor-parallel plans, static-cache generate(), and the caching allocator's byte count were fixed. Tied LM heads in composite models now get their own TP plan entry, enable_expert_parallel applies the EP plan alone again, TPU loading uses the `tor

Generation

This release fixes static-cache generate() bugs with host-side inputs and under tensor parallelism. It also speeds up Whisper generation on TPU by cutting device syncs and recompilation, and fixes Whisper batched timestamp decoding for tensor and NumPy inputs. Pipelines get a cleanup of generation-argument handling that removes spurious warnings.

Vision

Vision processing was updated in several places. Image decoding now uses torchcodec, Molmo2's image and video kwargs follow the library's standard conventions, and Pixtral Flash Attention no longer detects false image boundaries after the axial RoPE refactor, which had caused CUDA illegal memory access errors and incorrect output. Several CI and Docker changes also shipped: GPU images moved to Python 3.11, a numpy/pyarrow conflict

Kernels

This release improves kernel support across hardware. flash_attention_2 now falls back to the AITER kernel on ROCm when flash-attn isn't installed, and T5-style models get a contiguous SDPA position-bias mask so they use fused kernels again. The Hub rotary kernel is now limited to CUDA inference because its backward pass gave incorrect gradients. The Atlas Inference Gated DeltaNet kernels for Strix Halo and

Cache

This release fixes several cache-related bugs: static-cache generate() now handles host-side nested inputs and tensor parallelism correctly, the offloaded cache prefetch no longer races with the offloading copy, and Phi3 LongRoPE cache rebuilds at the context boundary now receive the full prefix. It also counts the query token in MiMo-V2-Flash's sliding window and corrects the quantized cache (hqq/quanto) documentation exam

Bugfixes and improvements

Significant community contributions

The following contributors have made significant changes to the library over the last release:

  • @itazap
    • Add MOSS-Transcribe-Diarize model (#48386)
    • handle empty precompiled charsmap in spm converter (#49211)
    • 🚨🚨🚨 luke fixe (#48408)
  • @ydshieh
    • Add MIC contract lock files for PR CI integration (#49631)
    • Combine Nvidia+AMD run-slow CI comments into one (#49598)
    • Fix CI slack reporting: surface crashed/missing model jobs in model_results.json (#49427)
    • [Qwen3VLMoe] Update expected text in test_small_model_integration_test_expand (#49301)
    • Complete PR #49228: remove unused device=None from rope init in 11 model families (#49285)
    • Revert "[CI] Temporarily disable AMD scheduled CI trigger" (#49433)
    • [CI] Temporarily disable AMD scheduled CI trigger (#49394)
    • [tests] Fix flaky CMVN tests in Speech2TextFeatureExtractionWithoutTorchaudioTest (#49374)
    • [Docker] Upgrade to torch 2.14/CUDA 13.0 and optimize FA2 build from source (#49381)
    • [tests] Skip onnx.exactness for BitForImageClassification (ORT fp32 drift) (#49366)
  • @icavan
    • Fix a port allocation race in FSDP tests (#49455)
    • [Model] Add native Ling 3.0 Flash support (#49446)
  • @tarekziade
    • Build docs with the Python version from .python-version (#49603)
    • Fix flaky streamer timeout tests (#49583)
    • CI: Move GPU images to python 3.11 (#49352)
    • Fix Gemma3nVision2TextModelTest::test_model_parallelism split (#49562)
    • Fix PR slow CI suggestion: install transformers in a venv (#49558)
    • Use one static Transformers CI badge in PR descriptions (#49471)
    • fix: treat every declared field of the config class as a default fixes #49456 (#49466)
    • PR CI: grant actions: read to the reusable workflow (#49417)
  • @lucamotz
  • @SangbumChoi
  • @maxiaolong03
    • [Model] Add PP-DocLayoutV4 Model Support (#48387)
Check out latest releases or
releases around transformers 5.20.0

Don't miss a new transformers release

NewReleases is sending notifications on new releases.

Get notifications