github huggingface/transformers v5.23.0
Release v5.23.0
on GitHub

41 minutes ago

Release v5.23.0

New Model additions

Inkling2

image

Inkling 2 Small is a multimodal model from Thinking Machines Lab that accepts text and image inputs and generates text outputs, and it is intended for use in English and other languages and across multiple coding languages. Its language backbone has 48 mixture-of-experts decoder layers, each routing tokens to 8 of 384 experts, for 594B total and 17B active parameters. It keeps the Inkling decoder (learned relative attention, short causal convolutions, mixture-of-experts layers) and adds block attention residuals, gated attention with a learned sink, and dynamic key/value convolutions. Images are encoded by a Perception Encoder-based vision transformer and projected into the language model embeddings, one embedding per 40x40 pixel patch.

Links: Documentation | Blog Post

JEV

JEV is a decision head that scores single-token labels with a linear layer at the last token of the prompt, soft-capped like the logits of a language model head. JevForDecision is the model of the checkpoint and its head, and it reads the head files shipped next to the weights (head.safetensors and judge_config.json for JEV). The same head is also used by GEV and reads the 255 answer codes of pplx-decider and d3, and it can be run through the experimental decision pipeline or directly with decide.

Note: JEV and the decision pipeline are currently experimental. Their APIs, configuration, and behavior may change as development continues. They should be used with caution in production environments.

Links: Documentation

LFM3

image

LFM3 is the third generation of Liquid Foundation Models from Liquid AI. Like LFM2, it is a hybrid architecture that combines gated short convolutions with grouped-query attention, with the attention layers split between full-attention and sliding-window layers. The attention layers apply RMSNorm to the query and key heads and gate their output with a per-head sigmoid. Every layer applies a sandwich norm to the operator and feed-forward outputs before the residual addition.

Links: Documentation

Dots 3 Note Preview

Dots 3 Note Preview is a mixture-of-experts causal language model that accepts native text, image, video, and audio inputs. It uses a shared vision encoder for images and videos and a Whisper-style audio encoder, and both encoders project their outputs into the language model's hidden space before autoregressive text generation. BF16 and FP8 inference are supported.

Links: Documentation

LFM3-VL

LFM3-VL is a vision-language model family from Liquid AI, built on the LFM3 language model. It takes images and videos at their native resolution together with text. It combines a hybrid vision encoder (a linear patchifier, gated depthwise 2D convolution layers, and bidirectional grouped-query attention layers with 2D rotary embeddings) with a projector and an LFM3 language model. The projector merges each 2x2 block of patch features into one language token through a layer norm and a two-layer MLP.

Links: Documentation

IQuestCoder

IQuest-Coder-V1 is a family of code language models developed by IQuest Lab, available in 7B, 14B and 40B sizes with base, instruct and thinking variants. Its architecture is the same as Llama, so the Transformers auto classes load it with LlamaModel and LlamaForCausalLM, and only IQuestCoderConfig is specific to IQuestCoder. The looped IQuest-Coder-V1-*-Loop-* checkpoints use a different architecture and aren't supported.

Links: Documentation

InternLM3

InternLM3 is an 8B instruction-tuned language model developed by Shanghai AI Laboratory (checkpoint: internlm/internlm3-8b-instruct). Its architecture is the same as Llama, so the auto classes load it with the Llama implementation (LlamaModel and LlamaForCausalLM), and only InternLM3Config is specific to InternLM3. The config names the attention biases qkv_bias and the output projection and MLP biases bias. InternLM3Config loads these as attention_bias and mlp_bias and requires them to be equal, as they are in the released checkpoints.

Links: Documentation

Breaking changes

Exporters: export, share and run any model on its backend (#50029, #50032, #50309, #50311)

exporters.mp4

Every export now returns an ExportedModel. You call it like the original model (same kwargs, same output class), push it to the Hub, and load it back on the backend's own runtime. The same API covers Dynamo, ONNX Runtime, OpenVINO and ExecuTorch. 

from transformers import AutoModel, AutoTokenizer
from transformers.exporters import ExportedModel, OnnxConfig, OnnxExporter

model = AutoModel.from_pretrained("google/embeddinggemma-2")
tokenizer = AutoTokenizer.from_pretrained("google/embeddinggemma-2")
inputs = tokenizer(["Exporters are here!"], return_tensors="pt")

exported = OnnxExporter().export(model, inputs, OnnxConfig(dynamic=True, ort_fusions="cuda"))
exported.push_to_hub("my-org/embeddinggemma-2-onnx")

exported_model = ExportedModel.from_pretrained("my-org/embeddinggemma-2-onnx", device="cuda")
outputs = exported_model(**inputs)  # same output class as model(**inputs)
  • Fused for the runtime: OnnxConfig(ort_fusions="cpu" | "cuda" | "webgpu" | "dml") rewrites the graph into ONNX Runtime's fused transformer ops at export time: attention, GroupQueryAttention over the KV cache, rotary, norms and MoE. OpenVINO exports now hit OpenVINO's fused CPU kernels for attention, RoPE, MoE experts and compressed int8/int4 weights, which makes decode 2.4–3.9× faster than before.
  • Faster than torch.compile: on EmbeddingGemma2 (8 × 128 tokens), ONNX Runtime with the TensorRT EP is 7.6× faster than eager PyTorch and 1.4× faster than torch.compile on an A100. OpenVINO is 4.1× faster than eager and 2.2× faster than torch.compile on a Xeon Platinum 8570 (4.9× and 2.6× with int8 weights).
  • Shareable: an export saves and loads like a model, optionally in a subfolder of the model repo. Its export.json records the config, input shapes, package versions and I/O names. Quantized exports (PT2E, NNCF, ONNX Runtime) save and run the same way.

🚨 export() returns an ExportedModel, and an ONNX program is an onnx_ir.Model. OnnxConfig's and OpenVINOConfig's output_path options are gone in favour of save_pretrained. Custom backends register with register_backend.

📚 Docs: https://huggingface.co/docs/transformers/main/en/exporters

Vision

This release fixes a range of vision-model bugs: classification pipelines now respect problem_type, and several models get corrected behavior (Step3.7 RoPE axis order, MiniMax-M3-VL packed attention isolation, Qwen VL positions with reused caches, PaddleOCR-VL interpolation alignment, Kosmos2 SDPA masking, and Emu3 float image handling). Chunked prefill now keeps image inputs on the

Parallelization

This release improves distributed training and inference. Expert parallelism gains a selectable ep_implementation (all-reduce for CUDA-graph-friendly inference, all-to-all for training). The Trainer now reduces FSDP/HSDP gradients only on the last micro-batch, and the token-count normalization is fixed to use true division. Tensor-parallel fixes cover attention sinks (including Beam attention) and tied-weight comparisons on TPU

Cache

This release fixes several cache correctness issues: static-cache prefill attention in Gemma4's KV-shared layers, Qwen VL positions when a cache is reused with a new image or video, and DeepSeek-V4 compression-state reordering during beam search. It also improves cache sharing for Mistral Large 4 and adds OpenVINO export fusion for the KV cache and attention. Smaller changes cover CI and test fixes, including vLLM CI memory

Kernels

This release fixes several hub kernel issues: the MPS RMSNorm kernel repo id was corrected, the MegaBlocksMoeMLP mapping was restored for MXFP4 GPT-OSS, and a failed flash-attention kernel load is now retried instead of leaving models unregistered. It also raises the minimum kernels version (fixing ROCm issues and a multi-GPU bug), bumps mamba-ssm kernels to v4

Attention

This release fixes a range of attention bugs: vision-attention isolation in MiniMax-M3-VL, SDPA decode under torch.compile, eager NaNs in Qwen4-Exp with left padding, an unintended causal mask in Kosmos-2's SDPA projection, and default full-attention mask selection that wrongly picked linear layers. It also retries failed attention kernel loads, excludes padding from the repetition penalty, and lets the O

Generation

This release fixes several generation bugs: padded generation in RecurrentGemma, batched decoding in Falcon-H1, beam search cache reordering in DeepSeek-V4, and batched metadata and beam scores in TextGenerationPipeline. It also lets an unchanged GenerationConfig be saved without strict-validation errors, avoids copying the KV cache when decoding with grouped-query attention (GQA), and fixes an `Attribut

Quantization

This release fixes several quantization bugs: FP8 modules_to_not_convert entries are now renamed like weight names, so modules such as f_a_proj and f_b_proj are no longer wrongly quantized. MXFP4 experts now dequantize to the requested dtype instead of always using bfloat16. The MegaBlocksMoeMLP hub kernel mapping is restored for MXFP4 GPT-OSS, which fix

Audio

This release makes audio handling more correct and robust. AudioClassificationPipeline now respects the model's problem type (raw outputs for regression, sigmoid for multi-label, softmax otherwise), and VibeVoice and VibeVoice-ASR avoid GPU syncs when removing audio padding and gain a sample argument to disable VAE noise for deterministic outputs. Chat templates can also take audio decoded from video via `{"type": "aud

Bugfixes and improvements

Significant community contributions

The following contributors have made significant changes to the library over the last release:

  • @tarekziade
    • Fix the MPS RMSNorm hub kernel repo id (#50714)
    • QA: Make make run-mic easier to read and to target (#50652)
    • QA: move to transformers-mlinter 0.1.6 (#50071)
    • QA / Add a mic runner in Make (#50407)
    • [tests] Pin the chat-template date in Llama's FA static-cache test (#50296)
    • Remove the GitHub API cache-bust from the AMD docker images (#49351)
    • [Kosmos2] Fix SDPA applying a causal mask in the image-to-text projection (#50118)
    • [LwDetr] Initialize encoder heads like the detection heads (#50041)
    • [OneFormer] Fix loss computation when is_training=False (#50039)
    • Add CI dashboard badge when a pull request opens (#50106)
  • @vasqu
    • [Gemma4] Fix FA slicing (#50681)
    • [Offloading] Fix tests for some models (#50631)
    • [FA] Fix FA tests for unique models (#50632)
    • [GptOss] Fix the kernels autodispatch test (#50630)
    • [kernels] Bump min version (#50509)
    • [Colpali] Fix offload tests (#50432)
    • [SDPA] Fix decode under compile (#50388)
    • [Rope] Fix per layer rope in head dim manipulation (#49864)
    • [FA] Fix head dim manipulation for more special models (#49862)
    • Revert linguistic changes (#50345)
    • 🚨 [kernels] Bump version (#50307)
    • [Beam] Fix TP on attn (#50302)
    • [FA] Less flaky FA tests (#49861)
    • [FA] Fix old models that do not use a proper interface (#49859)
    • [Moe] Fix grouped mm fallback (#49857)
  • @nassarofficial
    • Granite for Docling (#49012)
  • @miraclezqc
    • Add dots3-note Preview model support (#47844)
  • @taking-lying-flat
    • Fix Inkling router precision in BF16 and autocast (#50621)
    • Fix grouped RMSNorm in Mamba2's non-fused path (#49744)
    • Fix Falcon-H1 gated RMSNorm shape during batched decoding (#50507)
    • Fix MiniMax-M3-VL packed vision attention isolation (#50346)
    • Fix DeepSeek-V4 compression cache reordering for beam search (#50348)
    • Fix Qwen4-Exp eager attention NaNs with left padding (#49923)
  • @ydshieh
    • Fix XVector outputs of the wav2vec2 family when the TDNN layers are offloaded (#50646)
    • Fix RecurrentGemma padded generation by declaring its layer_types (#50577)
    • Fix flaky ONNX export test for wav2vec2-style ForPreTraining models (argmax near-tie) (#50575)
    • Gemma4: fix static cache attending to empty cache slots in KV-shared layers at prefill (#50528)
    • Fix flaky test_causal_lm_loss_honors_shift_labels (#50533)
    • Fix flaky test_eager_matches_batched_and_grouped_inference in fp16 / bf16 (#50500)
    • Fix flaky backend equivalence tests with Pillow 13 (follow-up of #50105) (#50510)
    • [FSDP tests] Fork CPU ranks from a forkserver instead of spawning them (#50444)
    • [torch 2.15] Update qwen3_vl_moe FA2 integration test expected values (#50481)
    • [Mistral Large 4] Fix flaky test_left_padding_compatibility (#50479)
    • [torch 2.15] Update integration test expected values for A10 (cuda 8.6) (#50478)
    • [Docker] Update Nvidia CI and Flash Attn CI images to torch 2.15 / cu132 (#50467)
    • vLLM CI: fix install and KV-cache memory, add failure report job (#50436)
    • [Mistral Large 4] Fix flaky test_cached_decode_matches_cacheless (#50435)
    • Fix flaky Gemma4Audio2TextModelTest::test_training_gradient_checkpointing_use_reentrant_true (#50410)
    • Fix flaky T5GemmaModelTest::test_decoder_model_past_with_large_inputs (#50409)
    • Fix flaky Owlv2ModelTest::test_eager_matches_sdpa_inference_01_fp16_pad_left (#50403)
    • Fix flaky PocketTTSAudioTokenizerModelTest::test_batching_equivalence on CPU (#50402)
    • Update PR CI to use torch 2.15 (#50361)
    • fix(fetcher): case-sensitive submodule check on case-insensitive filesystems (Windows) (#50326)
    • fix(cosmos3_edge): bump backend_equivalence_atol for Pillow 13.x (miss from #50105) (#50327)
    • feat(ci): runtime dep recorder — track transformers imports per test file (#50312)
    • perf(pipelines): lazy-load pipeline submodules in pipelines/init.py (#50173)
    • Fix EncoderDecoderCache missing .layers in torch export tests (#50263)
    • fix(tests_fetcher): handle X as X re-export syntax and duplicate module entries (#50253)
    • fix(tests_fetcher): use posix paths throughout, eliminate all os.path.sep/os.path.join patterns (#50219)
    • fix(glm4v): bump backend_equivalence_atol to 0.5 for Pillow 13 compatibility (#50170)
    • test: lazy-load pipeline classes to eliminate collection-phase noise (#50158)
    • Fix AttributeError on LinearAttentionLayer in export generate test (#50140)
    • feat(ci): surface pre-existing failures in PR run-slow comment (#50122)
  • @eustlb
    • 🚨 Slim down [audio] to torchaudio + soundfile + soxr (drop librosa and phonemizer) (#50610)
    • 🚨 Deprecate Wav2Vec2ProcessorWithLM and pyctcdecode/kenlm LM decoding (#50408)
    • Don't pick torchcodec when it can't load or has no FFmpeg (#50362)
    • Import torchcodec lazily in processing_utils (#50325)
  • @ebezzam
    • Revert "Add transcription input arg to apply_transcription_request (#49869)" (#50587)
    • Add transcription input arg to apply_transcription_request (#49869)
  • @paulpak58
    • [Model] LFM3 Family: LFM3-VL (#50392)
    • [Model] LFM3 Family: LFM3-Text (#50322)
    • [Video] Require one metadata entry per video and refuse return_metadata=False (#50060)
    • [Masking] Skip linear layers for the default full-attention mask (#50059)
  • @ArthurZucker
    • Add Jev + decision core (#50054)
    • [compressed-tensors] better path through conversion mapping (#50238)
    • Use repo-wide linguist-generated globs so new models don't touch .gitattributes (#50181)
    • Mark generated files as linguist-generated (#50061)
    • v5.23.0-dev0 (#50155)
  • @harshaljanjani
    • fix(gemma3n): Clamp the AltUp magnitudes instead of using a CPU epsilon tensor (#50457)
    • [vLLM] Add Omni token counting and text replacement offsets for the Transformers backend (#49435)
  • @remi-or
    • [Improvement] Formalize KV sharing (#47290)
    • [Fix] Remove WIP doc for Ml4 (#50124)
Check out latest releases or
releases around huggingface/transformers v5.23.0

Don't miss a new transformers release

NewReleases is sending notifications on new releases.

Get notifications