github microsoft/onnxruntime v1.32.0
ONNX Runtime v1.32.0
on GitHub

one hour ago

ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases.

Highlights

  • Reintegrated ONNX 1.23.2 with opset 28 operator support (SpaceToDepth, Mod, signed BitShift, and BFloat16 extensions) and updated DLPack to v1.3 across all implemented ORT numeric and boolean tensor types (#33134, #32362, #33848).
  • Added a session-scoped CPU/GPU hybrid MoE offload planner and execution backend for quantized QMoE (NVFP4, INT4, INT3, and INT2) and block-packed MoEPacked models, backed by shared-layout AVX-512/AVX2 CPU QMoE kernels and memory-mapped expert storage (#33450, #33451, #33452, #33973).
  • Expanded core APIs with OrtApi::EpDevice_GetMemoryCapacity, explicit multi-mode weightless compilation (ModelCompilationOptions_SetWeightlessMode and ep.enable_weightless_mode), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in AddExternalInitializers (#33447, #32899, #33640, #33877).
  • Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading (#33407, #33939, #34009, #33839).
  • Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (GatedDeltaNet per-key-dimension decay), right-padded chunked prefill in GroupQueryAttention, native GGUF/block-packed operators (GatherPacked, MatMulPacked, MoEPacked), and sparse-attention index sharing (#33817, #33503, #33953, #33224).

Announcements & Compatibility

  • CUDA Plugin EP v0.4.0 and WebGPU Plugin EP v0.6.0 are matched with ONNX Runtime 1.32.0 (#33214). For contrib-op compatibility, core and the plugin must use the same contributed-operator schemas; building both from the same ONNX Runtime revision is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration.
  • ACL EP removed: Following its deprecation in 1.31.0, the Arm Compute Library (ACLExecutionProvider) execution provider, its build options, and its standalone C registration API have been removed. Python's get_all_providers() no longer lists ACLExecutionProvider; applications should remove it from explicit provider lists (#33851).
  • Legacy provider-bridge NvTensorRTRTXExecutionProvider removed: In-tree provider-bridge NvTensorRTRTXExecutionProvider sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via OrtApi::RegisterExecutionProviderLibrary. NvTensorRtRtxProviderOptions is retained as a deprecated declaration for source compatibility, and legacy AppendExecutionProvider calls now report a deprecation error directing callers to the plugin EP (#32276).
  • ONNX 1.23.2 integration: ONNX has been upgraded to 1.23.2, raising supported opset ceilings to 28 and restoring ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3 and ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2 public element-type mappings and Cast conversions (#33134).
  • DLPack v1.3 boolean contract change: DLPack has been upgraded to v1.3 with native kDLBool metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five float8 formats, packed float4, int4/uint4, and int2/uint2 (float6 remains unsupported in DLPack). The low-level is_bool_tensor override was removed, and uint8 DLPack metadata now always imports as uint8 (#33848).
  • Weightless compilation API migration: OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled and the ep.enable_weightless / ep.enable_weightless_ep_context_nodes session options are deprecated in favor of ModelCompilationOptions_SetWeightlessMode and ep.enable_weightless_mode ("0" = NONE, "1" = EXTERNAL_ONLY, "2" = ALL). EPs can now advertise multi-mode capability via OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY and weightless_supported_modes (#32899).
  • Training CI retirement: Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage (#33836).

New Features

Core APIs & Runtime

  • Added OrtApi::EpDevice_GetMemoryCapacity (C++ EpDevice::GetMemoryCapacity()) and OrtEpFactory::GetDeviceMemoryCapacity to query device-global total memory and estimated currently available memory (DEFAULT or HOST_ACCESSIBLE), with built-in and plugin CUDA EP support (#33447).
  • Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (weightless_supported_modes), applications request an explicit mode (ModelCompilationOptions_SetWeightlessMode / ep.enable_weightless_mode), and compiled EPContext models record weightless_mode and external_dependencies (source_model, external_weights, ep_context_binary) in model metadata (#32899).
  • Extended session.use_external_initializer_file_buffers_directly="1" to AddExternalInitializers for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via OrtModelEditorApi::AddInitializerToGraph with data_is_external=true (#33877, #33717).
  • Added lock-free multi-worker pinned-memory staging (session.external_data_loader.*) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs (#33839).
  • Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (session.moe.offload.*), generic session.kernel_dispatch_policy (auto, latency, memory, safe) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types (#33451, #33042, #33311, #33841).
  • Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (<profile_file>_compile) in onnxruntime_perf_test when -p is used with --compile_ep_context (#33053, #33694).
  • Made onnxruntime_DISABLE_EXCEPTIONS (--disable_exceptions) usable without onnxruntime_MINIMAL_BUILD and added the onnxruntime_DISABLE_STREAM=ON CMake option to disable device-stream support in non-minimal builds (#33120, #33482).

ONNX Opset 28 & Shared Contrib Operators

These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins.

  • Added ONNX opset 28 operator support: native SpaceToDepth (DCR and CRD) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus Mod on CPU, CUDA, and DML; opset-28 BFloat16 support for Compress, OneHot, ReverseSequence, Unique, Celu, and Einsum on CPU (plus Compress, OneHot, and ReverseSequence on CUDA); and BFloat16 Pad (opsets 21–25) on CPU and CUDA (#32362, #32369, #32378, #34062).
  • Extended GatedDeltaNet with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended GatedDeltaNet and CausalConvWithState state/gate handling for multi-token speculative decoding (#33817, #33471).
  • Added an optional query_seqlens input to GroupQueryAttention to skip right-padded prefill tokens and idle batch entries (query_seqlens[b] = 0) on CPU and CUDA, and added q_norm_epsilon / k_norm_epsilon attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion (#33503, #33505, #33832, #33363).
  • Defined the INT3 QMoE v1 packed-code contract, added opt-in post-SiLU clipping (swiglu_limit_after_activation) and tanh-GEGLU (activation_type="geglu") support to MoE/QMoE schemas, and added GatherPacked (CPU/CUDA), MatMulPacked (CUDA, including native Bolt global_scale), and MoEPacked (CUDA) to consume native GGUF/block-packed quantized weights directly (#33394, #33954, #33633, #33953, #33962).
  • Added PackedSparseAttentionIndexerMerge for multi-token prediction index sharing and optional state_status / mid-state outputs on PackedSparseAttentionIndexer across CUDA and WebGPU; extended GatedAdd with an optional projected gate_weight across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU MatMulNBits (#33224, #33383, #33472, #33470, #33559).
  • Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including SwiGLU, RMSNormRotary, HyperConnectionLowRankMix, RouterGateUpProj, SpeculativePrefixCommit, GatedDeltaNetStateReplay, ConvStateReplay, and MatMulNBitsQLoRA (#32126, #33469, #33384, #33650, #33474, #33213, #33493, #33370).

Plugin Execution Provider Infrastructure

  • Fixed an OrtMemoryInfo ABI mismatch across the ORT/plugin EP boundary by converting OrtMemoryInfo instances exclusively through the C API (#33571).
  • Zero-initialized OrtEpFactory structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries (#33117, #33701).
  • Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and shared_assets path confinement and symlink validation (#32790, #33826).

Execution Provider Updates

CUDA EP

CUDA updates include CPU/GPU hybrid expert offloading (session.moe.offload.*) for QMoE (NVFP4, INT4, INT3, INT2, and mixed-width) and MoEPacked; offline K-packed NVFP4 and Tensor-Core INT4/INT3/INT2 QMoE kernels; native GGUF/block-packed GatherPacked, MatMulPacked, and MoEPacked execution; long-context (up to 256K) and speculative-decoding acceleration across DynamicSparseAttention, PackedSparseAttentionIndexer/Merge, GatedDeltaNet, HyperConnectionLowRankMix, SwiGLU, RMSNormRotary, RouterGateUpProj, SpeculativePrefixCommit, H512 XQA decode, and H256 BF16/INT8 paged speculative XQA; an opt-in persistent fpA_intB MatMulNBits tactic cache and deterministic tactic selection; route-complete Level-1 workspace estimation; cuDNN 9 frontend Conv plan caching; opt-in EP-owned runtime fusions; standard ONNX integer bitwise/BitShift and BFloat16 ArgMax/ArgMin/Pad kernels; and numerical and synchronization fixes (#33452, #33973, #33094, #33689, #33802, #29588, #33078, #33123, #33757, #34021).

Detailed CUDA features, configuration options, limitations, and fixes will be covered in the separate CUDA Plugin EP v0.4.0 release notes. Most CUDA-specific PRs are covered there rather than repeated here. Use the matched core/plugin combination and observe the contrib-op schema compatibility requirement above.

WebGPU EP

  • Expanded generative AI and quantized inference with GatedDeltaNet KDA, GatedDeltaNetStateReplay and ConvStateReplay for speculative decoding, PackedSparseAttentionIndexerMerge, fused MatMulNBitsQLoRA, dense and routed block-scaled FP8 MatMul/QMoE, 16x16x16 and NVIDIA shared-memory subgroup-matrix MatMul, AMD wave64 and wide-tile Intel subgroup-matrix MatMulNBits, Qualcomm wide-subgroup DP4A, subgroup-matrix Conv, Intel dense-prefill FlashAttention, and M=1–8 FP16 GEMV (#33213, #33493, #33370, #33074, #33075, #32937, #33639, #32796, #32252, #32702).
  • Improved native host integration, runtime efficiency, and W3C WebNN conformance with Win32/D3D12 accelerated external weight loading on Windows, host Dawn shader-cache callbacks (OrtDawnCacheInterface), enableFeatures/disableFeatures EP feature controls, MatMulExecutionPlanner, batched transfers and graph-capture copy/clear replay, synchronized per-session command recording, bundled D3D12 Agility SDK Python wheel DLLs, and expanded QuantizeLinear/INT4/INT8/FP16 RNN/WebNN operator coverage (#33055, #33936, #33481, #32630, #33491, #33362, #33082, #32795, #33683, #33792).

Detailed WebGPU kernel additions, optimizations, and correctness fixes will be covered in the separate WebGPU Plugin EP v0.6.0 release notes.

Other Execution Providers

  • Enabled OpenVINO CPU EP device discovery on non-Intel x86 and ARM CPUs (#31981).
  • Fixed TensorRT subgraph cycle removal to map cycle vertex IDs directly to TensorRT subgraph indices (#34025).
  • Fixed JSEP Reshape and Unsqueeze schema registrations for opset 25+ models and flushed pending JSEP command encoders before CPU-to-GPU uploads onto reused buffers (#33749, #33361).

CPU & Core Optimizations

Session Initialization, MLAS, and CPU Kernels

  • Replaced linear free-buffer scans in the allocation planner with a compatibility-keyed reuse index and eliminated superlinear scans across constant sharing, graph I/O mapping, element-type lookup, and Kahn's topological sort, cutting session creation time on 100k-node constant-heavy graphs from over 10 minutes to ~5 seconds (#33407, #33939, #34009, #34068).
  • Added direct CPU QMoE execution over shared-layout INT4, INT3, INT2, and FP4 expert banks with AVX-512 and AVX2+FMA+F16C kernels, register-spill elimination, byte-preserving memory-mapped FileExpertStore reads, and opt-in ep.cpu.qmoe_deferred_scale (#33450, #33624, #33785, #33935).
  • Added transposed-B support to KleidiAI SME/SME2 FP16 GEMM on Arm and verified Linux x64 AMX tile-state preservation across CPU migration before enabling MLAS AMX QGEMM dispatch (#33130, #33258).
  • Enabled CPU MatMulNBits execution for block sizes above 256 (including 16-bit activation QDQ conversion), supported 1-D kernels in float channels-last NhwcFusedConv, improved MLAS NCHWc convolution thread load balancing, and parallelized CPU MultiHeadAttention QKV transposes (#32885, #33690, #32335, #33887).
  • Supported independent input/scale types and the opset-23 precision attribute in CPU QuantizeLinear, hardened AVX2 M=1 SQNBit kernels, and moved GetCpuPreferredNodes element-type checks to a compile-time switch (#32735, #29886, #29647).

Graph Optimization & Quantization Tools

  • Introduced an experimental, opt-in C++ graph pattern rewriter (onnxruntime::rewrite) and region-scoped EP capability rewriting infrastructure (#33631, #33652).
  • Extended Slice + Concat to SpaceToDepth fusion for YOLOX focus blocks (#32335).
  • Fixed PadFusion changing MaxPool results on negative inputs, MatmulBNFusion when intermediate Reshape/Transpose moves the channel axis, GemmTransposeFusion folding public graph outputs, repeated NHWC ConvActivationFusion, MatMulAddFusion on zero-sized dimensions, and lossy DOUBLE-to-float constant conversions in LayerNorm and MatMulScale fusions; reverted an unnecessary constant-folding determinism change (#33206, #33392, #33484, #33666, #33766, #32983, #32471).
  • Hardened QDQ and NCHWc transformations by skipping QLinear* and QDQ-drop rewrites for float8/float4 quantized types, validating quantization parameter shapes and types before QDQ Add/Mul/Concat/MatMul, MatMulIntegerToFloat, and QLinearSoftmax fusions, and handling zero-channel Conv and empty-batch Add in NCHWc transforms (#33663, #33397, #33616, #33485, #33568, #33346, #33518).
  • Preserved subnormal float16 quantization scales for small-range weights, fixed dynamic Gemm(transB=1) quantization with shared or shape-annotated weights, reused Cast nodes for shared always-float inputs in float16.py, and fixed Windows file-locking and relative external-weight resolution during float16 model-path conversion (#33239, #33415, #33090, #33193).
  • Added semantic and null-safety guards to training QDQ, transpose, scaled-sum, and SoftmaxCrossEntropyLoss graph optimizers, and updated training CUDA stream and runtime cuDNN loader call sites (#32982, #32987, #32981, #33744).

Security & Reliability

Model Loading and Graph Validation

  • Validated ORT-format graph edge slots, enforced a maximum call depth (100) for model-local functions, validated inline sparse payloads and index types before dense allocation, and checked equivalence of serialized and cross-session shared prepacked weights (#32200, #32339, #32863, #33499).
  • Preserved 256-byte prepacked-weight alignment across save and reload, handled kernel type-string resolution errors via Status instead of throwing under ORT_NO_EXCEPTIONS, and included the originating C API function name in catch-all error messages (#33700, #33828, #29561).
  • Fixed GroupQueryAttention and MulInteger type/shape inference for growing KV caches, shape-derived int32 sequence lengths, and omitted zero points (#32717, #33884, #33623).

Runtime and Operator Correctness

  • Fixed premature release planning for cross-stream aliased buffers, allowed scalar/{1} single-element tensor buffer reuse across rank representations, avoided module-load dynamic initialization for OrtEnv's mutex, and resolved Windows 1DS telemetry shutdown deadlocks and late-worker SDK lifetime races (#33881, #29863, #33419, #33291, #33321, #33993).
  • Strengthened dimension and bounds checks across Attention (projection/cache dimensions and positive head counts), PagedAttention, PackedAttention, LinearAttention, MoE hidden dimensions, MatMulNBits FP16 prepacking, Resize linear index bounds, Concat axis overflow, Whisper timestamp token bounds, TreeEnsemble category set IDs, and runtime tensor ranks in CPU Conv/ConvInteger/QLinearConv/CropAndResize/Einsum/AffineGrid (#32946, #33498, #31651, #33330, #33821, #33820, #33375, #33497, #33235, #33824, #33825, #33517).
  • Fixed GroupQueryAttention reading device-produced total_sequence_length tensors, rank-reduced broadcast scales in CUDA LayerNormalization/RMSNormalization, Split with num_outputs and an empty optional split input, zero-divisor validation in CUDA integer Mod, and Arm64 FP16 grouped Conv packed-weight offsets and per-group bias (#33945, #28971, #32972, #32546, #33598).
  • Fixed CPU MoE input corruption on aliased buffers, ordered NaN as the largest value in CPU TopK, fixed fractional floating-point indices in CPU OneHot, prevented QLinearSoftmax requantization overflow and empty-input failures, and fixed integer Range boundary counts, zero-K Gemm signed-zero/bias scaling, empty negative-axis reductions, and empty batch-transposed or zero-rank MatMul operands (#33885, #33543, #33273, #33277, #33401, #33399, #33669, #33232, #33512, #33171).

Language Bindings & Web

  • Added Python bindings for authoring, loading, and selecting variants from portable model packages (ModelPackage, ModelPackageContext, ModelPackageOptions, ModelPackageComponentContext), along with ModelCompiler.set_ep_context_binary_information (#33640).
  • Retained Node.js WebGPU output resources across session execution (#33741).
  • Excluded graph construction, ONNX schema/shape inference, and Protobuf parsing from Emscripten Asyncify instrumentation in JSEP and non-JSPI WebGPU WebAssembly builds (#33563).

Build, Packaging & Dependencies

  • Expanded CUDA 13 architecture coverage and reduced package sizes: added sm_87 (Jetson Orin), sm_100, sm_103 (GB300), and sm_110 (Jetson Thor) to Linux aarch64 CUDA 13 packages; switched the Linux aarch64 wheel to Blackwell family targets (100f, 120f, saving ~82 MB); reused sm_80 FlashAttention cubins on sm_86/sm_89 by default (saving ~30 MB in x64 GPU packages); added native sm_120 SASS to the CUDA plugin LLM target under MSVC; and added an experimental unified multi-ABI CUDA 13 Python wheel pipeline (#33369, #33315, #33649, #33301, #34089, #34088, #33173, #33034, #33771).
  • Added CMake package config files to Windows C API artifacts, isolated volatile ORT_BUILD_INFO provenance from shared onnxruntime_config.h, and fixed CMAKE_CUDA_ARCHITECTURES=all cache handling, Windows --disable_contrib_ops CUDA builds, system cuDNN include-directory filtering, Windows GPU package validation, and Clang/no-exceptions portability (#28897, #33687, #28019, #33702, #33736, #33889, #33408, #33532, #33583, #33830).
  • Updated Dawn, cpp_client_telemetry to 3.10.309.1 and the vcpkg baseline, libprotobuf-mutator, Python tooling pins (transformers 5.10.1, onnx >= 1.21.0 / 1.22.0, Pillow >= 13.0.0, setuptools >= 84.2.1), and JavaScript/Web dependencies including security updates for browserslist, shell-quote, nanoid, undici, fast-uri, next, electron, and follow-redirects; removed the stale Intel MKL third-party notice (#33390, #33949, #33210, #33501, #33591, #33734, #33656, #33903, #33587, #33588, #33589, #33608, #33610, #33233).

Contributors

Thanks to our 94 human contributors for this release!

@aciddelgado, @adrastogi, @akasakariko, @AlekseiNikiforovIBM, @alexk201, @apsonawane, @arnej27959, @aswanth-07, @baijumeswani, @BBaditi, @bmehta001, @cboseak, @cbourjau, @chilo-ms, @ConnorBaker, @cursoragent, @daijh, @danielsongmicrosoft, @DavidLuong98, @DevChuriwala, @edgchen1, @El1as42, @eserscor, @galleonli, @GopalakrishnanN, @gramalingam, @gyagp, @hanbitmyths, @haoxiz-nvidia, @HuzaifaChaudary, @iBreaker, @javier-intel, @Jayashree-mcw, @Jayashree-VR, @jchen10, @jiafatom, @Jiawei-Shao, @jmcarcell, @justinchuby, @KafKafrnZ, @keshavv27, @kunal-vaishnavi, @kylo5aby, @leninworld, @LinCanNerd, @lukas-zaborsky, @Manogna-Sree, @maxim-davgalev, @MayureshV1, @mei1127, @mirounga, @mohdmahmodi, @Mohit-25-tech, @MokiMeow, @mustjab, @n1harika, @nazanin-beheshti, @nietras, @ninglinLiu, @patryk-kaiser-ARM, @prathikr, @preetha-intel, @professor-moody, @qjia7, @rookieLiu2018, @rui-ren, @rvandermeulen, @samridh10exe, @sankalpsthakur, @sayanshaw24, @selenayang888, @SharmaRithik, @silviot, @sisleyli, @sushraja-msft, @swetha097, @TANGBUDU, @the0cp, @tianleiwu, @timothyanderson096-ocdealcheck, @titaiwangms, @Trisha-Sarvam, @TrishaWadhwani, @usmanmateen, @vinay-852, @xadupre, @xenova, @xgbtxy, @xhcao, @xiaofeihan1, @xiaoyu-work, @yairbenavraham, @ykhrustalev, @Yuhx141

Full Changelog: rel-1.31.1...rel-1.32.0

Release highlights were prepared with AI assistance.

Check out latest releases or
releases around microsoft/onnxruntime v1.32.0

Don't miss a new onnxruntime release

NewReleases is sending notifications on new releases.

Get notifications