cuDNN Frontend v1.32.0 Release Notes

cuDNN Frontend v1.32.0 adds Mamba-2, Qwen sparse attention training, Rubin triangle operations, and a CuTe DSL normalization backend. FROST SDPA gains more native forward and backward configurations, broader paged-cache support, and lower setup overhead. This release also adds FROST plan serialization, expands JAX grouped GEMM support, and updates Python bindings and dependencies.

New: Mamba-2 linear attention 🚀 🚀

Mamba-2 joins the FROST linear-attention family with forward, backward, and state-summary operations on SM100, SM103, and SM107. The Python entry point is cudnn.linear_attention.ops.mamba2 (#1577).

• Supports packed sequences, BF16/FP16 inputs, initial and final recurrent states, optional skip connections, output gating, and step-size controls. The supported state dimension is 128.
• State summaries expose the per-sequence end state and transition needed to compose sequence segments. Summary backward is also available.
• Checkpoints can use checkpoint_dtype=torch.float32; Mamba-2 checkpoint intervals are multiples of 128 tokens (#1694).

The wider linear-attention family also gains persistent caching for common forward kernels, a faster KDA backward pipeline, and fixes for multistream workspace reuse, GDP interleaved layouts and cached summaries, and GDN-2 NaNs under steep per-channel decay (#1708, #1699, #1569, #1664, #1695, #1644).

New: Qwen sparse attention training 🚀 🚀

cudnn.ops.qsa_forward and qsa_backward provide explicit packed QSA training calls for Blackwell SM100/SM103 and Rubin SM107. They support BF16 D256 attention with 12Q/1KV or 24Q/2KV heads and four-token blocks. Forward returns the BF16 output, an FP32 saved output, and natural-log LSE for backward. These functions do not install an autograd wrapper (#1765, #1778).

• Indexed training accepts existing token selections through qsa_indexed_forward and qsa_indexed_backward, initially for BF16 12Q/1KV D256 on SM100/SM103. At 128K tokens with selection width 2052, compact IDs use 256.5 MiB in either direction, compared with 16 GiB for bitmap-forward IDs or 8 GiB for paired bitmap-backward IDs. The indexed path also avoids the 4 GiB selection bitmap. These figures describe selection metadata, not total memory (#1774).
• Packed causal top-k backward is available through selection_mode="causal_topk", with explicit sequence boundaries and a bounded-workspace two-pass implementation. This mode has a stricter selection contract and requires the CUDA 13 CuTe DSL compiler (#1781, #1787).
• QSA attention and selection functions are consolidated under cudnn.ops.qsa_*. Indexed attention consumes caller-supplied selections; the separate selector APIs compute them.

Gated attention block: serving and packed training 🚀 🚀

The experimental Rubin SM107 D256 gated attention block gains paged-cache EXTEND and DECODE modes. These combine projection, optional QK normalization, RoPE, cache writes, attention, gating, and output projection through the block API. Weight loaders cover the documented Hugging Face and vLLM layouts (#1471, #1606, #1619).

• Qwen sparse attention serving: QsaSpec selects the index-list sparse attention path. Supported modes include caller-supplied lists and an integrated compression, scoring, and top-k indexer, including paged decode (#1639, #1789).
• Packed quantized execution: the unfused MXFP8 forward and per-tensor FP8 backward gain packed-sequence support. Packed half forward can fuse the output gate, and supported packed backward plans accept an externally computed delta (#1502, #1505, #1617, #1497).
• Fewer launches: MXFP8 backward fuses small operations and reuses block-scale dQ work. Serving paths add small-row projection kernels, fused preparation/cache writes, and programmatic dependent launch where supported (#1466, #1680, #1763).
• Compiled blocks can update quantization scales between steps through update_quant_scales. The fully fused MXFP8 projection also accepts MXFP4 weights (#1506, #1487).

SDPA: wider native forward support 🚀 🚀

Blackwell SM100/SM103

• Adds native FP16/BF16 Q/K576, V512 attention with dense queries and dense or paged KV, including absorbed-MLA cache layouts. It supports partial GQA packing, split-KV, independent K/V tables and layouts, masks, sinks, and optional Stats. Packed queries remain outside this new flavor (#1902).
• Adds native paged per-tensor FP8 D256 execution and physical-origin support for paged FP8 THD inputs. Paged FP8 attention can use sinks (#1887, #1895, #1874).
• Expands packed GQA prefill and D256 THD split coverage. Paged K and V can use independent physical layouts (#1468, #1533, #1697).

Rubin SM107

• Extends MXFP8 to packed D256 queries and paged K/V pools with sinks for supported dense D128/D256 queries (#1488, #1481).
• Adds native D64 per-tensor FP8 attention, D128 GQA packing, and D512 packed/split plans. Half-precision sinks also work with split-KV (#1878, #1830, #1863, #1858).
• MXFP8 and per-tensor FP8 forward rows are offered by default. The selected engine remains dependent on graph features and measured shape ranges (#1478, #1684).

SM120 and Ampere

• SM120 gains specialized MXFP8 prefill kernels, including D512, plus FP16/BF16 decode tiles for small packed query counts. Split-KV supports attention sinks, and GQA packing no longer requires the full head group to divide the query tile (#1599, #1636, #1768, #1731, #1740).
• SM80 forward serves native packed THD graphs. Its backward reads the current ragged offsets on the device (#1155, #1058).

Across the native THD paths, metadata setup is fused for small batches, tensor-map construction is parallelized, and setup overlaps kernel initialization. Context-parallel work lookup and paged scheduling use work prefixes and request lengths to reduce wasted work (#1831, #1851, #1886, #1696, #1709).

SDPA: backward and numerical fixes 🚀 🚀

• D128 backward on SM100/SM103: adds FROST dense and packed execution, with supported GQA/MQA paths. Dense BF16/FP16 MHA can request a deterministic, fixed-order recompute implementation with zero workspace; grouped and THD deterministic requests remain unsupported (#1688, #1783, #1875).
• More dense geometries: adds native D64 backward and left sliding windows for dense D64/D128 MHA, plus native dense D192×128 half-precision backward on SM100 (#1879, #1883).
• D256 addressing: supports batch-pitched backward and packed inputs with independent physical origins. Context-parallel backward reduces workspace and improves vector loads (#1690, #1872, #1629).
• Fixes negative attention-scale handling on supported SM100/SM107 and SM120 forward paths, masked leading KV tiles on Rubin, and D192 FP8 probability rounding (#1445, #1448, #1544, #1810).
• Fixes D512 shared-ring phase reuse, split-KV softmax barrier ownership, scheduler back-edges, and terminal dQ barrier reuse (#1521, #1541, #1547, #1698).

New: Rubin triangle operations 🚀 🚀

cudnn.triangle_attention and cudnn.triangle_multiplicative_update provide experimental forward operations with cuEquivariance 0.12.0-compatible argument signatures (#1607).

• Triangle attention supports BF16/FP16 activations, pair bias, key masks or key lengths, and head dimensions divisible by 8 up to 64. Rows with no live keys return zero output and negative-infinity LSE.
• Triangle multiplicative update supports BF16 pair representations with channel widths 128 or 256, both directions, and optional pair masks.
• Both target SM107 only and require the documented FROST opt-in. Backward is not implemented.

New: CuTe DSL normalization backend 🚀 🚀

SM100 gains forward and backward kernels for LayerNorm, RMSNorm, GroupNorm, InstanceNorm, and BatchNorm, with FP32, FP16, and BF16 I/O. They are available through graph engines and the direct cudnn.norm.norm_fprop / norm_bprop APIs (#1509).

GroupNorm, InstanceNorm, and BatchNorm have native channels-last paths that preserve layout through forward and backward. LayerNorm and RMSNorm reduce the last logical dimension and use the contiguous path for channels-last inputs.

Graph normalization APIs also accept optional RMSNorm scale, optional LayerNorm scale/bias, and optional AdaLayerNorm bias. RMSNorm and LayerNorm backward accept optional scale; LayerNorm validates matching statistics dimensions (#1542, #1548, #1555, #1552).

Sparse attention: DSA, BSA, and CSA ✨✨

• Deterministic DSA: adds deterministic indexer backward and standalone top-k, H32 indexer backward, and two-CTA H128 backward. A clustered reduction bounds workspace for long-context deterministic backward, including 1M-context workloads (#1641, #1637, #1723, #1666, #1770).
• DSA indexer and execution: adds MXFP4 scoring on SM100, native H96 target recomputation, and faster scale packing. Prepared DSA execution uses caller-owned outputs/workspace and preserves 64-bit addressing for long sequences (#1775, #1859, #1564, #1182, #1767).
• Sage FP8 BSA: adds native SM120 block-128 forward, SM90 block-64/block-128 forward, and SM100/SM103 block-128 forward. SM100 backward also supports padded KV blocks (#1163, #1752, #1785).
• BSA correctness: fixes SM90 shared-V lifetime, SM100 softmax stability for large logits, backward tail zero-fill, and FP32 computation of O·dO products (#1498, #1221, #1682, #1713).
• CSA: adds deterministic dAPE reduction and parallelizes it. Ratio-128 forward handles NaN chunks and negative-infinity positions explicitly (#1605, #1717, #1692).

GEMM, MoE, and JAX ✨✨

• Canonical JAX grouped operations expand across MXFP8, NVFP4/MXFP4, and BF16 wherever the underlying kernel supports the format. Coverage includes GLU, dGLU, weight gradients, quantization, SReLU/dSReLU, and Hadamard fusions. Packed FP4 travels as uint8 with two values per byte (#1399, #1535, #1600).
• Grouped BF16, dGLU, gradient quantization, weight-gradient, and Hadamard paths accept canonical layouts. BF16 grouped execution no longer reads offsets back to the host (#1422, #1601, #1602).
• FROST GEMM gains SM120-specific tile scoring and split-K selection. The SM120 family adds MoE COMBINE and mixed MXFP4-weight/MXFP8-activation execution; callers can also own scheduler-counter initialization (#1460, #1500, #1638, #1881).
• Fixes compiled-object cache reuse across CUDA devices in grouped GLU, Hadamard, and quantization. Also fixes the SM100 GLU/Hadamard/quant epilogue deadlock (#1808, #1813, #1904, #1901).

Graph API, serialization, and execution ✨✨

• FROST plan serialization: pygraph.serialize() records the selected Python engine, public knobs, graph IR, and optionally compiled kernels. Deserialization reuses kernels only when their recorded environment matches; otherwise it recompiles. enforce_precompiled=True rejects compilation. Python-plan blobs cannot be loaded by C++ Graph::deserialize (#1862).
• Binary serialization: serialize(as_bytes=True) returns bytes, and deserialize() accepts bytes. The default serialized return type remains list[int] (#1725).
• CUDA Graph controls: opt-in populate_cuda_graph_direct() / update_cuda_graph_direct() avoid an extra child graph for eligible plans. Applications that manage resource lifetimes can disable automatic retention with cuda_graph_retention=NONE; the default retains resources. Cleanup moves out of execution and into plan/graph destruction (#1140, #1495, #1676).
• Shared compilation: concurrent processes using the same compiled cache avoid duplicate compilation. SDPA plans also cache immutable function metadata (#1707, #1844).
• PyTorch integration: cudnn.torch.install_sdpa_overrides() returns a removable handle for the cuDNN SDPA overrides, supporting PyTorch's Python-provider switch (#1776).
• Unified SDPA compute precision is forwarded to the backend with cuDNN 9.29 header/runtime gates. Explicit indexed execution remains independent of the selected plan, and legacy autotuning keeps metadata aligned with reordered plans (#1746, #1786, #1806, #1701).

Compatibility and API changes ✨✨

• CuTe DSL minimum: the package now requires nvidia-cutlass-dsl[cu13]>=4.8.0. Windows ARM64 remains excluded from this dependency and apache-tvm-ffi, because those runtimes do not provide packages for that platform (#1820, #1837).
• Bindings: Python bindings move from pybind11 to nanobind. Builds require CMake 3.26 or newer. Optional CUDNN_FRONTEND_PYTHON_STABLE_ABI=ON targets GIL-enabled CPython 3.12+; free-threaded Python continues to require version-specific builds (#1523).
• Experimental aliases removed: explicit KDA/BSA operations move to cudnn.ops, and grouped SwiGLU aliases are removed. Differentiable JAX KDA/BSA entry points remain in cudnn.jax (#1727, #1597).

• Amax_S is deprecated. FROST FP8 plans do not produce this output; a materialized Amax_S requires a backend plan (#1684).
• Windows: adds CUDA 13 dynamic loading, correct runtime-major dispatch for changed CUDA entry points, and Windows ARM64 packaging fixes (#1499, #1566, #1714).

Documentation, benchmarks, and tests 📊

• Redesigned documentation navigation and indexed the lazy public API. FlashInfer-facing capability queries and runtime helpers are now public (#1587, #1663, #1592).
• Attention benchmarks now include the default frontend engine choice, a report of selection cost versus the fastest measured plan, context-parallel THD training, and serving-style paged KV with mixed cache lengths (#1848, #1856, #1867).
• Added QSA training benchmarks, normalization benchmarks, and Mamba-2 examples and measurements. Expanded CUDA Graph replay, deterministic execution, long-sequence, and multi-device regression coverage.
• cudnn-repro handles malformed tensor-dump lines and missing log files, and restores Python 3.9 import compatibility for that standalone tool (#1584, #1582).

Source

Release source: 56a5d0c1215e. Changes since the source tree shipped in v1.31.0.

Acknowledgements 🙏

Thanks to everyone who contributed to this release:

@0z5a, @Adnios, @alfredclwong, @Anerudhan, @Aneureka, @brandonfzhang, @Butterfingrz, @dingqingy-nv, @egilliam-nv, @hwanseoc, @icavan, @jhjpark, @JiaxuanBai, @jiayus-nvidia, @Levius-Fubuki, @liujane-dev, @luolie0577, @miyanyan, @msalasooNV, @NVIDIA-JerryChen, @rmhaskarnvidia, @RomanAnders90, @scottyokim, @Sujendra-23, @thynics, @tiffany940107, @tongke6, @vedaanta, @wplf, @xiufanl, @YangXu1990uiuc, @yanqinz2, @yanzhuo607, @yekerr, @yihuawei, @Zoe923.