cuDNN Frontend v1.31.0 Release Notes

cuDNN Frontend v1.31.0 extends FROST attention from inference into more training workloads, expands paged and ragged execution, and makes repeated graph launches use shared native argument binding. This release also adds graph-attention APIs, JAX Kimi Delta Attention, and fixes for CUDA Graph lifetime, multi-GPU execution, and installed-wheel packaging.

New: native SDPA execution and ordered tensor bindings 🚀 🚀

v1.30.0 introduced prepared launches. v1.31.0 extends that work across the forward families: dense, packed THD, staged conversions, paged attention, split-KV, block-scaled outputs, and supported output gates now enter through a shared native binder. The graph, standalone, and staged paths use the same prepared-host contract; production Python forward framing and its fallback branches have been removed (#1431, #1432, #1434, #1438).

Ordered graph bindings (#1222) let a caller pass a tuple or list of tensors with tensor_uids, avoiding reconstruction of a UID dictionary on each invocation. Both the cuDNN backend and FROST accept the same form, and existing mapping calls remain supported. execute_plan_at_index accepts it too.
Native half-backward binding (#1433) unifies the shared half-precision backward entry points. Quantized backward and standalone compact-GQA backward retain their explicit binder ownership; this is not a claim that every backward path has migrated.
Current buffers remain current. Prepared launches reuse immutable geometry and binding metadata while reading the buffers supplied on each call. The migration includes physical 64-bit stride/Stats addressing, device-context handling, and CUDA Graph replay with changed inputs.

graph.execute(
    (q, k, v, out), workspace, handle=handle,
    tensor_uids=(Q.get_uid(), K.get_uid(), V.get_uid(), O.get_uid()),
)

An in-process compiled-plan memo now sits in front of the persistent cache (#1456). Repeated plans can reuse the compiled object for the same kernel, options, and target/live devices without tracing, compiling, or reading a disk artifact again. The persistent-cache format is unchanged; CUDNN_FRONTEND_COMPILED_CACHE_INPROCESS_MEMO=0 disables the process-local memo.

New: gated attention block backward 🚀 🚀

The experimental SM107 gated attention block introduced in v1.30.0 now supports training. A training forward produces an explicit saved record spanning projection, optional QK-RMSNorm, RoPE, SDPA, sigmoid gate, and output projection; GatedAttentionBlockBwd differentiates that record (#1302, #1324).

BF16 / FP16 backward computes input, projection-weight, and normalization-weight gradients. Optional gate-backward fusion supplies SDPA's delta directly; optional weight-gradient overlap places the two weight-gradient GEMMs on a side stream with explicit event dependencies (#1303, #1304, #1330, #1331).
Per-tensor FP8 backward uses the FP8 training record as written, with on-device gradient quantization, current/delayed scale modes, and FP8 projection GEMMs. MXFP8 backward uses block-scaled projection GEMMs and the Rubin MXFP8 SDPA backward row. Their GQA gradient folds accumulate FP32 partials before the final conversion (#1395, #1409, #1417, #1429).
FP4 projection weights can train. An MXFP4 QKVG projection, and an NVFP4 or MXFP4 output projection, are supported through the MXFP8 training record. Data-gradient GEMMs consume the caller's transposed FP4 artifacts; weight-gradient GEMMs continue to use 8-bit operands. The NVFP4 output-gradient conversion uses a two-level scale to avoid losing small gradients at the block-scale floor (#1465).
Packed sequences are supported for BF16/FP16 forward and backward. The per-tensor FP8 forward also writes a packed training record that can be differentiated through the documented BF16 backward route. Native FP8/MXFP8/FP4-weight backward remains dense-only (#1350, #1351).

The block remains an experimental API targeting Rubin SM107 with D256 attention. Supported quantization, saved-record, and packing combinations are checked explicitly.

SDPA: paged KV, split-KV, and engine selection 🚀 🚀

Paged attention expands across precisions and head dimensions, while split selection accounts for packed tokens, GQA packing, launch capacity, and bounded shape overrides.

SM100 per-tensor FP8 D128 paged KV (#1106), with prepared paged execution and independent K/V page-table handling added through the launch migration (#1231, #1254).
SM100/SM103 MXFP8 paged KV (#1214) supports dense queries for D128, D192×128, D256, and D512. K/V scale factors are page pools in F8_128x4 layout, and page sizes must be divisible by 128. Packed queries over those pools, block-scaled output over pools, and the SM107 MXFP8 paged path remain outside this support surface.
SM100 D512 FP16/BF16 paged KV (#1093) extends the large-head forward flavor to paged caches.
Rubin FP16/BF16 split and paged execution adds D128/D192 split-KV, packed MLA, paged half execution, and D256 packed split-KV. Single-CTA D128 paged prefill and packed-GQA D256 prefill also become available (#1359, #1366, #1443, #1447, #1450).
Decode and chunked-prefill selection gains prepared paged D128 THD split-KV, small-Q MLA splits, Blackwell D64 paged THD splits, and broader bounded D128 splits. Selection uses physical launch capacity and packed/ragged work rather than assuming a dense envelope (#1317, #1320, #1362, #1403, #1449, #1452, #1455, #1463).

Default engine availability changes too (#1421): the SM90 and SM107 half-forward rows, SM100 FP8 forward, and SM100 large-head backward are offered without the FROST opt-in flag. Placement remains shape-dependent; being offered does not mean that every graph selects FROST.

SDPA: Rubin backward, large heads, and softmax controls 🚀 🚀

Rubin D256 backward now covers BF16/FP16, per-tensor FP8, and MXFP8. The series adds per-batch KV lengths, externally supplied delta, and THD/ragged execution; quantized backward graph nodes gain max_total_seq_len_q and max_total_seq_len_kv for declared packed capacities (#1212, #1327, #1328, #1333, #1349, #1352, #1355).
A 2x2 datapath adds D256 backward and D512 forward/backward variants on the supported Blackwell/Rubin rows. D512 half-forward uses the new datapath by default where eligible; other configurations retain their existing path (#1323).
Hopper D512 prefill gains a specialized FP16/BF16 FROST kernel and prepared host launches (#1159, #1326).
Rubin softmax controls extend softmax_precision=HALF across supported FP8/MXFP8 forward flavors and add attn_scale_prefolded to supported graph operations. Prefolded scale requires the documented caller-side Q scaling and is not supported by every dtype/layout. These are numerical operation attributes, not interchangeable autotuning settings; unsupported engines decline them (#1424, #1436).

New: GAT and GATv2 Python APIs 🚀 🚀

cudnn.gnn.mha_gat and cudnn.gnn.mha_gat_v2 expose cuDNN graph attention with first-order PyTorch autograd (#1021).

They accept homogeneous or bipartite CSC graphs, optional edge features and attention outputs, caller-provided dropout masks, configurable activations, and concatenated or averaged heads. Deterministic backward uses reverse-CSC metadata; CscGraph.with_reverse_csc() can construct it. Gradient-precision controls allow supported FP16/BF16 operations to compute gradients in FP32.

These APIs require a loaded cuDNN backend 9.28 or newer, SM80 or newer, and a non-Windows platform. Entry points are resolved dynamically, so cuDNN 9.28 headers are not required when building the frontend. Higher-order gradients are not supported.

New: JAX Kimi Delta Attention 🚀 🚀

Experimental cudnn.jax.kimi_delta_attention, kimi_delta_attention_fwd, and kimi_delta_attention_bwd support jitted forward, explicit backward, and first-order reverse-mode differentiation through FROST's existing KDA computation (#975).

The bridge supports packed THD inputs, recurrent state, optional gate gradients, log/linear gates, INT32/INT64 sequence offsets, and coarse checkpoints. XLA owns outputs, the stream, and per-call workspace; the adapter does not require Torch. The initial scope is the supported SM100/SM103 FROST configurations, with other engine families and architectures deferred. JAX KDA and BSA also gain fixes for execution in multi-GPU processes (#1410, #1383).

Sparse attention: DSA, selection attention, and HCA 🚀 🚀

Native DSA training composition (#1227) adds DSA.sparse_attention with first-order PyTorch autograd and score recomputation aligned to the caller's original sparse-index slots. Safe metadata handling covers invalid indices, inactive suffixes, holes, and duplicates. This uses the native FE DSA wrappers for supported BF16 SM100 shapes; it does not require an external FlashMLA installation.
FP32 indexer head weights (#1311) are accepted with BF16 Q/K for supported SM90/SM100 forward scoring, preserving FP32 projection weights without an intermediate BF16 cast.
Standalone top-k tie policies (#1332, #1413) can prefer smaller or larger original column indices at an exact cutoff tie. This controls the selected set; it does not promise sorted output slots.
Block-128 causal selection attention (#804) supports GQA ratio 16 and token-level causal masking inside selected blocks. Fully masked future blocks contribute zero, allowing arbitrary selected-block order. Selection metadata is still supplied by the caller.
Aligned HCA backward (#1198) adds an opt-in API for GB300 and Rubin with CP4/8/16 and aligned local chunks. The supported contract is BF16 H128/D512, window/compression 128, and one sequence in the declared padded layout, with caller-owned scratch. The Triton dependency is 3.7 or newer on GB300 and 3.8 or newer on Rubin.

GEMM and MoE ✨✨

SM120 MoE multi-GEMM and GLU fusion (#1269) extends the FROST GEMM source family. Gather/scatter routing is available on supported SM100/SM120 paths (#1282), and explicit final-group sizes are supported (#1295).
Weighted COMBINE (#1380) on the FROST SM100 family fuses the second expert projection, routing-score multiplication, and accumulation into token outputs. It starts from zero on each execution. Contributions accumulate atomically in the declared output dtype, so this mode is not bitwise deterministic; native cuDNN and SM120 engines decline it.
Deterministic dGLU dprob (#1319, #1414) is opt-in for supported SM100 block-scaled grouped GEMM with dense or discrete weights. Fixed-order partial reductions replace the atomic accumulation for that output; unsupported combinations such as dbias and Rubin are rejected.
DSv4.1 BF16 expert backward gains fused checkpoint-activation recomputation in dGLU and corrected clamped dGeGLU derivative masks (#1187). Native N-major discrete BF16 weights can be reused for input gradients without materializing a transposed copy (#1199).
Grouped wrapper setup reuses metadata and routes grouped SwiGLU through the unified GLU implementation (#1233). NVFP4 QAT backward prefers the supported FROST path with automatic Triton fallback (#1003).

New: prepared NVFP4 conversion and fused Conv3D 🚀 🚀

Standalone NVFP4 conversion (#814) provides BF16-to-packed-E2M1/E4M3 conversion and dequantization under cudnn.ops on SM100 or newer. Reusable plans fix K while admitting runtime M; callers own the packed tensors, scales, and prepared outputs. The operation requires CuTe DSL 4.7.0 or newer, while the package-wide dependency floor remains unchanged.

Experimental Conv3D post-operations (#1329) target selected BF16 video-VAE inference shapes on SM100/SM103. Direct Python APIs combine convolution with bias, optional residual addition, normalization, SiLU, and padding; raw/causal convolution and standalone normalization variants are included. They require CUTLASS DSL 4.9 or newer and selected 3×3×3 unit-stride/dilation shapes. They are not automatically selected by cudnn.pygraph and have no backward implementation.

Graph API and Python execution ✨✨

Automatic handles are isolated by thread and CUDA device (#1357). Switching GPUs or concurrent callers no longer reuse one process-wide handle, and cleanup permits subsequent recreation.
Autotuning preserves plan identity (#1340): configurations, numerical/behavior notes, and filter state move with reordered plans, keeping reported engine/knob identities aligned with execution and replay.
Build-all honors the complete candidate list (#1342), preserves an eligible selection, and restores the corresponding native selection for workspace queries, serialization, and CUDA Graph APIs. Unrelated candidate declines do not invalidate a successfully built selected plan.
Failed autotuning reports failure (#1338) when no candidate was successfully timed, preserving the original plans for correction/retry. Workspace queries skip unbuilt plans, and failure diagnostics retain the relevant plan information (#1336, #1334).

Compatibility and numerical behavior ✨✨

Linear-attention Q/K L2 normalization now uses x * rsqrt(sum(x*x) + 1e-6) in FROST, replacing the previous clamped-norm convention. The SM90 CUDA KDA implementation uses the same additive epsilon. Results can change, particularly for small vectors (#1454, #1474).
An omitted graph attn_scale is 1.0, matching the cuDNN backend (#1426). An explicit zero is preserved rather than treated as a request for a default scale; kernels unable to compute it decline the request (#1430).
Direct half split-attention calls require caller workspace (#1374). SM100-family direct callers allocate scratch_workspace_bytes() and pass workspace; omission raises before launch. Public Torch wrappers already provide the required buffer.

Documentation 📖

Documented native execute-time shape-override boundaries and ragged-backward workspace capacity (#1148, #1358).
Added grouped GEMM modules to the API index and documentation for the new operation families (#1346).
Added Microsoft Azure Linux RPM installation guidance, including the published Azure Linux Arm CUDA repository (#1458, #1459).

Tooling, CI, Benchmarks, and Tests 📊

Reorganized Python tests by operation and backend (#1232).
Cross-architecture Rubin compile probes select an explicit supported target in a fresh process and disable the compiled cache, so an inherited SM80 target no longer breaks this source-level check. The probe skips only for the named missing DSL intrinsic; other compiler failures remain failures (#1494).
Added isolated per-run FROST routing directories, shared-GPU memory controls, committed smoke/nightly test tiers, and opt-in reference caching (#1457). These are harness changes; they do not change the library's computation.
Expanded explicit-stream, device-context, CUDA Graph lifetime, wide-address, normalization, and determinism regression coverage (#1149, #1287, #1321, #1322, #902, #1406).
Added sparse-attention benchmark configurations for DeepSeek-V4.1 and GLM-5.3, and refreshed benchmark artifacts (#1205, #1219, #1285).

Bug Fixes 🐛

SM90 KDA installed wheels now include the CUDA kernel sources read by NVRTC, fixing the FileNotFoundError that source-tree tests could miss. Package-data coverage checks guard the omission (#1473).
CUDA Graph lifetime retains required runtime-compiled execution-plan resources, the RmsNorm+SiLU engine, and host-copy sources for graphs that reference them (#1195, #1289, #1293, #1299).
Non-reentrant checkpointing works for linear-attention and causal-convolution autograd bridges by unpacking saved tensors once per backward invocation (#958).
Packed Stats addressing stays 64-bit through prepared host arguments and final stores, including D512 2x2 kernels (#1377, #1427).
SM80 attention scale handling fixes large positive scales and nonpositive scales (#1442, #1446).
SM100 FP8/MXFP8 forward compilation works when quack-kernels has been imported into the process. Quack's process-wide DSL preprocessor patch exposed a loop-variable scoping conflict in masked and split-KV builds, producing TYPE_UNSTABLE_JOIN. Distinct loop-local names fix the affected FP8 D256 and MXFP8 D256/D512 kernels without changing their API, knobs, or eligibility (#1483).
Workspace/device validation handles DLPack and CUDA Array Interface buffers on a different GPU from the caller's current device, restoring the caller's context; missing required graph workspace is rejected before execution (#1381, #1363). SM80 direct forward also handles a foreign current CUDA device (#1437).
DSA score recomputation fixes ring phases and scratch synchronization on SM100 (#1225).
Grouped GEMM correctness fixes SwiGLU TMA buffer lifetimes, WGrad descriptor visibility, dGLU dbias/amax initialization ordering and cache keys, and SM110 SwiGLU reduction/scale-output ordering (#1276, #1283, #1325, #1280).
Quantized SDPA output checks fix the SM107 D512 FP8 THD Amax row bound and add Amax assertions to the quantized harnesses (#1462).

Known issues

Negative effective attention scales remain a limitation of SM100/SM107 FROST forward. Negative attn_scale with masking can produce NaNs; per-tensor FP8 with a negative descale_q * descale_k can produce incorrect outputs or nonfinite Stats. This limitation applies to the FROST implementation, not all cuDNN backend plans.
SM107 BF16 packed gated-attention backward has an observed numerical validation failure for causal attention with sequence lengths (5, 0, 0), head dimension 256, and 8 query / 2 KV heads: the K RMSNorm weight gradient (dw_k_norm) exceeds the existing tolerance against the FP64 reference. The cause remains unresolved.
SM100 dGeGLU backward has observed d_row_tensor gradient mismatches against an FP64 autograd reference in grouped_gemm_dglu_wrapper_sm100(act_func="dgeglu"), for BF16 inputs and FP32 output at (E, M, N, K) = (2, 512, 128, 128). The affected (geglu_alpha, glu_clamp_min, glu_clamp_max, linear_offset) configurations are (1.702, -7, 7, 1) and (1, -10, 10, 0), with both vector_f32 settings. These failures recur with CuTe DSL 4.7.0 and the tested 4.8.0a0 prerelease; the cause remains unresolved.
CuTe DSL 4.8 prerelease validation also reports two FROST test assertions: a Rubin causal cross-compile produces four LDTM instructions where the test expects six, and a deliberately pre-fix negative-control kernel completes instead of reproducing an expected hang. The shipping fixed-form contention test passes. These assertions do not establish a shipping-kernel hang or numerical defect.

Source

Release source: 8ab41a3b8fd8. Changes since the source tree shipped in v1.30.0.

Acknowledgements 🙏

Thanks to everyone who contributed to this release:

@0z5a, @Adnios, @Anerudhan, @baudouinMNC, @brandonfzhang, @Butterfingrz, @egilliam-nv, @harshithkantamneni, @hwanseoc, @icavan, @jhjpark, @layalir, @msalasooNV, @pkuwangh, @pmdavies-nv, @RomanAnders90, @tingyu66, @vedaanta, @vthumbe1503, @YangXu1990uiuc, @yanqinz2, @yanzhuo607, @yihuawei, @ZhiyuLi-Nvidia, and @zianglih.

NVIDIA/cudnn-frontend v1.31.0 v1.31.0 release on GitHub