flashinfer-ai/flashinfer v0.7.2

Release v0.7.2

on GitHub

These highlights are also published at flashinfer.ai/releases.

v0.7.2 Highlights

FlashInfer v0.7.2 brings MoELayer experts to Rubin through the cuDNN Frost backend, widens Rubin coverage across MLA, GEMM, linear attention and expert-parallel MoE, and adds fused-epilogue Prims-TS GEMMs on B200, B300 and Rubin. On Blackwell datacentre parts, DeepSeek-V4 NVFP4 sparse MLA reaches B200 and B300 and speculative-decode verification picks cuDNN automatically; expert-parallel MoE adds Hopper BF16 and SM120 W4A8 backends.

MoELayer experts run on Rubin

MoELayer on Rubin SM107 can run its experts on cuDNN Frost, a CuTe DSL grouped-GEMM backend for BF16, MXFP8, NVFP4 and MXFP8-activation/MXFP4-weight experts; the BF16 kernels also run on Blackwell SM120. Autotuning admits Frost alongside the existing backends, and CudnnFrostBf16Config, CudnnFrostMxfp8Config, CudnnFrostNvfp4Config and CudnnFrostMxfp8Mxfp4Config in flashinfer.fused_moe select it explicitly. Frost needs PyTorch 2.14 (CUDAGraph.retain_object) and, on SM107, a PTXAS or CuTe DSL assembler from CUDA 13.4 or newer.

• #6035 Yanqinz/move frost out from experiment
• #5762 Sm120 cudnn frost moe bf16

DeepSeek-V4 and Kimi K3 serving kernels run on Rubin

BatchMLAPagedAttentionWrapper(backend="auto") plans on B300 and Rubin, and DeepSeek-V4 sparse MLA runs on SM107. Rubin also runs the CuTe DSL GEMM family, mm_mxfp8(backend="cute-dsl") on an SM107-specific kernel, the CuTe packed Kimi Delta Attention decode fast path, the CuTe DSL Mamba2 SSDCombined prefill, hardware stochastic rounding in the Mamba selective state update, and silu_and_mul_mxfp8_quantize. SM107 MegaMoE through MoEEpLayer adds MXFP4 weights with MXFP8 activations, prequantized NVFP4 and MXFP8 checkpoints, and Kimi K3's SiTU activation, functionally validated on Rubin.

• #5984 Enable MLA auto-selection on SM103 and SM107
• #5391 feat(mla): enable TRTLLM-GEN DSv4 sparse MLA/RopeQuant on SM107 (Rubin)
• #5394 feat(gemm): enable TGV, CuTe DSL low-latency block-scaled and contiguous-grouped GEMMs on SM107 (Rubin)
• #6136 fix(gemm): use the SM107 CuTe-DSL kernel for mm_mxfp8(backend="cute-dsl") on Rubin
• #5396 feat(kda,mamba): enable CuTe packed KDA decode, cvt.rs stochastic rounding and Mamba2 SSD on SM107 (Rubin)
• #5267 perf(mamba): enable native SM107 stochastic rounding
• #5392 feat(quant): enable fused gated MXFP8 quantization (SwiGLU + MXFP8) on SM107 (Rubin)
• #5699 feat(moe_ep): support Rubin MXFP4 weights with MXFP8 activations
• #5662 feat(moe_ep): add Rubin SiTU, prequantized weights, and NVFP4 scaling

GEMMs fuse SwiGLU and QKV-norm-RoPE epilogues on B200, B300 and Rubin

flashinfer.prims_ts adds dense FP8 and NVFP4 GEMMs with fused SwiGLU (fp8_linear_swiglu, fp4_linear_swiglu) and fused QKV + QK-norm + RoPE (fp8_qkv_qknorm_rope, fp4_qkv_qknorm_rope) epilogues on B200, B300 and Rubin SM107. The NVFP4 SwiGLU and QKV variants accept an opt-in tmem_overlap=True tactic, which the autotuner also considers. The module needs the CuTe DSL experimental primitives shipped with nvidia-cutlass-dsl.

• #5758 feat(prims-ts): add FP8/NVFP4 GEMMs with SwiGLU and RoPE fusion
• #6255 feat(prims-ts): allow TMEM overlap for NVFP4 SwiGLU epilogues
• #6286 feat(prims-ts): allow TMEM overlap for NVFP4 qkv overlap

DeepSeek-V4 NVFP4 sparse MLA reaches B200 and B300 and adds prefill on RTX PRO 6000

DeepSeek-V4 serving with an NVFP4 KV cache runs sparse-MLA decode on B200 and B300 through trtllm_batch_decode_sparse_mla_dsv4(kv_cache_format="nvfp4", backend="cake"). On RTX PRO 6000 Blackwell and RTX 5090 the same entry point gains a backend="cake" decode and prefill path for the NVFP4 cache and a decode path for the DeepSeek-V4.1 mixed FP8/FP4 cache (SM120 and GB10/SM121), plus fused RoPE, NVFP4 quantize and paged-insert cache writers. These backends are opt-in.

• #6103 feat(cake_dsv4): NVFP4 DeepSeek-V4 sparse-MLA decode on SM100/SM103 (backend="cake")
• #5914 feat(cake_dsv4): SM120 DeepSeek-V4 NVFP4 sparse-MLA decode backend for RTX PRO 6000 / RTX 5090 (#5690)
• #5959 feat(cake_dsv4_sparse_mla): add SM120 DeepSeek-V4 NVFP4 sparse-MLA prefill backend for RTX PRO 6000 / RTX 5090
• #5983 feat(cake_dsv4_sparse_mla): SM120/SM121 DeepSeek-V4.1 mixed-cache sparse-MLA decode (528 B FP8 SWA + 288 B FP4 compressed, BF16 Q)
• #6033 feat(cake_dsv4): SM120 DeepSeek-V4 NVFP4 fused GPT-J RoPE + quantize + paged-insert writers

Speculative-decode verification runs faster on B200 and B300

BatchDecodeWithPagedKVCacheWrapper(backend="auto") on B200 and B300 routes multi-token decode rows (two to four query tokens per request, as in speculative or multi-token-prediction verification) of fp16/bf16 head-dim-128 grouped-query-attention models to cuDNN, which lowers verification decode latency. The route needs nvidia-cudnn-frontend[cutedsl]>=1.31.0. Single-token decode and runs with explicit determinism controls stay on FA2; FLASHINFER_DECODE_AUTO_CUDNN=0 pins FA2.

• #5508 feat(cudnn): backend="auto" takes cuDNN for the multi-token paged-decode rows its default FROST engine serves faster (cudnn-frontend 1.30+)

Expert-parallel MoE comes to H100 in BF16 and to SM120 in W4A8

H100 serves an expert-parallel MoE round entirely in BF16 through MoEEpLayer with Sm90_Bf16_Bf16_Bf16_PushCake_MegaMoeConfig: BF16 dispatch, fused SwiGLU, a deterministic combine over a pre-reduced BF16 wire, CUDA Graph capture and up to 32 ranks on one node. SM120 workstations get a W4A8 MegaMoE backend (Sm120_Mxfp4_Mxfp8_Bf16_Cutedsl_MegaMoeConfig) with CUDA Graph replay and uneven token counts across ranks. SplitConfig(comm=..., kernel=...) picks an NVLink one-sided, CAKE or NVLink two-sided communication backend, or the NCCL-EP or NIXL-EP transports, independently of the expert kernel.

• #5958 feat(cake_mega_moe): SM90 native BF16 MegaMoE expert compute through MoEEpLayer (sm90_bf16_bf16_bf16_push_cake)
• #6058 feat(cake_mega_moe): pre-reduced BF16 combine wire and fixed-cost levers for the SM90 MegaMoE backend
• #4632 [SM120] Add W4A8 CuTeDSL MegaMoE backend
• #5532 refactor(moe_ep): add a MoE-level communication interface for EP dispatch/combine

⚠️ Upgrade notices

• Install nvidia-cudnn-frontend>=1.31.0. The cuDNN linear-attention backend and the automatic cuDNN decode route need the cutedsl extra: pip install -U 'nvidia-cudnn-frontend[cutedsl]>=1.31.0'.
• After upgrading flashinfer-python, run flashinfer download-kernels to replace mismatched flashinfer-cubin and flashinfer-jit-cache wheels; --sm sm90a (repeatable) limits it to named architectures.
• Rename callers of cutlass_fused_moe_prepare_workspace to cake_fused_moe_prepare_workspace, replace prepare_bgmv_moe(..., backend="blackwell") with backend="cake", and import the cuDNN Frost MoE configs from flashinfer.fused_moe (the flashinfer.experimental path is a one-release shim).
• Sampling: batch-length seed and offset tensors apply one value per output row; callers passing distinct per-row values sample differently from before.

⚠️ Upgrade notices (Experimental)

• flashinfer.source_mega_moe (MegaMoEPlan, prepare_mega_moe) is removed; use flashinfer.mega_moe_v3.
• prepare_msa_nvfp4_sparse_decode drops workspace_buffer; msa_nvfp4_decode_workspace_size, workspace_layout and bind_decode_payload are removed. DeepGEMM-family prepare_* entry points drop their descriptor arguments.

What's Changed

• perf(cake_sparse_mla): round-6 DSv4 sparse-MLA Cake programs for SM100/SM103 (exact numerics; stacked on #5686) by @yyihuang in #5700
• refactor(cake_fused_moe): rename the Cake SiTU workspace preparation helper to cake_fused_moe_prepare_workspace by @yyihuang in #5710
• chore: bump nvidia-cudnn-frontend floor to >=1.30.0 by @Anerudhan in #5702
• test: guard tests/moe architecture requirements on the T4 lane by @taylor-yb-lee in #5465
• ci: treat a widened Literal annotation as a compatible public API change by @yyihuang in #5713
• feat(cake_mm_fp4): add per-token NVFP4 quantize + mm_fp4 per-token alpha backend for SM100 and SM103 by @yyihuang in #5697
• perf(cake_gemm): two-fragment TMEM drain and routing-aware route selection for the fused grouped FP8 gate_up + SwiGLU + quant program by @yyihuang in #5714
• perf(cake_sampling): round-5 kernels (software-pipelined sampled pass, ballot-gated filter, two-level cluster select, two-warp fused tail, ept-32 streams, per-architecture stage-2/3 forms) and chain-aware dispatch by @yyihuang in #5684
• fix(cute_dsl): wait on grid dependencies before the finalize-fusion kernel reads routing outputs by @yyihuang in #5682
• fix(cake_kda): repack the BF16 inverse fragment before block composition in the native prefill facade bodies by @yyihuang in #5659
• perf(cake_kimi_k3_latent_moe): round 9 -- release-scoped stream-K publication in both prefill GEMMs, symmetric two-way fix-up for the front's trailing-wave K halves (SM100a/SM103a) by @yyihuang in #5721
• fix(moe): initialize TRTLLM routing padding for TMA gathers by @byshiue in #5444
• feat(kda): kda for kda cutedsl impl by @Lyken17 in #5612
• feat(cake_dsa): add native 64-query-head DSA sparse-attention kernels for SM100/SM103 (#5657) by @yyihuang in #5704
• fix(moe): guard Humming MXFP4xFP8 runner behind ENABLE_FP4 by @x41lakazam in #5086
• fix(moe): skip unrouted (-1) expert ids in the SM12x b12x fused MoE by @SamMausberg in #5451
• feat(prims-ts): 8-bit Q/K/V sage attention for the PrimTS block sparse decode kernels by @heyuhhh in #5123
• ci: stage benchmarks for nightly package tests by @dierksen in #5698
• feat(mla): fall back to FA2 for unsupported SM90 auto requests by @saltyminty in #5712
• fix: log expected SM107 GEMM candidate rejections at debug level by @yufeiwu-nv in #5689
• Update Docker CI tags to 20260930-b1420f1 by @flashinfer-bot in #4673
• feat(cake_bgmv_moe): SM90/SM100/SM103 support and cake_bgmv_moe naming for the deterministic multi-LoRA BGMV MoE backend by @yyihuang in #5727
• feat(kda): kda-for-kda CAKE PTX impl by @Lyken17 in #5665
• perf(mla): deepen FP8 and FP16/BF16 decode load pipelines on SM107 by @zcnrex in #5703
• feat(cake_sampling): round-6 radix sampling bundle: coarse-sample stage-1 twins, row-span filter arm and small-k dispatch re-fit by @yyihuang in #5731
• feat(quant): enable fused gated MXFP8 quantization (SwiGLU + MXFP8) on SM107 (Rubin) by @Vinnie6167 in #5392
• perf(cake_mm_fp4): tactic rules and regenerated programs for the per-token NVFP4 route by @yyihuang in #5744
• perf(cake_kimi_k3_latent_moe): round 10 -- final-item staged TMA store for the 256-wide prefill tail GEMM instances (SM100a/SM103a) by @yyihuang in #5734
• feat(kda,mamba): enable CuTe packed KDA decode, cvt.rs stochastic rounding and Mamba2 SSD on SM107 (Rubin) by @Vinnie6167 in #5396
• perf(cake_deepgemm): faster two-CTA prologue, early TMA load start and descriptor prefetch on the mixed FP8xFP4 and native FP4 GEMM routes (SM100a, SM103a) by @yyihuang in #5739
• perf(cake_deepgemm): one bulk TMA store per 16-row slab on the FP32 K-grouped FP4 GEMM swapped route (SM100a, SM103a) by @yyihuang in #5754
• perf(cake_deepgemm): per-group leader pre-ranking on the M = 1 routing-gate cluster split-K route (SM100a, SM103a) by @yyihuang in #5766
• perf(cake_vision_backend): round 6 — split-KV attention with exact fixed-order merge, wide unit table + prefetch, per-row cluster trim, half-N GEMM tail twins (sm_100a/sm_103a) by @yyihuang in #5746
• perf(cake_gqa): serve BF16/page-16 and E4M3/page-64 (head_dim 128/256) DCP speculative decode on SM100/SM103 with the on-device load-balanced programs by @yyihuang in #5750
• feat(kda): kda for kda TIRx impl by @Lyken17 in #5613
• feat(cake_lm_head_loss): add chunked LM-head + fused loss training kernels for SM100/SM103 (#5680) by @yyihuang in #5752
• perf(cake_kda): read packed-row strided q/k/v in place in every prefill body by @yyihuang in #5846
• refactor(cake_comm): fold the MoE all-reduce union to one kernel per route, one launcher and a small route table by @yyihuang in #5848
• refactor(cake_moe_finalize_allreduce_fusion): drop the SM103 copy, fold the PDL/binding clones and replace the import manifest by @yyihuang in #5852
• refactor(cake_comm): drop the unreachable all-reduce-output kernels from the MoE all-reduce fusion bundle and build it through gen_jit_spec by @yyihuang in #5849
• perf(cake_fmha): round-4 balanced DCP E4M3 bodies and deterministic packed-MTP fold on SM100/SM103 by @yyihuang in #5855
• feat(cake_moe_grouped_gemm): ragged BF16 grouped GEMM for MoE experts (forward, dX, dW) on SM100/SM103/SM107 (#5678) by @yyihuang in #5751
• perf(cake_sampling): round 7 -- speculative-sample stage-1 twin, whole-CTA tail for cluster-8 streams, capability-keyed host policy by @yyihuang in #5847
• fix(tests): scale cake/cute SSDCombined parity atol by the output scale by @kahyunnam in #5756
• fix(tests): skip unsupported groupwise GEMM architectures by @saltyminty in #5759
• fix(aot): include CUTLASS fused MoE in SM110 providers by @janbernloehr in #5695
• Add native W4A16 prefetch and grouped-tile tactics on SM121 by @stecasta in #5648
• Add SM12x NVFP4 support to the CuTe DSL GEMM backend by @stecasta in #5100
• feat(cake_kimi_k3_attn_res): add Kimi-K3 AttnRes backend for SM100/SM103 by @yyihuang in #5853
• fix(autotuner): preserve profiling policy in saved configs by @lightseek-bot in #5705
• perf(cake_mm_fp4): half-M pair tile for the narrow-N 2-CTA rows of the per-token NVFP4 route (sm_100a, sm_103a) by @yyihuang in #5869
• refactor(cake_comm): retire the isolated MoE all-reduce fusion bundle; serve calls without the all-reduce output from the union by @yyihuang in #5856
• fix(gemm): keep narrow block-scaled N tiles inside the first 32-token SFB group by @yyihuang in #5890
• fix(tests): skip unsupported cuTile groupwise GEMM architectures by @saltyminty in #5883
• fix(cake_comm): retain replaced MoE all-reduce scratch tensors recorded by captured graphs by @yyihuang in #5894
• revert(cake_fused_moe): remove the Kimi-K3 W4A8 MXFP4 SiTU changes to the CuTe DSL MoE path (#5430) by @yyihuang in #5681
• fix(gemm): fall back from stale MXFP8 split-K tactics by @ZeldaHuang in #5269
• perf(gemm): cute-dsl bf16-nvfp4 dense gemm fix to include no-autotune fallback tactic by @jimmyzho in #4941
• update-frost-kernel-and-name by @yanqinz2 in #5891
• feat(moe_ep): add Rubin SiTU, prequantized weights, and NVFP4 scaling by @akaashrp in #5662
• perf: optimize CuTe DSL W4A4 and W4A16 unfused finalize by @zianglih in #5319
• feat(cake_dsv4): SM120 DeepSeek-V4 NVFP4 sparse-MLA decode backend for RTX PRO 6000 / RTX 5090 (#5690) by @yyihuang in #5914
• fix(gdn): keep CP prefill correct when max_seqlen is omitted by @lucifer1004 in #5661
• refactor(cake kda): one host shim and one argument plan for the SM103a PTX KDA programs by @yyihuang in #5903
• refactor(cake kda): render the generated prefill selectors from Python and compact the module table by @yyihuang in #5919
• refactor(cake kda decode): drop the seven never-selected instances and serve each family from one binding header by @yyihuang in #5901
• refactor(cake mega_moe): drop the smoke-shape programs, the SM-count override and per-call preparation work from both generated MegaMoE families by @yyihuang in #5904
• refactor(cake tinygemm2_sm100): fold the six STAGES x PDL sections into one templated kernel by @yyihuang in #5910
• feat(cake_kimi_k3_attn_res): regenerate the Kimi-K3 AttnRes programs after the schedule-selection round (SM100 / SM103) by @yyihuang in #5916
• perf(cake_mm_fp4): stream-K stripe tail for the partial last wave of the per-token NVFP4 2-CTA GEMM (SM100) by @yyihuang in #5944
• refactor(moe_ep): add a MoE-level communication interface for EP dispatch/combine by @bobboli in #5532
• fix(mla): prefer measured SM12x backends and guard cuTile CTA tiles by @saltyminty in #5896
• Canonical-layout NVFP4 W4A16 MoE in CuTe DSL on SM12x by @stecasta in #5735
• feat(moe): refresh the SM107 CuTe DSL gather+activation MoE kernel from TensorRT-LLM (SiTU, Relu2) by @Vinnie6167 in #5462
• fix(sampling): honor per-row seed and offset tensors by @wdykas in #5745
• fix(gdn): point the gated_delta_rule_mtp deprecation marker at 0.8.0 by @kahyunnam in #5306
• feat(mla): enable TRTLLM-GEN DSv4 sparse MLA/RopeQuant on SM107 (Rubin) by @Vinnie6167 in #5391
• perf(cake_sampling): round 8 -- slab-tail twins of the sample builds, pushed-coarse-sums twin of the cluster-8 default build, Hopper whole-CTA tail policy by @yyihuang in #5950
• feat(cake_dsv4_sparse_mla): add SM120 DeepSeek-V4 NVFP4 sparse-MLA prefill backend for RTX PRO 6000 / RTX 5090 by @yyihuang in #5959
• feat(attention): enable CuTe DSL VSA/BSA and Sage-quant tests on SM107 (Rubin) by @Vinnie6167 in #5397
• fix(cake_dsv4_sparse_mla): SM-count-aware split and head-tile planners for GB10 (48 SMs) by @yyihuang in #5949
• feat(cake_bgmv_moe): fall back to the portable BGMV MoE path for unsupported inputs and launch the portable kernels on the current stream by @yyihuang in #5809
• feat(cake_dsa): SM107 (Rubin) generated programs and strided / packed GLM-5.2 trainer layouts for the native 64-head DSA sparse-attention training kernels (#5675) by @yyihuang in #5737
• feat(moe_ep): support Rubin MXFP4 weights with MXFP8 activations by @akaashrp in #5699
• feat(cake_fused_qk_rope_append): fused BF16 QK RMSNorm + NeoX RoPE + paged KV append for SM90 / SM100 / SM103 by @yyihuang in #5952
• docs(cake_dsv4_sparse_mla): add Parameters / Returns sections to the documented SM120 sparse-MLA functions by @yyihuang in #5986
• feat(cake_bgmv_moe): generic-shape Cake BGMV MoE bundles (hidden % 8 == 0, rank 8/16/32/64) and a token->pair route index for arbitrary pair order by @yyihuang in #5850
• feat(cake_dsa_indexer): fused DSA indexer scoring + deterministic exact top-k for training (SM100/SM103/SM107) by @yyihuang in #5972
• perf(cake_dsa): single-pass key ranges for packed multi-segment rows, GB200-trace bench rows and fused varlen index glue for the native 64-head DSA sparse-attention kernels (#5657) by @yyihuang in #5865
• perf(kda): stop probing Cake for explicit T=1 cu_seqlens decode under "auto" by @kahyunnam in #5760
• feat(cake_mega_moe): SM90 native BF16 MegaMoE expert compute through MoEEpLayer (sm90_bf16_bf16_bf16_push_cake) by @yyihuang in #5958
• feat(cake_fused_qk_rope_append): Cake backend for the fused QK RMSNorm, RoPE, FP8 quantize and paged KV append entry (SM90, SM100) by @yyihuang in #5956
• perf(prims-ts): bf16 dense fmha context perf bringup by @harrisonzhy in #5475
• feat(cake_dsv4_sparse_mla): SM120/SM121 DeepSeek-V4.1 mixed-cache sparse-MLA decode (528 B FP8 SWA + 288 B FP4 compressed, BF16 Q) by @yyihuang in #5983
• feat(cake_kimi_k3_attn_res): small-M Kimi-K3 AttnRes programs for SM100 / SM103 by @yyihuang in #5994
• perf(cake_dsv4_sparse_mla): split-count-guarded batched merge for 8-split SM120 decode calls and GB10 head-tile refinements by @yyihuang in #6000
• refactor(cake_kda): one TF32 prefill kernel source per schedule with a shared host shim and compact registry by @yyihuang in #5909
• refactor(cake_kda): remove the exact-shape indexed and evolution prefill portfolios by @yyihuang in #5927
• refactor(cake_sampling): top-k/top-p radix sampling from probabilities on SM90/SM100/SM103/SM12x: drop the dead whole-CTA-tail blocks and the undispatched stage-2/3 forms, lean loader and host dispatch by @yyihuang in #5985
• refactor(cake_warp_decode): fused-MoE warp-decode expert GEMM on SM100/SM103: compile from a per-target source list, share the identical SM100/SM103 kernels, add a GPU parity test by @yyihuang in #5908
• refactor(cake_minimax_h3): MiniMax-H3 MXFP8 pre-attention (norm + quantize stages before attention) on SM100/SM103: one generated source per stage, one loader, thin launcher by @yyihuang in #5918
• refactor(cake_sage): block-sparse Sage attention (FP8 on SM100/SM103, SM120 variants): share the SM120 launcher, merge the SM100/SM103 sources, drop the manifests by @yyihuang in #5907
• refactor(cake_dsa): GLM-5.2 64-head DSA sparse-attention training kernels (forward and backward) on SM100/SM103/SM107: six architecture-neutral programs with by-value TMA descriptors and a thin launcher host by @yyihuang in #5957
• refactor(cake_dsv4_sparse_mla): DeepSeek-V4 sparse-MLA decode on SM100/SM103: one launcher-helper header, shared split-merge reducers, data-only loader, route rejections up front by @yyihuang in #5920
• refactor(cake_megamoe_topk_reduce): MegaMoE top-k expert-output reduce (BF16, hidden 4096, top_k 6) on SM100/SM103: one generated source, only the helpers the kernel calls by @yyihuang in #5911
• refactor(cake_router_gemm): DeepSeek-V3 MoE router GEMM (M 1..16, K 6144/7168) on SM100/SM103: standard JIT path, one binding, dense-view validation by @yyihuang in #5915
• refactor(cake_deepgemm_fp8_gemm): regenerate the FP8 1D1D GEMM as CUDA source with runtime shapes by @yyihuang in #5912
• refactor(cake_deepgemm_sparse_mqa): one metadata program per layout with arch-neutral logits and a registry loader by @yyihuang in #5947
• refactor(cake_deepgemm_mega_mhc): one generated source per program across SM100 and SM103 with a lean loader by @yyihuang in #5955
• refactor(cake_bgmv_moe): BGMV (batched gather matrix-vector) MoE expand/shrink kernels on SM90/SM100/SM103: host and binding cleanup by @yyihuang in #5989
• refactor(cake_deepseek_fused_routing): DeepSeek-V3 fused MoE expert routing (scores + bias top-k) on SM100/SM103: fold 36 routing kernels into two templates and check contiguity by @yyihuang in #5906
• refactor(cake_minimax_h3): MiniMax-H3 SM120 FP8/NVFP4 GEMMs, norm/quant kernels and FP8 packed-varlen attention: share the host helpers and norm/quant kernels, bound the caches by @yyihuang in #5905
• refactor(cake_minimax_h3): MiniMax-H3 packed-varlen BF16/NVFP4 self-attention (prefill and split-KV) on SM100/SM103: one Cake-generated source per program by @yyihuang in #5969
• refactor(cake_minimax_h3): MiniMax-H3 NVFP4 pre-attention and QKV quantize-and-pack kernels on SM100/SM103: one generated source per program by @yyihuang in #5938
• perf(cake_kda): serve explicit T=1 cu_seqlens decode on the frozen Cake route and memoize schedule selection by @yyihuang in #6005
• refactor(cake_deepgemm_mega_gate): take the token count at runtime with one program per schedule template by @yyihuang in #5961
• fix(prefill): stage cuDNN paged-prefill lengths without a host sync (v0.7.1 regression from #5350) by @aleozlx in #5976
• refactor(cake_deepgemm_kgroup_gemm): accept any group layout with one program per schedule and output mode by @yyihuang in #5925
• refactor(cake_deepgemm_mixed_gemm): one runtime-shape kernel family instead of per-shape programs by @yyihuang in #5941
• fix(moe_ep): avoid fixed-port rendezvous for standalone EP1 by @akaashrp in #6002
• perf(cake_fused_qk_rope_append): per-architecture / per-shape program selection for the fused BF16 QK RMSNorm + NeoX RoPE + paged KV append (SM90 / SM100 / SM103) by @yyihuang in #6008
• fix(gemm): fall back from stale narrow-tile / split-K tactics in cute-dsl mm_mxfp8 and mm_fp4 (follow-up to #5890) by @aleozlx in #5730
• fix(prefill): allow plan-once CUDA-graph use of the CUTLASS ragged backend (v0.7.1 regression from #5133) by @aleozlx in #5975
• fix(comm): construct FusedOp after the PDL grid dependency sync by @yya007 in #5081
• Sm120 cudnn frost moe bf16 by @yihuawei in #5762
• refactor(cake_moe_grouped_gemm): re-export with deduplicated sources and thin launchers by @yyihuang in #6017
• perf(cake_dsa): direct accumulation into the caller's packed fp32 rows, key-range pass cap, in-kernel row lengths and fused index preparation for the native 64-head DSA sparse-attention kernels (#5657) by @yyihuang in #6019
• perf(mla): prefetch split partials in the CuTe DSL MLA decode reducer by @zcnrex in #5709
• perf(cake_dsv4_sparse_mla): speed up the SM120 DeepSeek-V4 NVFP4 sparse-MLA prefill kernels with exact-equivalent schedule changes by @yyihuang in #6004
• refactor(cake_xqa): regenerate Thor (sm110) XQA as one templated source per program by @yyihuang in #5872
• refactor(cake_kimi_k3_vision_tower): regenerate as one source per program by @yyihuang in #5884
• refactor(cake_msa): regenerate the Blackwell MSA family as one source per program by @yyihuang in #5893
• refactor(cake_mla_varq_dcp_decode): regenerate as one source per schedule by @yyihuang in #5868
• refactor(cake_fp8_projection): for kimi_k3, share kernel sources across SM100 and SM103 by @yyihuang in #5888
• refactor(cake_mla): for kimi_k3, share kernel sources across SM100 and SM103 by @yyihuang in #5871
• refactor(cake_all_gather_matmul): share kernel sources across SM100 and SM103 by @yyihuang in #5881
• refactor(cake_nvfp4_attention): regenerate without dead code and accept strided inputs by @yyihuang in #5866
• refactor(cake_fused_norm_combine): regenerate as one source per variant by @yyihuang in #5874
• refactor(cake_mamba_ssd_combined): one source per kernel, one host launcher, table-driven loader (SM100 / SM103) by @yyihuang in #5924
• refactor(cake_mxfp8_megamoe_ep16): one fused source behind a return-protocol flag, admit SM100 (SM100 / SM103) by @yyihuang in #5922
• refactor(cake_nvfp4_svdquant): one source set, shape-rule routing, by-value tensor maps (SM100 / SM103) by @yyihuang in #5960
• refactor(cake_deepgemm_batched_gemm): one source per schedule, dispatch by schedule rule (SM100 / SM103) by @yyihuang in #5931
• refactor(cake_mm_fp4): regenerate the per-token NVFP4 programs as architecture-neutral sources (SM100 / SM103) by @yyihuang in #5971
• refactor(cake_selective_state_update): regenerate the checkpoint, MTP and FP32 STP programs as architecture-neutral sources, keep the shipped BF16 STP kernels (SM100 / SM103) by @yyihuang in #6003
• refactor(cake_vsa_sm90): drop the planner-dead split stages, regenerate the Hopper VSA kernels with shared headers and thin launchers (SM90) by @yyihuang in #5923
• refactor(cake_vsa): one device source per profile, shared host shim, JIT loader, plan-time FP16 metadata (SM100 / SM103) by @yyihuang in #5926
• refactor(cake_deepgemm_dense_mqa): one program set with runtime shapes and KV-range routes (SM100 / SM103) by @yyihuang in #5988
• refactor(cake_deepgemm_fp4_gemm): one runtime-shape program per schedule, no per-arch copies or per-shape catalog (SM100 / SM103) by @yyihuang in #5935
• refactor(cake_gdn): collapse constant-clone kernels; merge SM100/SM103 copies by @yyihuang in #5939
• refactor(cake_deepgemm): share one catalog loader; add random-input reference checks by @yyihuang in #5928
• refactor(cake_gdn): regenerate SM100/SM103 context-parallel prefill kernels as architecture-neutral programs by @yyihuang in #6025
• refactor(cake_bf16_fp4_gemm): regenerate SM100/SM103 BF16 x FP4 GEMM kernels from one generator by @yyihuang in #5990
• refactor(cake_kda): regenerate fused KDA decode kernels as generic stride-taking programs by @yyihuang in #6026
• refactor(cake_kda): drop import manifests and tools from the unbounded-softplus prefill kernels by @yyihuang in #5917
• refactor(cake_grouped_fp8_gemm): regenerate grouped FP8 GEMM and fused SiLU-quant kernels for SM100/SM103 by @yyihuang in #5993
• refactor(cake_bf16_bmm): share one source set across SM100/SM103; drop per-call device queries by @yyihuang in #5937
• refactor(cake_grouped_mxfp8_quantize): use one dtype-templated kernel; cache target resolution by @yyihuang in #5951
• feat(cake_lm_head_loss): fuse the last chunk's dW scale + cast into its GEMM epilogue; per-geometry GEMM programs (#5680) by @yyihuang in #5897
• refactor(cake_msa_nvfp4_decode): regenerate as one source per program by @yyihuang in #5875
• fix(cake_dsv4_sparse_mla): regenerate the SM120 decode kernels with the 32-bit E2M1 unpack for CUDA 12.9 ptxas by @yyihuang in #6027
• refactor(cake_nvfp4_mla_decode): share kernel sources across SM100 and SM103 by @yyihuang in #5867
• feat(cake_mega_moe): add the FP4 1024- and 4096-token model routes (SM100 / SM103) by @yyihuang in #6029
• fix(cake_selective_state_update): threshold the softplus of dt in the MTP-short and checkpoint programs, register the family in the AOT build (SM100 / SM103) by @yyihuang in #6030
• refactor(cake_mega_moe): retire source_mega_moe in favour of mega_moe_v3 by @yyihuang in #6031
• refactor(cake_mega_moe): fold mega_moe_v3 into one architecture-neutral program per schedule (SM100 / SM103) by @yyihuang in #6034
• perf(cake_sparse_mla): rounds 7-17 DSv4 sparse-MLA Cake programs for SM100/SM103 on the #5920 layout (cumulative) by @yyihuang in #6045
• feat(cake_gemm): masked grouped FP8 batch DeepGEMM MoE GEMM backend for SM100 / SM103 (backend="cake") by @yyihuang in #6044
• feat(cake_minimax_h3): MiniMax-H3 stage operators take the diffusion engine's native operands (strided AdaLN tables, int64 indices, RoPE cache + positions, strided q/k/v views) on SM100/SM103 by @yyihuang in #6039
• feat(cake_kimi_k3_attn_res): mid-M routing of the small-M Kimi-K3 AttnRes programs for SM100 / SM103 by @yyihuang in #6036
• feat(cake_selective_state_update): promote the headdim-64 / dstate-128 single-token decode row (Nemotron-H, granite-4.0-h) by @yyihuang in #6040
• feat(cake_dsv4): SM120 DeepSeek-V4 NVFP4 fused GPT-J RoPE + quantize + paged-insert writers by @yyihuang in #6033
• feat(moe_ep): add Rubin GenPhase and quantized combine options by @akaashrp in #5991
• refactor(cake_moe_finalize_allreduce_fusion): regenerate the 12 kernels as one templated source by @yyihuang in #5870
• feat(cake_backend): generated backend="cake" for nvfp4_sparse_mla_decode on SM100 / SM103 by @yyihuang in #5970
• feat(cake_nvfp4_attn): opt-in uncompensated (nodelta) NVFP4 varlen attention variant for MiniMax-H3 SM120 by @yyihuang in #5660
• perf(cake_fmha): round-5 balanced DCP speculative-decode bodies (BF16, E4M3 head_dim 256) on SM100/SM103 by @yyihuang in #5887
• refactor(cake_alphamoe_nvfp4): drop dead kernels and exports, share the device helper library, set the smem opt-in once by @yyihuang in #5930
• refactor(cake_kimi_k3_tp12_tail): one source per program, K23 keyed by capacity, runtime rank for the one-shot K1 by @yyihuang in #5940
• refactor(cake_kimi_k3_fused_router): regenerate as arch-neutral programs with shared headers by @yyihuang in #5962
• refactor(cake_balanced_gqa_decode): regenerate the export as arch-neutral programs by @yyihuang in #5954
• refactor(cake_ulysses): arch-neutral generated kernels, drop the 36 unused bindings and the examples-hosted test fixture by @yyihuang in #5953
• refactor(cake_gqa_decode): sm110, one registry, thin launcher bindings, no per-call host sync by @yyihuang in #5929
• refactor(cake_dcp_alltoall): one arch-neutral generated kernel per CP size, drop the unused bindings by @yyihuang in #5948
• refactor(cake_comm): regenerate the MoE all-reduce union (route decisions, unread parameters) by @yyihuang in #5945
• refactor(cake_mla): regenerate the Blackwell MLA decode programs as runtime-shape families by @yyihuang in #5968
• refactor(cake_fmha): split the registry and host helpers per component by @yyihuang in #5932
• perf(cake_kimi_k3_latent_moe): round 11 -- programs regenerated by the current generator (no kernel change; unused device helpers dropped), SM100a/SM103a by @yyihuang in #5851
• feat(cake_ssd_combined): one exact-scan family with FP16 delta, FP32 state, token-major out, any seqlen and varlen without initial states (SM100 / SM103) by @yyihuang in #6051
• perf(cake_dsa): per-target key-range-pass rule for the 64-head DSA sparse-attention training backward (SM100 / SM103 / SM107) and regenerated programs by @yyihuang in #6007
• feat(cake_rmsnorm_train): add fused BF16 RMSNorm training forward/backward (FP32 rstd, deterministic dw) for SM100/SM103/SM107 by @yyihuang in #5741
• refactor(kimi_k3_latent_moe): regenerate as arch-neutral programs with compile-line TP constants and a flat registry by @yyihuang in #5974
• perf(cake_bgmv_moe): round-5 BGMV MoE kernels: pair-grouped exact pipeline, programmatic dependent launch, per-arch selectors by @yyihuang in #6016
• refactor(cake_fmha): drop unreachable members and merge one-line per-arch copies by @yyihuang in #5933
• refactor(cake_dcp): one program per split-KV variant, batch size as a kernel argument by @yyihuang in #6043
• refactor(cake_fmha): remove unreachable request-ordered decode modules and benchmark-row routing by @yyihuang in #5934
• feat(cake_grouped_fp8_gemm): block-scaled UE8M0 family with native -1 padding and per-call operand rebinding (SM100a/SM103a) by @yyihuang in #6048
• feat(cake_fused_moe): add the Kimi-K3 W4A8 MXFP4 SiTU routed MoE as Cake-exported backends (cake, cake_cute) by @yyihuang in #5783
• feat(cake_minimax_h3): two-launch BF16 pre-attention stage consuming the engine-resident QKV weight (SM100a / SM103a) by @yyihuang in #6055
• feat(cake_all_gather_matmul): accept the engine's [N,K] weight view, any local M and any N % 256 == 0 by @yyihuang in #6054
• feat(cake_mega_moe): pre-reduced BF16 combine wire and fixed-cost levers for the SM90 MegaMoE backend by @yyihuang in #6058
• perf(cake_nvfp4_attn): warp-specialised attention form for the MiniMax-H3 SM120 NVFP4 varlen attention, selected per call on RTX PRO 6000 by @yyihuang in #6060
• Enable MLA auto-selection on SM103 and SM107 by @saltyminty in #5984
• perf(mamba): enable native SM107 stochastic rounding by @roigarnettNvidia in #5267
• Recover SM107 (Rubin) test coverage lost to stale architecture gates by @Vinnie6167 in #5308
• fix(prims-ts): support CuTe DSL 4.7 and 4.8 by @PerkzZheng in #5478
• fix(moe): skip unrouted (-1) expert ids in the SM12x fp8/mxfp8_mxfp4 grouped MoE route by @aleozlx in #5977
• perf(cake_sampling): round 9 -- leader-push exchange twins of the multi-CTA streams, graph-replay host fix by @yyihuang in #6065
• feat(cake_deepgemm): DeepSeek-V3.2 DSA indexer logits with the DeepGEMM signatures -- 64-head dense fp8_mqa_logits routes for any Q/K and paged FP8 fp8_paged_mqa_logits + metadata over the block table, admitted per arch (SM100a, SM103a) by @yyihuang in #6059
• feat(gemm): enable TGV, CuTe DSL low-latency block-scaled and contiguous-grouped GEMMs on SM107 (Rubin) by @Vinnie6167 in #5394
• docs: document environment variables and cake RMSNorm by @kangbintNV in #6041
• docs: document cutlass fused MoE backend parameter by @kangbintNV in #6042
• feat(prims-ts): add FP8/NVFP4 GEMMs with SwiGLU and RoPE fusion by @nekorobov in #5758
• fix(autotuner): re-rank tactics when the profiling policy changes by @aleozlx in #5715
• feat(cake_warp_decode): cover the sharded Qwen3.5-397B TP2/TP4 and MiniMax-M2 TP2/TP4 slices (SM100 / SM103) by @yyihuang in #6067
• chore: add @dierksen to catch-all CODEOWNERS by @dierksen in #6084
• fix(autotuner): reuse managed winners with matching profiling policy by @lucifer1004 in #5495
• ci: guard --full option registration to fix nightly PYTEST_ADDOPTS conflict by @alex-ameen in #5880
• limit SM architecture kernel selection for download-kernels by @Gregory-Pereira in #5876
• perf(cake_fused_moe): pre-shuffled scale-factor copy for the 16384-token FC1 of the Kimi-K3 NVFP4 SiTU experts (SM100/SM103) by @yyihuang in #5755
• perf(cake_fused_moe): fused quant+router and epilogue-triggered FC1 programs for the Kimi-K3 NVFP4 SiTU small rows (SM100/SM103) by @yyihuang in #5736
• perf(kda): one-wave dispatch and single-pass RMSNorm for multi-token KDA decode on SM10x by @djmmoss in #5902
• perf(cake_moe_finalize_allreduce_fusion): route-pair prefetch, DSM slot-ring RMSNorm reduction, occupancy-sized grid by @yyihuang in #6057
• fix(sampling): index per-request top_k / min_p by probs row when indices is set by @JulianZJN in #5340
• perf(cake_fused_moe): Kimi-K3 NVFP4 SiTU experts SM100 + SM103 claim8 FC2 2 CTAs/SM by @yyihuang in #6015
• feat(cake_dsv4): NVFP4 DeepSeek-V4 sparse-MLA decode on SM100/SM103 (backend="cake") by @yyihuang in #6103
• perf(cake_fused_moe): thread-block-cluster routing kernel for the 512- and 1024-token Kimi-K3 NVFP4 SiTU rows (SM100/SM103) by @yyihuang in #5895
• perf(cake_fp8_projection): Kimi-K3 KDA/MLA projection GEMM round 6 (L2 prefetch, weight multicast, TMA-store epilogue, cluster split-K, 192-wide / stream-K GEMM, PDL), shared-source export (SM100a, SM103a) by @yyihuang in #5672
• feat(cake_dsv4_sparse_mla): variant-indexed generated family for the SM120 DSv4.1 mixed-cache sparse-MLA decode by @yyihuang in #6071
• perf(cake_all_gather_matmul): one native host sequence per route (SM100a, SM103a) by @yyihuang in #6099
• perf(cake_nvfp4_mla_decode): faster than the FP8 CuTe-DSL MLA decode on every shipped shape (SM100a, SM103a) by @yyihuang in #6107
• perf(cake_fused_moe): pre-shuffled scale-factor route with a register-only writer kernel for the 8192- and 16384-token Kimi-K3 NVFP4 SiTU rows (SM100/SM103) by @yyihuang in #5942
• perf(cake_sampling): round 10 -- CTA-local select twins, integer-tested whole-CTA tail, R200 one-wave re-pick (SM90, SM100a, SM103a, SM107a) by @yyihuang in #6105
• feat(cake_kimi_k3_fp8_projection): mid-M dispatch buckets, per-weight cached launcher, SM103 152-SM plan (round 7) by @yyihuang in #6096
• perf: reduce KDA prefill output-overlap validation overhead by @Anerudhan in #6073
• feat: Add SM100 cuteDSL ReplaySSM GDN MTP decode kernels by @Tom-Zheng in #5314
• Yanqinz/move frost out from experiment by @yanqinz2 in #6035
• fix(cake_sparse_mla): trtllm-gen sparse-validity semantics, workspace contract and FP8 probability scale for the SM100/SM103 DSv4 sparse-MLA Cake programs by @yyihuang in #6066
• fix(comm): size the allreduce fusion two-shot buffer by the element size by @yekerr in #6092
• fix(comm): order the two-shot allreduce barrier so a growing grid cannot pass it early by @yekerr in #6091
• feat(experimental): migrate b12x to experimental submodule by @lukealonso in #5767
• chore(artifacts): bump trtllm-gen BMM cubins for SM107 2x/4x MMA-K MoE kernels by @Vinnie6167 in #6090
• perf(cake_moe_finalize_allreduce_fusion): poll the owner segment at world size 8 too by @yyihuang in #6128
• perf(cake_nvfp4_attn): block-relative P codes and a selector upper bound for the MiniMax-H3 SM120 NVFP4 varlen attention by @yyihuang in #6068
• perf(cake_nvfp4_mla_decode): pin the decode kernel's launch bounds so its register budgets apply (SM100a 1.6-2 % faster) by @yyihuang in #6127
• refactor(cake_kimi_k3_situ): regenerate the SiTU MoE backend from its producer, drop dead bindings and sequences, fold per-arch copies by @yyihuang in #6135
• perf(cake_fp8_projection): Kimi-K3 KDA/MLA projection GEMM round 7 (FP8 token ring on the M=256 cluster split-K cell, parallel-issue DSM partial exchange), shared-source export (SM100a, SM103a) by @yyihuang in #6146
• perf(cake_deepgemm_mega_gate): 16-way split-K cluster reduction with a compensated token-parallel combine on the small-M routing-gate routes (SM100a, SM103a) by @yyihuang in #6129
• feat(cudnn): backend="auto" takes cuDNN for the multi-token paged-decode rows its default FROST engine serves faster (cudnn-frontend 1.30+) by @vedaanta in #5508
• fix(cake_kimi_k3_mla): bound the lazy E4M3 reference drop on the row-tile kernels (Inf/NaN rows with attention-sink keys); regenerate SM100 / SM103 by @yyihuang in #6126

New Contributors

• @byshiue made their first contribution in #5444
• @Lyken17 made their first contribution in #5612
• @SamMausberg made their first contribution in #5451
• @janbernloehr made their first contribution in #5695
• @lightseek-bot made their first contribution in #5705
• @ZeldaHuang made their first contribution in #5269
• @wdykas made their first contribution in #5745
• @yya007 made their first contribution in #5081
• @yihuawei made their first contribution in #5762
• @roigarnettNvidia made their first contribution in #5267
• @alex-ameen made their first contribution in #5880
• @Gregory-Pereira made their first contribution in #5876
• @JulianZJN made their first contribution in #5340

Full Changelog: v0.7.1rc5...v0.7.2