flashinfer-ai/flashinfer v0.7.0

Release v0.7.0

on GitHub

What's Changed

• fix(sampling): reject unsafe multi-CTA top-k launches on low-SM GPUs by @bkryu in #4595
• feat(cake_kda): optimize small-BH recurrent-KDA prefill by @yyihuang in #4571
• test: enable the unified MoE fuzzer by default and prune legacy UTs by @feih-nv in #4475
• feat(gdn): add pooled state and state checkpointing and dtype support for feature parity by @guangyunh-nv in #4436
• feat(attention): add PrimTS Q64/KV256 and paged GQA block-sparse attention by @heyuhhh in #4474
• add backward-compatible aliases for bsa_attn_fwd and bsa_attn_blk64_fwd by @hsr1234563 in #4590
• fix(moe): restore SM12x MoE kernels broken by self-resolved helper in borrowed dense methods by @lucifer1004 in #4602
• Add @Anerudhan to CODEOWNERS for core review by @aleozlx in #4622
• fix(moe_ep): fix in_kernel_fc2_reduce livelock on zero-token launches (MXFP8 + NVFP4) by @mhoqueanik in #4531
• feat(moe_ep): SM100 BF16 CuTeDSL MegaMoE kernel by @mhoqueanik in #4386
• feat: CuTe DSL kernels for Rubin (SM107) and batched FP8 GEMM for Blackwell by @Vinnie6167 in #4526
• Add back TRTLLM Gen MoE split-K by @jiahanc in #4617
• feat(kda): add CuTe DSL recurrent prefill backend by @Observer007 in #4605
• feat(cake_backend): accelerate DeepSeek fused routing by @yyihuang in #4587
• Stop legacy nightly release publishing by @dierksen in #4623
• feat(cake_backend): add Blackwell Router GEMM by @yyihuang in #4594
• [feat]custom all reduce kernel by @qsang-nv in #4393
• fix(attention): handle extreme negative logits in masked softmax by @shoutoutuoadi325 in #4401
• feat: trtllm-gen FMHA features for sm107 (spcompress, fp16softmax) by @jimmyzho in #4596
• feat(moe_ep): SM100 W4A8 (MXFP8xMXFP4) CuTeDSL split kernel backend with MXFP8 packed dispatch by @mhoqueanik in #4529
• Add Qwen fused GDN decode step for sm120 by @nv-yunzheq in #4481
• feat: collect a union of TEST_PATH targets in unit CI by @kahyunnam in #4641
• fix(moe): correct unified fuzzer references by @feih-nv in #4639
• [perf] split TRT-LLM Gen routing kernels to reduce compile time by @jiahanc in #4635
• feat(moe): add SiTU-GLU activation to the CUTLASS fused-MoE backend by @xuanyu-mistral in #4460
• feat(cake_mamba): add Blackwell selective state update backend by @yyihuang in #4616
• ci: consolidate CUDA coverage and validate candidate images by @dierksen in #4469
• feat(kda): add SM120a CuTe DSL prefill backend by @JimpleMa in #4633
• feat(moe): standalone trtllm-gen routing op + decomposed tests/moe routing matrix by @aleozlx in #4082
• feat(cake_vsa): add optimized SM100/SM103 block-sparse attention (VSA) by @yyihuang in #4593
• feat(cake_kda): add paired recurrent training for SM100a and SM103a by @yyihuang in #4636
• perf(gemm): optimize CuTe DSL W4A16 dense GEMM by @zianglih in #4686
• fix(sm120): align MXFP8 plain tactic and FP8 moe stage policy by @CarstyYou in #4660
• feat(moe): align unified MoE do_finalize behavior with flat API by @feih-nv in #4614
• perf(gdn): reduce non-CP CuTeDSL launch overhead by @guangyunh-nv in #4699
• feat(cake_mamba): add Blackwell Mamba SSDCombined by @yyihuang in #4576
• feat: SM120 NVFP4 SVDQuant Gemm in CuteDSL by @rosenrodt in #4420
• ci: disable sccache for cu134 nvcc by @dierksen in #4682
• Add paged MQA logits (attn_scores) kernels for Blackwell SM100 by @dhiraj113 in #4365
• perf(sm120): optimize NVFP4 attention with N64 score-slot reuse by @tiffany940107 in #4502
• feat(cake_msa): add Blackwell minimax sparse attention source kernels by @yyihuang in #4355
• perf(cake_kda): further optimize recurrent-KDA prefill on Blackwell by @yyihuang in #4675
• feat(moe): allow B12xMoEWrapper to share pre-allocated workspaces by @lucifer1004 in #4603
• (perf) add fused_GDN_step support for Qwen 3.6 35B A3B on sm120 by @nv-yunzheq in #4708
• fix: SageAttention support block size doesn't divide sequence; support K-smoothing by @xrq-phys in #4654
• ci: coordinate CUDA dependency policy by @dierksen in #4711
• perf(msa): chunked top-k and in-kernel causal offsets for the SM12x indexer by @yichengj0 in #4030
• perf(cake_kda): add recurrence-piece persistent M128 prefill by @yyihuang in #4728
• feat(moe): add remaining CUTLASS unified MoE runners by @feih-nv in #4610
• fix: Correctly wire scale_qkvo to cute-dsl fmha backends by @xrq-phys in #4665
• fix(kda): fall back to Cake when CuTe DSL predates cutlass.experimental by @kahyunnam in #4667
• fix(ci): skip source-only CUDA config test in nightlies by @dierksen in #4750
• perf(activation): cap act_and_mul_kernel block size for ~17-19% speedup at large hidden dims by @yekerr in #4733
• fix(fmha): select CGA reduction for MLA H512 decode by @yihwang-nv in #4702
• feat(norm): fused Add+RMSNorm+1x128 fp8 block-quant producer by @NVShreyas in #4480
• chore(aot): exclude single prefill/decode modules from jit-cache prebuilds by @bkryu in #4760
• build(jit): reduce JIT-cache wheel size by @dierksen in #4757
• feat(moe): add unified activation parity by @feih-nv in #4613
• Support per-token NVFP4 ReLU2 MoE by @xuantengh in #4618
• feat: CUB DeviceBatchedTopK top-k backend with variable-length support by @NaderAlAwar in #4442
• feat(cake_fmha): add native Blackwell DCP speculative decode by @yyihuang in #4518
• fix(cute_dsl): consult the arch gate in the GEMM and GDN dispatchers by @Vinnie6167 in #4649
• feat(cake_activation): add fused Blackwell SwiGLU MXFP8 quantization by @yyihuang in #4638
• Adding var-window FMHA context support for PrimsTS. by @mingxu1067 in #4599
• Add bias support to cublast gemm backend and as fallback backend for cutedsl gemm backend by @jiahanc in #4772
• misc: multi-arch cubins (sm100, 103, 107) in a single artifact by @jimmyzho in #4648
• refactor(mla): isolate planned FA2, FA3, and CUTLASS backends by @saltyminty in #4697
• Enable CuTe DSL MLA benchmarks for low head counts by @lunarz-dev in #4656
• fix(gemm): never move the shared cuDNN GEMM workspace by @yanqinz2 in #4666
• feat(cake_kda): add recurrent training template dispatch by @yyihuang in #4726
• fix(norm): limit add RMSNorm FP4 launch config heuristics to SM100 and SM103 by @soodoshll in #4494
• perf(cake_vsa): refresh Blackwell block-sparse WS kernel by @yyihuang in #4804
• feat(decode): add prims-ts backend and is_causal to paged decode by @elwhyjay in #4739
• [Bugfix] Skip .item() readback for trtllm_ragged_attention_deepseek during CUDA graph capture (#4609) by @zhang-keliang in #4703
• feat(cake_comm): Add a Cake Blackwell all-gather matmul backend by @yyihuang in #4722
• feat(cake_bgmv): add deterministic prepared Blackwell MoE backend by @yyihuang in #4821
• feat(cake_backend): add grouped MXFP8 quantization by @yyihuang in #4820
• feat(attention): add variant_owns_mask for JIT variants that own the full mask by @lesj0610 in #4695
• feat(cake_kda): add native unbounded-softplus Kimi-Linear kernels by @yyihuang in #4535
• feat(cute_dsl): replace SM100 blk64 BSA C++/CUTLASS kernel with CuTe-DSL by @hsr1234563 in #4612
• feat(mla): support variable-Q decode with DCP in CuTeDSL by @Observer007 in #4719
• fix: support class method re-exports in API checker by @saltyminty in #4717
• perf(sm12x): optimize and unify static MoE by @EricChen02 in #4718
• fix(attention): make PrimTS paged block-sparse metadata live by @heyuhhh in #4664
• feat(MoE): FP8 MoE per-channel quant support by @raayandhar in #2809
• fix(ci): skip subprocess torch.compile tests when kernels are absent from jit-cache by @bkryu in #4783
• Unify MoE CuTe DSL dispatch to be dtype agnostic by @PetersonGuo in #4793
• fix(cute_dsl): make the optional-dependency guard independent of cutlass by @Vinnie6167 in #4753
• perf(attention): cut NVFP4 KV dequant instructions on targets without native FP4/FP8 conversion by @lesj0610 in #4746
• feat(moe): add cuTile fused MoE backend for BF16 and NVFP4 Unified MoE by @bkryu in #4646
• fix: skip SM107 low-latency GEMM cubins on Blackwell (#4773) by @kahyunnam in #4848
• feat(prims_ts): support no-padding MLA query rows by @PerkzZheng in #4830
• Fix TRTLLM MOE per-token NVFP4 TileN 192 and update cubins by @jiahanc in #4741
• fix(benchmark): align MLA metadata with backend contracts by @lunarz-dev in #4835
• Add PR labeler rules for op: linear attention and op: misc by @kahyunnam in #4852
• [bug fix] Issue 3700 sparse mla sm121 hang by @kahyunnam in #4732
• feat(jit): log cold nvcc module builds by @mgoin in #4844
• docs: fix 0.6.19 document check failures by @kangbintNV in #4871
• Rubin open issue fixes by @Vinnie6167 in #4787
• fix(attention): count NVFP4 scale-factor staging in the batch smem budget by @lesj0610 in #4767
• fix(attention): scope the NVFP4 split-KV workaround to the arch it was found on by @lesj0610 in #4747
• fix: honor sparse MLA KV page strides by @bvolpato in #4362
• feat(gemm): enable tinygemm2_sm100 on SM107 (Rubin) by @Vinnie6167 in #4849
• test(moe): generate activation matrix and add backend contract fuzzers by @feih-nv in #4805
• refactor(moe): bind TRTLLM launch state to each call by @feih-nv in #4797
• fix(cudnn): return base-2 LSE from prefill (+ paged head_dim_qk != head_dim_vo O offset) by @egilliam-nv in #4663
• feat: Add cute-dsl-prims backend for SM120 FP8 GQA prefill by @Tom-Zheng in #4714
• Enable top_k_varlen on Rubin (SM107) and add the DKG filtered-radix backend (radix_filter) by @dhiraj113 in #4621
• chore: update TRTLLM-Gen GEMM cubins by @bobboli in #4840
• feat(moe): add CuTe DSL SM90 BF16 MoE backend by @Aneureka in #4878
• fix(gemm,moe): restore bmm_fp8 auto fallback and drop an over-strict SM107 tactic guard by @Vinnie6167 in #4853
• docs(moe): regenerate the Unified MoE activation matrix (unblocks CI on main) by @Anerudhan in #4903
• feat(moe): enable tile_size=256 for Rubin MoE autotuning by @Vinnie6167 in #4851
• perf(sparse mla): Update sparse MLA half-Q-tile heuristic to apply to short queries only by @emascarenhas-nv in #4752
• fix(gdn): stream-scope WY staging buffers and prefill launch workspace by @elwhyjay in #4476
• fix(moe): disable PDL in the trtllm-gen fused-MoE path on SM107 (Rubin) by @Vinnie6167 in #4806
• feat(moe_ep): SM120 MXFP8 swap-AB CuTeDSL MegaMoE kernel by @mhoqueanik in #4387
• feat(moe_ep): make the nccl_ep split path CUDA-graph capturable by @Anerudhan in #4795
• feat(gemm): add router GEMM ops for Kimi-K2 / Kimi-K3 and bf16 output by @Vinnie6167 in #4630
• feat(prims_ts): accept SM107 (Rubin) in the attention-ts device gate by @Vinnie6167 in #4755
• feat(moe): expand cuTile fused MoE activation support by @bkryu in #4888
• fix(gemm): remove SM120 MXFP8 tile configs that exceed the SM12x shared-memory limit by @yichengj0 in #4013
• Add CuteDSL Low Latency GEMM tactic to mm_fp4, mm_fp8, mm_mxfp8, tgv_gemm_sm100 backends (SM100/SM103) by @PetersonGuo in #4685
• feat(cake_warp_decode): add SM103 NVFP4 warp-decode backend by @yyihuang in #4855
• feat(cake_comm): extend Blackwell all-gather matmul to TP8 by @yyihuang in #4822
• Summary improvement by @righthandabacus in #4895
• fix(kda): accept SM107 (Rubin) in the FlashKDA family guards by @Vinnie6167 in #4710
• ci: enable patched sccache for CUDA 13.4 by @dierksen in #4808
• Modify CODEOWNERS - Anik by @aleozlx in #4937
• feat(kda): output-only KDA decode + drop-in for vLLM Kimi K3 recurren… by @ameynaik-hub in #4709
• Refactor Sparse MLA SM120 by @lucifer1004 in #4802
• Add ReplaySSM autotuning by @askliar in #4545
• feat(attention): cuTile paged/ragged prefill + paged/MLA decode by @yifeis-nv in #4018
• docs: teach agents the PR guidelines in CLAUDE.md by @aleozlx in #4948
• [feat] Add warp level split k bf16 gemm by @jiahanc in #4908
• feat: add experimental API/backend policy, @flashinfer_experimental_api, and flashinfer.experimental namespace by @bkryu in #4880
• fix(ci): publish the test scope with GITHUB_TOKEN, not the bot PAT by @aleozlx in #4962
• test: skip MonoMoE on GPUs with fewer than 128 SMs by @cindyzxq in #4954
• feat(topk): self-sampling GVR V2 backend (SM100/103/107), oracle-tracking auto, and the V1 threshold repair by @dhiraj113 in #4811
• ci: provision cuTile compiler in CUDA 13 images by @saltyminty in #4939
• Add option to skip checking active rows in TRTLLM ragged prefill by @wzhao18 in #4931
• fix(ci): anchor bot command matching to line start, outside code fences by @aleozlx in #4956
• feat(autotuner): Autotuner v2 (autotune_v2) — managed persistence, deployment-matched measurement, runner contract by @YangXu1990uiuc in #3861
• feat(cake_concat_mla): add SM100/SM103 concat MLA K backend by @Kathryn-cat in #4860
• feat(cake_kda): add fp32 state & long context kda prefill kernels by @yyihuang in #4845
• feat(cake_sage): add SM120 block-sparse attention backend by @yyihuang in #4951
• feat(prims-ts): support proxy-compensated block-sparse attention by @heyuhhh in #4872
• feat(prims-ts): support decode GQA ratios up to 128 by @PerkzZheng in #4915
• feat(mla): expose DSv4 TRTLLM-GEN RopeQuant by @PerkzZheng in #4918
• fix(moe): address review findings for PR #4610 by @cindyzxq in #4914
• perf(attention): repack NVFP4 KV tiles to 16-bit on targets without a native E2M1 convert by @lesj0610 in #4769
• feat(sm120): add NVFP4 sparse MLA support for DeepSeek V4 Flash by @tiffany940107 in #4955
• Refactor SM120 grouped GEMM as SM12x GEMM by @CarstyYou in #4838
• feat: add NVFP4 quantize-append path for the MLA paged KV cache by @elwhyjay in #4676
• docs: resolve documentation check failures by @kangbintNV in #4927
• perf(moe): phase NVFP4 dispatch payload copies at EP4/EP8/EP16 by @samnordmann in #4067
• fix(docs): deploy stable documentation from release tags by @cindyzxq in #4945
• fix(kda): compile direct serving modules as C++20 by @migarci2 in #5017
• feat(cake_mega_moe): add optimized MXFP8 MegaMoE EP16 backend by @hzfan in #4970
• feat: Add valid_hidden_size and valid_intermediate_size params t… by @aleozlx in #2482
• Nccl extensions dependency update by @Anerudhan in #5016
• fix(trace): model the SM100/103 cute-dsl mm_bf16_fp4 prepared layout by @kahyunnam in #4620
• fix(mla): scope NoPE sparse_mla_top_k_lens requirement to non-sparse backends by @lucifer1004 in #4947
• fix(moe): unify the NVFP4 profiler workspace predicate and fix gated fc1 SF sizing by @yichengj0 in #4010
• feat(gdn): adopt the CuTe-DSL disk cache for TVM-FFI GDN kernels by @elwhyjay in #4912
• fix(gdn): support WY ucache kernels on SM121 by @hebo1221 in #4528
• feat(cake_gdn): add gdn prefill decode mtp Cake backend for SM100 and SM103 by @yyihuang in #4581
• perf(prims-ts): Optimize&refine PrimsTS block sparse attention by @heyuhhh in #5002
• feat(attention): opt-in LDTM.STAT for PrimTS context row_max by @kris1025 in #4839
• chore: remove unused benchmark test by @yufeiwu-nv in #5046
• feat(moe): wire unified UnpackedPrecomputed for TRTLLM BF16 and block-FP8 by @feih-nv in #4909
• feat(kda): support graph-safe CuTe DSL prefix checkpoints by @Observer007 in #5021
• feat(benchmarks): measure TRTLLM ragged row-check overhead by @yufeiwu-nv in #4998
• test(kda): restore the frozen module-ident consistency check by @kahyunnam in #5035
• feat(cake_gqa): Add an experimental SM110 GQA decode kernel by @yyihuang in #5052
• feat(moe): split QuantConfig into weight/activation/output format axes by @feih-nv in #4952
• fix(decode): reserve max(K+V, st.o) smem for FA2 FP8+GQA decode by @ir1ka in #5038
• fix: apply calibration scales to ragged FP8 KV prefill by @yuchenwang3 in #4984
• perf(attention): Refresh DSL_FMHA cubins by @xrq-phys in #4997
• fix(bench): quote benchmark CSV fields by @lunarz-dev in #5045
• fix(attention): enable SM121 FMHA v2 prefill by @aeichler-ac in #4661
• fix(mla): prevent intermittent hang in SM120 sparse-MLA swapAB prefill on DGX Spark by @bkryu in #5048
• refactor(attention): unify PrimTS plan/run and page-table contracts by @yuxianq in #4829
• Replayssm prefix materialize by @hnover-nv in #4815
• fix(bench): use float32 cos_sin_cache in apply_rope_with_cos_sin_cache by @aeichler-ac in #5066
• feat(cake_kernel): add MiniMax-H3 MXFP8 pre-attention for SM100a/SM103a by @yyihuang in #5060
• fix(jit): avoid invalidating bgmv_moe cache on source staging by @200lz in #4798
• ci: log installed Python packages before tests by @dierksen in #5062
• fix(jit): preserve source mtimes when staging rmsnorm_silu and monomoe sources by @bkryu in #5063
• fix(kda): make recurrent_kda backend="auto" decode fall back to CuTe DSL by @kahyunnam in #5037
• feat: Test runner summary prints cudnn backend version by @righthandabacus in #5068
• feat(jit-cache): split cache into architecture provider wheels by @dierksen in #4514
• bump version to 0.7.0 by @aleozlx in #4938
• fix(release): drop the stale diffusion_ops.minimax_h3 trace-registry entry by @aleozlx in #5216

New Contributors

• @shoutoutuoadi325 made their first contribution in #4401
• @JimpleMa made their first contribution in #4633
• @NaderAlAwar made their first contribution in #4442
• @mingxu1067 made their first contribution in #4599
• @PetersonGuo made their first contribution in #4793
• @bvolpato made their first contribution in #4362
• @emascarenhas-nv made their first contribution in #4752
• @Kathryn-cat made their first contribution in #4860
• @migarci2 made their first contribution in #5017
• @kris1025 made their first contribution in #4839
• @yufeiwu-nv made their first contribution in #5046
• @yuchenwang3 made their first contribution in #4984
• @aeichler-ac made their first contribution in #4661
• @hnover-nv made their first contribution in #4815
• @200lz made their first contribution in #4798

Full Changelog: v0.6.18rc10...v0.7.0