Highlights
This release ships Expert Parallelism (moe_ep) in the default install and extends the TRTLLM-GEN MoE stack for large-model serving. It broadens Blackwell model coverage — Gemma 4 / MiniMax-M3 MoE on consumer / DGX-Spark SM12x, DeepSeek-class MLA decode on B300, and context-parallel GDN on SM120 — and adds Video Sparse Attention plus CUDA-graph-safe FP8 all-reduce fusion for distributed inference.
Unified MoE API with Expert Parallelism (Experimental API to try!)
FlashInfer's unified MoE compute API is now wired into expert parallelism. A new flashinfer.moe_ep.MoEEpLayer runs one MoE layer split across ranks as dispatch → per-expert grouped GEMM → combine, over pluggable transport (NCCL-EP via nccl.ep/nccl4py, and NIXL-EP). The expert GEMM reuses the unified flashinfer.fused_moe.MoELayer as a pure per-expert grouped GEMM — routing lives in dispatch/combine. moe_ep is part of the default install (CUDA 13+), with vLLM-facing APIs and checkpoint-safe MoE all-to-all graph VAs. This EP API is also the integration surface for MegaMoE kernels in upcoming releases — try it out and let us know how it works for you.
Gemma 4 and MiniMax-M3 NVFP4 MoE now run on Blackwell SM12x
Gemma 4 and MiniMax-M3 NVFP4 MoE now run on Blackwell SM12x (consumer / DGX Spark), enabled by two new NVFP4 MoE activation functions — gelu_tanh and swiglu_oai — added to the SM12x MoE path.
TRTLLM-GEN MoE adds shared experts, DeepSeek-V4 routing, and low-latency FP8
TRTLLM-GEN MoE gains shared-expert fusion for FP8 paths, hash-based DeepSeek-V4 routing (hash_topk), and a fused FP8 blockwise megakernel that cuts latency for small batches (BS ≤ 8).
Video Sparse Attention and DeepSeek MLA decode on Blackwell B300
Video Sparse Attention (VSA) is now integrated into the block-sparse attention API, bringing efficient long-context attention for video diffusion models to FlashInfer. DeepSeek-class MLA decode extends onto Blackwell B300 (SM103) with cluster-aware CUTLASS split_kv, and CuTe-DSL GQA decode adds sliding-window and attention-sink masking for newer attention variants.
Linear attention: context-parallel GDN on SM120 and faster KDA decode
Gated delta-rule (GDN) linear attention adds SM120 context-parallel delta rule support, extending the CuTe-DSL GDN rewrite from 0.6.14 onto Blackwell SM120 prefill, alongside SM90 context-parallel prefill optimizations. KDA recurrent-decode kernels are also optimized for lower decode latency.
Distributed comm: FP8 fusion and CUDA-graph checkpoint restore
Dynamic per-token FP8 quantization fuses into allreduce + residual + RMSNorm (TRT-LLM and MNNVL backends, CUDA-graph safe). All-reduce workspaces support checkpoint_prepare / checkpoint_restore so physical backing can be released and remapped at stable VAs across serving checkpoints.
What's Changed
- [feat] Optimize TRTLLMGEN MoE routing by @jiahanc in #3751
- fix(bench): make fused_dit_layernorm and FP8 quant refcheck work correctly by @lunarz-dev in #3758
- test: cut unit-test CI wall time by @bkryu in #3601
- bump version to 0.6.14 by @aleozlx in #3752
- fix(mla): make cute-dsl decode eligibility gate impl-aware by @zcnrex in #3664
- docs: add missing arg docs for output_scalar_scale and activation by @kangbintNV in #3749
- feat: add sm120 cp delta rule by @guangyunh-nv in #3659
- fix(comm): clarify MoE finalize allreduce input shape by @samuellees in #3754
- fix(aot): exclude head_dim=512 FA2 modules from AOT jit-cache wheel by @bkryu in #3769
- feat: Fuse shared experts into trtllm_gen moe (fp8) by @nv-yunzheq in #2625
- feat: [MoE] remove redundent trtllm-gen moe tensor allocation by @IwakuraRein in #3721
- MoE-EP: wire unified MoE compute into NCCL-EP / NIXL-EP expert parallel (LL + HT) by @Anerudhan in #3686
- feat(moe): add gelu_tanh and swiglu_oai activations to b12x NVFP4 MoE for SM12x by @bkryu in #3744
- fix(moe): reject local tactic ID for gemm2 in flashinfer cutlass_fused_moe by @Aneureka in #3661
- feat: Expose disabling finalize fusion in TRTLLM cutlass moe backend by @djns99 in #3598
- fix: int64 Offsets in Grouped MXFP8 Quantization by @philipphack in #3743
- fix: limit rotating buffer to avoid OOM on large problem size by @guangyunh-nv in #3662
- perf: optimize sm90 cp delta rule by @guangyunh-nv in #3788
- feat(bench): adds variable-length top-k transform benchmark for decode/prefill by @elstehle in #3772
- [model] add example model: wan by @forrestl111 in #3087
- Add caller-owned workspace sizing helper by @lesj0610 in #3741
- ci: derive long-running/solo test buckets from pytest markers by @jimmyzho in #3770
- feat: add dynamic FP8 allreduce RMSNorm fusion by @samnordmann in #3421
- Enable cuBLASLt BF16 GEMM on SM80+ by @askliar in #3804
- Enable Ampere FA2 large-head attention by @lesj0610 in #3739
- feat(gemm): CuTeDSL "cute_ext" TGV backend as default for mm_bf16 / bmm_bf16 by @Yang-YiFan in #3281
- Feat/vllm moe ep api by @Anerudhan in #3813
- Feat/cutedsl moe swiglu oai and use activation type to pass in by @vitamin-chaos in #3737
- Fix 0/0 NaN in GLM52 routing renorm on sigmoid underflow by @xianbaoqian in #3803
- integrate VSA (Video Sparse Attention) into block-sparse API by @hsr1234563 in #3250
- Add NVFP4 slot-mapping paged KV append by @lesj0610 in #3740
- Add paged NVFP4 KV dequantization helper by @lesj0610 in #3748
- Fix CuteDSL paged decode non-causal boundary mask by @elwhyjay in #3717
- chore: manual add to codeowners for fmhav2 by @jimmyzho in #3816
- chore: add @qiching to core maintainers in CODEOWNERS by @qiching in #3851
- fix(fmha_v2): two small perf/correctness wins on Hopper FP8 prefill by @jdebache in #3299
- Optimize FP4 V-scale dequant for h512 FA2 prefill by @lesj0610 in #3822
- Add trtllm-gen decode log2 bmm1 scale override by @yihwang-nv in #3583
- fix(gdn): reject invalid sm120 tma lowering by @guangyunh-nv in #3844
- docs: add missing Parameters sections to workspace_size and docstring to is_gated_activation by @kangbintNV in #3827
- fix(attention): add LEFT_SLIDING_WINDOW drain loop to FP8 Hopper prefill (#3578) by @whycoming in #3682
- doc: Adding owners for moe_ep by @mhoqueanik in #3872
- perf(kda): optimize recurrent decode kernels by @djmmoss in #3766
- Fused FP8 blockwise MoE megakernel For BS <= 8 by @yugong333 in #3424
- feat: make moe_ep (EP) part of the default install; drop nccl submodule by @Anerudhan in #3821
- feat: offline tactics blocklist for GPU-aware autotuner pruning by @qiching in #2997
- Enables CUDA Graphs and Cold L2 cache in MLA Decode Autotuning by @Vinnie6167 in #3726
- fix-grouped-mm-cudnn-backend-mismatch by @yanqinz2 in #3797
- perf(gdn): optimize delta rule cp prefill for ultra low parallelism case by @guangyunh-nv in #3866
- fix(moe): return bf16 expert_weights from trtllm_fp4_block_scale_moe (#3595) by @waynehacking8 in #3644
- fix(wan example): timestep embedding sin/cos halves were swapped by @forrestl111 in #3819
- Add cold-L2 and CUDA graph to mm_bf16 API by @b8zhong in #3789
- [feat] Add cutedsl split K for MXFP8 dense gemm by @jiahanc in #3847
- feat(mnnvl): preserve MoE all-to-all graph VAs across checkpoint restore by @galletas1712 in #3727
- Add separate trtllm-gen KV counter buffer by @yihwang-nv in #3582
- docs: fix 8 new documentation issues from nightly doc-check (v0.6.14) by @kangbintNV in #3884
- Revert GDN prefill regression by @jhjpark in #3889
- feat(cute-dsl): sliding window and attention sinks for GQA decode (non-paged + paged) by @richardmcai in #3794
- fix(comm): make TRT-LLM reductions partial-warp safe by @samnordmann in #3880
- Fix AutoTuner memory leak for MoE kernels by @jdebache in #3687
- fix: make the cutlass MoE gemm profiler MXFP8-aware (autotune crash on MXFP8xMXFP8) by @waynehacking8 in #3614
- feat(moe): add DSv4 hash-based MoE routing (MOE-01-HASH) by @leonardHONG in #3561
- fix(mla): enable cute-dsl/auto on SM103 + cluster-aware cutlass split_kv by @yekerr in #3888
New Contributors
- @zcnrex made their first contribution in #3664
- @Aneureka made their first contribution in #3661
- @elstehle made their first contribution in #3772
- @forrestl111 made their first contribution in #3087
- @lesj0610 made their first contribution in #3741
- @Yang-YiFan made their first contribution in #3281
- @vitamin-chaos made their first contribution in #3737
- @xianbaoqian made their first contribution in #3803
- @hsr1234563 made their first contribution in #3250
- @whycoming made their first contribution in #3682
- @yugong333 made their first contribution in #3424
- @yekerr made their first contribution in #3888
Full Changelog: v0.6.14rc1...v0.6.15