What's Changed
- feat(mega_gate): experimental SM120 CuTe DSL backend for prepare_mega_gate by @ruoqianguo in #8479
- fix(attn_scores): order the MQA logits kernels' TMEM dealloc after every math warp's tcgen05.ld by @bastianhagedorn in #8855
- fix: rename the alphamoe router local small, a Windows SDK macro by @ranx2026 in #9303
- fix(jit): embed cubins as COFF objects on Windows by @yyihuang in #9415
- fix(attention): reject ignored sinks in standard FA paged adapters by @YangXu1990uiuc in #9410
- test(gdn): bound state_dtype prefill output and state errors below the signal by @kahyunnam in #8405
- bump version to 0.7.5 by @kahyunnam in #9418
- test: fix RMSNorm cache warmup and split EP CUDA guard by @akaashrp in #9412
- fix(bgmv_moe): restore BGMVMoEBlackwellPlan and backend="blackwell" as deprecated aliases by @yyihuang in #9388
- fix(fused_topk_deepseek): exp-form sigmoid in the default and Cake routing kernels by @yyihuang in #9393
- fix(cake_grouped_mxfp8_quantize): keep nonzero blocks whose E8M0 scale byte is 0 by @yyihuang in #9416
- fix(cake_kda): keep non-finite V of one packed sequence out of the others in KDA prefill by @yyihuang in #9407
- perf(cake_msa): blackwell_msa BF16-Q E4M3-KV sparse prefill visits only the selected KV blocks by @yyihuang in #9394
- feat(cake_dsv4): NVFP4 sparse-MLA prefill for SM100/SM103 on a packed NVFP4 query by @yzh119 in #6851
- fix(cake_dsv4): FP8 route refuses widths over 1152; faster bodies for low-head and SGLang decode rows by @yzh119 in #9049
- fix(cake_mega_moe): address each warp's own TMEM lane quarter in the BF16 rank-major FC1/FC2 epilogues by @yzh119 in #6890
- perf(cake_stepfun_moe): resident-grid routed FC1 / FC2 programs for the StepFun fused MoE (sm_100a, sm_103a) by @yzh119 in #7203
- Add batch-invariant BF16 MegaMoE for Blackwell and Rubin by @yanqinz2 in #9417
- feat(moe): support routing weights before FC2 for NVFP4 by @amitz-nv in #9260
- bench(moe_ep): cover MegaMoE variants and input policies on SM107 by @akaashrp in #9339
- feat(batch_invariant):
sampling_from_probs/sampling_from_logitsunderbatch_invariant=Trueby @bkryu in #9365 - fix(cake_gdn): GDN prefill accepts element-aligned initial/output state and checkpoints by @yyihuang in #9434
- fix(ci): prefetch metadata before parallel provider spec lint by @YangXu1990uiuc in #9441
- feat(attention): support explicit cuDNN paged sliding windows by @YangXu1990uiuc in #9411
- Enable workload-bounded H100 D256 paged cuDNN auto dispatch by @YangXu1990uiuc in #9451
- perf(gemm): let the mm_mxfp8 autotuner rank CuTe-DSL and cuDNN on SM107 by @zcnrex in #9020
- Enable FP16 in the qualified H100 paged attention auto domain by @YangXu1990uiuc in #9453
- test(attention): update paged cuDNN capability checks by @akaashrp in #9459
- fix(testing,autotuner): flush L2 by reading so cold-L2 timing excludes dirty writeback by @vadiklyutiy in #9374
- feat(trace_apply): detect in-place rewrites of setup inputs, re-prepare or refuse by @akshaver in #9408
- perf(attention): bound Hopper ragged cuDNN batch declarations by @YangXu1990uiuc in #9469
- fix(gemm): raise on CUTLASS failure in SM120 groupwise GEMM launchers by @sunboaooo-blip in #7421
- fix: respect speculative masks in decode benchmark by @saltyminty in #9296
- test(batch_invariant): the sampling draw suite's precompile workers inherit the compiler environment by @bkryu in #9430
- Yanqinz/megamoe bi variants autotune by @yanqinz2 in #9454
- feat(attention): admit H100 noncausal paged prefill to cuDNN by @YangXu1990uiuc in #9473
- perf(cake_sage_block_sparse_attention): faster FP8 Sage block-sparse attention on SM100/SM103, combined quantize+attention API, auto backend default by @yyihuang in #9438
- perf(cake_sage_block_sparse_attention): TMA-ring Sage-FP8 quantizer for small rows on SM100/SM103 by @yyihuang in #9500
- feat(mla): fuse FP8 SM120 sparse prefill by @guangyunh-nv in #8747
- fix(quantization): zero-row launches return early and the launches are checked (fp4_quantize, mxfp8_quantize) by @feih-nv in #9377
- test: cover empty ranks in NVLink two-sided communication by @cindyzxq in #9491
- test(attention): restore cuDNN FP8 replay and independent numerics coverage by @YangXu1990uiuc in #9516
- [Perf] Optimize SM90 DOTS3 MLA sliding-window decode by @DylanChen-NV in #9511
- test: cover cuDNN empty-request prefill across graph replay by @YangXu1990uiuc in #9519
- test(attention): cover cuDNN paged auto boundaries and replay by @YangXu1990uiuc in #9520
- perf(prims_ts): optimize NVFP4 decode conversion and scheduling by @PerkzZheng in #9315
- test: compare both cuDNN packed-query layouts to reference by @YangXu1990uiuc in #9522
- test: cover cuDNN D256 paged auto boundaries and replay by @YangXu1990uiuc in #9523
- feat(moe): optimize SM12x gather and add low-parallelism SplitK FC1 by @CarstyYou in #9312
- feat(gemm): fuse gated activation into masked grouped GEMM epilogue by @garry569 in #7850
- test: cut full-suite pytest collection time by @dierksen in #9424
- feat(batch_invariant): batch-invariant fp8 per-tensor unified
MoELayer(BatchInvariantFp8PerTensorConfig) by @bkryu in #9406 - fix(attention): skip the kv length copy fa2 paged prefill never reads by @lesj0610 in #9397
- feat(sparse): graduate qsa_sparse_paged_attention to a stable API by @lesj0610 in #9474
- fix(comm): Lamport clear size in 16-bit units shared by every writer; fence the Helix sender before refilling shared memory by @feih-nv in #9379
- fix(moe): four upstream fixes to the CUTLASS MoE / GEMM sources by @feih-nv in #9380
- fix(moe): guard the CuTe-DSL act-fusion SFC store at the N boundary; align offset Rubin pointers by @feih-nv in #9383
- perf(fused_moe): faster routing and finalize for precomputed top-k by @bobboli in #8495
- trtllm-gen: raise on cubin load failure, skip unloadable cubins on Windows by @ranx2026 in #6436
- fix(gemm): read the mm_bf16 CuTe-DSL warp split-K bias after the PDL wait by @almightychang in #7591
- docs(gemm): state that DeepGEMM grouped FP8 GEMMs need power-of-two scales on SM100 by @yekerr in #8889
- perf(moe): prims-TS epilogue optimizations by @Nyrio in #9404
- chore: add @jimmyzho to autotuner CODEOWNERS by @jimmyzho in #9539
- feat(comm): support MNNVL packed group FP8 fusion by @samnordmann in #3850
- fix(gemm): handle unaligned N in b12x FP4 and MXFP8 stores by @Donkeyzhenbang in #7182
- fix(gemm): order the TMEM hand-offs of the TGV BF16 GEMM by @leejnau in #9337
- perf(gemm): For autotuning cute-dsl mm_mxfp8 GEMMs move from 1 to 16 launches per sample by @emascarenhas-nv in #8836
- fix(quantization): route batched NVFP4 quantize on SM107 by total rows by @kahyunnam in #9471
- feat(prims-ts): NVFP4 KV cache for QToken-KvBlock-Sparse-Attention by @PerkzZheng in #9375
- fix(prims_ts): acquire at cluster scope before reducing the split-KV partials in the decode correction by @bastianhagedorn in #9224
- fix(triton): keep GEMM epilogues in FP32 until the output store by @nulllt2 in #9095
- feat(sm120_dsa_indexer): DeepSeek-V4.1 decode in the paged form: strided page caches and candidate blocks by @lucifer1004 in #8740
- fix(gemm): keep SegmentGEMM lengths as integers by @nulllt2 in #9097
- fix(kda): isolate non-finite V past the sequence end in small-bh prefill by @kahyunnam in #9286
- perf(moe): one persistent DeepSeek FP8 activation kernel for the trtllm-gen MoE by @dhiraj113 in #9329
- fix(gemm,moe): order the TMEM release after the epilogue's tcgen05.ld in the TGV, low-latency blockscaled and frost GEMMs by @bastianhagedorn in #8787
- feat(batch_invariant):
mm_mxfp8_mxfp4, the MXFP8-activation x MXFP4-weight GEMM by @bkryu in #9366 - refactor!: remove the CAKE training kernels from FlashInfer by @aleozlx in #9326
- feat(batch_invariant): the training-forward surface of the sampling classes --
log_softmax(out_dtype=),sampling_from_logits(temperature=, return_logprob=)by @bkryu in #9535 - perf(cake_sage_block_sparse_attention): double-buffered S and tail split for many-block rows on SM100/SM103 by @yyihuang in #9536
- test(moe_ep): expect retuning after workspace capacity changes by @akaashrp in #9540
- fix(gdn): isolate non-finite next-sequence rows in SM12x tail chunk by @kmlx in #9373
- perf(moe): optimize SM90 mixed-input CUTLASS MoE backend for TP-dominant workloads by @StudyingShao in #9493
- feat(moe_ep): SM120 MXFP4 x MXFP8 nosync MegaMoE kernel as a small-batch option of MoEEpLayer by @YijiaZhao in #8806
- perf(cake_gdp): static decode programs for 2, 3, 5, 6, 7 and 16 tokens by @yyihuang in #9613
- test: cover large-vocab chain correction across scan chunks by @seongyun1104 in #7572
- perf(batch_invariant): prefill placements of the batch-invariant attention class by @Vinnie6167 in #9301
- docs(gemm): correct the TGV epilogue's TMEM-load fence comment by @leejnau in #9545
- feat(moe_ep): refresh SM107 MegaMoE sources and adapter by @akaashrp in #9426
- perf(cake_sage_block_sparse_attention): overlapped tile epilogue for mid-sparsity rows and faster tail split on SM100/SM103 by @yyihuang in #9605
- fix: read PDL-produced inputs after griddepcontrol.wait in cluster top-k, mm_fp4 and mm_bf16_fp4 CuTe-DSL kernels by @almightychang in #7288
- fix(gemm): reject K not divisible by 128 for cute-dsl mm_bf16 at M <= 32 by @zw-g in #8033
- fix(gemm): reject K not divisible by 256 for trtllm gemm_fp8_nt_groupwise by @zw-g in #6881
- feat(sdsa): convert NVFP4 KV tokens to FP8 for token-sparse attention, once per key by @amitz-nv in #9527
- feat(gemm): quantize masked grouped GEMM output to NVFP4 in the epilogue by @garry569 in #7851
- fix(utils): keep _get_stream_cache_buf traceable under torch.compile(dynamic=True) by @nvpohanh in #9354
- fix(autotuner): retune when cached winners lack runner coverage by @losewayy in #9144
- Port sharding runner process control to native Windows by @dierksen in #9466
- ci: seed the PR AOT sccache after merges to main by @dierksen in #9465
- ci: add a Windows arm64 wheel and JIT-cache build driver by @dierksen in #9462
- test: guard POSIX-only calls for native Windows by @dierksen in #9461
- fix(moe): size the CUTLASS MoE autotuner workspace for the FINALIZE fusion on SM >= 90 by @feih-nv in #9507
- fix(moe): index the trtllm-gen MoE permute and finalize kernels with int64 by @feih-nv in #9512
- fix(moe): Llama4 routing warp kernel lays out CTAs and padding from the local experts only by @feih-nv in #9513
- fix(moe): pad an empty EP rank's FP8 block-scale grouped GEMM input with one zero row by @feih-nv in #9514
- fix(common): CUDA_LAUNCH_BLOCKING is read on the first call; fmtstr ends its va_list copies by @feih-nv in #9521
- fix(gemm): order the stage release, output stores and loader exit of the warp split-K BF16 GEMM by @akhilg-nv in #8081
- feat(comm): destination-major DCP inputs and token masks by @dongha-yoon in #6483
- fix(autotuner): support stateless process groups by @itayalroy in #9472
- fix(gemm): narrow the masked grouped GEMM arm to SM100/SM103/SM107 by @xulunfan in #9252
- fix(gemm): normalize base_waves by CTA slots in mm_bf16_fp4 split-K gate by @melcheikh in #7746
- fix(gemm, grouped_mm): keep BF16 x FP4 and MoE cuDNN graphs used by captured CUDA graphs alive across cache eviction by @yuy-blip in #8188
- test: accept the Windows device-assert text in test_replayssm_tree_verify by @ranx2026 in #9480
- fix(jit): find nvcc.exe for the cake_gdn cubin compile on Windows by @ranx2026 in #9481
- feat(prims-ts): add merge-compatible softmax statistics and MLA KV bounds by @yuxianq in #5489
- feat(mla): add FP8 split-KV decode and wave-aware mega dispatch by @guangyunh-nv in #9274
- refactor(infra): move infra scripts and tests into scripts/infra and tests/infra by @dierksen in #8993
- fix(prims_ts): exact row max for E4M3 V in fmha context by @harrisonzhy in #8549
- fix(cudnn): keep SM120 prefill kernel off explicit-length token-offset graphs by @kahyunnam in #9458
- fix(prims-ts): select weight cache policy per expert by @bobboli in #8496
- feat(gdn): run GDN prefill on SM110 (Jetson Thor) by @yeh-sudo in #7827
- feat(cake_gdn): opt-in exact sequence split for the Cake GDN prefill on SM103 by @yyihuang in #9628
- feat(cake_ssd_combined): explicit state_dtype, caller-owned final_states, no final-state storage without final states by @yyihuang in #9550
- feat: Windows arm64 build support with the MSVC host compiler by @ranx2026 in #8431
- docs: fix API contracts, Returns rendering and examples by @kangbintNV in #9358
- feat(moe): add locality domains for trtllm-gen NVFP4 MoE by @jimmyzho in #9538
- fix(ci): restore documentation checker imports after infra script move by @kangbintNV in #9639
- fix(gemm, norm, attention): acquire at cluster scope before reading peer-written DSMEM in the blockscaled split-K GEMMs and the cluster reductions by @bastianhagedorn in #8853
- feat(cake_mega_moe): add SM107 NVFP4 MegaMoE EP backend by @zxybazh in #9603
- fix(decode): reject unsupported CUDA-core split disabling by @Lewis-panda in #6458
- feat(moe-ep): opt-in multimem (NVLS) NVLink barrier for MegaMoE by @x41lakazam in #9389
- feat(moe): optional fc2_input_out for the per-token NvFP4 trtllm-gen MoE by @zhangyimi in #9232
- test(moe): the Llama4 warp routing path reports local expert indices by @feih-nv in #9638
- fix(ci): run @flashinfer-bot commands posted by the bot account itself by @aleozlx in #9650
- fix(moe): reject non-contiguous CUTLASS MoE operands by @jiahanc in #9640
- feat(batch_invariant):
chunk_gated_delta_rule(batch_invariant=True), the GDN classgdn.bf16.c64by @kahyunnam in #9405 - Add missing backends for unified MoE benchmark by @SzymonOzog in #9604
- test: isolate MegaMoE CPU tests from visible GPU architecture by @cindyzxq in #9655
- feat(gemm): offer Cake as the last autotuning candidate for bmm_bf16, svdquant and per-token mm_fp4 by @yyihuang in #9656
- perf(cake_gdp): retune five n = 3 static decode programs on SM90 and SM103 by @yyihuang in #9654
- fix(autotuner): keep previously loaded cache files when entering autotune(cache=...) by @Orien-Xu in #8766
- perf(cake_batch_deepgemm_fp8): locality-domain serving GEMM for dual-die SM107a by @yyihuang in #9660
- feat(batch_invariant): recurrent_kda(batch_invariant=True), the KDA class kda.bf16.c16 by @kahyunnam in #9294
- feat(cake_ssd_combined): read and write states in a caller state pool by slot by @yyihuang in #9644
- feat(cake_gdn): several sequences per call in the opt-in Cake GDN sequence split by @yyihuang in #9663
- perf(cake_sage_block_sparse_attention): overlapped tile epilogue for many-block rows and the SM103 tail split by @yyihuang in #9648
- perf(batch_invariant): Rubin placements of the batch-invariant attention class -- the decode launch grid, PDL and more by @bkryu in #9622
- feat(cake_gdn): opt-in Cake GDN sequence split on SM100 and SM107 by @yyihuang in #9669
- perf(gemm): swap-AB small-M tiles for the SM120 FP8 groupwise GEMM by @AndreasBis in #9464
- fix(decode): reject off-device metadata in fast_decode_plan by @aniruddh-krovvidi in #6711
- perf: reuse FP4 compile workers during autotuning by @righthandabacus in #9327
- feat(gemm): opt-in flag to keep the SM120 FP4 GEMM autotuner off stream-K tactics by @Theodore-Liu in #7819
- feat(gemm): add a bias to bmm_fp8's cuBLAS backend by @gkswns0531 in #8811
- perf(cake_ssd_combined): state-slot claim and wrap programs on SM107 by @yyihuang in #9679
- fix(cake_locality): keep the caller's thread state across devices, sync the freeing context, retry pools after a PyTorch memory-limit refusal by @yyihuang in #9673
- test(moe_ep): fix W8A16 guards and scalar reset expectations by @akaashrp in #9671
- perf(batch_invariant): cut the host launch overhead of the batch-invariant GEMM entries (bit-identical) by @bkryu in #9620
- fix(jit): match object paths to Ninja output names by @akaashrp in #9681
- fix(moe): Cake warp-decode MoE kernels drop out-of-range expert ids in-kernel by @yyihuang in #9682
- feat(batch_invariant): batch-invariant top-k / top-p / min-p sampling by @bkryu in #9619
- perf(cake_gdn): sm_107a locality-domain GDN decode for homed states (B>=48) by @yyihuang in #9706
- fix(comm): load malloc and free from ucrtbase on Windows by @ranx2026 in #9478
- test: accept the mangled sentinel kernel name in cake_profiler by @ranx2026 in #9479
- feat(moe_ep): expose prepared weights for expert migration by @akaashrp in #9621
- fix(cake_gdn): reject unaligned q/k/state in the nontranspose decode; check the twin runs in its race gate by @yyihuang in #9712
- feat(cake_gdp): FP16 state pool with stochastic rounding for the decode routes on SM100 and SM103 by @yyihuang in #9720
- fix(ci): resolve CUTLASS DSL setup from the checkout root by @cindyzxq in #9688
- fix(cake_gdn): reject unaligned inputs in the pretranspose decode by @yyihuang in #9725
- Optionally fuse padding in MXFP8 quantize kernel by @AdityaKane2001 in #9429
- fix(b12x): admit 64-wide V heads in BF16 decode traits by @rootsbymenda in #7180
- feat(prims_ts): VC-Attention QK-INT8 for context fmha by @harrisonzhy in #8551
- perf(batch_invariant): more Rubin placements of the batch-invariant attention class by @bkryu in #9726
- perf(prims-ts): QToken sparse group sizes 16 and 32 and a latency-model group policy for NVFP4 prefill by @PerkzZheng in #9659
- ci: add NVSkills CI request workflow for skills/ by @jwu1980 in #9698
- feat(batch_invariant): stateless call of the GDN class
gdn.bf16.c64(a trainer's forward pass) with its boundary states by @kahyunnam in #9677 - fix: narrow CuTe-DSL cache dependencies for named imports by @righthandabacus in #9738
- fix(prims-ts): fix VC-attention prepass: V gather, k-means initialization, restricted token grouping by @harrisonzhy in #9549
- fix: let PyTorch select DeepGEMM's C++ standard by @righthandabacus in #9744
- test: skip exhaustive search in SM107 memory accounting test by @righthandabacus in #9739
- Revert "fix(autotuner): serve lookups above the largest tuned bucket from it (#7626)" by @ji24077 in #9534
- fix(gemm): reject K not a multiple of 8 for the TGV and TinyGEMM bf16 backends by @arpera in #9610
- feat(attention): expand Blackwell paged auto for FP16 and full prefills by @YangXu1990uiuc in #9463
- build: require cuDNN Frontend 1.33.0 or later by @Anerudhan in #9672
- perf(cake_batch_deepgemm_fp8): full serving grid on 212-SM devices from expected_m 36 by @yyihuang in #9687
- feat(cake_gdn): opt-in Cake CP and "auto" backends for chunk_gated_delta_rule_paged (SM100/SM103/SM107) by @yyihuang in #9704
- perf(prims_ts): faster MXFP4 x MXFP8 MoE GEMMs and tuning by @bobboli in #8497
- perf(moe_ep): finalize the split layer's experts into the combine input by @bobboli in #8498
- feat(moe): MXFP8 locality-domain intermediate split + add on Rubin by @jupiterepoch in #9541
- refactor(prims-ts): derive MLA CP bounds from global sequence lengths by @yuxianq in #9641
- perf(cake_gdp): shared-memory noise programs for stochastic FP16 snapshot calls of n = 3 on SM100 and SM103 by @yyihuang in #9763
- refactor(prims-ts): hoist paged context helper imports by @yuxianq in #9446
- fix(prims-ts): keep persistent decode sink barriers in step on empty-KV requests by @yuxianq in #9693
- perf(batch_invariant): fewer and cheaper GEMM compiles: runtime schedule arguments, kernel-source cache keys, engine-less export by @bkryu in #9727
- perf(mla): build the Cake Blackwell MLA decode row plan on the device by @yyihuang in #9718
- fix(cake_batch_deepgemm_fp8): route no twin on a device whose memory runs below its full clock by @yyihuang in #9757
- fix(ci): prepare selected source for provider spec lint by @cindyzxq in #9753
- feat(cake_gdp): decode programs on SM107 (sm_107a) by @yyihuang in #9774
- fix(cudnn): accept the fourth linear-attention output from cuDNN Frontend 1.33 by @jimmyzho in #9765
- fix(autotuner): apply the tactics blocklist to cache-file entries by @Orien-Xu in #8768
- perf(mla): Cake MLA decode in backend="auto" and a split-KV route by @yyihuang in #9779
- feat(moe): routing tile query for routed FP4 MoE routing metadata by @wookjeHan in #9745
- perf(sampling): single-launch cluster top-p renorm for SM103 serving shapes by @nv-yunzheq in #9598
- feat(batch_invariant): stateless call of the KDA class
kda.bf16.c16(a trainer's forward pass) by @kahyunnam in #9676 - Track opt-in tactic pools in autotuning cache coverage by @leejnau in #9289
New Contributors
- @sunboaooo-blip made their first contribution in #7421
- @garry569 made their first contribution in #7850
- @Donkeyzhenbang made their first contribution in #7182
- @nulllt2 made their first contribution in #9095
- @kmlx made their first contribution in #9373
- @YijiaZhao made their first contribution in #8806
- @dongha-yoon made their first contribution in #6483
- @itayalroy made their first contribution in #9472
- @xulunfan made their first contribution in #9252
- @melcheikh made their first contribution in #7746
- @yeh-sudo made their first contribution in #7827
- @zxybazh made their first contribution in #9603
- @Lewis-panda made their first contribution in #6458
- @zhangyimi made their first contribution in #9232
- @SzymonOzog made their first contribution in #9604
- @Orien-Xu made their first contribution in #8766
- @AndreasBis made their first contribution in #9464
- @aniruddh-krovvidi made their first contribution in #6711
- @gkswns0531 made their first contribution in #8811
- @AdityaKane2001 made their first contribution in #9429
- @rootsbymenda made their first contribution in #7180
- @jupiterepoch made their first contribution in #9541
Full Changelog: v0.7.5rc4...v0.7.6rc1