github flashinfer-ai/flashinfer v0.6.16
Release v0.6.16
on GitHub

3 hours ago

What's Changed

  • feat(comm): preserve all-reduce graph VAs across checkpoint restore by @galletas1712 in #3745
  • fix(attention): clamp CTA_TILE_Q for fp8 h512 ragged prefill on 99KB-smem GPUs by @qsang-nv in #3863
  • bump version to 0.6.15 by @kahyunnam in #3900
  • fix(moe): fail loudly instead of IMA when no routing tier covers (numExperts, topK) by @jinzh-nvidia in #3840
  • Add set_autotune_process_group to synchronize tactic choice across ranks by @thanhhao98 in #3187
  • test(moe): align EP-offset tests and docs with global expert ids (gh #3547) by @waynehacking8 in #3591
  • docs: install flashinfer-cubin from flashinfer.ai index; drop cubin PyPI upload by @aleozlx in #3853
  • Tune SM90 FP8 (e4m3) warp-specialized prefill by @HongminTan in #3760
  • feat: E2E MoE LoRA integration with bgmv backend by @JyChang012 in #3746
  • perf(moe): support caller-provided combine output by @samnordmann in #3776
  • fix(attention): correct SM120 NVFP4 qk_correction layout, row-sum reduction, and lse by @waynehacking8 in #3838
  • feat(gdn): add output-only BF16-state WY MTP decode kernel by @ameynaik-hub in #3720
  • [Bugfix] Skip autotuning for small token counts on SM100 to prevent TMA crash by @InfoSage05 in #3837
  • [CuTe DSL][MoE] Optimize PDL startup and balance GEMM2 partial-tile stores by @dishengbin in #3756
  • fix (test): release CUDA cache before prefill compile subprocess by @cindyzxq in #3914
  • [fix] fix autotuner non tensor guard by @jiahanc in #3918
  • Add BF16 unified MoE conformance coverage by @feih-nv in #3867
  • Fix: graph-safe uniform multi-token decode on FA2 tensor-core path by @lucifer1004 in #3871
  • Add mm_fp8 into benchmark by @sychen52 in #3924
  • feat(attention): enable SM121 (GB10/DGX Spark) for NVFP4 attention by @bkryu in #3897
  • feat: MegaMoE Kernel integration in moe_ep by @mhoqueanik in #3852
  • [feat] New CuTe DSL JIT kernels for FMHA: Dense & MXFP8 & NVFP4 formats by @xrq-phys in #3857
  • fix(mla): allow native-width block tables for TRTLLM-GEN by @yihwang-nv in #3916
  • Add FP8 groupwise MoE GEMM entry (cute SM120 backend) by @CarstyYou in #3891
  • fix: autotuner memory leak follow up by @jdebache in #3912
  • Add per-token NVFP4 quantization to CuTe DSL MoE by @zianglih in #3645
  • test: unified GEMM/BMM fuzzer + convention auditor by @YangXu1990uiuc in #3539
  • perf(attention): reuse TRT-LLM-gen counter buffers by @saltyminty in #3949
  • perf(gemm): update mm_fp4 cute-dsl tactic heuristic and autotune over a top-N ranked list by @bkryu in #3948
  • fix (test): mock MoE EP arch validation in split unit tests by @cindyzxq in #3964
  • fix(cute_dsl): migrate off APIs removed in nvidia-cutlass-dsl 4.6 (keeps 4.5.x compat) by @YangXu1990uiuc in #3922
  • Fix cuDNN batch prefill test by @jhjpark in #3784
  • Update moe cubins to fix 2CTA hanging issue by @IwakuraRein in #3973
  • perf: optimize MXFP4xBF16 & INT4xFP8 and add MXFP4xFP8 CUTLASS MoE backend for SM90 by @StudyingShao in #3738
  • docs: add fused_moe LoRA delta APIs to rst by @kangbintNV in #3941
  • docs: cover remaining @flashinfer_api symbols missing from rst by @kangbintNV in #3942
  • feat(attn): Enable MiniMax Sparse Attention (MSA) for Consumer Blackwell GPUs (SM120/121) by @yichengj0 in #3655
  • feat(jit): JitSpec ABC + disk cache for JIT-compiled CuTe-DSL kernels by @bkryu in #3874
  • Add BF16 weight preparation and unified fuzzer coverage by @feih-nv in #3940
  • Yanqinz/autotuner tactic by @yanqinz2 in #3707
  • chore: fix cute-dsl moe test by @IwakuraRein in #3979
  • feat(moe_ep): restructure MegaMoE kernel sources into kernel_src + latest CuTeDSL MegaMoE kernels with improved performance by @mhoqueanik in #3980
  • feat(attention): accept token-unit batch offsets on the cudnn prefill path by @egilliam-nv in #3921
  • Disable TMA kernel for nvfp4 quantize by @kahyunnam in #3977
  • perf: vectorize AppendPagedKVMlaCache to 128-bit access by @aws-jiadingg in #3524
  • fix(autotuner): time tactics with %globaltimer under Confidential Computing by @elvischenv in #3870
  • feat: expose trtllm-gen block-sparse attention for decode by @PerkzZheng in #3955
  • feat(moe): wire in-kernel routing (FromLogits) into unified MoE API + routing axes in fuzzer by @feih-nv in #3892
  • docs: fix 3 documentation issues from nightly doc-check by @kangbintNV in #3911
  • fix(comm): restore MNNVL flags under inference mode by @galletas1712 in #3950
  • fix(docs): mock optional quack dependency by @kangbintNV in #4021
  • feat: enable fmha_v2 HMMA attention for SM120 standard shapes by @blake-snc in #3016
  • Autotuner mem leak follow up 2 by @jdebache in #3970
  • test(comm): add additional comm tests to multi-node CI by @nvamyt in #3968
  • Ameyn/gdn wy perf followup by @ameynaik-hub in #3908
  • fix(benchmark): don't fail non-trtllm mm_fp4 backends on unaligned n by @sychen52 in #4033
  • perf(moe): optimize one-sided EP4 NVFP4 dispatch by @samnordmann in #3846
  • perf(jit): drop dead SM12x gencode from SM10x-serving modules by @yichengj0 in #3947
  • fix(cudnn): include attn scale in the prefill/decode graph-cache keys by @YangXu1990uiuc in #4014
  • feat: SM100 CUTLASS NVFP4 SVDQuant fused GEMM (mm_nvfp4_svdquant) by @jingyu-ml in #3858
  • fix(moe): restore legacy fused_moe.core imports for interleave helpers by @StudyingShao in #4025
  • feat(moe): add deterministic CuTe DSL NVFP4 finalize mode by @zianglih in #3976
  • fix(moe): allocate trtllm-gen expert_weights as bf16 in all launchers by @aleozlx in #3925
  • feat(quantization): support configurable scale-factor layouts for MXFP4 by @huangzhilin-hzl in #3927
  • test(comm): make checkpoint worker importable by @galletas1712 in #4034
  • feat(moe): add B12x backends to unified MoE API and add corresponding tests by @Aneureka in #3983
  • feat: trtllm FP4 routed-MoE: accept fp32 topk_weights (copy-free) by @jdebache in #3763
  • feat: CuTe-DSL Fused SwiGLU NVFP4 Quantization by @philipphack in #4040
  • docs: fix MSA and GDN doc-check findings by @kangbintNV in #4058
  • Yanqinz/fix autotuner using autotuning plan from other runners by @yanqinz2 in #4004
  • perf(jit): prune arch gencode that dispatch can never load by @yichengj0 in #4073
  • mamba checkpointing SSU: two-kernel split + ring-buffer cache for checkpointing SSU by @ishovkun in #3975
  • perf(kda): use a one-warp/grouped recurrent decode hybrid by @djmmoss in #4001
  • perf(gemm): Improve mm_fp4 cute-dsl autotune time via disk-cache and parallel compilation by @bkryu in #4029
  • feat: add optional normed-output (y_out) to add_rmsnorm_fp4quant by @nv-guomingz in #4072
  • Code review guidance proposal by @aleozlx in #3790
  • Port the TensorRT-LLM one-sided A2A optimizations to Flashinfer by @b8zhong in #3697
  • feat(moe): add unified block-scale FP8 support by @feih-nv in #4026
  • feat: allow using NVL a2a kernels without MNNVL as long as all peers are within the same node / intra-node NVL domain by @jdebache in #3701
  • [feat] Add TRTLLM/SageAttention quantization routine by @xrq-phys in #3982
  • fix: Fix the MOE tests to properly scale NVFP4 bias by @djns99 in #3755
  • feat(moe): support caller-owned CUTLASS MoE workspace by @feih-nv in #4057
  • feat: add collect-env environment report tool + issue templates (bug, feature request) by @YangXu1990uiuc in #4007
  • feat(gdn): index GDN prefill state pool via state_indices (SM100/SM103) by @nv-guomingz in #4084
  • fix(fused_moe): avoid repeated runner setup in host dispatch by @YAMY1234 in #4045
  • feat(moe_ep): close the nixl_ep transport gaps blocking the vLLM Fleet/Handle adapters by @Anerudhan in #4075
  • fix: keep HEAD_DIM 256 prefill within 64 KiB smem on SM75/SM70 by @HuChundong in #3526
  • fix(gemm): prevent out-of-bounds read in MaskedScheduler by @dashanji in #4046
  • fix: bug-report issue template silently dropped by chooser (description over length limit) by @YangXu1990uiuc in #4100
  • perf(sm103): optimize NVFP4 epilogue stores and scaling by @tiffany940107 in #4063
  • test: compute mm_fp4 cosine similarity in fp32 to avoid bf16 rounding cliff by @Adnios in #4105
  • fix(comm): sync all sender threads before publishing the final FIFO step to fix the intermittent race issue by @Aneureka in #4092
  • fix: Add missing nvfp4 sizing branch in getProfilerWorkspaces by @jimmyzho in #4080
  • docs: fix flashinfer doc check findings by @kangbintNV in #4097
  • Revert "[Bugfix] Skip autotuning for small token counts on SM100 to prevent TMA crash" by @b8zhong in #4041
  • fix(trtllm-gen): select dynamic-page decode kernels for MHA (numHeadsQPerKv == 1) by @syuoni in #4095
  • fix: move JIT symlinks to writable gen dir by @namgyu-youn in #3468
  • feat(comm): Add FP8 quantized two-shot AllReduce via symmetric memory by @kailashbuki in #3759
  • fix: let FLASHINFER_CUBIN_DIR env var override flashinfer-cubin package path by @ianliuy in #3062
  • feat(moe): add unified per-tensor FP8 support by @feih-nv in #4091
  • Update MoE CODEOWNERS by @feih-nv in #4127
  • comm: multicast-free (IPC) workspace for TRT-LLM AR fusion under Confidential Computing by @elvischenv in #3993
  • feat: fused FP8 quantization output for LayerNorm by @elwhyjay in #3962
  • chore: Add Anerudhan to comm codeowner by @bkryu in #4135
  • CuTe-DSL modular prefill: sliding-window support + band-mask performance by @pgera in #3952
  • fix(gdn): compile SM12x CuteDSL kernels as sm_121a on DGX Spark by @nv-tusharma in #3960
  • fix(moe_ep): fix two NIXL-EP combine deadlocks under real serving load by @Anerudhan in #4139
  • Adds SM107 support by @Vinnie6167 in #4122
  • Split trtllm-gen cubin pins per arch by @Vinnie6167 in #4191
  • perf(sampling): gate launch_bounds to SM107 only (NVBug 6517769) by @kahyunnam in #4189
  • release-v0.6.16: cherry-pick #4167, #4137, #4187, #4178 and bump to 0.6.16rc2 by @aleozlx in #4197
  • release-v0.6.16: fix cubin checksum collision and unguarded Sm107a build break by @aleozlx in #4200
  • fix(moe): reject incompatible output-scale cubins by @aleozlx in #4213
  • fix(jit): pass map_sm107_to_100f in gen_moe_utils_module by @aleozlx in #4215
  • test: fix #4191 regressions in autotuner_core and symlink race test by @kahyunnam in #4225
  • cherry-pick: #4145 test(sm103): fix FP4 autotuner cache inspection by @kahyunnam in #4231
  • cherry-pick: #4199 fix(xqa): PDL load ordering and SM90 fp8 draft-mask dispatch by @kahyunnam in #4232
  • cherry-pick: #3882 fix mxfp8 gemm quantization / scale-layout warnings by @kahyunnam in #4233
  • fix(trtllm): restrict routed-MoE backends to supported architectures (release port for #4107) by @kahyunnam in #4230
  • fix(moe): name-filter output-scale-incompatible cubins until pinned packages carry mDtypeSfC by @Vinnie6167 in #4235
  • Make CuTe-DSL arch guard env-aware and gate norm's DSL dispatch on it by @Vinnie6167 in #4226
  • chore: bump version to 0.6.16rc4 by @kahyunnam in #4236
  • cherry-pick: #4129 and #4162 by @jimmyzho in #4245
  • revert: #3975 and #4129 (mamba checkpointing ssu 2-split + doc) by @jimmyzho in #4246
  • cherry-pick: #4185 (for issue 4090) by @jimmyzho in #4247
  • cherry-pick: #4152 ([perf] Optimize TRT-LLM routing for high-expert, high-top-k workloads) by @jimmyzho in #4251
  • test: skip NVFP4-KV decode tests on SM107 by @Vinnie6167 in #4257
  • artifacts: add DEEPGEMM_RUBIN artifact path and checksum by @jimmyzho in #4258
  • deep_gemm: arch-based split for DEEPGEMM_RUBIN cubins by @jimmyzho in #4261
  • cherry-pick: #4180 ([feat] Add SITU trtllmgen MOE) + conflict resolution by @jimmyzho in #4252
  • chore: bump version to 0.6.16rc5 by @jimmyzho in #4264
  • cherry-pick: #4237 (fix(moe): pad trtllm-gen route map by one element to avoid OOB read) by @jimmyzho in #4288
  • bump version to 0.6.16 by @jimmyzho in #4289

New Contributors

Full Changelog: v0.6.15rc4...v0.6.16

Check out latest releases or
releases around flashinfer-ai/flashinfer v0.6.16

Don't miss a new flashinfer release

NewReleases is sending notifications on new releases.

Get notifications