What's Changed
- feat: Add backend='auto' to mm_fp4 and enable autotune for backend='cudnn' by @bkryu in #1979
- fix: Fix bench_mm_fp8.py by @bkryu in #2129
- feat: Enable API Logging for Better Debugging POC by @bkryu in #2108
- fix: add a check for int32 indices in sampling.py by @raayandhar in #2127
- update autotuner input tensor random range by @jiahanc in #2116
- enable xqa speculative decoding by @qsang-nv in #2105
- Add custom communicator for trtllm_mnnvl_ar by @wenscarl in #2056
- fix: DeepSeek activation uninitialized data by @nekorobov in #2128
- chore: Update CODEOWNERS by @flashinfer-bot in #2135
- bugfix: fix unittest error introduced in #2056 by @yzh119 in #2136
- fix flaky xqa test by @qsang-nv in #2126
- fix: some bugs of headDim 256 trtllm-gen fmha kernels. by @PerkzZheng in #2137
- fix(trtllm): reset negative strideBatch to 0 for ragged KV layout to … by @YAMY1234 in #2134
- feat: add trtllm-gen per-tensor sparseMla kernels. by @PerkzZheng in #2138
- Use global TuningConfig, to fix memory leak caused by AutoTuner LRU cache and dynamic lambda TuningConfig by @juju812 in #2140
- feat: add seed offset args to sampler to allow cuda graph support by @ksukrit in #2132
- ci: Reduce test time by moving compilation off-line by @kahyunnam in #2089
- feat: TRTLLM FMHAv2 backend for ctx attention by @jimmyzho in #2142
- refactor: pass hopper deepgemm include directory through python by @yzh119 in #2090
- bugfix: add driver support to CUPTI benchmark function, issue #2145 by @nv-yunzheq in #2154
- Bump tvm ffi version to 0.1.4 by @cyx-6 in #2155
- Update Docker CI tags to 20251202-23ff744 by @flashinfer-bot in #2158
- misc: Label APIs for Logging by @bkryu in #2153
- Update nvidia-cutlass-dsl version to 4.3.1 by @aleozlx in #2161
- chore: Update CODEOWNERS by @flashinfer-bot in #2152
- feat: C++ side tensor validation by @raayandhar in #2160
- Update Docker CI tags to 20251203-4efb7bb by @flashinfer-bot in #2164
- ci: Install CUDA version specified torch first during container building. by @bkryu in #2167
- fix xqa mha_sm90.cu by @qsang-nv in #2157
- Update Docker CI tags to 20251203-1e15fed by @flashinfer-bot in #2172
- enable sm103 moe dsl backend by @aleozlx in #2149
- ci: Use stable Torch Release for cu130 by @bkryu in #2174
- tiny upd
mm_fp4docstring by @b8zhong in #2177 - fix: compile flags for trtllm fmha_v2 by @jimmyzho in #2175
- Fix/dsl smem query by @aleozlx in #2178
- Update Docker CI tags to 20251204-cdc5fb7 by @flashinfer-bot in #2176
- feat: MxInt4 x Bf16 TRT-LLM Gen MoE support by @nekorobov in #2159
- refactor: Move mla code from decode.py to mla.py and add to documentation by @bkryu in #2163
- Fix gemm allreduce two shot by @aleozlx in #2171
- Update Docker CI tags to 20251205-54c1678 by @flashinfer-bot in #2179
- Rename noauxtc to fused_topk_deepseek by @nv-yunzheq in #2181
- refactor: update fa3 codebase and fix hopper unittest [part 1] by @yzh119 in #2111
- Add data type check for deepseek fp4 moe by @samuellees in #2165
- benchmark: Make use_cupti the default in microbenchmarks. by @bkryu in #2180
- ci: Specify MPI implementation to mpich by @bkryu in #2182
- Update Docker CI tags to 20251206-185d63a by @flashinfer-bot in #2184
- test: Skip sm90 test in test_jit_warmup.py if not on sm90 by @bkryu in #2189
- ci: Update sm12X minimum cuda capability to 12.9 in aot.py by @bkryu in #2188
- Super tiny fix version by @fzyzcjy in #2199
- docs: Document CUDA version support in README and installation page by @bkryu in #2197
- docs: Fix inaccurate API docstrings for attention prefill by @bkryu in #2196
- feat: unit-test and api change, w4a8 grouped-gemm fused MoE for SM90 by @jimmyzho in #2193
- Permute page table in benchmarking by @jhjpark in #2194
- Fix for moe on sm110 by @jhalabi-nv in #2190
- chore: update authorized codeowners by @jimmyzho in #2210
- perf: bunch of features and optimizations for top-k (sampling + sparse attention) by @yzh119 in #2119
- Refactor trtllm_mnnvl_allreduce by @timlee0212 in #2118
- chore: Update CODEOWNERS by @flashinfer-bot in #2186
- feat: support more head dim in RoPE kernel by @raayandhar in #2109
- Port TRT-LLM communication kernels to flashinfer by @djns99 in #2102
- cicd: Add sanity test script by @kahyunnam in #2212
- feat: add memcpy and memset to CUPTI timing method by @nv-yunzheq in #2223
- Added an initial implementation of Q and KV Cache in fp8 and to use t… by @Anerudhan in #2035
- feat: Support unpadded output hidden size for trtllm_fp4_block_scale_moe by @elvischenv in #2217
- fix: Eliminate the usage of CUDA ARCH macro in host function. by @timlee0212 in #2228
- misc: support checks for gemm by @jimmyzho in #2214
- feat: Cold L2 Cache Benchmarking with Rotating Buffers by @bkryu in #2213
- Move the run function definition out of BatchedGemmInterface by @jhalabi-nv in #2211
- make DeepGEMM swapAB available for linear gemm SM90 by @katec846 in #2131
- misc: upgrade tvm-ffi dependency to 0.1.6 by @yzh119 in #2229
- A unified API for the MNNVL and single-node/multi-GPU AllReduce kernels. by @nvmbreughe in #2130
- Update Docker CI tags to 20251217-f059241 by @flashinfer-bot in #2231
- Rebase FP8 SM100 Cutlass FMHA Attention to main (original PR#1238) by @pavanimajety in #2047
New Contributors
- @YAMY1234 made their first contribution in #2134
- @juju812 made their first contribution in #2140
- @ksukrit made their first contribution in #2132
- @samuellees made their first contribution in #2165
- @jhjpark made their first contribution in #2194
- @jhalabi-nv made their first contribution in #2190
- @djns99 made their first contribution in #2102
- @katec846 made their first contribution in #2131
Full Changelog: v0.5.3...v0.6.0rc1