github flashinfer-ai/flashinfer v0.7.4
Release v0.7.4
on GitHub

one hour ago

These highlights are also published at flashinfer.ai/releases.

v0.7.4 Highlights

FlashInfer v0.7.4 brings DeepSeek-V4 sparse MLA decode and MXFP4 expert-parallel MoE to Rubin, and adds a batch-invariant mode for GEMMs and quantizers on Blackwell and Rubin that returns the same bits for a row at any batch size. Sparse-attention models get the SDSA indexer and a context-parallel MiniMax-M3 indexer, Gemma 4 decodes from an NVFP4 KV cache, and tensor-parallel layers fuse FP8 GEMM with all-reduce. Speculative-decode verification and sampling run faster on Hopper, and DeepSeek Sparse Attention indexing and Qwen3.5 speculative decode reach RTX PRO 6000.

DeepSeek-V4 sparse MLA decode and MXFP4 expert-parallel MoE run on Rubin

trtllm_batch_decode_sparse_mla_dsv4(backend="cake") runs its FP8 decode route on SM107 for DeepSeek-V4, and the experimental Prims-TS sparse MLA wrapper plans on Rubin, NVFP4 KV caches included. SM107 MegaMoE through MoEEpLayer adds MXFP4 activations with MXFP4 weights, kernel_variant="auto" selects between the inference and GenPhase variants through autotune_v2(), and the Lamport transport of NVLinkOneSidedAlltoAll for decode-sized steps builds for SM107. Rubin also gets a block-scaled contiguous grouped FP8 GEMM (prepare_group_gemm_fp8_nt_groupwise_contiguous), batch_deepgemm_fp8_nt_groupwise(backend="cake") for four weight shapes, the TinyGEMM backend of mm_bf16, and an SM107 kernel behind gated_delta_rule_mtp_ucache_flush for Gated DeltaNet MTP decode.

  • #8375 feat(cake_dsv4): SM107 runs the FP8 decode route of backend='cake'; the low-head prefill program takes PDL
  • #8186 perf(prims-ts): run sparse MLA on SM107 and trim the NVFP4 KV converter
  • #7944 feat(moe_ep): support MXFP4 W4A4 on SM107
  • #7911 feat(moe_ep): select Rubin MegaMoE variants with autotune v2
  • #8419 feat(moe_ep): build the NVLink one-sided Lamport transport for SM107
  • #8683 feat(cake_grouped_fp8_gemm): block-scaled contiguous grouped FP8 GEMM on SM107a
  • #8688 feat(cake_batch_deepgemm_fp8): masked BLOCK_N batch DeepGEMM FP8 on SM107a
  • #8227 fix(gemm): enable the TinyGEMM mm_bf16 backend on SM107
  • #6791 feat(gdn): Rubin (sm_107a) build of the UMMA ReplaySSM MTP u-cache flush kernel, with uGPU localization

Batch-invariant GEMMs and quantizers return the same bits for a row at any batch size

mm_bf16, bmm_fp8, mm_mxfp8 and mm_fp4 take batch_invariant=True on B200, B300 and Rubin: an output row's bits depend only on the input values and the weight shape, independent of the batch size, the row's position, the SM count and the configuration the dispatcher or autotuner picks. Reinforcement-learning pipelines can match the inference engine's logits with the trainer's bit for bit. mm_bf16 takes a bias in this mode, and mxfp8_quantize, mxfp4_quantize and nvfp4_quantize gain the same flag with numerics aligned to Transformer Engine. The flag requires backend="auto", and an unsupported input or device raises an error.

  • #8361 feat(batch_invariant): batch-invariant mode with mm_bf16(batch_invariant=True)
  • #8453 feat(batch_invariant): bmm_fp8(batch_invariant=True); the dispatcher becomes a ranking heuristic
  • #8535 feat(batch_invariant): bias for mm_bf16(batch_invariant=True)
  • #8536 feat(batch_invariant): mm_mxfp8(batch_invariant=True), the MXFP8 GEMM class
  • #8537 feat(batch_invariant): mm_fp4(batch_invariant=True), the NVFP4 and MXFP4 GEMM classes
  • #8565 feat(batch_invariant): batch-invariant quantizers: mxfp8_quantize, mxfp4_quantize and nvfp4_quantize(batch_invariant=True)

SDSA and MiniMax-M3 indexers select sparse-attention keys on Blackwell and Rubin

flashinfer.sdsa adds the indexer of SDSA (Simplified DSA), where a one-head indexer selects the Top-K keys of each query token for the token-sparse MQA decode: indexer_scores scores BF16 or NVFP4 paged keys for decode, quantize_write_index_qk_nvfp4 writes NVFP4 keys to the paged cache, and Top-K fusion (gather_key_sample, thresholds_from_sample, indexer_threshold_select, top_k_from_candidates) keeps only the keys above a per-row threshold for prefill. The APIs run on Blackwell SM100 and SM103, and indexer_scores with BF16 keys also runs on other GPUs such as H100. The experimental FP8MQATopKWrapper(backend="cutedsl") adds Selective TopK for single-request prefill with FP8, MXFP8 and MXFP4 queries and keys on B200 and B300. For MiniMax-M3, the experimental flashinfer.msa_ops index-context-parallel indexer covers TP2 and TP4 with page sizes 128 and 256 on B200, GB300 and Rubin, and dsa_indexer_det_topk(backend="cake") accepts FP8 queries and keys.

  • #8374 [SDSA] Add the SDSA (Simplified DSA) indexer for Blackwell, with BF16 scores on other GPUs
  • #7325 feat(attn_scores): add selective MQA TopK for prefill (indexer-throwaway topk)
  • #8122 feat(cake_msa_icp_indexer): MSA-compatible index-context-parallel indexer for TP2/TP4, P128/P256, SM100/SM103/SM107
  • #8749 feat(cake_dsa_indexer_det): opt-in FP8 (e4m3) form

Gemma 4 decodes from an NVFP4 KV cache on Blackwell, and tensor-parallel layers fuse FP8 GEMM with all-reduce

The experimental Prims-TS decode backend covers head dimension 512 with FP8 queries and an NVFP4 KV cache, the Gemma 4 attention geometry, with native BF16 output; Prims-TS attention also adds attention sinks, NHD caches, automatic runtime split-KV, and MXFP8 queries and keys for dense context attention on B200 and B300. On Blackwell, BatchPrefillWithRaggedKVCacheWrapper(backend="auto") serves sliding-window and attention-sink calls on cuDNN. For tensor-parallel serving, flashinfer.comm.fp8_rowwise_gemm_allreduce(backend="cake") runs a row-wise scaled FP8 GEMM and the all-reduce in one kernel at 2, 4 and 8 ranks on SM100 and SM103, and the CAKE NVFP4-transport all-reduce fuses the residual add and RMSNorm at four ranks for any row count. trtllm_fp4_block_scale_routed_moe(routing_metadata=...) runs the expert GEMMs from routing metadata the caller built earlier, for example on another stream.

  • #5720 feat(prim_ts): Add H512 FP8-Q/NVFP4-KV decode support
  • #6925 feat(prims-ts): broaden attention support and runtime split-KV
  • #7509 feat(prims-ts): QK-MXFP8 context FMHA
  • #5286 feat(prefill): teach the cuDNN ragged path sliding window, attention sinks and scalar scales
  • #8379 feat(cake_fp8_rowwise_gemm_allreduce): SM100 fused FP8 GEMM + all-reduce for TP2/4/8
  • #8605 feat(cake_fp8_rowwise_gemm_allreduce): SM103 fused FP8 GEMM + all-reduce for TP2/4/8
  • #8395 feat(cake_fp4_allreduce): four-rank fused all-reduce + residual + RMSNorm; any row count at four ranks
  • #8050 feat(moe): run the FP4 routed MoE expert body from caller-populated routing metadata

Mamba, Kimi K3 and Qwen hybrid models get packed causal-conv prefill, indexed state pools and a fused GDN decode block

flashinfer.mamba adds causal_conv1d_fn for packed variable-length prefill and causal_conv1d_update for decode and speculative verification on B200 and B300, with indexed state pools, fused SiLU and CUDA Graph replay. recurrent_kda(backend="small-bh") reads and updates an indexed state pool through ssm_state_indices for Kimi K3 prefill, and checkpointing_ssu accepts strided ReplaySSM trackers, so vLLM can keep them inside the per-slot Mamba state page. The experimental flashinfer.qwen_3_5_gdn_block runs the Gated DeltaNet sub-layer of Qwen 3.5-architecture models for one decode or MTP-verify step in a single call on B200, and the experimental Qwen 3.8 block adds extend and prefill entries for chunked prefill through the serving caches at any row count.

  • #8273 feat(mamba): add inference causal conv1d prefill and state update
  • #8287 feat(kda): indexed state pool (ssm_state_indices) for the small-BH prefill backend
  • #8548 feat(checkpointing_ssu): accept strided ring_start and prev_num_accepted
  • #8131 [Experimental] Qwen 3.5 GDN block: decode and MTP verify behind a backend registry, with an SM100 CuTe DSL backend
  • #7983 [Experimental] Qwen 3.8 Flash-Next block: the indexer's extend / prefill entries (chunked prefill through the serving caches) and the cuDNN Frontend 4c8e24355 re-vendor
  • #8058 [Experimental] Qwen 3.8 block: the extend / prefill entries take any row count (cuDNN Frontend 6b21be0e0 re-vendor)

Speculative-decode verification and sampling run faster on Hopper

On H100 and H200, BatchPrefillWithPagedKVCacheWrapper(backend="auto") serves batches whose requests carry at most 64 query tokens, such as MTP or speculative-decode verification, on FA2 with a query tile sized to the request. top_p_sampling_from_probs, top_k_sampling_from_probs and joint top_k_top_p_sampling_from_probs split each row across a thread-block cluster, which lowers sampling latency at decode batch sizes. The experimental qsa_sparse_paged_attention(backend="auto") runs faster for BF16, FP8 and NVFP4 caches on A100 and H100, and the SM90 BF16 MegaMoE backend adds opt-in compressed expert weights (wcx_max_tokens) for decode rounds.

  • #5188 prefill: use fa2 backend for small per-request query lengths
  • #5617 perf(sampling): split rows across thread block clusters in rejection samplers on SM90
  • #8423 perf(qsa): speed up BF16, FP8 and NVFP4 on SM80-SM90
  • #8089 feat(cake_mega_moe): SM90 BF16 opt-in compressed expert weights (WCX) for decode rounds

DeepSeek Sparse Attention indexing and Qwen3.5 speculative decode run on RTX PRO 6000

The experimental dsa_indexer_det_topk(backend="sm120") runs the DeepSeek Sparse Attention lightning indexer on SM120, scoring plus exact top-k in FP8 or MXFP4 with 32 or 64 heads, for DeepSeek-V3.2, DeepSeek-V4, DeepSeek-V4.1 and GLM-5.x in ragged prefill and paged decode forms. Gated DeltaNet ReplaySSM speculative decoding reaches SM120: gated_delta_rule_mtp(cache_replayssm=True) and gated_delta_rule_replayssm_commit run on RTX PRO 6000 in the Qwen3.5-397B geometry. BlockSparseAttentionWrapper(backend="vsa_sm120_blk64") runs on CAKE kernels.

  • #7085 feat(dsa_indexer): experimental SM120 backend for FP8 / MXFP4 prefill and decode
  • #7536 feat(gdn): add SM120 cuteDSL ReplaySSM GDN MTP decode kernels
  • #8799 perf(cake_vsa_sm120): run vsa_sm120_blk64 backend on Cake kernels

⚠️ Upgrade notices (Experimental)

  • To use APIs under flashinfer.experimental, install from source with pip install --no-binary flashinfer-python flashinfer-python==0.7.4 or pip install git+https://github.com/flashinfer-ai/flashinfer.git@v0.7.4, or use a nightly wheel from https://flashinfer.ai/whl.
  • Prims-TS attention: rename bmm1_scale and sm_scale to softmax_scale, bmm2_scale to output_scale, *_data_type to *_dtype, use_packed_q to packed_query, seq_lens to seq_lens_kv, max_seq_len to max_kv_len, and the MLA num_heads to num_qo_heads.

What's Changed

Check out latest releases or
releases around flashinfer-ai/flashinfer v0.7.4

Don't miss a new flashinfer release

NewReleases is sending notifications on new releases.

Get notifications