github NVIDIA/TensorRT-LLM v1.3.0rc26
on GitHub

pre-release5 hours ago

Highlights

  • Known Issues

    • LoRA adapters that target the routed experts of Qwen3 mixture-of-experts models are silently ignored and produce base-model output. Adapters targeting attention layers behave correctly.
    • Models using the V2 KV cache manager may fail to start with an insufficient-memory error even when enough GPU memory is free, because the KV cache size estimate is too conservative. GPT-OSS with Eagle-3 speculative decoding and sliding-window cache reuse is the most affected configuration. Lowering the maximum batch size or raising the free GPU memory fraction can allow startup to proceed.
    • Long-running servers using the FlashInfer attention backend may run out of GPU memory over time as attention planning resources accumulate across varying batch sizes. Using the default TRTLLM attention backend avoids this.
    • Image generation and image editing requests to a VisualGen deployment can fail with an HTTP 500 error, and video endpoints report inconsistent timing values. Text generation is unaffected.
    • Disaggregated serving requests may fail with a server error after a transient network interruption between the router and a context or generation worker, because the retried request is no longer matched to its in-flight counterpart.
    • Models using multi-head latent attention, such as DeepSeek, may hang or crash during generation on SM120 Blackwell GPUs, most often with multi-token prediction and chunked prefill enabled.
    • NVFP4 W4A16 mixture-of-experts models are unsupported on SM120 Blackwell GPUs and abort while preparing kernels. A future release will address this issue.
    • Enabling online expert-parallel load balancing with the TRTLLM-gen mixture-of-experts backend on unquantized models fails during model loading. The CUTLASS mixture-of-experts backend supports the same configuration.
    • Diffusion pipelines that offload the VAE component, such as Wan and Cosmos3, can fail to load. Running without VAE offloading works.
    • Kimi KDA models may crash when a scheduled batch contains no prefill work, which can happen under mixed prefill and decode scheduling.
    • Creating two models with different mixture-of-experts expert-parallel layouts in a single process fails. Use a separate process per configuration.

  • Model Support

    • Add DeepSeek-V4 Hopper support #16940
    • Add Qwen3.8-Flash-Next support #18585
    • Enable Kimi K3 on B200 #18164
    • Enable Nemotron Super 3.5 VL NVFP4 and video input #18526, #18370
    • Qualify Mistral and Qwen3 dense models for Model Express #17142, #18558

  • API

    • Remove TRTLLMSampler (BREAKING) #18232, #18233, #18532
    • Redesign VisualGen reference inputs and worker IPC (BREAKING) #17493
    • Complete the /v1/responses surface #18130
    • Add Kimi K3 API compliance in trtllm-serve #17845
    • Add authentication for RL control endpoints #17434
    • Support response_format='path' on /v1/images/edits #18444

  • Feature

    • Add Rubin SM107 GEMM, MoE, and CuTe DSL support #17485, #17707, #18197, #18369
    • Support custom masks in TRTLLM attention #16214
    • Add SkipSoftmax sparse attention to the CuTe DSL backend #17781
    • Enable CuTe DSL MLA with Helix #18131
    • Make MLA skip correction configurable #18118
    • Add an MLA post-attention o_proj gate hook #18375
    • Update trtllm-gen FMHA kernels and cubins #18183
    • Cache and extract FMHA selection in TrtllmAttention #18350, #18548
    • Add encoder CUDA graphs for fixed-shape encoders #17030
    • Refactor advanced-sampling CUDA graph capture #17936
    • Support breakable CUDA graph for the Kimi K3 KDA mixer #17816
    • Add NVFP4 KV cache for DSA and cold-page compression #17681, #18091
    • Add Mamba state snapshots for hybrid KV cache reuse #18272
    • Improve KV cache manager V2 stats and VMM allocation #16710, #18347
    • Make the Python KV cache transceiver the default #18134
    • Add pipelined KV cache transfer for disaggregated serving #15727
    • Add a DisaggTransferCoordinator and transceiver conformance fake #18595, #18178
    • Split connector KV save out of _send_kv_async #18186
    • Use msgspec msgpack for disaggregated worker bodies #16873
    • Add a selectable OpenEngine gRPC server #17084
    • Add exit code telemetry to the CLI and LLM API #17377
    • Consolidate DSpark draft paths and support standalone drafters #18043
    • Add emission-assisted GVR top-K decode for DeepSeek V4 #16953
    • Improve GVR V2 decode top-k and prior seeding #18410, #18646
    • Optimize DSA decode metadata, index remap, and paged MQA #18268, #18391, #18383
    • Add an env-gated BF16 DSA indexer projection #18264
    • Enable fused-A GEMM for the GLM MLA A-projection #18265
    • Optimize DSpark attention with Blackwell CuTe DSL kernels #18061
    • Optimize Kimi K3 weight loading and prefill metadata #17741, #18064, #18643
    • Optimize Kimi KDA decode and prefill convolution #18251, #17870, #17862
    • Convolve the Mamba2 prefill on a channel-last projection #18081
    • Optimize MiniMax-M3 GEMMs, decode q, and block selector #17238, #18474, #18154
    • Optimize Qwen3.6-35B-A3B-NVFP4 on SM120 #17976
    • Add split-K auto-tuning to the LoRA grouped GEMM #17315
    • Support uneven split-K and fix fused mHC coherence #18331
    • Fuse relu2 into a single kernel #18082
    • Patch DeepEP for top-k 16 #17984
    • Make the DeepGEMM barrier timeout configurable #18179
    • Add a locality domain Python layer #18317
    • Preserve MNNVL and MoE A2A graph addresses across restore #16635, #16632
    • Reuse FlashInfer plans for single-token decode #18596
    • Merge attention-DP host collectives #18293, #18639
    • Implement rank-striped checkpoint read-ahead #17393
    • Forward speculative-decoding state through the Nemotron VL wrapper #18647
    • Add offloading support for VisualGen #14095
    • Add VisualGen FP8 row-wise quantization #16847
    • Add Cosmos3 action generation to VisualGen #17325
    • Add a trtllm-eval pipeline for VisualGen #18199
    • Declare MoE backend capabilities instead of class checks #18409
    • Relocate MoE code under _torch/moe/ #17952
    • Extract multimodal encoder scheduling into engine/multimodal.py #18524
    • Remove unused KV cache draft-token rewind helpers #18533

  • Fix

    • Isolate FlashInfer JIT workspaces across MPI processes #18278
    • Normalize perf metrics clocks across frontend processes #17491
    • Guard CuTe DSL grouped-GEMM imports in fused MoE #18144, #18142
    • Bump CUTLASS DSL to 4.6.2 for FA4 split-KV #18145
    • Include the FINALIZE-fusion workspace in the MoE autotuner #18675
    • Size the SM90 fp8 block-scale MoE workspace correctly #18323, #18322
    • Reject NVFP4 MoE layers with unaligned gated FC1 #18520
    • Reject incompatible cached low-M GEMM tactics #18635
    • Support MXFP8 under MiniMax SwiGLU in CutlassFusedMoE #18257
    • Pass the runtime SM to the NVFP4 Marlin support check #17502
    • Fix barrier-divergence races in the GVR top-k decode kernel #18339
    • Fix self-sampling top-k in the register family #18625, #18501
    • Reject unsupported speculative outputs #18401
    • Use prompt lookahead for MTP Eagle chunked prefill #18295
    • Keep CuTe DSL MLA decode for Kimi K3 H=96 verify batches #18226
    • Apply the K3 prompt-token offset only when known #18509
    • Keep quantized KV context on TRTLLM-Gen FMHA #18545
    • Use fallback FMHA for SM103 context attention #18354
    • Fix Gemma4 FMHA fallback and video token counting #18544, #18274
    • Fix window vector layer indexing #17466
    • Fix MiniMax-M3 MSA KV cache mapping and vision meta init #17374, #18466
    • Remove an invalid GLM KV cache assertion #18374
    • Support fixed-only draft KV cache budgets #18459
    • Exclude mamba conv1d from quantization #18607
    • Fix Qwen3.8-27B NVFP4 checkpoint loading #18387
    • Fix LLM-only behavior for Mistral Large #18086
    • Fix attention-DP dummy token accounting #18308, #18157
    • Register beneficial-to-skip contributions after scheduling #18202
    • Recalculate PEFT cache capacity for the adapter dtype #18200
    • Guard Python KV receive ownership and publication #17720
    • Avoid sleeping out the KV transfer poll interval #18175
    • Keep disaggregated worker liveness within its TTL #18403
    • Fail-stop on distributed warmup errors #17667
    • Stabilize the telemetry lifecycle #18454
    • Fix lazy imports for profile matching and MPI worker submit #18236, #18284
    • Restore verl rollout after the sampler_type removal #18513
    • Restore Wan 5B per-token AdaLN with TeaCache #18209
    • Preserve prepared Cosmos3 video frames #18464
    • Allow sm_107a in the VisualGen CuTe DSL FMHA #18476
    • Raise the Starlette security floor #17650

  • Documentation

    • Split the LLM API reference into smaller pages #18440
    • Document the FMHA availability contract #18710
    • Update the GLM-5 docs for GLM-5.3 and disaggregated serving #18388
    • Fix stale Wide-EP EPLB documentation #18626
    • Clarify nightly release installation #18282
    • Fix the multimodal quick start clone step #18429
    • Give a runnable lm_eval install command #18431
    • Fix XGrammar structural tag doc links #18430
    • Remove the invalid html_inline myst extension #18427
    • Add a human-in-the-loop section to the modeling-bringup README #18358
    • Add a video generation optimization blog #16506

  • Test & Infra

    • Update flashinfer-python to 0.6.18 #18428
    • Declare and pin CI-imported dependencies #18033
    • Add a declarative extra import path for test sources #17884
    • Upload disaggregated lifecycle spans to OpenSearch #18445
    • Key perf-sanity case identity on the test case name #18408
    • Adjust perf-sanity ctx_only scheduling and warmup #18346, #18432
    • Add AgentX DeepSeek-V4-Pro-DSpark perf lanes on GB300 #18298
    • Add MiniMax-M3 disaggregated perf recipes #18416
    • Add GB300 multi-node Qwen3.8-2.4T-A95B accuracy stages #18363
    • Add Kimi K3 accuracy and speculative decoding tests #17922, #17921
    • Add KimiLinearForCausalLM helix mapping coverage #18491
    • Expand Gemma 4 MTP QA coverage #18218
    • Validate the GLM-5.2 feature support matrix #18470
    • Add hybrid MoE TP x EP modes to bench_moe #18285
    • Count async KV completions as benchmark fill progress #18456
    • Parametrize the combined FMHA test stub #18504
    • Support CBTS result reuse and OpenEngine coverage #18073, #17843
    • Improve CBTS anchors, follow-up, diagnostics, and caching #18543, #18573, #18422, #18309
    • Expand the CBTS Tier 2 pilot to QA #18442
    • Enable the BOLT profile overlay on published images #18279, #18291
    • Improve BoltProfileGen downloads, partitions, caching, and gating #18424, #18425, #18426, #18462, #18469, #18395
    • Enable BOLT premerge consume scaffolding #18622
    • Use an authenticated token for GitHub fetches #17788, #17686
    • Check GitHub links in documentation #18591
    • Run pre-commit on all files for large diffs #18441
    • Fix INFRA-DEFER status and extend infra fail-fast #18305, #17799
    • Upload results at the end of a stage #18355
    • Retry SLURM agent online timeouts #17586
    • Add a maintenance stage config #18508
    • Lower the GB300 16-GPU accuracy test timeout #18480
    • Revert the GB300 DeepSeek-V4-Pro KV transfer timeout to 60 s #17137
    • Fix disagg NIXL readiness timeout and UCX_TLS settings #18372, #18373, #18301
    • Mitigate disagg cancellation test hangs #18326
    • Narrow the Wan2.2 I2V fixture scope #18328
    • Relax the Cosmos3-Nano LPIPS gate and unwaive #18550
    • Unwaive test_disaggregated_qwen3_32b_fp8 #18319
    • Avoid MPI bootstrap in the RocketKV unit test #18593
    • Lower the MMMU reference accuracy threshold #18642
    • Drop redundant min_tokens from the token-boundary smoke test #18306
    • Re-enable the layer-wise benchmark test on A10 #15497
    • Move the Gemma4 perf test to post-merge #18089
    • Prune legacy Llama and Nemotron tests #17858
    • Re-enable previously waived tests #18121, #18189, #18203, #18221, #18286, #18304, #18344, #18371, #18394, #18400, #18503, #18603, #18606
    • Add perf-analyze and perf-optimize skills to agent-flow #18330, #18434
    • Remove the AutoDeploy agent skills #18572

What's Changed

  • [TRTLLM-15498][perf] fuse KDA beta preprocessing and isolate FLA fallbacks by @jiaganc in #17862
  • [TRTLLMINF-345][doc] clarify nightly release installation by @chzblych in #18282
  • [None][feat] k3 weight pipeline opt by @WeiHaocheng in #17741
  • [https://nvbugs/6337224][test] Unwaive test_config_database_tests_sync by @hyukn in #18221
  • [TRTLLM-15316][feat] Rubin sm107 trtllm-gen gemms by @farazkh80 in #18197
  • [TRTLLM-15894][feat] Responses API: complete the /v1/responses surface for agent CLIs by @JunyiXu-nv in #18130
  • [TRTLLM-14874][feat] Refactor advanced-sampling CUDA graph capture by @asfiyab-nvidia in #17936
  • [None][perf] Histogram top-k for MiniMax-M3 block selector by @brb-nv in #18154
  • [TRTLLMINF-339][infra] Declare BoltProfileGen's SCM pass-through params by @mlefeb01 in #18291
  • [https://nvbugs/6571410][fix] Move Gemma4 perf test to post-merge by @2ez4bz in #18089
  • [None][chore] Update blossom-ci allowlist by @yuanjingx87 in #18315
  • [https://nvbugs/6572838][fix] Fix LLM-only behavior for Mistral large by @2ez4bz in #18086
  • [TRTLLM-15036][test] Add Kimi K3 GSM8K/MMMU accuracy tests and register them in QA's weekly multinode list by @moraxu in #17922
  • [https://nvbugs/6644468][fix] Defer the four runner imports to their sole isinstance use site (reached… by @trtllm-agent in #18144
  • [https://nvbugs/6384357][fix] Fail stop distributed warmup errors by @pranav-nvidia in #17667
  • [https://nvbugs/6681216][infra] Waive DeepSeek V3 Lite NIXL on B200 and B300 by @chienchunhung in #18325
  • [https://nvbugs/6650463][fix] Let CutlassFusedMoE serve MXFP8 under MiniMax SwiGLU by @xxi-nv in #18257
  • [None][feat] Make the Python KV-cache transceiver the default runtime by @nv-xtf in #18134
  • [None][refactor] Split connector KV save out of _send_kv_async by @nv-xtf in #18186
  • [None][test] Update GB300 MiniMax-M3 waive by @jieli-matrix in #18332
  • [None][perf] Fuse relu2 into a single elementwise kernel by @nv-guomingz in #18082
  • [None][fix] Keep CuTe-DSL MLA decode for Kimi K3 H=96 speculative-verify batches by @moraxu in #18226
  • [None][infra] Cache merge request file changes for CBTS by @crazydemo in #18309
  • [None][feat] Add perf-analyze and perf-optimize workflows to agent-flow by @kaiyux in #18330
  • [https://nvbugs/6581071][fix] Pass the SM explicitly as is_nvfp4_marlin_supported_sm(get_sm_version()) so… by @trtllm-agent in #17502
  • [None][perf] SM120 optimizations for Qwen3.6-35B-A3B-NVFP4 by @amukkara in #17976
  • [None][feat] Enable CuTe DSL MLA with Helix by @mingyangHao in #18131
  • [TRTLLM-15314][fix] recalculate PEFT cache capacity for adapter dtype by @achartier in #18200
  • [TRTLLM-14116][feat] Add DeepSeek-V4 Hopper support by @lfr-0531 in #16940
  • [None][feat] Cosmos3 action generation by @ishovkun in #17325
  • [None][infra] Retry SLURM agent online timeouts by @Mgluhovskoi in #17586
  • [None][test] Don't force disable_overlap_scheduler for perf-sanity ctx_only tests by @chenfeiz0326 in #18346
  • [None][feat] Consolidate the DSpark draft paths and support standalone drafters by @dc3671 in #18043
  • [https://nvbugs/6670227][fix] Drop redundant min_tokens from DSv4-Pro token-boundary smoke by @zhaoyangwang-nvidia in #18306
  • [https://nvbugs/6337228][fix] In tests/unittest/tools/test_layer_wise_benchmarks.py, replace check_call with… by @tensorrt-cicd in #15497
  • [None][doc] Add human-in-the-loop section to modeling-bringup README by @WeiHaocheng in #18358
  • [TRTLLM-15498][perf] optimize Kimi KDA prefill convolution data flow by @jiaganc in #17870
  • [None][infra] Declare and pin CI-imported deps to prevent transitive drops by @brnguyen2 in #18033
  • [https://nvbugs/6571418][test] Unwaive DeepSeek-V4-Pro GSM8K accuracy by @lfr-0531 in #18189
  • [TRTLLM-15585][feat] Wire SkipSoftmax sparse attention into the CuTeDSL backend by @karljang in #17781
  • [#17016][feat] Add selectable OpenEngine gRPC server by @connorcarpenter15 in #17084
  • [https://nvbugs/6480621][test] Revert to 60-second KV transfer timeout for GB300 DeepSeek V4 Pro disaggregated perf-sanity by @chienchunhung in #17137
  • [TRTLLMINF-324][infra] Extend infra-scoped fail-fast to the build→consumer edge by @brnguyen2 in #17799
  • [https://nvbugs/6669206][test] Unwaive test_kimi_k3_gen_dep[1] fixed by #18226 by @brnguyen2 in #18371
  • [None][test] Unwaive test_trtllm_bench_llmapi_launch for nvbugs/6568058 by @allisonlim-nv in #18203
  • [https://nvbugs/6644644][fix] Replay both verified orphan payloads onto current origin/main as a single… by @trtllm-agent in #18142
  • [None][chore] Refine runtime CODEOWNERs by @mikeiovine in #18318
  • [TRTLLM-14604][fix] add auth for RL endpoints by @hchings in #17434
  • [TRTLLM-14764][feat] trtllm-serve: Kimi K3 API compliance for the Kimi Vendor Verifier (KVV) by @moraxu in #17845
  • [None][ci] Waive flaky TestGemma3_1BInstruct::test_auto_dtype[False] disagg test by @brnguyen2 in #18390
  • [https://nvbugs/6641268][fix] Use fallback FMHA for SM103 context attention by @yuxianq in #18354
  • [https://nvbugs/6581049][ci] Unwaive test by @2ez4bz in #18394
  • [https://nvbugs/6663281][fix] Fix Gemma4 video token counting by @2ez4bz in #18274
  • [TRTLLM-15376][test] expand Gemma 4 MTP QA coverage by @jieli-matrix in #18218
  • [TRTLLM-15316][feat] sm107 gemm + quant by @farazkh80 in #17485
  • [TRTLLMINF-336][infra] BOLT profile merge: skip runtime tensorrt import gate (libs-only) by @mlefeb01 in #18395
  • [TRTLLM-15405][refactor] BREAKING: Remove TRTLLMSampler and sampler_type by @zhaoyangwang-nvidia in #18232
  • [TRTLLM-14843][chore] Establish _torch/moe/ and relocate MoE modules, custom ops, and communication by @lori-ren in #17952
  • [https://nvbugs/6640776][fix] Bump CUTLASS DSL to 4.6.2 to unblock FA4 split-KV by @trtllm-agent in #18145
  • [None][test] Unwaive GB300 MiniMax M3 NVFP4 test by @jieli-matrix in #18400
  • [None][chore] Update trtllm-gen FMHA kernels and cubins by @kris1025 in #18183
  • [None][chore] Apply clang-format to FMHA kernel header by @chzblych in #18417
  • [https://nvbugs/6650388][chore] Unwaive TestDeepSeekV32 nvfp4 chunked_prefill and piecewise_cuda_graph on B200 by @zhaoyangwang-nvidia in #18304
  • [https://nvbugs/6647405][fix] Do not sleep out the KV transfer poll interval with no in-flight session by @chuangz0 in #18175
  • [None][feat] Support custom masks in TRTLLM attention by @yuxianq in #16214
  • [#18338][fix] Fix barrier-divergence races in the Blackwell CuTe DSL GVR top-k decode kernel by @zkyue in #18339
  • [None][test] Add MiniMax-M3 disaggregated perf recipes to QA multi-node list by @fredricz-20070104 in #18416
  • [https://nvbugs/6384625][fix] Unwaive testcase by @YihuiLu512 in #18286
  • [https://nvbugs/6633928][test] Unwaive Gemma3 FP8 accuracy test on H100 by @erictsai-nv in #18121
  • [None][test] key perf-sanity case identity on test case name by @chenfeiz0326 in #18408
  • [TRTLLMINF-339][infra] enable BOLT profile overlay on published container images by @mlefeb01 in #18279
  • [https://nvbugs/6402009][test] unwaive Qwen3-235B NVFP4 TRTLLM MoE test by @jiahanc in #18344
  • [None][feat] KVCacheManagerV2: suspend/resume observability stat + coverage by @eopXD in #16710
  • [TRTLLM-15883][infra] clarify CBTS fallback diagnostics by @crazydemo in #18422
  • [None][test] Add AgentX DeepSeek-V4-Pro-DSpark perf-sanity lanes on GB300 by @chenfeiz0326 in #18298
  • [None][feat] Prefer POSIX FD handle type for KV cache V2 VMM allocations by @chuangz0 in #18347
  • [None][chore] Bump version to 1.3.0rc26 by @yuanjingx87 in #18402
  • [TRTLLMINF-371][infra] Run pre-commits checks on all files if there are more than 2500 changed files by @yiqingy0 in #18441
  • [None][infra] Add GB300 multi-node post-merge stages for Qwen3.8-2.4T-A95B NVFP4 accuracy by @Wanli-Jiang in #18363
  • [TRTLLM-11412][feat] Add offloading support for visual_gen by @rahul-steiger-nv in #14095
  • [TRTLLM-12680][feat] add exit code telemetry by @Mgluhovskoi in #17377
  • [None][refactor] Harden the KvCacheTransceiver contract and add a conformance fake by @nv-xtf in #18178
  • [TRTLLM-14575][perf] Batch DSA cross-layer index remap into one kernel launch per indexer group by @xwang233 in #18391
  • [TRTLLM-14778][perf] Add feature-mode encoder CUDA graphs for fixed-shape encoders (Whisper) by @pranav-nvidia in #17030
  • [https://nvbugs/6698722][test] Waive Gemma3 1B disagg KV-cache-v2 NIXL flaky test by @brnguyen2 in #18458
  • [TRTLLM-14880][feat] qualify Qwen3 dense for MX by @chienchunhung in #17142
  • [TRTLLM-14705][fix] Kimi K3 B200 enablement: MLA decode dispatch fix, L0 wiring, docs by @moraxu in #18164
  • [None][doc] Remove invalid html_inline entry from myst_enable_extensions by @nv-guomingz in #18427
  • [https://nvbugs/6633929][fix] Fix XGrammar structural tag doc links and pin the source link by @nv-guomingz in #18430
  • [https://nvbugs/6676032][doc] Give a runnable lm_eval install command in the trtllm-eval guide by @nv-guomingz in #18431
  • [https://nvbugs/6676066][doc] Clone examples/llm-api before running the multimodal quick start by @nv-guomingz in #18429
  • [https://nvbugs/6626445][fix] Narrow Wan2.2 I2V test fixture scope to reduce peak GPU memory by @o-stoner in #18328
  • [None][doc] Update GLM-5 docs for GLM-5.3 and disaggregated serving by @brnguyen2 in #18388
  • [https://nvbugs/6384357][test] Unwaive DeepSeek-V3.2 DSA host cache offload mtp params by @pranav-nvidia in #18324
  • [TRTLLM-15800][feat] bench_moe: hybrid MoE TP x EP parallel modes by @guqiqi in #18285
  • [None][perf] optimize DSpark attention by @mingyangHao in #18061
  • [https://nvbugs/6676511][fix] Reject unsupported speculative outputs by @2ez4bz in #18401
  • [TRTLLM-14575][perf] Use native next_n for DSA paged-MQA on Blackwell by @xwang233 in #18383
  • [https://nvbugs/6644487][fix] Unwaive one case which might be fixed by @EmmaQiaoCh in #18192
  • [None][feat] Add perf-analyze and perf-optimize skills to agent-flow by @kaiyux in #18434
  • [None][perf] Convolve the Mamba2 prefill on a channel-last projection by @nv-guomingz in #18081
  • [TRTLLM-11446][feat] trtllm-eval visual-gen generation evaluation pipeline by @yibinl-nvidia in #18199
  • [None][infra] Lower GB300 16-GPU accuracy test timeout to 90 minutes by @Wanli-Jiang in #18480
  • [https://nvbugs/6649386][chore] Unwaive disaggregated deepseek_r1_v2_fp4_mtp_stress on B200 by @zhaoyangwang-nvidia in #18348
  • [TRTLLM-15820][feat] Enable Nemotron Super 3.5 VL video input by @eopXD in #18370
  • [None][fix] register beneficial-to-skip contributions only after scheduling by @eopXD in #18202
  • [https://nvbugs/6697142][fix] Wire the stub the way TrtllmAttention.forward does and parametrize over both… by @trtllm-agent in #18504
  • [TRTLLM-15405][refactor] Remove the C++ decoder stack behind TRTLLMSampler by @zhaoyangwang-nvidia in #18233
  • [None][perf] Merge per-iteration attention-DP host allgathers in input prep by @qiaoxj07 in #18293
  • [None][fix] Support MiniMax-M3 vision meta init by @peihu-nv in #18466
  • [https://nvbugs/6657468][fix] Set UCX_TLS for GB300 in disagg logprobs test by @moraxu in #18373
  • [https://nvbugs/6699646][fix] Preserve prepared Cosmos3 video frames by @ishovkun in #18464
  • [TRTLLM-15035][test] Wire Kimi K3 spec-dec and suffix-automaton tests into L0 CI by @moraxu in #17921
  • [https://nvbugs/6608382][test] Mitigate disagg cancellation test hangs by @chienchunhung in #18326
  • [None][fix] use prompt lookahead for MTP Eagle chunked prefill by @yizhang-nv in #18295
  • [None][perf] Cache FMHA library selection in TrtllmAttention by @yuxianq in #18350
  • [https://nvbugs/6705034][test] waive Kimi KDA empty prefill test on B200 by @Mgluhovskoi in #18549
  • [https://nvbugs/6681216][fix] Raise disagg nixl readiness timeout on Blackwell by @brnguyen2 in #18372
  • [None][feat] Add locality domain Python layer by @zhangcl in #18317
  • [https://nvbugs/6633931][fix] Size the SM90 fp8 block-scale MoE workspace for the buffers it actually uses by @pranav-nvidia in #18323
  • [None][perf] Optimize MiniMax-M3 MXFP8 GEMMs by @peihu-nv in #17238
  • [None][feat] Add NVFP4 as a cold-page KV Cache Compression Method by @Hudayday in #18091
  • [https://nvbugs/6661914][fix] restore Wan 5B per-token AdaLN with TeaCache by @chang-l in #18209
  • [TRTLLM-12680][fix] stabilize telemetry lifecycle follow-ups by @Mgluhovskoi in #18454
  • [TRTLLMINF-372][infra] BoltProfileGen: parallel chunked artifact downloads by @mlefeb01 in #18424
  • [TRTLLM-15316][feat] Rubin trtllmgen batchedGemm MoE by @farazkh80 in #17707
  • [None][perf] Widen MiniMax-M3 decode q in-register instead of in a copy by @brb-nv in #18474
  • [None][test] Unwaive qwen3.5 test cases by @nv-guomingz in #18511
  • [TRTLLM-15498][perf] optimize Kimi KDA decode data flow by @jiaganc in #18251
  • [None][fix] Minimize and account for attention-DP dummy tokens by @liji-nv in #18308
  • [https://nvbugs/6633931][fix] Chunk the MoE workspace on the SM90 branch of TestDeepSeekV32::test_fp8_blockscale by @pranav-nvidia in #18322
  • [TRTLLM-15040][test] Prune legacy Llama and Nemotron tests by @xinhe-nv in #17858
  • [None][infra] Expand CBTS Tier 2 pilot to QA team by @crazydemo in #18442
  • [None][fix] Enforce the count-crossing invariant in the self-sampling top-k register family by @zkyue in #18501
  • [None][test] Enable warmup request for disagg e2e and ctx_only perf sanity lanes by @chenfeiz0326 in #18432
  • [https://nvbugs/6698723][fix] Unwaive tests after model config fix by @ZhanruiSunCh in #18536
  • [https://nvbugs/6467691][fix] Raise Starlette security floor by @BowenFu in #17650
  • [TRTLLM-14575][perf] Fuse DSA decode metadata into a single Triton kernel by @xwang233 in #18268
  • [TRTLLM-14575][perf] Enable fused-A GEMM for the GLM MLA A-projection by @xwang233 in #18265
  • [None][test] Add coverage for KimiLinearForCausalLM._setup_helix_mappings and related paths by @StanleySun639 in #18491
  • [https://nvbugs/6668807][fix] Remove invalid GLM KV cache assertion by @yizhang-nv in #18374
  • [None][chore] Remove AutoDeploy (ad-) skills and agents by @kaiyux in #18572
  • [TRTLLM-14867][feat] Auto-tuning for split-K in LoRA grouped GEMM by @AlessioNetti in #17315
  • [None][fix] Fix INFRA-DEFER stage status and multi-GPU blocking by @ZhanruiSunCh in #18305
  • [None][doc] Split LLM API reference into smaller searchable pages by @chzblych in #18440
  • [None][perf] Emission-assisted GVR top-K decode for the DeepSeek V4 indexer by @siyidNV in #16953
  • [TRTLLMINF-353][infra] add maintenance stage config by @hanjingtian in #18508
  • [None][infra] Tighten CBTS testdef anchors for deletion-only test diffs by @yufeiwu-nv in #18543
  • [TRTLLMINF-160][infra] Support CBTS test result reuse by @hanjingtian in #18073
  • [TRTLLM-15889][fix] Proper lazy import for MPI worker submit by @tongyuantongyu in #18284
  • [https://nvbugs/6602094][fix] Avoid MPI bootstrap in RocketKV unit test by @lfr-0531 in #18593
  • [TRTLLM-14715][feat] preserve native MoE A2A graph VAs across restore by @hhzhang16 in #16632
  • [TRTLLM-15498][perf] prepare Kimi K3 prefill metadata by @jiaganc in #18064
  • [TRTLLM-15888][fix] Proper lazy import for post transform profile matching by @tongyuantongyu in #18236
  • [None][fix] Fix Qwen3.8-27B NVFP4 checkpoint loading by @jupiterepoch in #18387
  • [TRTLLM-12499][feat] Pipelined KVCache transfer for disaggregated serving in Python Cache Transceiver by @athena-nv in #15727
  • [None][feat] Add Rubin SM107 CuTe DSL foundation and BF16 kernels by @farazkh80 in #18369
  • [TRTLLM-15448][perf] Implement rank-striped checkpoint read-ahead by @chienchunhung in #17393
  • [TRTLLM-14959][refactor] declare MoE backend capabilities instead of checking exact classes by @xxi-nv in #18409
  • [https://nvbugs/6693989][fix] Apply the K3 prompt-token offset only when a prompt count is actually known… by @trtllm-agent in #18509
  • [TRTLLMINF-373][infra] BoltProfileGen: persistent version-keyed llvm-bolt cache by @mlefeb01 in #18425
  • [TRTLLMINF-374][infra] BoltProfileGen: log SLURM queue-vs-run timing by @mlefeb01 in #18426
  • [None][ci] waive pre-existing test failures on main by @brnguyen2 in #18460
  • [TRTLLM-15316][feat] Fix fused mHC Phase-4 coherence and support uneven split-K by @farazkh80 in #18331
  • [TRTLLMINF-376][infra] default BoltProfileGen aarch64 platform to gb300-flex-oci-jhb by @mlefeb01 in #18469
  • [TRTLLMINF-365][infra] BoltProfileGen: run the merge job on the CPU partition by @mlefeb01 in #18462
  • [None][feat] GVR V2 decode top-k goes hint-free by default: the bracket comes from the current row by @longcheng-nv in #18410
  • [TRTLLM-15701][feat] Add branch-point Mamba state snapshots for hybrid KV cache prefix reuse by @athena-nv in #18272
  • [TRTLLMINF-316][infra] Use authenticated token for GitHub fetch in build and test jobs by @weiminwang-nv in #17686
  • [None][infra] Avoid false CBTS follow-up for test lists by @crazydemo in #18573
  • [None][perf] Enable configurable MLA skip correction for TRT-LLM Gen FMHA by @yunruis in #18118
  • [https://nvbugs/6693811][test] Unwaive DeepSeekV3Lite disagg guided_decoding mtp test by @chuangz0 in #18503
  • [https://nvbugs/6693811][test] Unwaive DeepSeek V3 Lite guided decoding by @chienchunhung in #18606
  • [https://nvbugs/6700260][fix] Fix Gemma4 TRTLLM FMHA fallback selection by @yuxianq in #18544
  • [None][chore] Remove unused KV cache draft-token rewind helpers by @zhaoyangwang-nvidia in #18533
  • [TRTLLM-15160][feat] Add post-attention o_proj gate hook to shared MLA base and make trtllm::kda_decode inplace-only by @WeiHaocheng in #18375
  • [None][feat] Support response_format='path' on /v1/images/edits and report Server-Timing total by @luyiyun1021 in #18444
  • [None][fix] Support fixed-only draft KV cache budgets by @yizhang-nv in #18459
  • [None][perf] serve: always use msgspec msgpack for disagg orchestrator->worker body by @Tabrizian in #16873
  • [https://nvbugs/6646568][chore] Make DeepGEMM in-kernel barrier timeout configurable by @longlee0622 in #18179
  • [None][doc] Add video generation optimization blog by @bobboli in #16506
  • [TRTLLM-15277][feat] BREAKING: Redesign VisualGen reference inputs and server-to-worker IPC by @luyiyun1021 in #17493
  • [None][chore] Update flashinfer-python from 0.6.16 to 0.6.18 by @nv-guomingz in #18428
  • [None][test] check GitHub links in documentation by @StanleySun639 in #18591
  • [TRTLLMINF-356][infra] Upload results at the end of stage no matter whether the peridic uplo… by @yiqingy0 in #18355
  • [TRTLLM-14575][perf] Add env-gated BF16 DSA indexer projection by @xwang233 in #18264
  • [TRTLLM-15821][feat] Enable Nemotron Super 3.5 VL NVFP4 variants by @eopXD in #18526
  • [TRTLLM-13409][fix] Count async KV completions as benchmark fill progress by @qiaoxj07 in #18456
  • [TRTLLM-14645][fix] VisualGen: allow sm_107a in CuTe DSL FMHA and guard unsupported SAGE SM by @zhenhuaw-me in #18476
  • [https://nvbugs/6670614][fix] Set UCX_TLS for Ray disaggregated serving tests by @chuangz0 in #18301
  • [TRTLLM-15751][refactor] Extract the MM-encoder item-scheduling facet into engine/multimodal.py by @lori-ren in #18524
  • [None][fix] Normalize perf metrics clocks across frontend processes by @lishicheng1996-nv in #17491
  • [None][doc] Fix stale Wide-EP EPLB documentation by @xxi-nv in #18626
  • [https://nvbugs/6602094][chore] Unwaive RocketKV test_model tests by @lfr-0531 in #18603
  • [https://nvbugs/6693991][fix] Restore verl rollout after sampler_type removal by @zhaoyangwang-nvidia in #18513
  • [None][fix] reject incompatible cached low-M GEMM tactics by @Wanli-Jiang in #18635
  • [TRTLLMINF-316][infra] Use authenticated token for GitHub fetch in docker image build by @weiminwang-nv in #17788
  • [https://nvbugs/5859751][fix] Reject NVFP4 MoE layers with unaligned gated FC1 by @dc3671 in #18520
  • [None][feat] Add NVFP4 KV cache support for DSA by @Tracin in #17681
  • [https://nvbugs/6698606][fix] Keep quantized KV context on TRTLLM-Gen FMHA by @yuxianq in #18545
  • [None][ci] Add focused CBTS coverage for OpenEngine by @tanmayv25 in #17843
  • [None][refactor] Add DisaggTransferCoordinator skeleton and loop transcript tests by @nv-xtf in #18595
  • [https://nvbugs/6566734][test] Unwaive test_disaggregated_qwen3_32b_fp8 and accept both greedy completions by @moraxu in #18319
  • [https://nvbugs/6563482][fix] Always exclude mamba conv1d from quantization by @aswinvisva in #18607
  • [TRTLLM-14616][feat] add VisualGen fp8 row-wise quant + autotuner by @o-stoner in #16847
  • [TRTLLM-14881][feat] qualify Mistral dense for MX by @moraxu in #18558
  • [TRTLLMINF-336][infra] enable BOLT premerge consume scaffolding (default off) by @mlefeb01 in #18622
  • [None][fix] Self-sampling top-k drops +inf from the top-k (register family) by @longcheng-nv in #18625
  • [None][perf] Seed the GVR prior from device-side prefill lengths by @longcheng-nv in #18646
  • [https://nvbugs/6655359][test] Shrink the Cosmos3-Nano T2V LPIPS gate to 9 frames, relax T2V/V2V thresholds, and unwaive by @ishovkun in #18550
  • [https://nvbugs/6655987][fix] Isolate FlashInfer JIT workspaces across MPI processes by @chienchunhung in #18278
  • [None][fix] Guard Python KV receive ownership and publication by @chienchunhung in #17720
  • [TRTLLM-15822][feat] Forward speculative-decoding state through the Nemotron VL wrapper by @eopXD in #18647
  • [None][test] validate GLM-5.2 feature support matrix by @cascade812 in #18470
  • [https://nvbugs/6692197][perf] Reuse FlashInfer plans for single-token decode by @lfr-0531 in #18596
  • [None][fix] Handle refused attention DP pad dummy on legacy ranks by @brb-nv in #18157
  • [None][feat] Patch DeepEP Commit To Support TopK 16 by @jloftin-nv in #17984
  • [None][fix] Use HND mapping for MiniMax-M3 MSA KV cache by @peihu-nv in #17374
  • [None][perf] Give the K3 KDA prefill conv input its layout without a repack by @ziyixiong-nv in #18643
  • [TRTLLM-14715][feat] preserve MNNVL all-reduce graph VAs across restore by @hhzhang16 in #16635
  • [None][test] Declarative extra import path for test sources by @tongyuantongyu in #17884
  • [https://nvbugs/6686534][fix] Keep disagg worker liveness within its TTL window by @trtllm-agent in #18403
  • [https://nvbugs/6581049][fix] Slightly lower accuracy threshold by @2ez4bz in #18642
  • [None][infra] Add blossom-ci authorized users by @yiqingy0 in #18708
  • [None][chore] Extract FMHA manager from TrtllmAttention by @yuxianq in #18548
  • [None][feat] Kimi k3 Support bcg by @GuanhuaWang2001 in #17816
  • [TRTLLM-15947][refactor] BREAKING: Remove C++ state left dead by the TRTLLMSampler removal by @zhaoyangwang-nvidia in #18532
  • [None][doc] Document FMHA availability contract by @yuxianq in #18710
  • [None][feat] perf-sanity: upload per-request disagg lifecycle spans to OpenSearch by @chenfeiz0326 in #18445
  • [None][perf] Use single-op int64 exchanges for per-iteration attention-DP host collectives by @qiaoxj07 in #18639
  • [None][fix] Fix window vector layer indexing by @yuxianq in #17466
  • [None][feat] add Qwen3.8-Flash-Next support by @Wanli-Jiang in #18585
  • [https://nvbugs/6193837][fix] Include FINALIZE-fusion workspace for SM>=90 in the MoE autotuner by @farazkh80 in #18675

New Contributors

Full Changelog: v1.3.0rc25...v1.3.0rc26

Check out latest releases or
releases around NVIDIA/TensorRT-LLM v1.3.0rc26

Don't miss a new TensorRT-LLM release

NewReleases is sending notifications on new releases.

Get notifications