github NVIDIA/TensorRT-LLM v1.4.0rc1
on GitHub

pre-release4 hours ago

Highlights

  • Known Issues

    • Disaggregated serving with the default Python KV cache transceiver may fail generation requests with a KV cache transfer error after streaming requests are cancelled (observed on A100).
    • The OpenAI JSON-schema serving example (examples/serve/openai_completion_client_json_schema.py) may return an unbounded response with Qwen3 models.

  • Model Support

    • Add DeepSeek-V4.1-Flash support #19727
    • Support NVFP4 GLM-5.3-Flash checkpoints #20127
    • Add Qwen3.8-Flash-Next NVFP4 as an experimental Modeling V2 target #20143
    • Support LiLiCorr draft checkpoints in DFlash #19856
    • Remove the automatic W4A16 fallback for Nemotron-H NVFP4 checkpoints on Hopper #15772

  • API

    • Default to KV cache manager V2 and the Python KV cache transceiver (BREAKING) #17495
    • Remove unused LlmRequest fields (BREAKING) #19967
    • Replace the C++ ArrayView API with std::span #20205
    • Select Modeling V2 through LLM configuration and arguments #20077
    • Publish serving configuration schemas #19906
    • Add a KV cache lending API for transfer backends #19722
    • Expose native KV event streaming and routing load #19440
    • Support runtime sleep and wakeup through LLM and trtllm-serve, including MiniMax-H3 #19016, #19604
    • Unify compiled-artifact cache configuration through TRTLLM_CACHE_DIR #18897
    • Make KV cache V2 pool rebalancing parameters configurable #19077
    • Reorganize VisualGen attention modules, keeping deprecated import paths and lazy exports #20214, #20215, #20218

  • Feature

    • Integrate the Mooncake connector with KV cache manager V2 #19171
    • Fill the KV cache V2 host tier progressively #19650
    • Add RoPE precision options for NVFP4 cold-page compression #19391
    • Honor per-request seeds in batched TorchSampler and one-model speculative acceptance #19719, #20147
    • Add ragged DSpark verification with confidence-based policy and calibrated scoring #19707, #19818, #20071
    • Add deterministic FP32 GVR V2 top-K decoding and reduce its overflow overhead #20100, #20290
    • Add an opt-in setting to compile only piecewise CUDA graph forwards #18366
    • Support Rubin locality-domain execution and extend SM107 and multi-node MegaMoE initialization #19272, #19183
    • Optimize Kimi K3 on Rubin and avoid cuBLAS fallback in KDA prefill #19960, #20084
    • Support packed query heads and compact inputs in Blackwell CuTe DSL MLA #19338
    • Add PrimTS dense BF16, mixed BF16/FP8, and FP8 context attention paths #20006
    • Use paged attention for MiniMax-M3 dense layers and cut NVFP4 KV host overhead #20159, #19974
    • Limit GLM-5.3-Flash indexer scoring to visible pools with BF16 tensor cores #20251
    • Add token-sharded tensor parallelism for Wan in VisualGen #19713
    • Support MiniMax-H3 parallel tiled VAE decoding and Ulysses parallelism #19506
    • Speed up MiniMax-H3 video-VAE decoding and fuse AdaLN RMSNorm modulation #19774, #19772
    • Unify multimodal encoder caching and reduce encoder output copies #17876, #19799
    • Optimize Qwen vision encoders and decode JPEGs through torchvision #19797, #19794
    • Reduce per-response overhead in LLM API streaming #20244
    • Enable SwiGLU clamping and update NVFP4 FC1 weight layout in Rubin CuTe DSL MoE #19488, #20181
    • Overhaul the NVLink one-sided all-to-all #19610

  • Fix

    • Align MiniMax-M3 and GLM reasoning and tool parsing between streaming and non-streaming #20442, #19935
    • Flush the Hugging Face streaming detokenizer on the final response #20424
    • Restore trtllm-serve streaming throughput by constraining uvloop below 0.23.0 #19992
    • Constrain FastAPI to versions below 0.140.0 #20161
    • Keep the overlap scheduler responsive while waiting for disaggregated KV receives #19857, #19940
    • Reserve retiring-request slot headroom for overlap scheduling without attention data parallelism #20250
    • Correct draft KV scheduling and prompt-tail recomputation on cache hits with overlap speculation #19745, #19748
    • Size draft-probability buffers by batch capacity and fix speculative beam-search races #19490, #20064
    • Size Blackwell speculative-decoding tree masks for the attention kernel layout #20223
    • Guard Whisper cross-KV reuse when encoder tokens are unavailable #19302
    • Preserve LoRA cache contents when too few pages can be evicted #20212
    • Reclaim idle eager CUTLASS MoE workspace #19981
    • Correct CuTe DSL MoE autotuning inputs and restore FC2 N-tile autotuning #17410, #20277
    • Fix W4A16 NVFP4 MoE dequantization and expert-load-balancing migration, including Nemotron-H #19929
    • Restore routing-weight dtype after NCCL expert-parallel dispatch #20081
    • Enable SM107 NVFP4 CuTe DSL dispatch and fix CUDA-core GEMM for wide FP8 outputs #20204, #20271
    • Enable 96-head MLA in the FMHA autotuner and fix CuTe MLA page-table bounds #18599, #20256
    • Prevent the SM120 DSA KV append crash from a missing stride factor #20062
    • Bound packed indexer prefill logits and use 64-bit offsets for indexer K-cache #20164, #20190
    • Keep DeepSeek-V4 tensor-parallel decode all-reduces off NCCL #20233
    • Avoid deep-copying pending media loads in the DeepSeek-V4 tokenizer #20191
    • Fence Helix all-to-all shared writes before asynchronous bulk copies #20221
    • Match KDA decode fallback layout to prefill #19987
    • Preserve indexed GDN state during Triton autotuning and align recurrent Q/K normalization #19859, #19860
    • Propagate client import paths to remote MPI workers and refresh stale worker environments #19494, #20151
    • Drive MPI progress from a native thread on pipeline-parallel ranks #20030
    • Apply phase-appropriate MoE communication timeouts during warmup #20246
    • Keep single-rank device-mesh topology separate from the job-wide mesh #20021
    • Stop the statistics RPC server when its executor worker shuts down #20158
    • Route FLUX.2 linear_in quantization scales correctly when splitting gate and up weights #20403
    • Fix correctness issues on Rubin and in Kimi K3, KDA, MNNVL, and DFlash paths #19959, #19988
    • Allocate DLPack managed tensors in one block to fix lifetime and shutdown issues #18310
    • Reject paged-context FMHA configurations without the required fused context kernel #19746
    • Honor models that require KV cache manager V2 instead of falling back to V1 #20093
    • Raise a clear error for prompt files with no usable prompts #17733

  • Documentation

    • Add Qwen3.5 GB300 agentic-serving guidance and curated configurations #20514
    • Add the MoE Performance Guide and performance dashboard #19546
    • Add a KV cache compression technical blog #20278
    • Document NGC PyTorch wheel installation #20281
    • Describe PyTorch as the sole execution backend and remove stale guidance #19986, #20350
    • Correct the VisualGen constructor in the sparse-attention example #20549

  • Test & Infra

    • Upgrade the NGC PyTorch base image to 26.09, retaining the 26.08 forward-compatibility driver #19679, #20252
    • Update dependency constraints for base-image vulnerability fixes #20175
    • Upgrade the Modeling V2 infrastructure #19977
    • Disable BOLT consumption in pre-merge and stop emitting cortex-a53 veneers in BOLT builds #20496, #20024
    • Fix build-system reconfiguration and move C++ tests next to their components #20037, #19886
    • Update trtllm-gen FMHA artifacts and refresh the PrimTS vendor source #20054, #20155
    • Simplify TorchSampler strategy grouping around the fused sampling kernel #20188
    • Add agent-flow orchestration for parallel performance and modeling agents #18876
    • Add GR100 single-GPU post-merge coverage #19894
    • Add Kimi K3 and MiniMax-M3 AgentX performance-sanity coverage and single-phase benchmark lanes #20163
    • Add ModelExpress post-merge accuracy canaries #18561
    • Warm up KV-transfer connections before disaggregated performance-sanity measurements #18433
    • Migrate model, attention, Nemotron, Mamba, Kimi, and disaggregated tests to KV cache manager V2 #20098, #20097, #20046, #20050, #19387
    • Harden disaggregated and multi-node tests with isolated outputs, UCX/NCCL settings, and NVLink-domain placement #20140, #20031, #20189, #19492, #20418
    • Run VisualGen Wan, Qwen-Image, and key-padding tests in CI and split LPIPS goldens by model #20004, #20063, #20198, #19237
    • Run guardwords and release checks in nightly releases and pass source info to PLC scans #20135, #19594, #20449
    • Skip test stages during maintenance, classify CI failures from step logs, and fix rerun JUnit reporting #18353, #19258, #20153

What's Changed

  • [None][fix] Refuse paged-context FMHA configurations whose fused context kernel is absent by @brnguyen2 in #19746
  • [None][feat] Split VisualGen LPIPS golden media by model family and refresh goldens by @yibinl-nvidia in #19237
  • [None][chore] update bench_moe situ support list by @leslie-fang25 in #20009
  • [None][fix] Enforce KV correction when overlap scheduling runs with speculation by @brnguyen2 in #19748
  • [None][infra] Retry the mypy type-check step in the build stage by @weiminwang-nv in #19589
  • [None][infra] Add blossom-ci authorized users by @trtllm-agent in #20025
  • [None][fix] Ignore virtualenv and editable-install entries in the pytest sys.path check by @lori-ren in #19872
  • [https://nvbugs/6812588][fix] Bound Qwen3.5 batch/seq len in multi-node disagg tests by @reasonsolo in #20019
  • [None][fix] GLM tool and reasoning parsers: streaming matches the whole-text parse by @JunyiXu-nv in #19935
  • [TRTLLM-10804][infra] Restore release checks in nightly releases by @niukuo in #19594
  • [https://nvbugs/6842946][test] Unwaive Qwen3-VL disaggregated KV cache tests by @yizhang-nv in #19895
  • [TRTLLMINF-474][infra] Upgrade NGC PyTorch base image to 26.09 by @EmmaQiaoCh in #19679
  • [https://nvbugs/6849051][fix] The shared mock helper sets get_fp4_mla_page_table_spec = None and both… by @trtllm-agent in #19790
  • [None][feat] Rubin merge-back: SM107 MoE, CFT release op, MegaMoE multi-node bootstrap, core runtime fixes by @reasonsolo in #19183
  • [TRTLLM-16667][feat] Add Minimax H3 Parallel Tiled VAE Decoding and Ulysses Support by @yibinl-nvidia in #19506
  • [TRTLLM-14620][feat] Add ragged DSpark verification semantics by @zhaoyuanh-nvidia in #19707
  • [None][fix] Admit unpaired draft KV mirrors at scheduling time and recompute a prompt tail on cache hits by @brnguyen2 in #19745
  • [None][perf] Format the AUX-delivery debug message only when debug lo… by @roborluo in #19939
  • [TRTLLM-16770][fix] Fix W4A16 NVFP4 MoE dequantization and EPLB migration, and add Nemotron-H to EPLB allowlist by @eopXD in #19929
  • [https://nvbugs/6791675][fix] Propagate client sys.path to remote MPI workers by @trtllm-agent in #19494
  • [https://nvbugs/6890897][test] Fix startup telemetry architecture expectation and unwaive test by @Mgluhovskoi in #20015
  • [TRTLLM-16793][feat] Support multiple OpenEngine gRPC frontend processes by @tanmayv25 in #19870
  • [TRTLLM-14620][feat] Add DSpark confidence verification policy by @zhaoyuanh-nvidia in #19818
  • [https://nvbugs/6278337][fix] Unwaive DeepSeekV3Lite bf16 PP4 tests by @YihuiLu512 in #20020
  • [https://nvbugs/6700265][test] Unwaive TestDeepSeekV3Lite test_fp8_block_scales by @allisonlim-nv in #20012
  • [https://nvbugs/6718911][fix] Fix stale disagg rendezvous after requeue and CI watchdog cancelling long perf-sanity runs by @chenfeiz0326 in #20000
  • [TRTLLM-12891][feat] Integrate Mooncake connector into KVCMv2 by @brb-nv in #19171
  • [None][chore] remove orphaned process_streaming_events helper by @lonexreb in #16262
  • [TRTLLM-16857][feat] Add RoPE precision options to NVFP4 cold-page compression by @Hudayday in #19391
  • [TRTLLM-16708][feat] Honor per-request seeds in batched TorchSampler sampling via the fused kernel by @lori-ren in #19719
  • [None][ci] waive pre-existing test failures on main by @brnguyen2 in #20074
  • [TRTLLM-13996][feat] Multimodal encoder cache unification by @yechank-nvidia in #17876
  • [TRTLLM-16621][feat] Support DeepSeek-V4.1-Flash by @ziyixiong-nv in #19727
  • [None][perf] use GEMM patch projection and reuse fused QKV in Qwen vision encoders by @yechank-nvidia in #19797
  • [None][infra] Waive 1 failed cases for main in pre-merge 63586 by @trtllm-agent in #20107
  • [None][test] Update MiniMax M3 EAGLE3 draft model directory by @fredricz-20070104 in #20076
  • [https://nvbugs/6818106][fix] Compare disaggregated logprobs on the KV reuse path by @yifeizhang-c in #19928
  • [None][perf] Decode RGB and grayscale JPEGs with torchvision in ImageMediaIO by @yechank-nvidia in #19794
  • [None][feat] Wire Rubin locality-domain execution through PyTorch models by @peaceh-nv in #19272
  • [https://nvbugs/6661863][fix] Allocate one PyMem_RawCalloc'd _DLPackBlock (managed tensor + inline… by @trtllm-agent in #18310
  • [None][feat] Upgrade Blackwell cuteDSL MLA kernel for packed q heads and compact input by @pengbowang-nv in #19338
  • [None][infra] Waive 1 failed cases for main in post-merge 3014 by @trtllm-agent in #20130
  • [None][feat] BREAKING: Default to KV cache manager V2 and Python transceiver by @yizhang-nv in #17495
  • [None][test] Prune Kimi-K2 accuracy tests by @xinhe-nv in #20026
  • [None][infra] Add TensorRT-LLM Release 1.4 section to release notes by @EmmaQiaoCh in #20142
  • [TRTLLM-16593][fix] Size expanded draft probabilities by batch capacity by @zhaoyangwang-nvidia in #19490
  • [https://nvbugs/6865617][fix] Poll the disagg gen-init KV receive fence instead of blocking the overlap scheduler by @trtllm-agent in #19857
  • [TRTLLMINF-481][fix] handle outdated unresolved review threads by @ZhanruiSunCh in #20148
  • [https://nvbugs/6838020][fix] Tolerate near-tie greedy flips in KV pool rebalance accuracy test by @thorjohnsen in #19809
  • [None][feat] Let a model require KV cache manager v2 instead of being demoted to v1 by @brnguyen2 in #20093
  • [TRTLLM-16956][feat] Modeling V2 dynamic switch with LLM config and args by @WeiHaocheng in #20077
  • [None][test] Split QA test lists by GPU marker requirements by @crazydemo in #20150
  • [None][fix] Keep a one-rank DeviceMeshTopology off the job-wide device mesh by @ishovkun in #20021
  • [TRTLLM-16901][fix] SID-GR optimization 1/N: Fix speculative beam d2h races by @mikeiovine in #20064
  • [None][chore] Update trtllm-gen FMHA artifacts by @achartier in #20054
  • [None][refactor] overhaul NVLink one-sided all-to-all by @zhangcl in #19610
  • [None][chore] enable cutedsl rubin fc1 swiglu clamp by @leslie-fang25 in #19488
  • [None][fix] bench_moe: release each candidate's MoE module before next one by @leslie-fang25 in #20123
  • [None][infra] Waive 2 failed cases for main in pre-merge 63757 by @trtllm-agent in #20179
  • [None][perf] Speed up MiniMax-H3 video-VAE decode losslessly by @karljang in #19774
  • [TRTLLM-16304][feat] Upgrade the infrastructure of modeling V2 by @WeiHaocheng in #19977
  • [https://nvbugs/6891541][fix] Stop the stats RPC server on GenerationExecutorWorker shutdown by @lori-ren in #20158
  • [TRTLLM-16602][fix] Make the waiting-for-feedback stale PR lifecycle bot-aware by @karljang in #19739
  • [None][infra] Waive 2 failed cases for main in post-merge 3014 by @trtllm-agent in #20118
  • [https://nvbugs/6877253][fix] Disagg MTP KVCache Transfer AL Test by @allisonlim-nv in #20122
  • [None][feat] Deterministic mode for the self-sampling GVR V2 top-K decode (fp32) by @longcheng-nv in #20100
  • [None][perf] reduce multimodal encoder output copies by @yechank-nvidia in #19799
  • [None][chore] Refresh PrimTS vendor source onto FlashInfer main by @yuxianq in #20155
  • [None][fix] Restore routing-weight dtype after NcclEP dispatch by @peaceh-nv in #20081
  • [None][test] Migrate model and executor fixtures to KV cache manager V2 by @yizhang-nv in #20098
  • [https://nvbugs/6658015][fix] Pin uvloop<0.23.0 to restore trtllm-serve streaming throughput by @chenfeiz0326 in #19992
  • [TRTLLM-17005][refactor] Component-local C++ Tests by @Funatiq in #19886
  • [None][infra] Waive 3 failed cases for main in pre-merge 63827 by @trtllm-agent in #20219
  • [https://nvbugs/6908030][test] Use IPv4 loopback for VisualGen test rendezvous by @luyiyun1021 in #20203
  • [None][fix] raise clear error for prompt file with no usable prompts by @lonexreb in #17733
  • [None][fix] Set use_kv_cache_manager_v2="auto" in required-V2 resolution tests by @thorjohnsen in #20177
  • [None][fix] Align GDN recurrent Q/K normalization with prefill by @shikicloud in #19860
  • [None][fix] Preserve indexed GDN state during Triton autotuning by @shikicloud in #19859
  • [None][infra] Waive 8 failed cases for main in pre-merge 63914 by @trtllm-agent in #20234
  • [https://nvbugs/6812588][fix] Apply the cluster UCX rules in multi-node disagg tests by @reasonsolo in #20189
  • [https://nvbugs/6762594][test] Run VisualGen key padding mask parity test in CPU CI by @luyiyun1021 in #20198
  • [https://nvbugs/6762590][test] Run Qwen-Image VisualGen unit tests in CI by @karljang in #20063
  • [https://nvbugs/6902353][chore] waive test_transfer_lifecycle_gpu by @tburt-nv in #20235
  • [TRTLLM-16212][feat] Unified caching via TRTLLM_CACHE_DIR environment variable by @AlessioNetti in #18897
  • [https://nvbugs/6891758][fix] Fence Helix all-to-all shared writes before async bulk copy by @JunyiXu-nv in #20221
  • [None][infra] switch mirrors from URM to PDX by @tburt-nv in #19862
  • [TRTLLM-16104][test] Add ModelExpress post-merge accuracy canaries by @moraxu in #18561
  • [None][feat] CUTEDSL: switch the Rubin NVFP4 FC1 weight layout to gate16/up16 by @leslie-fang25 in #20181
  • [https://nvbugs/6852440][fix] Set constraint to fastapi version by @HuiGao-NV in #20161
  • [TRTLLM-14620][feat] Integrate calibrated DSpark confidence scoring by @zhaoyuanh-nvidia in #20071
  • [https://nvbugs/6911105][fix] Remove sparse metadata preparation waiver by @ziyixiong-nv in #20220
  • [None][test] Complete encoder-decoder V2 test migration by @yizhang-nv in #20096
  • [None][feat] Support NVFP4 GLM-5.3-Flash checkpoints by @ruocheng-nv in #20127
  • [None][perf] Avoid cuBLAS's fallback in the KDA prefill projection by @ruocheng-nv in #20084
  • [None][infra] Vulnerability fix by @yuanjingx87 in #20175
  • [None][perf] Fuse MiniMax-H3 AdaLN RMSNorm with per-token modulation by @karljang in #19772
  • [None][fix] Fix issues in the build system during reconfiguration by @tongyuantongyu in #20037
  • [https://nvbugs/6762593][fix] Add untracked Wan test files to CI test lists by @o-stoner in #20004
  • [None][test] Migrate Nemotron and Mamba validation to KV cache V2 by @yizhang-nv in #20046
  • [TRTLLMINF-336][fix] Stop emitting cortex-a53-843419 veneers in BOLT builds by @mlefeb01 in #20024
  • [None][feat] Stream native KV events and expose routing load by @tanmayv25 in #19440
  • [TRTLLM-16315][infra] relax CBTS import-time coverage bounds by @crazydemo in #19013
  • [TRTLLM-16316][feat] Generalize Runtime Sleep/Wakeup for LLM and trtllm-serve by @Funatiq in #19016
  • [TRTLLM-16012][feat] Restrict torch.compile to piecewise graph capture only by @AlessioNetti in #18366
  • [TRTLLM-15863][feat] Export the Attention module-layer classes from _torch.attention lazily by @YihuiLu512 in #20218
  • [None][chore] Remove permanently skipped DeepSeek-V3-Lite CUTEDSL FP8 tests by @trtllm-agent in #20131
  • [TRTLLM-15864][chore] Consolidate the VisualGen attention domain under _torch/visual_gen/attention/ by @YihuiLu512 in #20214
  • [TRTLLMINF-381][fix] classify CI failures from originating step logs by @hanjingtian in #19258
  • [None][test] Drop fastapi upper-bound assertion from web framework security floors test by @HuiGao-NV in #20274
  • [https://nvbugs/6564113][test] Warm up KV-cache transfer connections before disagg e2e perf sanity by @chuangz0 in #18433
  • [TRTLLM-12288][feat] revert all the changes of the PR#14775 by @JadoTu in #15772
  • [https://nvbugs/6812332][fix] Reshape the KDA decode fallback core to match prefill by @yifeizhang-c in #19987
  • [None][doc] Add KV cache compression tech blog by @Hudayday in #20278
  • [None][fix] Kimi K3 / KDA / MNNVL / DFlash: fixes for defects on main by @vsabavat in #19988
  • [None][fix] Enable VMM CUDA IPC in Kubernetes GPU CI by @yizhang-nv in #20042
  • [TRTLLM-16993][fix] Preserve LoRA cache when evictable pages are insufficient by @zhaoyangwang-nvidia in #20212
  • [TRTLLM-16186][feat] Add KV cache lending API for transfer backends by @Shixiaowei02 in #19722
  • [None][infra] Waive 2 failed cases for main in pre-merge 64127 by @trtllm-agent in #20296
  • [None][infra] Waive 1 failed cases for main in pre-merge 64154 by @trtllm-agent in #20302
  • [TRTLLM-16770][feat] Support LiLiCorr draft checkpoints in DFlash by @eopXD in #19856
  • [None][infra] Waive 2 failed cases for main in pre-merge 64155 by @trtllm-agent in #20304
  • [None][test] Migrate ordinary disaggregated coverage to KV cache manager V2 by @yizhang-nv in #20050
  • [None][infra] Reuse PLC pipeline result if commit not changed by @yuanjingx87 in #19562
  • [None][fix] Fix SM120 DSA KV append crash on missing _cached_stride_factor by @nvchenghaoz in #20062
  • [None][fix] Merge back #19593 from release/1.3: Make MoE communication timeouts follow the warmup phase by @xxi-nv in #20246
  • [None][perf] Cut per-response overhead in the LLM API streaming result path by @jthomson04 in #20244
  • [TRTLLM-15864][chore] Add deprecated forwarding shims for the old VisualGen attention paths by @YihuiLu512 in #20215
  • [None][perf] MiniMax-M3: remove the host overhead that made NVFP4 KV slower than FP8 by @zheyuf in #19974
  • [None][test] Add a timeout to the subprocess.run call in test_bolt by @chenfeiz0326 in #20305
  • [https://nvbugs/6650817][fix] Drive MPI progress from a native thread on PP ranks by @YihuiLu512 in #20030
  • [https://nvbugs/6921137][fix] Fix CuTe MLA page-table batch bounds by @pengbowang-nv in #20256
  • [https://nvbugs/6728119][fix] Keep NCCL off PCIe P2P in disagg tests on pre-Hopper nodes by @chuangz0 in #19492
  • [TRTLLM-16628][refactor] BREAKING: Remove unused LlmRequest fields by @Funatiq in #19967
  • [TRTLLMINF-488][infra] Fix rerun junit reporting by @yiqingy0 in #20153
  • [https://nvbugs/6905381][fix] Fix CUDA-core GEMM launch for wide FP8 outputs by @eopXD in #20271
  • [None][feat] Develop agent flow orchestration and support parallel execution of modeling agent based on it by @WeiHaocheng in #18876
  • [TRTLLM-16460][fix] deepseek_v4 tokenizer: do not deep-copy the OpenAI server's pending media loads by @Wanli-Jiang in #20191
  • [None][test] Migrate attention fixtures and prims-ts validation to KV cache V2 by @yizhang-nv in #20097
  • [None][perf] Do not block the executor on KV bounce recv-region backp… by @roborluo in #19940
  • [None][fix] Rubin merge-back: correctness fixes by @reasonsolo in #19959
  • [None][perf] Score only visible pools on BF16 tensor cores in the GLM-5.3-Flash indexer by @ruocheng-nv in #20251
  • [TRTLLM-16460][fix] indexer K-cache gather/scatter: compute the contiguous-side offset in 64 bits by @Wanli-Jiang in #20190
  • [TRTLLM-10804][doc] Document NGC PyTorch wheel installation by @niukuo in #20281
  • [https://nvbugs/6878353][test] Unwaive DeepSeekV3Lite nvfp4 tp4 torch_compile test on DGX_B300 by @YihuiLu512 in #20282
  • [None][fix] Restrict large-model four-GPU perf cases to VR200 by @yufeiwu-nv in #20121
  • [TRTLLM-16336][feat] Add the qwen4_exp modeling_v2 target for Qwen3.8-Flash-Next-NVFP4 by @Wanli-Jiang in #20143
  • [None][test] Patch moe_comm_serving_timeouts in the phase-selective warmup test by @chenfeiz0326 in #20362
  • [None][chore] Remove ArrayView and use std::span instead by @yihwang-nv in #20205
  • [TRTLLM-16944][refactor] Simplify TorchSampler strategy grouping and fast path around the fused sampling kernel by @lori-ren in #20188
  • [None][test] Validate Kimi default KVCM V2 accuracy and disaggregated transfer by @yizhang-nv in #19387
  • [TRTLLMINF-353][infra] skip test stages during maintenance by @hanjingtian in #18353
  • [TRTLLM-14146][fix] Correct CuteDSL MoE autotune routing inputs by @Barry-Delaney in #17410
  • [None][infra] Waive 1 failed cases for main in pre-merge 64375 by @trtllm-agent in #20427
  • [None][feat] Publish serving configuration schemas by @Mgluhovskoi in #19906
  • [https://nvbugs/6800223][fix] Fix stale KvCacheCreator fixtures in budget split tests by @lowsfer in #20399
  • [TRTLLMINF-486][infra] Get sonar-scanner-cli from internal Artifactory and verify its SHA-256 by @sharabeshj-nvda in #20240
  • [None][infra] Reuse rank processes across visual_gen multi-GPU tests by @ishovkun in #20239
  • [https://nvbugs/6919428][fix] Keep DeepSeek-V4 TP decode all-reduces off NCCL by @chenfeiz0326 in #20233
  • [https://nvbugs/6919819][perf] Autotune the CuTeDSL FC2 N tile again when the override is unset by @chenfeiz0326 in #20277
  • [https://nvbugs/6920456][fix] Describe PyTorch as the sole execution backend in the architecture overview by @trtllm-agent in #20350
  • [TRTLLM-16992][perf] Use paged attention for MiniMax-M3 dense layers by @zhaoyangwang-nvidia in #20159
  • [None][feat] make KVCacheManagerV2 pool rebalancing parameters configurable by @thorjohnsen in #19077
  • [None][perf] primsTS dense bf16, qk-bf16/pv-fp8, fp8 fmha context bringup by @harrisonzhy in #20006
  • [None][feat] VisualGen: token-sharded tensor parallelism (tp_layout) for Wan by @chang-l in #19713
  • [None][test] Update the gemma4 and Qwen3.6 case for spark by @JennyLiu-nv in #20210
  • [https://nvbugs/6888251][test] Disable THP for KV cache V2 host tiers in disagg tests by @chuangz0 in #20032
  • [None][fix] Reduce deterministic GVR V2 top-K overflow overhead by @longcheng-nv in #20290
  • [TRTLLMINF-439][infra] Add GR100 post-merge single-GPU stage by @yiqingy0 in #19894
  • [https://nvbugs/6836304][fix] Forward-port stale MPI worker env fix to main and unwaive multimodal tests by @sunnyqgg in #20151
  • [https://nvbugs/6836097][test] Restrict UCX TCP to loopback in single-node disagg tests by @chuangz0 in #20031
  • [None][fix] Enable SM107 NVFP4 CuTe DSL dispatch by @peaceh-nv in #20204
  • [None][infra] Add blossom-ci authorized users by @trtllm-agent in #20461
  • [https://nvbugs/6691772][fix] Update trtllm-gen fmha autotuner to enable 96 head MLA by @pengbowang-nv in #18599
  • [None][test] Exclude Inductor subproc threads from integration threadleak check by @HuiGao-NV in #20294
  • [None][doc] Add MoE Performance Guide and Perf Dashboard by @guqiqi in #19546
  • [https://nvbugs/6913517][fix] Drop launcher spawn-proxy env in the disagg cancel marathon by @Shixiaowei02 in #20420
  • [https://nvbugs/6902353][chore] Waive test_transfer_lifecycle_gpu[destination_fatal] by @lowsfer in #20457
  • [https://nvbugs/6926547][fix] Route FLUX.2 linear_in quantization sca… by @yingguo-trt in #20403
  • [https://nvbugs/6893965][fix] Extend disagg server start timeout in multi-node disagg test by @reasonsolo in #20453
  • [TRTLLM-16780][feat] Honor per-request seeds in one-model speculative decoding's acceptance step by @lori-ren in #20147
  • [TRTLLM-15891][infra] Refactor CBTS Collection by @tongyuantongyu in #18299
  • [TRTLLM-16972][fix] Reclaim idle eager CUTLASS MoE workspace by @zhaoyangwang-nvidia in #19981
  • [TRTLLM-16619][doc] Remove stale execution backend guidance by @Funatiq in #19986
  • [https://nvbugs/6900756][fix] Bound packed indexer prefill logits by grouping context requests by @thorjohnsen in #20164
  • [None][feat] Add opt-in MiniMax-H3 sleep/wake support by @rystewart-nvidia in #19604
  • [https://nvbugs/6713231][fix] Guard Whisper cross-KV reuse without encoder tokens by @pranav-nvidia in #19302
  • [https://nvbugs/6902355][fix] Keep TopK rows in the DSA sparse MLA test KV pool by @lfr-0531 in #20283
  • [https://nvbugs/6811958][chore] Unwaive test_build_capture_manifest_matches_committed_golden by @farazkh80 in #20057
  • [None][chore] Remove is_first_draft dead code by @mikeiovine in #19867
  • [None][fix] disable bolt by @tburt-nv in #20496
  • [None][infra] Waive 1 failed cases for main in pre-merge 64667 by @trtllm-agent in #20478
  • [None][infra] Waive 1 failed cases for main in pre-merge 64680 by @trtllm-agent in #20500
  • [TRTLLM-10804][infra] Run guardwords scan in nightly releases by @niukuo in #20135
  • [https://nvbugs/6843484][fix] Keep multi-node aggregated perf-sanity jobs in one NVLink domain by @chenfeiz0326 in #20418
  • [None][feat] Kimi K3 optimizations on Rubin by @reasonsolo in #19960
  • [https://nvbugs/6713424][fix] Flush the HF streaming detokenizer on the final response; fix the AgentX aiperf install check by @chenfeiz0326 in #20424
  • [None][doc] Document Qwen3.5 GB300 agentic serving by @Ankur-singh in #20514
  • [None][infra] Waive 4 failed cases for main in pre-merge 64735 by @trtllm-agent in #20505
  • [https://nvbugs/6919405][fix] Keep the 26.08 CUDA forward-compat driver on the 26.09 image by @chenfeiz0326 in #20252
  • [https://nvbugs/6919819][fix] Give non-attention-DP overlap runs the retiring-request seat headroom by @chenfeiz0326 in #20250
  • [None][infra] Waive 1 failed cases for main in pre-merge 64813 by @trtllm-agent in #20523
  • [None][test] Kimi-K3 and MiniMax-M3 AgentX perf-sanity cases, AgentX single-phase lanes, drop gpt-oss by @chenfeiz0326 in #20163
  • [None][fix] Keep MiniMax-M3 streaming and non-streaming reasoning parsing in step by @pranav-nvidia in #20442
  • [None][perf] Fill the KVCM2 host tier progressively by @lowsfer in #19650
  • [None][feat] Add spec_dec_variant to telemetry by @mikeiovine in #20432
  • [None][fix] Pass source repository and commit to PLC container scans by @niukuo in #20449
  • [None][fix] Size the Blackwell spec-dec tree mask for the SwapsMmaAb kernel layout by @lori-ren in #20223
  • [https://nvbugs/6933917][docs] Fix VisualGen constructor in sparse at… by @yingguo-trt in #20549
  • [https://nvbugs/6899682][fix] Isolate disaggregated client outputs per test by @chuangz0 in #20140
  • [None][infra] Waive 1 failed cases for main in pre-merge 64813 by @trtllm-agent in #20563
  • [None][infra] Waive 1 failed cases for main in post-merge 3026 by @trtllm-agent in #20542

New Contributors

  • @harrisonzhy made their first contribution in #20006
  • @rystewart-nvidia made their first contribution in #19604
  • @Ankur-singh made their first contribution in #20514

Full Changelog: v1.4.0rc0...v1.4.0rc1

Check out latest releases or
releases around NVIDIA/TensorRT-LLM v1.4.0rc1

Don't miss a new TensorRT-LLM release

NewReleases is sending notifications on new releases.

Get notifications