github NVIDIA/TensorRT-LLM v1.4.0rc2
on GitHub

pre-release3 hours ago

Python package version

The Python wheels for this release are published as 1.4.0rc2.post1. Container images and documentation remain at 1.4.0rc2. Use 1.4.0rc2.post1 when installing the Python package.

The wheel-only patch removes a direct Git dependency that prevented PyPI publication (#21128) and updates the Boost download mirror used during builds (#21133). Patched wheel source: v1.4.0rc2.post1, commit 43f8f70da6415a3b06f91f9ebaa4fb7611ab62cc.

Highlights

  • Known Issues

    • Tensor-parallel serving with RPC orchestration may hang during model initialization when reusing an MPI worker pool across threads.

  • Model Support

    • Add Cosmos3-Edge Reasoner support for visual generation workloads #20307
    • Add GLM-5.2 NVFP4 low-latency decode on Rubin through Modeling V2 #20517
    • Enable MiniMax-M3 on Rubin with NVFP4 KV exemptions and paged FMHA #20167
    • Expand Qwen3.8-Flash-Next Modeling V2 support for MTP, video, and quantized KV caches #20276
    • Add SM103 Modeling V2 defaults for GPT-OSS-120B and DeepSeek-R1-0528 NVFP4 #20267
    • Qualify dense Phi3 models for ModelExpress integration #19997
    • Support static FP8 Cosmos3 parallelism and four-step distilled checkpoints #20007
    • Support FLUX.1-dev static NVFP4 with per-tensor FP8 attention on SM103 #18396

  • API

    • Migrate LlmRequest fields from C++ to Python (BREAKING) #20388
    • Drop KV cache manager V1 support from transceiver V2 (BREAKING) #20129
    • Remove obsolete executor backend compatibility paths (BREAKING) #20043
    • Enable zero-copy mode for Mooncake Store (BREAKING) #20108
    • Deprecate n-gram and user-provided speculative decoding modes #20438
    • Add configurable FP32 LM-head logits through lm_head_output_dtype #19850
    • Consolidate prompt rendering into standalone serve/render support for Dynamo #19964
    • Define backend contracts and share lifecycle enforcement for KV cache transfers #20117, #19822
    • Add a blob-store driver boundary with an in-memory test implementation #20217

  • Feature

    • Add native retention and cross-turn generation-block reuse to KV cache V2 #20086, #19710
    • Commit sliding-window prefixes at fork points and account for connector-provided reuse blocks #19877, #20180
    • Add first-N/last-N mixed-precision KV caching using native sliding-window attention #20452
    • Support FP4 MLA in disaggregated serving deployments #20544
    • Add MXFP8 refitting with piecewise CUDA graphs and Ray collectives #20075
    • Optimize DeepSeek-V4 and DSpark on Rubin, including fused RMSNorm/RoPE kernels #19961, #20366
    • Add Kimi K3 fabric gathering, TCPStore rendezvous, and managed KDA replay buffers #20576
    • Integrate FlashInfer KDA prefill with fused convolution #20653
    • Enable shared CUDA graph setup for Qwen3-family vision-language encoders #20213
    • Schedule multimodal encoders by prefill chunk and reduce Gemma attention-mask memory #19884, #20405
    • Improve Qwen-Image Ulysses performance and add MiniMax-H3 asynchronous Ulysses execution #17147, #20333
    • Add PrimsTS SageAttention for VisualGen VSA and SOL workloads #20224
    • Parallelize MiniMax-M3 indexer scoring and use BF16 tensor-core router GEMMs #19975, #20699
    • Accelerate GLM-5.3-Flash sparse attention, verification offsets, mHC, quantization, and routing #20454, #20389, #20390
    • Improve DSA paged-MQA logits kernels and default indexing to CuTe DSL #19976
    • Optimize one-model speculative decoding host scheduling, target logits, and KDA draft records #19989
    • Optimize GVR V2 for large batches and short sequences #20106
    • Fuse SwiGLU into BF16 gate/up GEMM epilogues on SM100 #19773
    • Compile Mamba replay-input preparation and vectorize SID-GR final beam assembly #20688, #20166
    • Support required tool choice for Nemotron Super 3.5 VL #20422

  • Fix

    • Prevent global stop-word buffer growth in the PyTorch sampler #20679
    • Preserve original errors when lazily importing serving modules and submodules #20948, #20660
    • Retry wrapped connection resets within the transient retry budget #20889
    • Respect Gemma4 encoder budgets and fix startup on non-SM100 GPUs #20663, #18547
    • Preserve admitted LoRA adapter formats and restrict multimodal adapters to decoders #20774, #20605
    • Bound LoRA parameter-fill reads to valid module output sizes #20477
    • Correct MiniMax-M3 rendered prompts, video counts, multimodal speculation, and KV reuse #20626
    • Fall back to tokenizer chat templates when multimodal processors provide none #20880
    • Update live-token metadata before constructing sparse-attention fields #19721
    • Apply NVFP4 AWQ pre-scales and expert input scales in MoE loading #20823, #20547
    • Preserve MXFP8 fused weight-scale alignment when loading weight rows #20443
    • Support MegaMoE CuTe DSL TEP across models and CUTLASS MXFP8 on SM107 #20055, #19953
    • Restore CUTLASS MoE compatibility, feature checks, and INT8 tensor-parallel padding #19753, #16198
    • Correct CuTe DSL input-quantization synchronization for prefill and multi-stream execution #20779, #20808
    • Keep low-M GEMM outputs distinct during CUDA graph capture #20334
    • Honor separate value strides and support FP8 custom-mask context attention #20242, #20387
    • Stabilize deterministic GDN decode recurrence and pad DeepSeek-V4 compressed-FP8 scratch #20226, #20119
    • Retain GLM-5.3-Flash MTP draft layers in the target KV cache #20742
    • Recover executor loops after failed forwards and avoid leader-precheck deadlocks #19465, #20272
    • Keep serving metrics responsive and preserve MPI shutdown requests during teardown #20698, #20608
    • Resolve attention-DP dummy roles and flush overlap KV events on every rank #20325, #20314
    • Pass image inputs through the Responses API to multimodal models #20231
    • Use consistent video hashing and avoid GIL-held copies during VisualGen encoding #19920, #20575
    • Clean up cancelled video encodes and remove TRTLLM_VIDEO_ASYNC_ENCODE #19401
    • Resolve Wan quantization per transformer and handle per-token I2V TeaCache timesteps #20413, #20594
    • Upgrade PyNvVideoCodec to version 2.2.0 or later #20738
    • Repair indexer-Q prewarming imports and lazy attention package exports #20734, #20606
    • Use string cache salts with KV cache manager V2 #20048

  • Documentation

    • Document LoRA performance tuning and clarify unified TRTLLM_CACHE_DIR behavior #20611, #20607
    • Explain Nemotron tool parsing and correct sliding-window and long-sequence guidance #20525, #20347, #20348
    • Repair stale documentation links and constrain generated Sphinx page exceptions #20598, #20756, #19405
    • Refresh MoE performance dashboard data and clarify post-merge CI descriptions #20798, #20475
    • Remove the issue-approval requirement from contribution guidelines #20630

  • Test & Infra

    • Upgrade NIXL to 1.5.0, UCX to 1.23.x, and CuTe DSL to 4.8.0 #20099, #20034
    • Build tritonrelease images from source and unify Docker orchestration and nightly wheel selection #10574, #20596
    • Enforce container wheel version and source provenance; align nightly BOLT behavior #20564, #20612
    • Retain CPU-only unit tests in NGC image sanity checks #20818
    • Add PrimTS vendor promotion tooling and promote the updated vendor source #19048, #20870
    • Synchronize release/1.3 changes back to main #20232
    • Narrow runner interfaces, simplify engine state, and split decoder execution by phase #20270, #20134
    • Remove obsolete warmup paths, runtime switches, and two-model speculative-decoding leftovers #20253, #20464
    • Standardize CUTLASS MoE identities and reorganize domain kernels and component-local tests #20741, #19275, #20658, #20222
    • Expand CBTS coverage, dataclass tracking, docstring analysis, and recorded shadow decisions #20583, #20411, #20410, #20265
    • Classify Slurm failures and preserve post-merge testing after test-list checks fail #19537, #10695
    • Determine changed files from the base commit in merge-queue builds #20456
    • Restore B200 multi-GPU stages after temporary queue relief and shard GB200 coverage #20828, #20899, #20780
    • Add Kimi K3 context-only AgentX and new-model performance-sanity coverage #20613, #20680
    • Add DeepSeek-V4.1 NVFP4 DSpark, Gemma4 assistant, and Qwen3 model coverage #20202, #20671, #20593
    • Restore LTX-2 worker CUDA graph regression coverage #19080
    • Correct forward-error mocks, Qwen token expectations, and KV-cache stored-event assertions #20883, #20673, #20758, #20740
    • Exercise mid-generation KV rebalancing and migrate legacy disaggregated transceiver tests #20479, #19472
    • Use MSA in MiniMax-M3 multi-node evaluation and move hang-detector tests to CPU #20701, #20559
    • Correct performance baseline cutovers, MLA routing thresholds, and benchmark prompt limits #20297, #20652, #20101
    • Report benchmark readiness rejections and repair the performance-triage image reference #20749, #20729
    • Remove DeepSeek-R1 from CI performance-sanity lists #20677
    • Refresh serving configuration schemas and clean up attention-test KV managers on failures #20493, #20522
    • Mirror UCXX downloads, retry wheel builds, and synchronize agent tooling #20080, #19980

What's Changed

  • [https://nvbugs/6948011][fix] Shut down attention test KV cache managers on failure paths by @HuiGao-NV in #20522
  • [https://nvbugs/6784181][fix] Add an exact-path SPHINX_GENERATED_DOC_PAGES allowlist keyed off helper.py's… by @trtllm-agent in #19405
  • [TRTLLM-5432][infra] Fetch UCXX's downloads from the mirror and retry build_wheel.py by @weiminwang-nv in #20080
  • [TRTLLM-15766][refactor] Decouple decoder tests from private methods and split DecoderRunner by phase by @lori-ren in #20134
  • [https://nvbugs/6713424][chore] Unwaive GB300 DSpark AgentX con1156 e2e by @chenfeiz0326 in #20579
  • [https://nvbugs/6939562][fix] Bound module_out_sizes read in LoRA param-fill kernel by @chenfeiz0326 in #20477
  • [https://nvbugs/6939212][fix] Unwaive unittest/_torch/attention after the KVCM2 teardown fix by @chenfeiz0326 in #20471
  • [None][perf] GLM-5.3-Flash C++ kernel speedups: vectorized FP8 1x128 quant and 288-expert router GEMM by @ruocheng-nv in #20390
  • [None][test] Add coverage for Qwen3ForCausalLM by @StanleySun639 in #20593
  • [None][chore] Sync Claude Code agents and skills from trtllm-agent-toolkit by @kaiyux in #19980
  • [TRTLLM-17109][perf] Reduce Gemma multimodal attention mask construction peak memory by @zhaoyangwang-nvidia in #20405
  • [None][fix] Load MXFP8 fused weight scales with the rows of their weights by @pranav-nvidia in #20443
  • [None][doc] Fix CI overview description of post-merge pipeline by @sharabeshj-nvda in #20475
  • [None][fix] Use string cache salts in V2 and migrate LLM API coverage by @yizhang-nv in #20048
  • [https://nvbugs/6942724][fix] Refresh serving configuration schema snapshots by @achartier in #20493
  • [None][fix] Pass every expert's input_scale to the glm5_next NVFP4 MoE loader by @ruocheng-nv in #20547
  • [None][perf] GLM-5.3-Flash: share KDA verify offsets, fuse mHC in every phase, BF16 router GEMM by @ruocheng-nv in #20389
  • [TRTLLM-15863][fix] Address review follow-ups on the lazy attention package exports by @YihuiLu512 in #20606
  • [None][infra] save cbts shadow run decision to artifact by @crazydemo in #20265
  • [https://nvbugs/6906428][fix] Stop a leader-only precheck failure from deadlocking the instance by @chuangz0 in #20272
  • [TRTLLM-16975][infra] compose CBTS docstring analysis and expand no operation rules by @crazydemo in #20410
  • [TRTLLM-14924][feat] Reuse generation KV blocks across turns with KV cache manager V2 by @erictsai-nv in #19710
  • [TRTLLM-16984][feat] modeling_v2: default targets for gpt-oss-120b and DeepSeek-R1-0528 NVFP4 on sm_103 by @erictsai-nv in #20267
  • [https://nvbugs/6890342][fix] Fix fp4 mla test and update cutedsl to 4.8.0 by @Tracin in #20034
  • [None][feat] Account for connector-provided blocks in KV reuse by @brb-nv in #20180
  • [TRTLLM-15717][fix] Remove TRTLLM_VIDEO_ASYNC_ENCODE and clean up the encode a delete cancels by @luyiyun1021 in #19401
  • [None][docs] Explain Nemotron's use of the Qwen3-Coder tool parser by @eopXD in #20525
  • [None][infra] Add PrimTS vendor promotion maintainer tool by @yuxianq in #19048
  • [TRTLLM-14558][chore] Split cute_dsl_kernels/ into the domain kernel trees and consolidate the thop unit tests by @YihuiLu512 in #19275
  • [TRTLLM-14691][doc] Add documentation on LoRA performance tuning knobs by @AlessioNetti in #20611
  • [TRTLLM-16212][doc] Clarify documentation on TRTLLM_CACHE_DIR unified caching by @AlessioNetti in #20607
  • [https://nvbugs/6933914][fix] Handle per-token timesteps in Wan I2V TeaCache by @yingguo-trt in #20594
  • [https://nvbugs/6581067][test] Unwaive test_fp4_gemm_bias_per_backend mnk2/mnk3 by @luyiyun1021 in #20581
  • [None][test] Add DeepSeek V4.1 NVFP4 DSpark core coverage by @yufeiwu-nv in #20202
  • [#20365][fix] Use the fused DSpark RMSNorm/RoPE kernels on SM107 by @lucifer1004 in #20366
  • [https://nvbugs/6799208][test] Unwaive test_disaggregated_overlap by @chuangz0 in #19887
  • [None][feat] Add the blob store driver boundary and an in-memory test driver by @nv-xtf in #20217
  • [None][test] Migrate legacy disaggregated E2E transceiver paths by @chuangz0 in #19472
  • [None][fix] Enforce container wheel version and source commit by @chzblych in #20564
  • [https://nvbugs/6948129][fix] Test H=96 single-token MLA routing at its measured batch floor by @trtllm-agent in #20652
  • [TRTLLM-16404][fix] restore CUTLASS MoE compatibility and early feature checks by @achartier in #19753
  • [https://nvbugs/6613027][fix] Responses API: pass image input to multimodal models by @JunyiXu-nv in #20231
  • [https://nvbugs/6892493][fix] Support MegaMoE CuteDSL TEP for all models by @peihu-nv in #20055
  • [None][fix] Encode VisualGen video without a GIL-held frame copy by @luyiyun1021 in #20575
  • [None][feat] Add native KV cache retention to KVCM V2 by @yizhang-nv in #20086
  • [None][doc] Remove issue-approval requirement from CONTRIBUTING.md by @karljang in #20630
  • [None][perf] schedule multimodal encoder items by prefill chunk by @yechank-nvidia in #19884
  • [https://nvbugs/6812424][fix] Enable CUTLASS MXFP8 MoE on SM107 by @peaceh-nv in #19953
  • [TRTLLM-16315][infra] bound new dataclasses through CBTS consumers by @crazydemo in #20411
  • [TRTLLM-17054][refactor] Remove obsolete warmup paths and test-only runtime switches by @zhaoyangwang-nvidia in #20253
  • [None][fix] hash offline-loaded videos like the serve path by @yechank-nvidia in #19920
  • [TRTLLM-16480][perf] Speed up GLM-5.3-Flash sparse attention by @ruocheng-nv in #20454
  • [https://nvbugs/6913951][fix] Support FP8 custom-mask context attention by @yuxianq in #20387
  • [None][infra] Point the PerfSanityTriage job at an image that exists by @chenfeiz0326 in #20729
  • [None][test] Remove DeepSeek-R1 from the CI perf-sanity lists by @chenfeiz0326 in #20677
  • [None][test] Add coverage for Gemma4AssistantForCausalLM by @StanleySun639 in #20671
  • [https://nvbugs/6925906][fix] Floor gen_only device-step-time baselines at the expected-bucket cutover by @chenfeiz0326 in #20297
  • [TRTLLM-9636][infra] Keep run all test stage when check test list failed in post-merge by @ZhanruiSunCh in #10695
  • [None][perf] Port the DKG paged MQA logits kernel improvements and default the DSA indexer to CuTe DSL by @dc3671 in #19976
  • [TRTLLM-16186][feat] Define backend contract for KV cache transfers by @Shixiaowei02 in #20117
  • [None][feat] FLUX.1-dev NVFP4 Static Quant with per-tensor FP8 quantized MHA (sm103 CUTEDSL) by @jhaotingc in #18396
  • [None][fix] Do not lose the MPI Comm Server shutdown request while workers tear down by @chenfeiz0326 in #20608
  • [None][perf] BREAKING: Enable zero-copy mode for Mooncake Store by @brb-nv in #20108
  • [TRTLLM-17060][refactor] BREAKING: remove obsolete executor backend compatibility by @Funatiq in #20043
  • [https://nvbugs/6700677][fix] Fix Gemma4 startup on non-SM100 GPUs by @lfr-0531 in #18547
  • [None][fix] Flush overlap KV cache events on every attention DP rank by @dongfengy in #20314
  • [None][perf] MiniMax-M3: BF16-split tensor-core router GEMM by @pranav-nvidia in #20699
  • [None][fix] match nightly release BOLT behavior to postmerge by @tburt-nv in #20612
  • [TRTLLM-17242][feat] merge two renders into a standalone serve/render for dynamo by @hchings in #19964
  • [TRTLLM-15893][fix] Fix the dead import in the indexer-Q CuTe DSL prewarm by @YihuiLu512 in #20734
  • [None][fix] Resolve attention DP dummy roles from queued requests by @dongfengy in #20325
  • [None][perf] MiniMax-M3: parallelize MSA indexer scoring over cached prefix dimension by @zheyuf in #19975
  • [None][fix] Honour --num-prompts in the VisionArena benchmark dataset by @v0ropaev in #20101
  • [None][feat] FP4 MLA disagg by @Tracin in #20544
  • [None][chore] Move hang detector test to CPU stage by @tongyuantongyu in #20559
  • [TRTLLM-10804][fix] Unify Docker image orchestration and nightly wheel selection by @niukuo in #20596
  • [None][infra] Upgrade NIXL to v1.5.0 and UCX to v1.23.x by @zackyoray in #20099
  • [None][chore] Share KV transfer lifecycle enforcement primitives by @chienchunhung in #19822
  • [TRTLLM-14848][chore] Move the MoE-owned thop unit tests into tests/unittest/_torch/moe/kernels/ by @lori-ren in #20658
  • [https://nvbugs/6932341][fix] Resolve Wan quantization per transformer by @yingguo-trt in #20413
  • [None][chore] Remove two-model speculative decoding leftovers by @lori-ren in #20464
  • [TRTLLM-16975][infra] expand CBTS Python coverage collection by @crazydemo in #20583
  • [TRTLLM-16186][refactor] Breaking: Drop KV manager V1 support from transceiver V2 by @Shixiaowei02 in #20129
  • [https://nvbugs/6942751][test] Use MSA backend in MiniMax-M3 multi-node eval test by @brb-nv in #20701
  • [None][infra] Keep cpu_only unit tests in NGC image sanity tests by @EmmaQiaoCh in #20818
  • [TRTLLM-16957][perf] SID-GR Optimization 2/N: Vectorize final beam assembly by @mikeiovine in #20166
  • [None][infra] pause B200 multi-GPU stages to relieve the CI queue by @sharabeshj-nvda in #20828
  • [TRTLLM-17134][fix] Survive a failed forward step in the executor loops by @brnguyen2 in #19465
  • [https://nvbugs/6951802][fix] Fix stale doc links failing test_http_url_validity by @trtllm-agent in #20756
  • [None][chore] Mass merge release/1.3 branch back to main by @SimengLiu-nv in #20232
  • [https://nvbugs/6838020][test] Rebalance mid-generation in KV pool rebalance accuracy test by @thorjohnsen in #20479
  • [None][feat] Run static FP8 Cosmos3 with TP/Ulysses/CFG and support the FP8 4-step distilled checkpoints by @ishovkun in #20007
  • [None][fix] Keep GLM-5.3-Flash MTP draft layers in the target KV cache by @ruocheng-nv in #20742
  • [None][doc] Update MoE Perf Dashboard data to 1.4.0rc1 (0372799) by @leslie-fang25 in #20798
  • [None][test] Add Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash and GLM-5.3-Flash perf-sanity cases; reset perf-sanity timeouts by @chenfeiz0326 in #20680
  • [https://nvbugs/6933668][fix] Fix dead links failing test_http_url_validity by @trtllm-agent in #20598
  • [https://nvbugs/6758245][fix] Cherry-pick #20429: Keep /metrics from blocking the serving event loop by @JunyiXu-nv in #20698
  • [https://nvbugs/6960771][fix] Expect decode blocks in V2 KV cache stored events by @erictsai-nv in #20740
  • [None][feat] Add lm_head_output_dtype for FP32 LM head logits by @shikicloud in #19850
  • [None][perf] One-model speculative decoding: host work between steps, worker-produced target logits, KDA drafts' records by @vsabavat in #19989
  • [https://nvbugs/6631760][fix] Bump PyNvVideoCodec to >=2.2.0 to addre… by @EmmaQiaoCh in #20738
  • [TRTLLM-17015][feat] VisualGen: wire async_ulysses for MiniMax-H3 by @karljang in #20333
  • [TRTLLMINF-455][infra] Add support to get changed files from base_commit by @weiminwang-nv in #20456
  • [https://nvbugs/6812381][fix] Split GB200-4_GPUs-PyTorch-Post-Merge into two shards by @trtllm-agent in #20780
  • [https://nvbugs/6958731][chore] Report benchmark readiness request rejections by @BowenFu in #20749
  • [https://nvbugs/6796296][fix] Pad DSv4 NVFP4 compressed FP8 scratch by one FMHA KV tile by @peaceh-nv in #20119
  • [None][fix] Keep GDN decode recurrence stable in deterministic mixed batches by @shikicloud in #20226
  • [None][feat] Support PrimsTS SageAttention for VisualGen VSA and SOL by @heyuhhh in #20224
  • [https://nvbugs/6932723][feat] Support required tool choice for Nemotron Super 3.5 VL by @eopXD in #20422
  • [None][feat] Kimi K3: DWDP fabric gather and TCPStore rendezvous, KVCM-managed KDA replay buffers by @reasonsolo in #20576
  • [None][feat] qualify Phi3 dense for MX by @moraxu in #19997
  • [TRTLLM-17005][refactor] Component-local C++ Tests (Part 2) by @Funatiq in #20222
  • [TRTLLM-16628][refactor] BREAKING: Migrate LlmRequest fields to Python by @Funatiq in #20388
  • [TRTLLMINF-384][fix] classify failures from Slurm job logs by @hanjingtian in #19537
  • [None][chore] promote flashinfer-prims-ts source from #20224 by @yuxianq in #20870
  • [None][infra] Add blossom-ci authorized users by @trtllm-agent in #20885
  • [TRTLLM-10155][infra] Update trtllm-release image stage to tritonrelease, and make this build from source, not devel by @ZhanruiSunCh in #10574
  • [https://nvbugs/6920458][fix] Correct the stale tokens_per_block default in the long-sequence guide by @trtllm-agent in #20348
  • [https://nvbugs/6891162][fix] Ban the tokens Qwen3 generates in test_generate_with_bad_words by @trtllm-agent in #20758
  • [TRTLLM-17033][feat] Qwen3.8-Flash-Next modeling_v2: CUTEDSL, MTP 1..7, video, FP8/NVFP4 KV, bf16 state, canonical QSA selection, breakable prefill graph, GB300 fast paths by @Wanli-Jiang in #20276
  • [None][fix] Honor separate V token stride in context FMHA by @achartier in #20242
  • [TRTLLM-15863][fix] Raise ImportError with the cause when a lazily imported submodule raises AttributeError by @YihuiLu512 in #20660
  • [https://nvbugs/6913865][fix] Scope multimodal LoRA to the decoder by @eopXD in #20605
  • [None][feat] Enable MiniMax-M3 on Rubin (SM107): NVFP4 KV exemption and 128-token paged FMHA kernels by @peihu-nv in #20167
  • [https://nvbugs/6920459][fix] Point the Sliding Window Attention figure at its own image by @trtllm-agent in #20347
  • [TRTLLM-16754][feat] Commit SWA windows at prefix fork points for sibling reuse by @erictsai-nv in #19877
  • [None][test] Add the Kimi-K3 AgentX ctx_only lanes to GB300 post-merge perf-sanity by @chenfeiz0326 in #20613
  • [TRTLLM-14960][refactor] Give the CUTLASS MoE implementations canonical identities by @xxi-nv in #20741
  • [None][fix] Keep low-M GEMM outputs distinct during CUDA graph capture by @dongfengy in #20334
  • [https://nvbugs/6448152][fix] Keep CuteDSL NVFP4 input quantization synchronous for prefill batches by @trtllm-agent in #20779
  • [None][chore] Deprecate ngram/user-provided spec dec by @mikeiovine in #20438
  • [None][perf] Add torch.compile decorator for mamba replay input prep by @mikeiovine in #20688
  • [None][perf] Fuse SwiGLU into the BF16 gate/up GEMM epilogue on SM100 by @karljang in #19773
  • [https://nvbugs/6891162][fix] Stop test_generate_with_stop_words on the token Qwen3 generates for N by @trtllm-agent in #20673
  • [https://nvbugs/6958731][fix] Use the tokenizer chat template when the processor has none by @chenfeiz0326 in #20880
  • [TRTLLM-16958][feat] MXFP8 refit with PCG changes, recompute_active_requests, Ray collectives by @hchings in #20075
  • [TRTLLM-17204][feat] Add First-N/Last-N mixed-precision KV caching using native SWA in KVCM V2 by @Hudayday in #20452
  • [None][perf] optimize gvr v2 under large batchsize and small seqlen by @yweng0828 in #20106
  • [None][perf] Integrate FlashInfer KDA prefill with fused convolution by @jiaganc in #20653
  • [None][feat] Qwen Image Ulysses perf enhancement by @yibinl-nvidia in #17147
  • [None][test] Fix the _validate_request fakes in test_forward_error_survival by @moraxu in #20883
  • [None][feat] Add Cosmos3-Edge Reasoner support by @ConstBob in #20307
  • [None][fix] Pad INT8-woq MoE intermediate to 64 for tensor-parallel sharding by @Dorijan10 in #16198
  • [https://nvbugs/6848775][fix] Set metadata._num_tokens = live_tokens before _build_msa_fields() (what… by @trtllm-agent in #19721
  • [https://nvbugs/6963167][fix] MiniMax-M3 VL: serve-rendered prompts, video token counts, speculative decoding and KV reuse on multimodal requests by @pranav-nvidia in #20626
  • [TRTLLM-15763][refactor] Narrow the runner surface, slim engine state, and make the engine's executor interface explicit by @lori-ren in #20270
  • [None][feat] Staircase: GLM-5.2 NVFP4 low-latency decode target for Rubin (modeling_v2, TP16) by @dongfengy in #20517
  • [None][fix] Apply the NVFP4_AWQ pre-quant scale in MegaMoE CuteDSL by @xxi-nv in #20823
  • [https://nvbugs/6858520][fix] Reload a request's LoRA adapter in the format it was admitted with by @trtllm-agent in #20774
  • [https://nvbugs/6921831][fix] Schedule Gemma4 multimodal encoder items under the encoder budget by @jasxu-nvidia in #20663
  • [TRTLLM-15840][test] Restore LTX-2 worker CUDA graph regression by @chang-l in #19080
  • [https://nvbugs/6973221][fix] Fork CuteDSL MoE input quantization only under multi-stream by @reasonsolo in #20808
  • [None][feat] DeepSeek V4 / DSpark optimizations on Rubin by @reasonsolo in #19961
  • [None][infra] restore B200 multi-GPU stages by @hanjingtian in #20899
  • [None][fix] retry wrapped connection resets with transient budget by @jiaganc in #20889
  • [TRTLLM-15863][fix] Raise ImportError with the cause when a lazily imported serve module raises AttributeError by @YihuiLu512 in #20948
  • [None][perf] enable Qwen3.*-VL encoder CUDA graphs via shared setup by @yechank-nvidia in #20213
  • [TRTLLM-17184][fix] Avoid global stop-word buffer growth in the PyTorch sampler by @zhaoyangwang-nvidia in #20679

New Contributors

  • @v0ropaev made their first contribution in #20101
  • @ConstBob made their first contribution in #20307
  • @jasxu-nvidia made their first contribution in #20663

Full Changelog: v1.4.0rc1...v1.4.0rc2

Check out latest releases or
releases around NVIDIA/TensorRT-LLM v1.4.0rc2

Don't miss a new TensorRT-LLM release

NewReleases is sending notifications on new releases.

Get notifications