github NVIDIA/TensorRT-LLM v1.3.0rc23
on GitHub

pre-release4 hours ago

  • Known Issues

    • Deepseek-V4-Pro can hang on GB300 disagg setups.
    • DeepSeek-R1 NVFP4 multi-GPU (PP4 + MTP) can crash with an MPI worker exit during execution (seen on GB300).
    • DeepSeek-V3-Lite BF16 + chunked prefill (Python scheduler) can hang / fail to complete.
    • Qwen MoE multi-LoRA (routed experts, varying ranks, eager) can fail during sampling.
    • Gemma3-1B FP8 prequantized with torch.compile can fail during CUDA graph capture (PyTorch CUDA allocator internal assert).
    • Qwen3.5-35B-A3B BF16 (CUTLASS, TP1 on A100) can hit CUDA OOM.

  • Model Support

    • Load DeepSeek V4 mixed-precision NVFP4 checkpoints (#16433)
    • Enable W4A8 checkpoint loading for Gemma4 K=V layers (#16797)

  • VisualGen

    • Implement uneven TP linear for VisualGen models (#14875)

  • API

    • Support multi-process HTTP frontends on the classic IPC executor path in trtllm-serve (#16523)
    • Emit initial KV cache stats at startup for external metric scrapers (#12596)

  • Feature

    • Default GPT-OSS and GLM-5 to the Python KV-cache transceiver (#16479, #16524)
    • Add MiniMax-M3 MSA sparse attention backend and migrate MiniMax M3 to loader v2 for TP8 support (#16291, #16468)
    • Support MiniMax M3 for disaggregated serving (#16017)
    • Support top_p_decay in the PyTorch TorchSampler (#16184)
    • Support GDN MTP replay (#16464)
    • Support Nemotron dynamic-tree MTP decoding (#15582)
    • Support rejection sampling under attention DP (including LM-head TP) (#16544)
    • Support MARLIN MoE with MTP and attention DP + EP (#16597)
    • Support fine-grained context chunk management (#16427)
    • Add TensorRT-LLM runtime integration for KV cache compression (#15697)
    • Add per-conversation KV cache block reuse (#16115)
    • Improve PyTorch encoder-decoder support and performance (#16158)
    • Optimize fused QKNormRoPE kernel (#16633)

  • Fix

    • Account for resume utilization in KV constraints and fix KV cache estimation capacity (#16484, #16545)
    • Make auto host-tier sizing rank-aware in KVCacheManagerV2 (#16106)
    • Fix DSpark all-reduce of draft MoE output under non-attention-DP TP (#16617)
    • Fix DSpark rolling-window slot collision in disaggregated serving (#16772)
    • Move Qwen-VL mRoPE seq-slot delta cache correctly (#16537)
    • Avoid attention workspace resize during CUDA graph capture (#16319)
    • Make FlashInfer sampling op wrappers opaque to Dynamo (#16732)
    • Allow trtllm-gen MoE autotuner when local_num_experts < top_k (#16653)
    • Use attention cache dtype for disaggregated transfer (#16505)
    • Fall back to disagg_request_id in Python NIXL decode receiver (#16720)
    • Fix MPI world heap corruption during teardown (#16550)
    • Stop thinking-budget processor from re-forcing the reasoning end tag (#16785)
    • Fix GPT-OSS router token identity (#16760)
    • Exclude ADP/CUDA-graph dummy requests from speculative-decode acceptance stats (#16571)
    • Use explicit errors instead of asserts for real-dataset input (#16257)

  • Documentation

    • Remove legacy Python relics and refresh docs after the TensorRT backend removal (#16612)

  • Benchmark

    • Support DeepSeek-V4 in layer-wise benchmarks (#16774)

  • Test & Infra

    • Assert MTP acceptance length in ADP + LM-head-TP accuracy tests (#16521)
    • Add DeepSeek R1/V3.2/V3-Lite disaggregated accuracy tests (#16344)
    • Update disaggregated GB200 test configs (#16481)
    • Raise BF16 Laguna DFlash memory skip threshold to prevent OOM (#16627)
    • Remove MiniMax-M2 TP16 multinode eval test case (#16542)
    • Add DeepSeek V4 Pro QA cases and DeepSeek-V4 B200/B300 single-node GPU tests (#16611, #16661)
    • Migrate Kimi disagg tests to Transceiver v2 and trim tests (#16482)
    • Split slow B300 attention unit tests (#16709)
    • Remove legacy TensorRT backend tests, examples, and CI plumbing (#16610)
    • Remove --trt_root and stop installing the TensorRT SDK into images (#16608)
    • Use main as default target branch in L0_MergeRequest_PR (#16480)
    • Preserve pytest progress with S3 capture (#16268)
    • Add support to run NGC container scanning in pre-merge (#15602)
    • Add per-stage option to cap or disable stage-level infra retries (#16551)
    • Require approval for PerfSanity wildcard runs (#16777)
    • Surface SLURM device faults to the failure classifier (#16557)
    • Update flashinfer-python from 0.6.14 to 0.6.15 (#16530)

What's Changed

  • [https://nvbugs/6245862][fix] Fix MoE EP divisibility check for EPLB configurations by @JacobHu-NV in #15410
  • [TRTLLMINF-127][infra] Upgrade dependencies for NGC PyTorch 26.05 stack by @EmmaQiaoCh in #15087
  • [None][test] Waive 7 failed cases for main in QA CI by @trtllm-agent in #16042
  • [None][feat] Add Laguna DFlash drafter support by @joerowell in #15666
  • [None][infra] Enable B300 stages by @yiqingy0 in #16359
  • [https://nvbugs/6428144][test] Unwaive GB300 DeepSeek-R1 gen-only disagg PerfSanity by @chienchunhung in #16267
  • [None][chore] Remove disagg-devs co-ownership of mla.py in CODEOWNERS by @Tabrizian in #16385
  • [TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation and safe cleanup by @chienchunhung in #15238
  • [https://nvbugs/6374873][fix] Allow fp4 KV Cache + non-FP4 Mamba State by @VALLIS-NERIA in #16304
  • [https://nvbugs/6435642][fix] detect killed MPI executor workers by @chienchunhung in #16338
  • [None][test] Fix the multinode test case on DGX-Spark(perf skipped and func hang) by @JennyLiu-nv in #16209
  • [TRTLLM-14024][feat] Prune CuTe DSL NVFP4 GEMM autotuner tactics with nvMatmulHeuristics by @peaceh-nv in #15909
  • [TRTLLM-12373][feat] RMSNorm nvfp4 quant fusion for DS V3.2 / Kimi-K2.5 by @JunyiXu-nv in #14848
  • [None][test] Refine CBTS logic for handling -Perf- stages by @yufeiwu-nv in #16328
  • [None][infra] Assign KV cache manager v2 test ownership by @jiaganc in #16372
  • [None][fix] fix missing torch_dtype_to_binding by @bo-nv in #16401
  • [https://nvbugs/6426860][fix] Stabilize compressor BF16 tolerance by @mingyangHao in #16376
  • [https://nvbugs/6403909][test] Remove attention backend test waiver by @yuxianq in #16400
  • [None][chore] Rename AutoDeploy standalone package to Paragraf by @bmarimuthu-nv in #16324
  • [None][test] Split test_moe_backend TRTLLM by quant=None to avoid in-process IMA cascade by @leslie-fang25 in #16068
  • [None][fix] Fix Gemma4 illegal memory access when max_seq_len is at most the sliding window size by @Hudayday in #16099
  • [https://nvbugs/6396422][fix] Disable kv-cache reuse for MiniMax-M2 by @tcherckez-nvidia in #16383
  • [None][test] Waive 1 failed cases for main in QA CI by @trtllm-agent in #16412
  • [None][fix] align KV slice token_range.end with transferred block count by @chuangz0 in #15824
  • [None][feat] Python transceiver support cpp cache manager + offload by @chuangz0 in #15245
  • [None][chore] add disagg-devs as code owner for disaggregated tests by @Shixiaowei02 in #16415
  • [TRTLLM-13212][refactor] Unify sampler ops and clean up dead code in TorchSampler by @zhaoyangwang-nvidia in #16365
  • [https://nvbugs/6260897][fix] Relax line-589 assertion to accept either IN_PROGRESS or SUCCESS (FAILURE… by @tensorrt-cicd in #15121
  • [None][infra] Waive 2 failed cases for main in pre-merge 47854 by @trtllm-agent in #16423
  • [https://nvbugs/6445332][test] Remove test waivers for nvbug 6445332 (code fix merged in #16326) by @trtllm-agent in #16316
  • [None][infra] Tail Slurm job logs when job is no longer active. by @mzweilz in #15455
  • [None][chore] KVCacheManagerV2: Python and test preparation for a C++ backend by @lowsfer in #16218
  • [https://nvbugs/6283537][fix] Unwaive TestQwen3_5_4B::test_bf16 by @amukkara in #16392
  • [None][infra] Waive 1 failed cases for main in pre-merge 47597 by @trtllm-agent in #16361
  • [None][infra] Waive 1 failed cases for main in pre-merge 47847 by @trtllm-agent in #16443
  • [None][infra] Waive 1 failed cases for main in pre-merge 47847 by @trtllm-agent in #16442
  • [https://nvbugs/6269388][fix] Raise error to enforce HMAC encryption in IPC by @yibinl-nvidia in #16249
  • [None][infra] Waive 1 failed cases for main in pre-merge 47953 by @trtllm-agent in #16450
  • [None][infra] Waive 1 failed cases for main in pre-merge 47930 by @trtllm-agent in #16452
  • [TRTLLM-12352][feat] add post-transform capability profiles by @chienchunhung in #16238
  • [https://nvbugs/6405760][fix] Do not load the multimodal encoder for text-only trtllm-bench runs by @thorjohnsen in #16250
  • [None][perf] Optimize Video Hashing Speed by @aswinvisva in #16340
  • [None][chore] Waive failing torch-compile test by @brb-nv in #16461
  • [https://nvbugs/6414762][fix] Minor fixes for helix tests by @brb-nv in #15999
  • [TRTLLM-14123][feat] add ParallelVAE_TrtllmWan for native Wan VAE by @taianz-nv in #16066
  • [None][feat] BREAKING: add per-model transceiver runtime auto selection by @nv-xtf in #16164
  • [None][fix] Skip commented-out lines in Groovy stage parsing by @yufeiwu-nv in #16414
  • [None][infra] Waive 14 failed cases for main in post-merge 2839 by @trtllm-agent in #16465
  • [None][perf] fuse gdn post-conv split, QK norm and gating and RMSNorm with FP8 quantization by @nv-guomingz in #16299
  • [TRTLLM-12838][infra] enhance function-level code coverage for catching subprocess data by @crazydemo in #16229
  • [None][infra] Waive 1 failed cases for main in pre-merge 48013 by @trtllm-agent in #16470
  • [TRTLLM-6905][feat] MM encoder cache by @2ez4bz in #15734
  • [TRTLLM-13614][fix] Enhance the dis-agg bounce buffer workflow by @Shixiaowei02 in #16116
  • [None][infra] Auto-label PRs by component from CODEOWNERS by @zhenhuaw-me in #16368
  • [None][feat] Add DeepSeek-V4-Pro curated configs by @lfr-0531 in #15919
  • [https://nvbugs/6327143][chore] Unwaive test cases by @marinayanov in #16367
  • [https://nvbugs/6412108][fix] AutoDeploy:Shard shared Qwen3.5 experts by @greg-kwasniewski1 in #16286
  • [TRTLLM-13639][perf] Migrate Kimi perf-sanity tests to Transceiver v2 by @Shixiaowei02 in #15966
  • [None][infra] Upgrade NIXL to v1.3.1 by @zackyoray in #16194
  • [TRTLLM-13321][feat] Extend rejection sampling to one-model speculative decoding (MTP/DRAFT_TARGET/PARD/DFLASH) by @zhaoyangwang-nvidia in #15775
  • [None][infra] Waive 3 failed cases for main in pre-merge 48086 by @trtllm-agent in #16495
  • [https://nvbugs/6435642][fix] handle session reuse worker registration by @chienchunhung in #16444
  • [TRTLLMINF-99][infra] Add SLURM frontend failover to L0 by @dpitman-nvda in #15674
  • [None][infra] Handle SBSA image tags for x86 CI agents by @chzblych in #16503
  • [https://nvbugs/6403920][fix] Move allreduce strategy test to smoke, use full-warp hidden_size by @MrGeva in #16389
  • [None][fix] Fix Qwen image CUDA graph with CFG by @yibinl-nvidia in #16391
  • [None][chore] Update API update missed in Dynamo test by @brb-nv in #16512
  • [None][test] disagg startup: detect and fail fast on worker OOM/crash by @brnguyen2 in #16456
  • [https://nvbugs/6422339][fix] unwaive disagg tests by @bo-nv in #16474
  • [None][test] add --keep-workspace pytest option to skip workspace cleanup by @brnguyen2 in #16451
  • [None][infra] Waive 1 failed cases for main in pre-merge 48139 by @trtllm-agent in #16519
  • [TRTLLM-14054][perf] Reduce per-step host preparation overhead in the PyTorch executor decode path by @kaiyux in #16313
  • [https://nvbugs/6404567][fix] Clamp piecewise cudagraph captures to the reachable ceiling instead of force-appending it by @dongfengy in #16256
  • [https://nvbugs/6272673][chore] Unwaive DeepSeek V3 Lite NVFP4 test by @yuxianq in #16398
  • [None][fix] DSA CuteDSL paged-MQA-logits: persistent output buffer (CUDA-graph stale-pointer IMA) by @Tabrizian in #16307
  • [None][fix] Fix unfused RoPE for yarn models: double rotation and GPT-OSS pairing by @Hudayday in #16378
  • [None][test] Update LLM performance test cases to use increased input/output lengths by @yufeiwu-nv in #16538
  • [https://nvbugs/6336747][fix] Honor GenerationResult timeout by @2ez4bz in #16453
  • [https://nvbugs/6460072][fix] Split the caller — for use_lm_head_tp_in_adp and is_all_greedy_sample, keep… by @trtllm-agent in #16440
  • [None][infra] Add blossom-ci authorized users by @yiqingy0 in #16541
  • [None][doc] Add DeepSeek-V4 optimization tech blog by @lfr-0531 in #16539
  • [https://nvbugs/6329052][fix] Change the config for DSV3 disagg conditional test by @Shixiaowei02 in #16528
  • [None][test] Promote disagg perf sanity tests to pre-merge for functional verification by @chenfeiz0326 in #16351
  • [None][test] Remove all 1k1k perf-sanity cases from CI by @chenfeiz0326 in #16535
  • [None][perf] Prewarm DeepGEMM paged_mqa_logits_metadata JIT buckets by @chenfeiz0326 in #16178
  • [https://nvbugs/6272644][fix] Stabilize and unwaive multi-GPU VisualGen LPIPS tests by @chang-l in #15730
  • [None][feat] Qwen3-VL: support mixed image+video modality requests by @aswinvisva in #15852
  • [#15344][fix] Add SM121 MLA cache reuse support by @peter941221 in #15434
  • [None][perf] Various Gemma4 related perf fixes by @2ez4bz in #16509
  • [None][feat] Add DeepSeek DSpark speculative decoding by @longlee0622 in #15808
  • [TRTLLM-12352][feat] integrate ModelExpress checkpoint loading by @chienchunhung in #15641
  • [TRTLLM-14509][feat] Disagg coordinator + multi-process orchestrator fleet by @reasonsolo in #15905
  • [None][chore] update allowlist 2026-07-17 by @tburt-nv in #16547
  • [None][feat] Append git commit hash to version for editable installs by @dc3671 in #16395
  • [None][fix] disagg: derive KV transfer layer offset from physical slot order by @lowsfer in #16429
  • [None][fix] Copy all AutoDeploy tests to Paragraf by @bmarimuthu-nv in #16408
  • [https://nvbugs/6476233][test] waive DeepSeek V3.2 tests on DGX H200 by @chzblych in #16573
  • [None][test] Consolidate test coverage for helix by @brb-nv in #16570
  • [https://nvbugs/6442074][fix] Make one-model spec-dec attn-metadata save/restore exception-safe by @dongfengy in #16382
  • [TRTLLM-14026][feat] BREAKING: Remove C++ modules for legacy TRT backend by @Wanli-Jiang in #16369
  • [TRTLLMINF-188][infra] Require approval for broad post-merge bot runs by @chzblych in #16318
  • [TRTLLM-5311][infra] Periodically query OpenSearch for test duration and push to main automatically by @EmmaQiaoCh in #13804
  • [https://nvbugs/6440089][test] Rework disagg mixed-stress accuracy metric and harden the test by @brnguyen2 in #16445
  • [TRTLLM-13235][feat] Support bad_words in TorchSampler by @zhaoyangwang-nvidia in #16052
  • [None][test] Add opt-in background prefetch of test MPI sessions and model page cache by @sunnyqgg in #15908
  • [None][fix] Fix DeepSeek V4 KV cache warmup handling and serveral other issues by @liji-nv in #16466
  • [None][infra] Waive 1 failed cases for main in pre-merge 48569 by @trtllm-agent in #16582
  • [None][test] Add DeepSeek-V4-Pro perf sanity cases on GB300 by @chenfeiz0326 in #16540
  • [https://nvbugs/6442074][fix] Unbreak main: DSparkWorker forward rename + test stub _forward_impl by @dongfengy in #16579
  • [TRTLLM-13409][fix] make proxy shutdown non-blocking when the engine is dead by @JunyiXu-nv in #16312
  • [TRTQA-3274][test] Waive 3 failed cases for main in QA CI by @trtllm-agent in #16585
  • [TRTQA-3273][test] Waive 4 failed cases for main in QA CI by @trtllm-agent in #16584
  • [https://nvbugs/6426852][fix] Fix intermittent cuda mapping error by @shuyixiong in #16226
  • [None][infra] Waive 5 failed cases for main in post-merge 2844 by @trtllm-agent in #16587
  • [None][fix] remove unsupported INT8 from trtllm-bench --quantization choices by @lonexreb in #16245
  • [https://nvbugs/6316983][chore] Unwaive TestQwen2_5_VL_7B::test_auto_dtype by @yihwang-nv in #16534
  • [None][infra] Waive B300 Nemotron SuperV3 MTP and Mistral Large3 timeout cases by @jieli-matrix in #16588
  • [None][chore] Remove all 1k1k cases from QA's disagg side by @fredricz-20070104 in #16586
  • [TRTQA-3281][test] Waive 2 failed cases for main in QA CI by @trtllm-agent in #16593
  • [None][test] Remove 33 closed-bug waive entries for main by @trtllm-agent in #16576
  • [TRTQA-3280][test] Waive 1 failed cases for main in QA CI by @trtllm-agent in #16599
  • [None][test] Assert MTP acceptance length in ADP + LM-head-TP accuracy tests by @qiaoxj07 in #16521
  • [TRTQA-3272][test] Waive 7 failed cases for main in QA CI by @trtllm-agent in #16602
  • [https://nvbugs/6314696] [fix] update disaggregated GB200 test configs by @reasonsolo in #16481
  • [TRTLLMINF-216][infra] use main as default target branch in L0_MergeRequest_PR by @weiminwang-nv in #16480
  • [TRTQA-3277][test] Waive 7 failed cases for main in QA CI by @trtllm-agent in #16605
  • [TRTQA-3279][test] Waive 8 failed cases for main in QA CI by @trtllm-agent in #16604
  • [TRTQA-3282][test] Waive 3 failed cases for main in QA CI by @trtllm-agent in #16600
  • [TRTQA-3276][test] Waive 3 failed cases for main in QA CI by @trtllm-agent in #16607
  • [TRTLLM-14345][feat] Support GDN MTP Replay by @JadoTu in #16464
  • [TRTQA-3278][test] Waive 6 failed cases for main in QA CI by @trtllm-agent in #16606
  • [None][feat] Add TensorRT-LLM runtime integration for KV cache compression by @Hudayday in #15697
  • [None][infra] Waive 1 failed cases for main in pre-merge 48660 by @trtllm-agent in #16615
  • [https://nvbugs/6305365][chore] Unwaive piecewise cudagraph related tests by @pengbowang-nv in #16591
  • [None][test] Waive 1 failed cases for main in QA CI by @trtllm-agent in #16622
  • [https://nvbugs/6473397][test] Raise BF16 Laguna DFlash memory skip threshold to prevent OOM by @DomBrown in #16627
  • [https://nvbugs/6438658][fix] Account for resume utilization in KV constraints by @jiaganc in #16484
  • [None][infra] Waive 2 failed cases for main in pre-merge 48727 by @trtllm-agent in #16631
  • [None][feat] Add per-conversation KV cache block reuse by @jiaganc in #16115
  • [None][perf] Improve PyTorch encoder-decoder support and performance by @cascade812 in #16158
  • [None][feat] Default GPT-OSS to the Python KV-cache transceiver by @nv-xtf in #16479
  • [None][feat] VisualGen: Add CuTe DSL JIT kernels for FMHA by @xrq-phys in #15831
  • [None][perf] Preallocate video decode buffer for HF zero-copy handoff by @aswinvisva in #16513
  • [None][chore] Bump version to 1.3.0rc22 by @tburt-nv in #16454
  • [TRTLLMINF-191][infra] Preserve pytest progress with S3 capture by @niukuo in #16268
  • [None][fix] DSpark: all-reduce draft MoE output under non-attention-DP TP by @longlee0622 in #16617
  • [https://nvbugs/6451032][fix] Reserve extra dflash context slot for dummy requests by @amukkara in #16548
  • [https://nvbugs/6475622][chore] Unwaive tests by @2ez4bz in #16638
  • [None][test] Waive 2 failed cases for main in QA CI by @trtllm-agent in #16654
  • [None][test] remove MiniMax-M2 tp16 multinode eval test case by @jieli-matrix in #16542
  • [TRTLLM-11875][feat] Support fine-grained context chunk management by @VALLIS-NERIA in #16427
  • [None][test] Waive hang issues by @xinhe-nv in #16625
  • [https://nvbugs/6395830][fix] Qwen-VL mRoPE: move seq-slot delta cach… by @nv-guomingz in #16537
  • [https://nvbugs/6120535][chore] Unwaive DeepSeek V3.2 disaggregated serving test by @peihu-nv in #16515
  • [None][perf] Close Mamba hybrid warmup gap in autotuner warmup by @chenfeiz0326 in #16177
  • [https://nvbugs/6463814][chore] Unwaive passing perf-sanity disagg tests by @chenfeiz0326 in #16655
  • [https://nvbugs/6468821][chore] Unwaive GPT-OSS B300 attention backend test by @yuxianq in #16616
  • [https://nvbugs/6428008][fix] Unwaive one test case of Qwen3_5 by @leslie-fang25 in #16493
  • [None][doc] Update tech blog by @juney-nvidia in #16659
  • [None][feat] Support rejection sampling under attention DP (incl. LM-head TP) by @zhaoyangwang-nvidia in #16544
  • [None][test] Add deepseek v4 pro cases on the qa side by @fredricz-20070104 in #16611
  • [None][fix] avoid attention workspace resize during CUDA graph capture by @yuxianq in #16319
  • [None][test] Add deepseek-v4 B200/B300 test cases for single node GPU. by @yufeiwu-nv in #16661
  • [None][infra] Waive 1 failed cases for main in pre-merge 48840 by @trtllm-agent in #16660
  • [None][fix] use explicit errors instead of asserts for real-dataset input by @lonexreb in #16257
  • [TRTLLM-14352][perf] Fuse Qwen3.5/3.6 attention preprocessing (QK-nor… by @nv-guomingz in #16469
  • [None][infra] fix lfs sync failure by @niukuo in #16665
  • [None][chore] Update flashinfer-python from 0.6.14 to 0.6.15 by @yihwang-nv in #16530
  • [None][chore] Add sparse cache manager code owner by @yuxianq in #16664
  • [TRTLLM-13639][test] Migrate Kimi dis-agg tests to Transceiver v2 and trim tests by @Shixiaowei02 in #16482
  • [https://nvbugs/6475930][fix] set small multi_round to save time by @bo-nv in #16663
  • [None][infra] Add support to run NGC container scanning in pre-merge by @yuanjingx87 in #15602
  • [None][test] update coderabbit prompt by @xinhe-nv in #16478
  • [TRTLLM-12838][infra] CBTS coverage db audit by @crazydemo in #16658
  • [None][infra] Waive 2 failed cases for main in pre-merge 30273 by @trtllm-agent in #16675
  • [TRTLLM-13117][feat] Implement Uneven TP Linear for VisualGen models by @belgarten-nv in #14875
  • [None][infra] Add per-stage opt to cap/disable stage-level infra retries by @dpitman-nvda in #16551
  • [None][infra] Waive 2 failed cases for main in pre-merge 48918 by @trtllm-agent in #16682
  • [None][infra] Waive 21 failed cases for main in post-merge 2850 by @trtllm-agent in #16700
  • [TRTLLM-14027][infra] Remove --trt_root and stop installing the TensorRT SDK into images by @Wanli-Jiang in #16608
  • [None][infra] Waive 1 failed cases for main in pre-merge 49101 by @trtllm-agent in #16698
  • [None][perf] Size the per-tensor FP8 dynamic-quant amax grid to the input by @hyukn in #16569
  • [None][feat] Raise the CuTE-DSL top-k decode limit to 16384 and support odd top_k by @Hudayday in #16546
  • [None][perf] Add batched-pybind fast-path in TorchSampler.update_requests for gpt-oss-120b by @chenfeiz0326 in #16564
  • [TRTLLM-14510][feat] serve: multi-process HTTP frontends on the classic IPC executor path by @lancelly in #16523
  • [https://nvbugs/6438658][fix] Fix KV cache estimation capacity by @jiaganc in #16545
  • [None][fix] Make auto host tier sizing rank-aware in KVCacheManagerV2 by @erictsai-nv in #16106
  • [None][feat] Support Nemotron dynamic-tree MTP decoding by @sunnyqgg in #15582
  • [https://nvbugs/6480574][fix] seed attrs bypassed by new in pool session shutdown test by @JunyiXu-nv in #16630
  • [None][perf] DSv4 GVR top-k (CuTe DSL): P4 histogram refinement + redundant-warp sync reduction by @siyidNV in #16424
  • [https://nvbugs/6468821][infra] Split slow B300 attention unit tests by @yuxianq in #16709
  • [https://nvbugs/6422318][fix] Cast cos_sin_cache to float32 at cat-time to satisfy flashinfer's contract… by @trtllm-agent in #16069
  • [TRTLLM-14473][chore] Remove legacy TensorRT backend tests examples and CI plumbing by @Wanli-Jiang in #16610
  • [TRTLLM-14019][feat] Add MiniMax-M3 MSA sparse attention backend [revised] by @brb-nv in #16291
  • [#12595][feat] Emit initial KV cache stats at startup for external metric scrapers by @BenjaminBraunDev in #12596
  • [TRTLLM-14255][fix] migrate MiniMax M3 to loader v2 for TP8 support by @peihu-nv in #16468
  • [TRTLLM-13231][feat] Support top_p_decay in the PyTorch TorchSampler by @zhaoyangwang-nvidia in #16184
  • [https://nvbugs/6442074][fix] Rename MTPEagleDynamicTreeWorker.forward to _forward_impl by @sunnyqgg in #16724
  • [None][feat] Add BaseMultimodalDummyInputsBuilder to minimaxm3_vl by @pcicotti in #16696
  • [TRTLLMINF-102][fix] Surface SLURM device faults to the failure classifier by @dpitman-nvda in #16557
  • [https://nvbugs/6499882][fix] Seed _multi_frontend_ipc_dir in bare proxy test helper by @pranav-nvidia in #16761
  • [None][infra] Preview/bump/main by @yuanjingx87 in #16758
  • [TRTLLM-13694][feat] Add IBDB recipe provenance and refresh configs by @Mgluhovskoi in #16254
  • [TRTLLM-14540][perf] Skip fp32 state round-trip in FlashInfer GDN pre… by @nv-guomingz in #16703
  • [None][chore] Remove attention backend test waivers by @yuxianq in #16723
  • [#11932][fix] Filter CUTLASS MoE GEMM tile configs by device shared memory on SM121 by @mihai-chiorean in #12704
  • [TRTLLM-13969][feat] Support MiniMax M3 for Disaggregated Serving by @peihu-nv in #16017
  • [TRTLLMINF-188][infra] Require approval for PerfSanity wildcard runs by @chzblych in #16777
  • [None][infra] Waive 1 failed cases for main in pre-merge 49424 by @trtllm-agent in #16781
  • [None][infra] Waive 1 failed cases for main in pre-merge 49424 by @trtllm-agent in #16780
  • [None][feat] ADP conversation router: configurable least-queued placement for new conversations by @lancelly in #16294
  • [None][fix] Load DeepSeek V4 mixed-precision NVFP4 checkpoints by @lfr-0531 in #16433
  • [None][infra] Waive 1 failed cases for main in pre-merge 49229 by @trtllm-agent in #16786
  • [https://nvbugs/6445456][fix] Restore inplace ops for functionalization v2 by @liji-nv in #16410
  • [https://nvbugs/6426850][test] Unwaive Qwen3.5 397B NVFP4 ADP4 TRTLLM test by @liji-nv in #16348
  • [None][chore] Add NVTX ranges to per-iteration ADP sync points in PyExecutor by @lancelly in #16422
  • [None][feat] Default GLM-5 to the Python KV-cache transceiver by @chuangz0 in #16524
  • [None][feat] top-k: route decode to CuTe DSL GVR top-k in e2e by @limin2021 in #16420
  • [None][fix] Make FlashInfer sampling op wrappers opaque to Dynamo by @qiaoxj07 in #16732
  • [None][infra] Waive 4 failed cases for main in pre-merge 49550 by @trtllm-agent in #16798
  • [None][feat] Bind SourceIdentity to checkpoint artifacts by @chienchunhung in #16159
  • [https://nvbugs/6448152][perf] make C++ context-transfer consensus asynchronous by @chienchunhung in #16634
  • [None][perf] Optimize fusedQKNormRope Kernel by @aswinvisva in #16633
  • [None][fix] Stage DSA indexer block_table H2D through pinned memory by @Tabrizian in #16757
  • [None][infra] Unwaive test_proxy_fast_death tests by @dpitman-nvda in #16808
  • [None][infra] Waive 2 failed cases for main in pre-merge 49489 by @trtllm-agent in #16819
  • [TRTLLM-14474][chore] Remove legacy python relics and refresh docs after the backend removal by @Wanli-Jiang in #16612
  • [https://nvbugs/6426834][fix] Deflake test_kv_transfer: cap NIXL progress threads, pin UCX env by @chuangz0 in #16708
  • [TRTLLM-14575][fix] MoE: fp32 accumulation in deferred MoEAllReduce finalize by @xwang233 in #16778
  • [None][feat] Support MARLIN MoE with MTP and attention DP + EP by @Wanli-Jiang in #16597
  • [https://nvbugs/6485885][fix] Stop thinking-budget processor re-forcing the reasoning end tag by @Wanli-Jiang in #16785
  • [#16767][fix] Fix DSpark rolling-window slot collision in disaggregated serving by @longlee0622 in #16772
  • [None][infra] Fix release check failure for .test_durations by @EmmaQiaoCh in #16784
  • [None][perf] spec one-model sampling: greedy rows via top_k=1 instead of unconditional vocab argmax by @Tabrizian in #16686
  • [None][perf] Avoid implicit device-scalar syncs in DeepSeek-V4 ctx sparse metadata by @qiaoxj07 in #16734
  • [TRTLLM-9920][feat] Add support for arbitrary KVCache transfer by @Tabrizian in #13055
  • [https://nvbugs/6473161][chore] unwaive 4 TestLlama3_1 tests by @lori-ren in #16824
  • [None][test] Declare supported attention test phases by @yihwang-nv in #16718
  • [TRTLLM-13948][test] Add DeepSeek R1/V3.2/V3-Lite disaggregated accuracy tests by @asfiyab-nvidia in #16344
  • [TRTLLM-13349][perf] Fuse gemma RMSNorm into AllReduce for Qwen3-Next/Qwen3.5… by @nv-guomingz in #15194
  • [None][perf] Fuse DeepSeek-V4 Indexer Q projection with CuTe DSL by @mingyangHao in #16532
  • [None][feat] Add support for MiniMax M3 and Qwen3.6 models in performance tests by @yufeiwu-nv in #16828
  • [NVBUG-6379624][fix] Enable W4A8 checkpoint loading for Gemma4 K=V layers by @Hudayday in #16797
  • [None][infra] Waive 14 failed cases for main in post-merge 2855 by @trtllm-agent in #16821
  • [None][feat] align time metrics between cpp and python cache transceiver by @chuangz0 in #15871
  • [TRTLLM-13948][test] Migrate DeepSeek R1/V3.2 disagg perf cases to transceiver V2 by @nv-xtf in #16669
  • [https://nvbugs/6482576][fix] Fall back to disagg_request_id in Python NIXL decode receiver by @Shixiaowei02 in #16720
  • [None][perf] Reordering torch.compile and cache-dit call by @BrianLi23 in #16508
  • [None][fix] Do not treat a bare fmha_v2_cu directory as completed generation by @brnguyen2 in #16743
  • [TRTLLM-14512][feat] multiprocess prometheus client for disagg-server by @reasonsolo in #16719
  • [None][test] Enable gen_only + ctx_only DeepSeek-V4-Pro perf-sanity on GB300 by @chenfeiz0326 in #16795
  • [https://nvbugs/6457853][fix] Allow trtllm-gen MoE autotuner when local_num_experts < top_k by @dongfengy in #16653
  • [https://nvbugs/6432953][fix] Fix MPI world heap corruption during teardown by @mikeiovine in #16550
  • [https://nvbugs/6020038][feat] Add NCCL-EP v0.1 MoE communication support by @nv-lschneider in #15597
  • [None][fix] Fix GPT-OSS router token identity by @SimengLiu-nv in #16760
  • [TRTLLM-14417][fix] Exclude ADP/cuda-graph dummy requests from speculative-decode acceptance stats by @xwang233 in #16571
  • [https://nvbugs/5948435][chore] Unwaive DeepSeekV3Lite test_nvfp4_4gpus CUTLASS ep4 fp8kv on RTXPro6000D by @xxi-nv in #16621
  • [https://nvbugs/6463822][fix] Fix LTX2 CUDA graph test leak issue by @yibinl-nvidia in #16775
  • [https://nvbugs/6465993][fix] use attention cache dtype for disaggregated transfer by @chienchunhung in #16505
  • [None][feat] Support DeepSeek-V4 in layer_wise_benchmarks by @ruodil in #16774

New Contributors

  • @joerowell made their first contribution in #15666
  • @peter941221 made their first contribution in #15434
  • @lonexreb made their first contribution in #16245
  • @weiminwang-nv made their first contribution in #16480
  • @erictsai-nv made their first contribution in #16106
  • @siyidNV made their first contribution in #16424
  • @BenjaminBraunDev made their first contribution in #12596
  • @asfiyab-nvidia made their first contribution in #16344
  • @BrianLi23 made their first contribution in #16508

Full Changelog: v1.3.0rc21...v1.3.0rc23

Check out latest releases or
releases around NVIDIA/TensorRT-LLM v1.3.0rc23

Don't miss a new TensorRT-LLM release

NewReleases is sending notifications on new releases.

Get notifications