github NVIDIA/TensorRT-LLM v1.3.0rc27
on GitHub

pre-release6 hours ago

Highlights

  • Known Issues

    • GPT-OSS Eagle3 speculative decoding may hang, produce non-terminating output, or lose accuracy when using multi-GPU execution.
    • GPT-OSS W4 inference with the Triton MoE backend may intermittently hang or lose accuracy in four-GPU data-parallel or expert-parallel configurations.
    • GPT-OSS Harmony tool calls may continue generating after the tool-call delimiter, resulting in malformed or repeated tool calls and an incorrect finish reason.
    • Qwen3-235B NVFP4 inference on B200 may intermittently lose accuracy when all-reduce autotuning is enabled. Set TLLM_DISABLE_ALLREDUCE_AUTOTUNE=1 to avoid the issue.
    • Qwen3-6B-A3B NVFP4 W4A16 inference may fail during initialization on Blackwell SM120 and SM121 GPUs.
    • MiniMax-M3 MXFP8 inference using piecewise CUDA graphs with the Triton sparse-attention implementation may crash during initialization on B300 and GB300 systems. Use the MSA implementation to avoid the affected path.
    • Kimi-K3-NVFP4 may fail to initialize its tokenizer with the Transformers version included in DLFW 26.08.
    • Nemotron-H weight updates through Ray may OOM or produce incorrect logits with DLFW 26.08 because compatible Mamba dependencies are unavailable.
    • mBART encoder-decoder inference may hang during the first generation request with DLFW 26.08.
    • Cosmos3 pipelines may fail to load when VAE CPU offload is enabled. Disable VAE CPU offload to avoid the affected path.
    • Multi-GPU disaggregated serving may hang during worker startup for TinyLlama deployments using pipeline or tensor parallelism, and for GPT-OSS-120B generation workers using TP4 on GB200.

  • Model Support

    • Add MiniMax-H3 VisualGen support for text-to-video with audio and first/last-frame-to-video with audio, with BF16 and FP8 blockwise configurations #18733
    • Enable Nemotron 3.5 Super VL serving with multimodal model delegation and automatic reasoning and tool-parser selection #18804
    • Add Cosmos3 Edge Policy DROID support for state-conditioned robot-policy rollout and action generation through VisualGen and trtllm-serve #18463

  • API

    • Remove deprecated two-model speculative decoding and related configuration fields (BREAKING) #18721
    • Remove unused legacy C++ runtime, batch-manager APIs, and Python bindings (BREAKING) #18846
    • Add Anthropic-compatible Messages, token-counting, and Message Batches APIs to trtllm-serve #18289
    • Add LLM profiling methods and serving endpoints across supported execution modes #13872
    • Enable streaming OpenEngine Generate RPC with disaggregated-serving request and control integration #17992

  • Feature

    • Integrate PrimTS FMHA and block-sparse attention with unified runtime inputs #17399, #18815
    • Add cuDNN as a selectable attention backend for supported workloads #18075
    • Add FlashInfer VisualGen attention and FP4 Conv3d for parallel Wan VAE #18174, #17262
    • Support Ray-orchestrated postprocessing workers for efficient distributed output handling #18399
    • Preserve sub-agent conversation affinity across disaggregated-serving requests and workers #18684
    • Add optional KV guard pages and fresh-page-fill diagnostics for debugging #18754
    • Derive per-layer KV-cache windows from hybrid model layer types automatically #18767
    • Publish KV-cache events through StreamingEvent for higher-throughput subscriber delivery #17023
    • Admit one context per uncached prefix, preventing duplicate prefix recomputation #18195
    • Add in-graph temperature, top-k, and top-p sampling for batched requests #18437
    • Add Kimi K3 KDA replay using KV-cache manager V2 #18294
    • Page DSpark drafter contexts through the dedicated draft KV-cache manager #18343
    • Add GVR V2 self-sampling prefill and two-level decode top-k dispatch #18702, #18446
    • Support image inputs through the Triton LLM API backend #18381
    • Optimize Qwen3.5 and Qwen3.8 Flash-Next generation, MoE, GDN, and sampling #18553, #17365, #18883, #18714, #18820, #18845
    • Accelerate long-sequence KV-cache resizing and asynchronous PP1 disaggregated transfers #18541, #18150
    • Optimize MiniMax-M3 sparse decode, HCA decode, and NVFP4 Marlin normalization #18611, #18869, #17685
    • Apply NUMA-aware CPU affinity across serving worker threads and processes #18701

  • Fix

    • Prevent generation workers from re-expanding multimodal placeholders during disaggregated serving #18886
    • Warn when tool-call markup cannot be parsed into a valid call #18865
    • Enable breakable prefill CUDA graphs for Nemotron-H hybrid models #18888
    • Fix KVCM V2 buffers, vocabulary sizing, page tables, draft reuse, and uploads #18824, #18957, #18885, #17643, #18093, #18694
    • Re-plan FlashInfer after page-table changes and bound eager plan-cache growth #18769, #18557
    • Stabilize MoE LoRA CUDA-graph storage and propagate per-layer adapter metadata #18972, #18527
    • Correct SM107 FMHA, FP8 GEMM, and NVFP4 KV-cache capability routing #18758, #18945, #18612, #18674
    • Fix quantized MoE sharding, providers, scales, and Kimi K3 SiTu formats #18870, #18770, #18772, #18709
    • Restore DSA TopK workspace lifetimes, row sizing, and warmup behavior #18624, #18683
    • Fix XQA synchronization and attention-sink numerical accuracy during decode #18141
    • Harden distributed execution across UserBuffers, VMM-backed DWDP, and MNNVL layouts #18037, #18644, #18715
    • Correct speculative token accounting and Eagle3 KV block-table rotation #18629, #18564
    • Fix disaggregated KV ownership, relay sends, MTP accuracy, and launch ordering #18041, #18135, #18554, #18613
    • Align generation-only request detection across Python, C++, scheduling, and disaggregation #18858
    • Correct custom-tokenizer alias resolution during model initialization in LLM arguments #18768
    • Densify unsupported SM120 BMM inputs and restore FlashInfer MLA dispatch #18678, #18653
    • Release MegaMoE buffers during teardown and page out DeepSeek-V4 weights #18182, #18832
    • Share Kimi auxiliary streams safely across model components during execution #18728
    • Fail guided-decoding requests cleanly when grammars reach dead-end states #18896
    • Stabilize VisualGen randomness, quantization, worker cleanup, and serving clocks #18477, #18750, #18030, #18686
    • Load mixed-precision Qwen3.8-Flash-Next NVFP4 checkpoints correctly across supported backends #18823
    • Support MoE two-sided preparation beyond 128 expert-parallel ranks in large deployments #18799
    • Preserve configured Mamba snapshot placement across runtime initialization and cache reuse #18724
    • Install built wheels using the active virtual-environment interpreter during development workflows #18773
    • Match fused finish-reason checks to the sampler's actual scratch row #18894
    • Update stale attention and KV-cache imports blocking backend initialization #18771
    • Improve errors for invalid roles, missing wheels, and EBADHANDLE config reads #17730, #18703, #18352
    • Update vendored PrimTS attention with FlashInfer follow-up corrections for stability #18996

  • Documentation

    • Add KV-cache compression guides, configuration guidance, and runnable user examples #18587
    • Add VisualGen quantized-attention, sparse-attention, and CUDA-graph implementation guides for developers #18792, #19023
    • Expand sparse-attention documentation with MQA and GQA developer guidance #18106
    • Document generation-side timing segments used in serving performance benchmarks #18213
    • Correct serving commands, configuration instructions, and broken example snippets #18500, #18901, #19009
    • Repair documentation links and images while reducing Sphinx build warnings #18984, #18992, #19007, #19050

  • Test & Infra

    • Upgrade build and testing infrastructure to NGC PyTorch 26.08 dependencies #18126
    • Expand coverage for BART, DeciLM, guided/speculative decoding, Kimi, Qwen, and Gemma #19017, #18948, #18908, #18905, #18699, #18655
    • Add performance coverage for Qwen, DeepSeek, Kimi, and disaggregated deployments #19012, #18853, #18875, #18806, #18633
    • Refine VisualGen serving benchmarks and performance-sanity validation coverage across models #18632
    • Add disaggregated lifecycle telemetry and checkpoint-startup measurements to performance tests #18990, #18608, #18856
    • Extend performance automation with parallel optimization, remote execution, and timeline analysis #18568, #18789, #18805, #18825
    • Align Hopper and multi-node disaggregated coverage with Python transceiver runtime #18941, #18828, #18404
    • Improve CI attribution using typed cleanup failures, timeouts, reporting, and retries #18049, #18812, #18590, #17630, #17069
    • Stabilize batching, Eagle VSWA, Kimi streams, LoRA scheduling, and LPIPS tests #19005, #18986, #19088, #18949, #18386, #18384
    • Add strict documentation-link and declarative-import validation coverage across CI stages #18718, #18586
    • Modernize CI with GB300 flex scheduling, deferral, Artifactory images, and labeling #19074, #18385, #18791, #18729

What's Changed

  • [TRTLLM-15011][infra] Unwaive TestDeepSeekV4Flash::test_auto_dtype by @lowsfer in #18000
  • [None][refactor] Table-drive the lazy-safetensors model-type check in HfWeightLoader by @moraxu in #18681
  • [https://nvbugs/6593378][baseline] Overwrite partial Slurm artifact retries by @BowenFu in #17630
  • [https://nvbugs/6674826][fix] Drop overlap hint from KV cache page-index upload by @lowsfer in #18694
  • [NVBUG-6689820][test] unwaive Llama disaggregated serving test by @chienchunhung in #18695
  • [https://nvbugs/6683840][fix] Join the encoder launch before the decoder forward by @pranav-nvidia in #18613
  • [None][feat] Add SM107 NVFP4 CuTe DSL fused MoE kernels and integration by @zhangcl in #18498
  • [None][feat] Enable Rubin (SM107) trtllm-gen FMHA features by @kris1025 in #18518
  • [None][fix] Revert #18445 by @tburt-nv in #18736
  • [None][feat] Add SM107 quantized dense and DSV4 CuTe DSL kernels by @farazkh80 in #18546
  • [TRTLLM-14558][chore] Consolidate the Attention domain: modules, backends and unit tests by @YihuiLu512 in #17968
  • [https://nvbugs/6707518][fix] Fix Kimi K3 spec dec test by @mikeiovine in #18682
  • [None][test] Add the 40-GPU AgentX DeepSeek-V4-Pro-DSpark case to post-merge and switch both cases to the NVFP4 checkpoint by @chenfeiz0326 in #18633
  • [https://nvbugs/6480110][fix] Fall back to FP8 KV cache when NVFP4 KV cache is requested on SM107 by @farazkh80 in #18674
  • [TRTLLM-14846][chore] Group KV Cache managers and reunite the Disaggregation transceiver halves by @lori-ren in #17899
  • [None][feat] Add Cosmos3 Edge Policy DROID support by @ishovkun in #18463
  • [https://nvbugs/6676352][doc] Fix missing backslash in prepare-dataset command block by @FrankD412 in #18500
  • [None][feat] Add FlashInfer VisualGen attention backend by @yumin066 in #18174
  • [TRTLLM-15752][refactor] Extract LoRA parameter construction into engine/lora.py by @lori-ren in #18652
  • [None][feat] integrate PrimTS FMHA kernels by @yuxianq in #17399
  • [TRTLLMINF-357][infra] Consolidate runBranchesWithInfraDefer into shared lib by @brnguyen2 in #18385
  • [https://nvbugs/6720250][fix] Use adjusted clock for VisualGen timing by @chienchunhung in #18686
  • [https://nvbugs/6676406][fix] Fix DSV4 disagg MTP accuracy by @mikeiovine in #18554
  • [TRTLLM-15316][fix] SM107 FP8 GEMM routing and runtime guards (follow-up to #17485) by @farazkh80 in #18612
  • [None][fix] Update stale attention and KV-cache import paths that break main by @brnguyen2 in #18771
  • [None][fix] Self-sampling top-k host: physical row-width envelope and exact-row warmup population by @longcheng-nv in #18683
  • [https://nvbugs/6594241][fix] increase timeout for test_disaggregated_gpt_oss_120b_harmony by @reasonsolo in #18649
  • [None][fix] validate the layer-wise replay request against the calibration before it runs by @fredricz-20070104 in #18260
  • [None][test] Enable strict checks for declarative extra import path by @tongyuantongyu in #18586
  • [None][feat] lm_eval: let the CLI override a task's stop strings by @brnguyen2 in #18751
  • [None][test] Fix layerwise replay bugs by @fredricz-20070104 in #18604
  • [https://nvbugs/6428069][fix] Drain only the due PP relay send before forward and unwaive the disagg PP tests by @lori-ren in #18135
  • [None][test] Add func and perf cases for Qwen3.6-35B-A3B and gemma4 on Spark and delete llm_spark_core.txt testlist by @JennyLiu-nv in #18655
  • [TRTLLMINF-250][infra] Migrate BoltProfileGen JNLP image to Artifactory by @chzblych in #18791
  • [None][fix] bench_moe: prune MNNVL comm methods that cannot span nodes by @guqiqi in #18785
  • [https://nvbugs/6682352][fix] rebuild MNNVL communicator on layout change by @xxi-nv in #18715
  • [None][test] Remove H20 in QA perf test by @yufeiwu-nv in #18648
  • [None][docs] Add KV cache compression documentation and examples by @Hudayday in #18587
  • [None][test] add Kimi K3 feature matrix coverage by @cascade812 in #18699
  • [None][test] check relative links in documentation by @StanleySun639 in #18718
  • [https://nvbugs/6621358][fix] Enable one-model draft KV reuse in cache manager V2 by @yizhang-nv in #18093
  • [None][feat] support Kimi K3 KDA replay with KV cache manager V2 by @VALLIS-NERIA in #18294
  • [None][feat] Add opt-in GPU keepalive to the benchmark fill gate by @qiaoxj07 in #18749
  • [None][chore] point PrimTS vendor source to dev branch by @yuxianq in #18833
  • [None][chore] Cleanup kv cache manager by @Funatiq in #18538
  • [None][fix] Install with the venv interpreter, not sys.executable by @brnguyen2 in #18773
  • [None][fix] Preserve configured Mamba snapshot placement by @liji-nv in #18724
  • [None][feat] Add --trtllm-custom-output-len to override OSL for the trtllm_custom dataset by @qiaoxj07 in #18790
  • [https://nvbugs/6642522][fix] Prevent orphaned VisualGen workers by @ishovkun in #18030
  • [None][feat] Two-level GVR decode top-K dispatch; remove the CUDA heuristic and temporal-only prior state by @longcheng-nv in #18446
  • [None][test] add a basic disaggregated multi-node tier by @fredricz-20070104 in #18404
  • [None][test] align perf disagg cases to the PYTHON transceiver runtime by @fredricz-20070104 in #18828
  • [https://nvbugs/6644226][fix] page out MoE safetensors during weight load regardless of EPLB by @chenfeiz0326 in #18832
  • [None][feat] add parallel feature for perf-optimize in agent flow by @GuanhuaWang2001 in #18568
  • [TRTLLM-14558][chore] Add the forwarding modules for the retired Attention paths by @YihuiLu512 in #17969
  • [None][test] Cleanup/multinode test list by @xinhe-nv in #18490
  • [TRTLLM-14963][TRTLLM-14966][refactor] Give the DeepGEMM MoE implementations canonical identities and add an impl_id pin by @xxi-nv in #18748
  • [TRTLLMINF-334][infra] Report FAILURE for unrerun test failures skipped due to duration/no-signature match by @yiqingy0 in #18590
  • [None][chore] Reuse datas in datasets to build test prompts with number larger than origin dataset by @HuiGao-NV in #15952
  • [None][feat] Add lightweight remote execution to performance workflows by @GuanhuaWang2001 in #18789
  • [None][chore] add attention team to vendor lock owners by @yuxianq in #18840
  • [None][fix] Let the kernel ledger record comm kernels and partial ncu captures without hiding the gaps by @kaiyux in #18574
  • [TRTLLM-15621][feat] Support in-graph sampling for temperature/top-k/top-p batches by @zhaoyangwang-nvidia in #18437
  • [None][fix] Make LlmRequest.is_generation_only_request a property to match the C++ binding by @eopXD in #18858
  • [https://nvbugs/6621358][test] Unwaive tests tracked by NVBug 6621358 by @yizhang-nv in #18829
  • [TRTLLM-15892][feat] Add Anthropic Messages API support to trtllm-serve by @JunyiXu-nv in #18289
  • [TRTLLM-13767][feat] integrate FP4 Conv3d into parallel Wan VAE by @taianz-nv in #17262
  • [TRTLLMINF-396][ci] Auto-label fully approved pre-merge PRs by @ZhanruiSunCh in #18729
  • [None][chore] Cleanup unused C++ algorithms by @Funatiq in #18540
  • [TRTLLM-15033][fix] Revert #17800 FlashInfer CuTeDSL MLA dispatch by @pengbowang-nv in #18653
  • [None][perf] speed up the block-scale MoE, GDN and sampler decode paths for Qwen3.8-flash-next by @Wanli-Jiang in #18714
  • [None][feat] Decompose the nsys timeline in the perf-analyze/optimize workflows by @kaiyux in #18805
  • [TRTLLM-11484][fix] Address Quantization Regressions for VisualGen CuTeDSL FMHA by @xrq-phys in #18750
  • [https://nvbugs/6682113][fix] give gb300 v4-pro con4301 ctx_only perf-sanity the 120min budget its workload already carries by @chenfeiz0326 in #18859
  • [None][test] update Coderabbit review by @xinhe-nv in #18857
  • [None][feat] perf-optimize: add elimination and overlap questions to the per-kernel coverage contract by @kaiyux in #18825
  • [https://nvbugs/6667807][fix] Support more than 128 EP ranks in MoE two-sided prepare cumsum by @dongfengy in #18799
  • [None][feat] Page the DSpark drafter context through the draft KV cache manager by @dc3671 in #18343
  • [None][test] Add Kimi K3 TEP8 performance coverage by @yufeiwu-nv in #18806
  • [https://nvbugs/6702267][fix] Stage kv_block_ids_per_seq unrotated so the block table matches… by @trtllm-agent in #18564
  • [TRTLLM-16103][feat] protect private model architecture names by @Mgluhovskoi in #18455
  • [None][test] Add sparse MQA/GQA coverage and support documentation by @lfr-0531 in #18106
  • [None][perf] Bypass static transfer admission for async Python Scheduler V2 by @chienchunhung in #18150
  • [https://nvbugs/6676844][test] Unwaive test_wan_t2v_example after CI checkpoint storage fix by @chang-l in #18732
  • [https://nvbugs/6721560][fix] Read the conditioning image in the three stale callers… by @trtllm-agent in #18745
  • [MX-299][feat] Delegate MX loading to ModelExpress strategies by @zhengluo-nv in #17029
  • [https://nvbugs/6572838][ci] Unwaive fixed test by @2ez4bz in #18911
  • [None][fix] MoE: keep separate NVFP4 gate/up global scales where supported by @brnguyen2 in #18772
  • [https://nvbugs/6655986][test] Raise LTX-2 LPIPS golden thresholds to 0.15 and unwaive by @chang-l in #18384
  • [None][test] retire Gemma 3 checkpoint tests by @xinhe-nv in #18580
  • [None][fix] match the sampler's scratch slot row in the fused finish-reason guard by @Wanli-Jiang in #18894
  • [TRTLLM-16182][fix] load the mixed-precision Qwen3.8-Flash-Next NVFP4 checkpoint by @Wanli-Jiang in #18823
  • [None][feat] Self-sampling GVR V2 prefill indexer top-K by @longcheng-nv in #18702
  • [None][fix] Kimi K3: admit every trtllm-gen SiTu quant format, not just MXFP4 by @xguannv in #18709
  • [None][docs] fix --config kv_cache_dtype samples and trtllm-serve positional model path by @nv-guomingz in #18901
  • [None][fix] Fix handling of hybrid FlashInfer page tables with KV cache V2 by @BoyueZ in #17643
  • [https://nvbugs/6737123][fix] Preserve PATH in vendor source offline checks by @yuxianq in #18930
  • [https://nvbugs/6655990][test] Gate WAN multi-GPU LPIPS against a within-build reference and unwaive by @chang-l in #18386
  • [TRTLLM-14645][fix] VisualGen: cross-hardware CPU RNG for pipeline noise by @zhenhuaw-me in #18477
  • [https://nvbugs/6625851][fix] Fail guided-decoding requests reaching a dead-end grammar state by @zhaoyangwang-nvidia in #18896
  • [https://nvbugs/6683837][fix] Thread lora_params to self.experts(); cache a per-(layer,module) rank… by @trtllm-agent in #18527
  • [TRTLLM-16163][feat] Enable Nemotron 3.5 Super VL serving paths by @eopXD in #18804
  • [None][fix] share Kimi auxiliary streams by @jiaganc in #18728
  • [https://nvbugs/6689016][fix] Limit eager FlashInfer plan cache growth by @2ez4bz in #18557
  • [https://nvbugs/6581063][fix] Release MegaMoE symm buffers on executor teardown by @sunnyqgg in #18182
  • [https://nvbugs/6695563][fix] Densify the bmm LHS with a.contiguous() gated on get_sm_version() in (120… by @trtllm-agent in #18678
  • [None][test] update qa test list by @crazydemo in #18942
  • [https://nvbugs/6641268][fix] Zero paged V-cache tails during context preprocessing by @yuxianq in #18925
  • [None][feat] Add a link mode to the precompiled editable-install path by @brnguyen2 in #18752
  • [https://nvbugs/6640875][test] Stabilize LoRA KV scheduler tests and re-enable seven cases by @yizhang-nv in #18949
  • [None][chore] Clean up unused code in C++ test files by @Funatiq in #18884
  • [https://nvbugs/6530090][fix] autoinstall media deps within test framework by @tburt-nv in #17069
  • [TRTLLM-15757][refactor] Land the ModelRunner contract with the no-KVCache families by @lori-ren in #18784
  • [None][test] expand speculative decoding model coverage by @cascade812 in #18905
  • [None][doc] Document the three generation-side time breakdown segments by @nv-yna in #18213
  • [None][fix] Make llm_args use the canonical custom-tokenizer alias table by @brnguyen2 in #18768
  • [None][chore] Clean up disagg transfer idle progress by @Tabrizian in #18267
  • [None][perf] Optimize HCA compressor decode by @mingyangHao in #18869
  • [TRTLLM-15331][feat] Improve KV event publish performance via StreamingEvent by @tanmayv25 in #17023
  • [https://nvbugs/6670516][fix] Call build_kv_page_indices with current signature by @brb-nv in #18980
  • [TRTLLM-15520][feat] Admit one context per uncached prefix instead of letting every duplicate recompute it by @eopXD in #18195
  • [None][fix] Resolve vocab_size through config within v2_KVCM instead of propagating depending on model family by @eopXD in #18885
  • [https://nvbugs/5836830][test] Remove DeepSeek-R1 W4AFP8 8-GPU quickstart test from QA list by @brnguyen2 in #18979
  • [None][doc] Fix broken images in tech blogs 27 and 28 by @karljang in #18984
  • [None][fix] bridge FP4 MLA disaggregated KV ownership by @chienchunhung in #18041
  • [None][perf] Apply NUMA-aware CPU affinity beyond the main thread by @qiaoxj07 in #18701
  • [https://nvbugs/6626640][test] Bound Eagle VSWA test batch capacity by @dongfengy in #18986
  • [TRTLLM-11484][doc] VisualGen separate out quantized-attention.md by @xrq-phys in #18792
  • [None][feat] Support image input in the Triton llmapi backend by @faradawn in #18381
  • [None][perf] Use indexed in-place state I/O for GDN prefill in verify batches by @nv-guomingz in #18845
  • [None][perf] fuse decode kernel seams for Qwen3.8-Flash-Next and de-vendor the low-M GEMMs by @Wanli-Jiang in #18883
  • [TRTLLMINF-368][infra] update cluster config for OCI-JHB by @tburt-nv in #18982
  • [None][fix] Add Linux EBADHANDLE error handling by @yufeiwu-nv in #18352
  • [None][chore] Update cutedsl to 4.8.0 dev by @rosong11 in #18808
  • [https://nvbugs/6709495][doc] Fix dead links flagged by test_http_url_validity by @nv-guomingz in #18992
  • [https://nvbugs/6530268][fix] Fix XQA race condition and attn sink numerical accuracy by @pengbowang-nv in #18141
  • [None][chore] promote PrimTS source from TRT-LLM #18808 by @yuxianq in #19011
  • [https://nvbugs/6676312][fix] Restore DSA CUDA TopK workspace lifecycle by @lfr-0531 in #18624
  • [None][fix] Re-resolve the MoE op provider once the per-layer quant config is known by @brnguyen2 in #18770
  • [None][chore] BREAKING: Remove unused C++ code by @Funatiq in #18846
  • [None][perf] optimize qwen3.5 perf by removing gather and scatter op by @nv-guomingz in #17365
  • [https://nvbugs/6707519][fix] Re-land attempt 1 verbatim on the current tip — add… by @trtllm-agent in #18629
  • [TRTLLM-15714][refactor] Move disagg send/reap/timeout/cancel orchestration into DisaggTransferCoordinator by @nv-xtf in #18827
  • [TRTLLM-15448][perf] Add checkpoint I/O experiment and startup telemetry by @chienchunhung in #18608
  • [None][fix] Harden VMM-backed DWDP lifecycle by @zhangcl in #18644
  • [None][fix] Raise SetupError instead of NameError when the precompiled wheel is missing by @David-Wu1119 in #18703
  • [TRTLLM-16179][test] Port detokenization stop-word tests to Qwen3-0.6B by @xinhe-nv in #18994
  • [None][feat] Add cuDNN attention backend by @xrq-phys in #18075
  • [None][fix] catch KeyError when parsing --server_role into ServerRole by @lonexreb in #17730
  • [https://nvbugs/6608795][fix] Fall back when UserBuffers multicast mapping fails by @cascade812 in #18037
  • [TRTLLM-15097][test] Prune MiniMax-M2 and MiniMax-M2.5 tests by @xinhe-nv in #19001
  • [https://nvbugs/6731971][doc] Fix broken relative paths flagged by test_relative_path_validity by @nv-guomingz in #19007
  • [https://nvbugs/6676352][doc] Fix two copy-paste-broken doc snippets by @BowenFu in #19009
  • [None][test] clean Llama-3.1-8B accuracy test coverage by @xinhe-nv in #18809
  • [None][refactor] Centralize FMHA availability and support capability checks by @yuxianq in #19008
  • [None][fix] Update vendored PrimTS with FlashInfer #4829 follow-ups by @yuxianq in #18996
  • [https://nvbugs/6737351][fix] Remove CGA smem-reduction clamp from trtllm-gen FMHA kernel selection by @kris1025 in #18945
  • [https://nvbugs/6644465][test] Unwaive WAN pipeline parallel test by @taianz-nv in #18928
  • [None][perf] Enable FlashInfer add-RMSNorm for NVFP4 Marlin by @rmeghwal-nv in #17685
  • [None][chore] promote PrimTS source from TRT-LLM #18996 by @yuxianq in #19047
  • [None][doc] Group VisualGen feature guides under one navigation entry by @chang-l in #19023
  • [None][feat] Derive per-layer KV cache windows from layer_types by @brnguyen2 in #18767
  • [None][fix] Narrow CBTS perf helper selection by @yufeiwu-nv in #18880
  • [https://nvbugs/6718910][chore] Unwaive deepseek perf sanity test by @mikeiovine in #19026
  • [https://nvbugs/6537568][fix] Support MXFP4/NVFP4 MoE TP shard padding by @liji-nv in #18870
  • [None][infra] Add blossom-ci authorized users by @trtllm-agent in #19062
  • [https://nvbugs/6739081][fix] Preserve per-layer KV page addressing for mixed head sizes by @yuxianq in #18957
  • [None][test] consolidate CBTS unit tests in CPU stage by @crazydemo in #19052
  • [None][test] Add DeepSeek V4 Pro-Base and NVFP4 DSpark B300 perf tests by @yufeiwu-nv in #18875
  • [None][perf] Speed up burst KVCM2 resize for very long sequences by @lowsfer in #18541
  • [None][test] Add timeouts to test subprocesses by @yihwang-nv in #18812
  • [None][doc] Reduce Sphinx build warnings from 1229 to 258 by @nv-guomingz in #19050
  • [None][feat] add generic PrimTS block-sparse FMHA and unify sparse attention runtime inputs by @heyuhhh in #18815
  • [None][perf] Build GDN verify-path intermediate state indices once per step by @nv-guomingz in #18820
  • [TRTLLM-12670][feat] add /start_profile and /stop_profile endpoints to trtllm… by @JunyiXu-nv in #13872
  • [None][feat] OpenEngine gRPC: wire Generate RPC to the engine with disaggregated serving by @tanmayv25 in #17992
  • [https://nvbugs/6608387][test] Unwaive test_overlap_scheduler cases tracked by 6608387 by @lori-ren in #19066
  • [None][test] add guided decoding architecture coverage by @cascade812 in #18908
  • [https://nvbugs/6662724][fix] Remove validated H100 waivers by @zhaoyuanh-nvidia in #19079
  • [None][fix] Map SM107 to the sm_100f NVRTC target in the trtllm-gen FMHA kernel loader by @farazkh80 in #18758
  • [https://nvbugs/6756996][fix] use llama_model_root fixture evaluation by @tburt-nv in #19031
  • [TRTLLM-15448][refactor] Add checkpoint catalog and shadow load planning by @chienchunhung in #18737
  • [None][infra] Waive failing GPT-OSS 4-GPU test case by @amukkara in #19089
  • [None][ci] Waive test_kimi_k3_trtllm_accepts_nvfp4_routed_experts by @brb-nv in #19092
  • [None][perf] Wire in custom decode kernels for MinimaxM3 by @brb-nv in #18611
  • [TRTLLM-16022][feat] Add sub-agent conversation affinity for disaggregated serving by @xwang233 in #18684
  • [None][fix] Stabilize MoE LoRA CUDA graph scratch by @achartier in #18972
  • [None][fix] Re-plan FlashInfer decode schedule on page-table changes under CUDA graphs by @brnguyen2 in #18769
  • [None][ci] Switch GB300 perf sanity stages to auto:gb300-flex by @chenfeiz0326 in #19074
  • [None][feat] Add no mcp flag for modeling agent by @WeiHaocheng in #18651
  • [None][fix] Shut down resources before GC assertions by @yihwang-nv in #18807
  • [None][feat] perf-sanity: upload per-request disagg lifecycle spans to OpenSearch by @chenfeiz0326 in #18990
  • [None][perf] DFlash draft latency for Qwen3.5-4B by @amukkara in #18553
  • [None][feat] Add SM107 CuTe DSL BF16 dense GEMM/BMM custom ops and dispatch by @farazkh80 in #18761
  • [None][fix] Filter zero-size buffers in KVCM V2 runtime wrapper by @yizhang-nv in #18824
  • [TRTLLM-16203][test] Restore disaggregated-serving coverage on Hopper by @xinhe-nv in #18941
  • [TRTLLM-16194][feat] Add MiniMax H3 support by @yibinl-nvidia in #18733
  • [TRTLLM-16122][feat] Refine VisualGen serve benching by @luyiyun1021 in #18632
  • [None][test] Supply auxiliary streams in Kimi K3 NVFP4 regression by @chang-l in #19088
  • [None][refactor] BREAKING: Remove the two-model speculative decoding path and dead C++ spec-dec code by @zhaoyangwang-nvidia in #18721
  • [None][infra] Classify SLURM resource-cleanup failures as typed infra by @brnguyen2 in #18049
  • [None][feat] Add opt-in KV guard page and fresh-page fill diagnostics by @brnguyen2 in #18754
  • [None][ci] Waive TestQwen3_8_Flash_Next::test_fp8_adp4_mtp3_trtllm_ple_offload on main by @brnguyen2 in #19119
  • [TRTLLM-15936][fix] Enable breakable prefill CUDA graphs (BCG) for Nemotron-H hybrid models by @dominicshanshan in #18888
  • [None][test] Add native PyTorch coverage for DeciLMForCausalLM by @StanleySun639 in #18948
  • [None][feat] Support num_postprocess_workers > 0 under the Ray Orchestration by @shikicloud in #18399
  • [None][feat] Add SM107 CuTe DSL quantized dense GEMM/BMM custom ops and dispatch by @farazkh80 in #18765
  • [None][chore] Refine QA code ownership by @LarryXFly in #18861
  • [None][test] Add coverage for BartForConditionalGeneration by @StanleySun639 in #19017
  • [None][test] Add single-node disagg DEP4 DSpark GSM8K test for DeepSeek-V4-Flash NVFP4 by @longlee0622 in #18853
  • [#17917][fix] Warn when a tool parser detects markup but extracts no tool calls by @100-JM in #18865
  • [#13949][test] Cover /v1/responses in per-request perf metrics tests by @100-JM in #18856
  • [None][test] Add Qwen 3.8 MAX and Flash-next performance coverage by @yufeiwu-nv in #19012
  • [None][fix] Under disagg, build the generation prompt from the context worker's tokens instead of the relayed media by @eopXD in #18886
  • [https://nvbugs/6727262][test] Bound batch size in default-backend smoke tests by @VALLIS-NERIA in #19005
  • [TRTLLMINF-397][infra] Update dependencies to NGC PyTorch 26.08 by @EmmaQiaoCh in #18126
  • [https://nvbugs/6621358][fix] Isolate DeepSeek NVFP4 LongBench MPI session by @yizhang-nv in #19134

New Contributors

  • @zhengluo-nv made their first contribution in #17029
  • @BoyueZ made their first contribution in #17643
  • @faradawn made their first contribution in #18381
  • @David-Wu1119 made their first contribution in #18703
  • @100-JM made their first contribution in #18865

Full Changelog: v1.3.0rc26...v1.3.0rc27

Check out latest releases or
releases around NVIDIA/TensorRT-LLM v1.3.0rc27

Don't miss a new TensorRT-LLM release

NewReleases is sending notifications on new releases.

Get notifications