github NVIDIA/TensorRT-LLM v1.3.0rc24
on GitHub

pre-release4 hours ago

  • Known Issues

    • Workloads combining torch.compile with CUDA graphs can fail during initialization or inference with allocator assertions, missing outputs, or illegal memory accesses.
    • MLA models using MTP with multi-stream or piecewise CUDA graphs on SM120 can asynchronously crash with an illegal memory access.
    • Workloads using KVCacheManagerV2 may intermittently report a teardown failure after otherwise successful inference; this is observed with DeepSeek-V4-Flash.
    • Diffusion image and video pipelines can show output-quality regressions when conditioning defaults or fused normalization change numerical behavior; this is observed with Cosmos 3 Nano T2I/T2V and Wan 2.2 T2V.
    • Multi-GPU low-precision MoE models using fused residual normalization and all-reduce can suffer severe accuracy loss; observed examples include Nemotron V3 Ultra NVFP4 and GPT-OSS W4.
    • Qwen3 MoE models may fail during startup when using FP8 block-scale quantization on Blackwell.
    • Mistral Large-family NVFP4 configurations can show materially reduced GSM8K accuracy, as observed with Mistral Large 3 675B.
    • Multi-rank hybrid models using shared FlashInfer JIT artifacts can fail to start when ranks compile the same kernel concurrently.
    • High-concurrency disaggregated serving over Python NIXL with UCX 1.21 can crash during KV-cache transfer or request cancellation.
    • Hybrid recurrent-attention models using Mamba/GDN state caches can be rejected at startup because required live and dummy slots are underestimated; observed examples include Nemotron V3 Ultra NVFP4 and Qwen3-Next BF16.

  • Model Support

    • Add Kimi K3 with KDA kernels, optimized MoE, parsers, speculative decoding, and disaggregated serving (#17054, #17190, #17225, #17269, #17327, #17334)
    • Add MiniCPM-V 4.6 image and video support to the PyTorch backend (#15976)
    • Add Whisper support to the PyTorch backend (#16141)
    • Add Qwen-Image-Layered and Qwen-Image-Edit-2511 pipelines with CFG parallelism and TeaCache/Cache-DiT (#15096, #16095, #16384, #16339)
    • Add distilled Cosmos3 four-step text-to-image, image-to-video, and video-to-video generation (#16563, #16690, #16155)
    • Add the FastWan2.2 TI2V-5B three-step text-to-video pipeline (#16162)
    • Enable reference-image conditioning for FLUX.2 (#16644)
    • Enable dual-topology parallelism for LTX-2 two-stage generation (#16502)
    • Support mixed image, video, and audio inputs for Qwen2-VL and Qwen3-VL (#16337)
    • Enable multimodal encoder caching, side-stream integration, and partial cache hits for Qwen3.x and Gemma4 (#16817, #16662, #16554, #17231)
    • Add Gemma4 MTP assistant support (#15833)
    • Expand Qwen3, Qwen3.5, and Qwen3-Next compatibility with Qwen3.5-397B BF16/FP8 refit, safer loading, MoE/GDN/MTP fixes, and W4A8 execution (#16642, #16936, #17051, #17120, #17196)
    • Improve Mistral checkpoint, tokenizer, chat, and multimodal format handling (#15123)
    • Improve Nemotron-H quantized loading (#16833)
    • Handle EXAONE 4.5 33B memory constraints (#16992)

  • API

    • Expand TorchSampler with repetition, frequency, and presence penalties, min-p, no-repeat n-grams, and per-request seeds; reject NaN sampling values and raise top_logprobs to 100 (#16485, #16590, #16594, #17172, #17159, #16851)
    • Honor compatible generation_config.json sampling defaults through generation_config="auto" and --generation-config auto (#17213)
    • Add per-request priority to OpenAI chat and completion APIs (#15329)
    • Improve Qwen3 tool-call and DeepSeek-R1 reasoning-parser completion handling (#16866, #17157)
    • Configure disaggregated-serving HTTP keep-alive with server_keep_alive_timeout (#16430)
    • Protect internal disaggregated metadata with shared-secret request signing (BREAKING) (#16002)
    • Refactor multiprocess per-request metrics with header/SSE responses and JSONL output; deprecate perf_metrics_max_requests (BREAKING) (#16668)
    • Replace Mamba cache management with KVCacheManagerV2-backed snapshots and move controls under mamba_state_config (BREAKING) (#16598)
    • Move kv_cache_config.block_reuse_policy to block_reuse_config.policy and add max_num_turns (BREAKING) (#16883, #17277)
    • Add TriAttentionKvCacheCompressionConfig for TriAttention KV-cache compression (#16957)
    • Add advanced_sampling_mode controls for one-model MTP sampling (#16561)
    • Expose prototype DSA MTP indexer top-k sharing through index_share_for_mtp_iteration (#15806, #17149)
    • Enable encoder and mixed encoder-decoder CUDA graphs through new LLM arguments (#16706)
    • Add fp8_context_mla_kv_len_cap for context-MLA workspace and KV-capacity tuning (#16399)
    • Add the MEGAMOE_CUTEDSL backend option and disable deprecated WIDEEP selection (#16190, #17119)

  • Feature

    • Introduce C++ KVCacheManagerV2 with paged Vanilla attention and FlashInfer block reuse (#14047, #16714, #17106)
    • Add batched physical compaction for KV-cache compression (#16836)
    • Enable VisualGen tensor parallelism with Attn2D (#16677)
    • Accelerate VisualGen with FLUX.2 fused NVFP4 SwiGLU, Wan layernorm/shift-scale/quant fusion and temporal VAE batching, and Qwen-Image fast paths (#16143, #15762, #17002, #16142)
    • Enable Marlin NVFP4 on Ada Lovelace and W4A16 NVFP4 on SM120 (#16749, #16511)
    • Expand Blackwell kernels with CuTe DSL FP8/FP16 MLA decode attention and optimized fused MHC half-MMA (#15138, #16799)
    • Optimize sparse-attention top-k and indexing with CuTe DSL radix/GVR kernels, selective DSA K-cache allocation, and reduced prefill work (#15756, #16457, #16558, #16789)
    • Optimize MiniMax M3 with fused projections, MoE routing, QK-norm/RoPE/quantization, SwiGLU and AllReduce-residual-RMSNorm, leaner Index-K caching, and deduplicated eager plans (#16904, #16859, #16906, #17093, #16905, #17091, #16856, #17092)
    • Improve low-latency GDN replay and eliminate state-reset host synchronization (#16768, #16716)
    • Reduce PyExecutor and sampler overhead by avoiding sequence-length host marshalling and logits copies (#16791, #16958)

  • Fix

    • Restore loading/API compatibility for Qwen RoPE dictionaries, composite speculative configs, MultimodalConfig, and deferred LoRA validation (#11575, #16762, #17112, #17138)
    • Correct DSpark NVFP4 mixed-precision drafting and one-model static EPLB (#16831, #16938)
    • Correct DeepSeek V4 FP8 MoE scaling and DSA request mapping during MTP drafting (#16881, #16925)
    • Preserve Qwen multimodal MRoPE and attention-cache correctness (#16944, #16568)
    • Honor Laguna YaRN scaling and use FP32 accumulation in fused all-reduce kernels (#16379, #16753)
    • Use 64-bit indexing for large block-scale MoE workloads (#15887)
    • Preserve accepted recurrent states and relay pipeline-parallel sampling state synchronously during speculative decoding (#16759, #16170)
    • Prevent CUDA graph capture state from leaking into sampling and enable graphs for final context tokens (#17033, #16746)
    • Prevent CUDA graph warmup OOMs and leaks through correct phase-one cleanup, padding preallocation, and partial KV-allocation cleanup (#16763, #16072, #17162)
    • Avoid FlashInfer autotuner mismatches and first-request sampling stalls (#17165, #17286)
    • Preserve KV-cache V2 pool sizing, partial-rewind reuse, and safe cache-reset ordering (#16783, #17122, #17163)
    • Correct disaggregated draft accounting, DSpark KV reservation, speculative window spill, and no-MTP overlap headroom (#16805, #16990, #17361, #17282)
    • Prevent disaggregated transfer failures by canceling pending UCX receives, using fabric memory, and filtering empty NIXL buffers (#16688, #16832, #16993)
    • Prevent MoE hangs by matching attention-DP chunk sizes, falling back from unsupported NCCL-EP paths, and applying phase-aware all-to-all timeouts (#16745, #16956, #17376)
    • Restore FP8 block-scale MoE fallback tactic selection (#17328)
    • Handle MNNVL partial allocations and invalid NVLink topology/probe indices without crashes (#17077, #16603, #17060)
    • Prevent attention-op cache races and torch.compile failures on mutable tensor-list communication ops (#16439, #16926)
    • Correct Cascade attention workspace setup regardless of multi-block mode (#16929)
    • Correct request admission, conversation DP routing, and ADP padding-slot handling (#16897, #16815, #17278)

  • Documentation

    • Add Kimi K3 quick-start, evaluation, performance-sweep examples, and a deployment guide (#17333)
    • Correct trtllm-bench prepare-dataset tokenizer usage and command documentation (#17076)
    • Fix figure and math rendering in tech blogs 25 and 26 (#17100)
    • Update the pinned container in the GPT-OSS deployment guide (#17228)
    • Make the sparse-attention LLM API example runnable (#17055)
    • Add sudo to the APT build prerequisites (#17161)

  • Test & Infra

    • Add Kimi K3 unit tests and GB300 L0 coverage (#17332)
    • Run THOP custom-op schema checks in CPU pre-merge (#17322)
    • Add single-device VisualGen feature and accuracy regression suites with LPIPS baselines (#16683)
    • Migrate runtime, generic, model, and disaggregated CPU-only tests into dedicated CPU stages (#16498, #17290)
    • Stabilize scaffolding worker, OpenAI worker, and LLM tests (#16853, #16964, #16966)
    • Route CUDA graph changes through multi-GPU tests and split slow B300 attention jobs (#17357, #16838)
    • Gate multi-GPU pre-merge stages behind explicit approval and clarify post-merge labeling (#16578, #16886)
    • Enable session prefetch across all test stages (#16770)
    • Add duration-based benchmark execution (#13385)
    • Enable PyTorch profiler traces for VisualGen (#16814)
    • Refresh benchmark recipe configurations and include aggregated perf-sanity recipes in config-database generation (#16809, #15484)
    • Extend MoE and layer-wise benchmarks with Kimi K2/GLM-5 routing and MTP shape metadata (#16830, #17247)
    • Expand performance coverage with cache-transceiver prechecks, Llama/Gemma sanity cases, key-model E2E cases, and overlap-scheduled disaggregated runs (#16674, #16355, #17047, #17390)
    • Harden perf-sanity execution with fail-fast diagnostics, complete-log sentinels, and gen-only warmups (#16403, #16717, #17098, #17140)
    • Bound gen-only benchmark queues and fill targets (#16915, #16961)
    • Improve performance-test launching with cluster-specific UCX settings, KV-cache V2 parsing, and pytest-shard alignment (#16725, #16931, #17134)
    • Right-size and prune performance cases to avoid OOMs and timeouts while retaining large-checkpoint coverage (#16894, #17183, #17197, #17303, #17423)
    • Automate nightly release jobs, embed source-commit wheel metadata, stabilize Docker-stage selection, and fix timer/manual GitLab status reporting (#17248, #17356, #17058)
    • Strengthen release security through nightly source scanning, container remediation, current OpenSSL installation, and removal of unused pytest GitLab credentials (#17394, #16694, #17096, #16946)
    • Add automated PLC risk triage and candidate reporting (#16510)
    • Harden SLURM CI with dispatcher finalization, terminal-failure handling, typed submission failures, and cleaner retry reporting (#16446, #15669, #17255, #17130)
    • Enhance CBTS coverage collection and prevent coverage-write failures from breaking result packaging (#16835, #17304)
    • Expand CI capacity for GB300 and migrate GB200 jobs off aws-dfw (#16977, #17133)
    • Harden build infrastructure with resource-ledgered Docker builds, Artifactory-hosted JNLP images, stale benchmark-copy removal, and the required FlashInfer source-build packaging dependency (#17136, #16996, #16884, #17075)
    • Enable container-local AutoTuner caching in CI (#16794)

What's Changed

  • [None][perf] Skip DeepGEMM clean_logits in DSA indexer prefill on custom top-k path by @dc3671 in #16789
  • [None][perf] Optimize Blackwell fused MHC half-MMA kernel by @MengmSun in #16799
  • [None][perf] prepare_inputs: avoid O(seq_len) get_tokens(0) marshalling on the host by @hyukn in #16791
  • [https://nvbugs/6507109][infra] Split slow DGX B300 attention unit tests by @yuxianq in #16838
  • [https://nvbugs/6479324][test] Remove waiver for fixed qwen3_5_4b_fp8_stress disaggregated stress test by @VALLIS-NERIA in #16878
  • [None][fix] Resolve NVFP4 mixed-precision base layers for the DSpark draft by @tianyuz-nv in #16831
  • [https://nvbugs/6433376][fix] Update the Dense test to mirror the MoE sibling — assert bfloat16 under… by @trtllm-agent in #16203
  • [TRTLLM-13642][feat] Add perf sanity tests for Llama-3.1-8B and Gemma-3-1B and verify cache transceiver V2 support by @moraxu in #16355
  • [https://nvbugs/6501376][fix] Test-only fix — drop the if hidden_size % 2 != 0: with pytest.raises(...)… by @trtllm-agent in #16844
  • [None][feat] Add kimi_k2/glm_5 grouped routing and fused router to bench_moe by @guqiqi in #16830
  • [https://nvbugs/6157892][fix] Mistral format refactor by @evezhier in #15123
  • [None][test] Adjust timeout cases in QA perf test by @yufeiwu-nv in #16894
  • [#15673][fix] Enable CUDA core fast path for SM89/SM120/SM121 by @mihai-chiorean in #12705
  • [TRTLLM-14475][chore] Self-sufficient transfer-agent dlopen and drop onnx/modelopt deps by @Wanli-Jiang in #16829
  • [None][fix] Clarify explicit post-merge stage CI label by @yibinl-nvidia in #16886
  • [None][fix] Drop stale benchmarks copies from Dockerfile.multi by @jieli-matrix in #16884
  • [https://nvbugs/6198785][fix] Unify phase-1 CUDA graph cleanup by @Mgluhovskoi in #16763
  • [TRTLLMINF-40][fix] Introduce a SLURM dispatcher pod "finalizer" by @dpitman-nvda in #16446
  • [https://nvbugs/6507081][fix] Refresh the fakes only — add reasoning_parser = None to… by @trtllm-agent in #16841
  • [None][fix] Fix disaggregated draft token accounting by @SimengLiu-nv in #16805
  • [None][feat] VisualGen TP with Attn2D by @belgarten-nv in #16677
  • [None][feat] Generic Mixed Modality Support by @aswinvisva in #16337
  • [None][perf] Enable FLUX2 VisualGen fused NVFP4 SwiGLU path by @pst2154 in #16143
  • [None][feat] Support MiniCPM-V 4.6 (image + video) on the PyTorch bac… by @hNSBQZ in #15976
  • [https://nvbugs/6479837][fix] Fix OOM of Qwen3_5_35B on a single a100 by @JadoTu in #16891
  • [None][perf] GVR top-K decode: enable R0 histogram-ladder admission by default by @longcheng-nv in #16457
  • [https://nvbugs/6479863][fix] Use scalar SwiGLU limit for DeepSeek V4 FP8 MoE by @lfr-0531 in #16881
  • [TRTLLM-14609][chore] Remove legacy MoE path in CuteDslFusedMoE by @xxi-nv in #16863
  • [TRTLLM-14609][chore] Remove legacy MoE path in DeepGemmFusedMoE by @xxi-nv in #16864
  • [None][feat] Add Qwen-Image-Layered baseline support by @yumin066 in #15096
  • [None][test] Enable session prefetch for all test stages by @sunnyqgg in #16770
  • [None][test] Add missing test durations for MiniMaxM3, Step3_7, and MoE LoR… by @xinhe-nv in #16930
  • [None][test] Fix test_perf.py to accept kv cache manager v2 format by @yufeiwu-nv in #16931
  • [None][fix] Fix nemotron-h quant and loading config by @Wanli-Jiang in #16833
  • [https://nvbugs/6450333][test] Unwaive DeepSeek V4 Flash auto dtype test by @lfr-0531 in #16890
  • [https://nvbugs/6435112][test] Unwaive Wan 2.2 I2V perf sanity test by @taianz-nv in #16409
  • [TRTLLM-11875][feat] BREAKING: MambaCacheManager based on KVCacheManagerV2 & agentic prefix caching by @VALLIS-NERIA in #16598
  • [None][test] Update CODEOWNERS to refine QA ownership by adding specific roles for performance/function/serving testing by @yufeiwu-nv in #16932
  • [https://nvbugs/6510284][fix] Cap gen-only benchmark queue size by @chienchunhung in #16915
  • [TRTLLM-14502][feat] LTX-2 two-stage: dual-topology parallel Stage 2 (cfg folds into ulysses) by @luyiyun1021 in #16502
  • [https://nvbugs/6240584][fix] Qwen3ToolParser: bare-JSON fallback for reasoning-preceded tool calls by @JunyiXu-nv in #16866
  • [None][feat] Batched physical KV-cache compaction for KV cache compression by @Hudayday in #16836
  • [https://nvbugs/6503293][fix] Restore whole-node GPU visibility for default packing by @JacobHu-NV in #16893
  • [None][fix] Update DeepSeek V4 Flash-Base MoE backend configuration in model YAML by @yufeiwu-nv in #16945
  • [TRTLLM-14609][chore] Remove legacy MoE path in CutlassFusedMoE by @xxi-nv in #16861
  • [None][fix] Keep chunked-MoE size vectors identical across attention-DP ranks by @dongfengy in #16745
  • [TRTLLM-13233][feat] Support no_repeat_ngram_size in TorchSampler and refactor token-ban handling into its own submodule by @zhaoyangwang-nvidia in #16594
  • [TRTLLM-11780][feat] Wan 2.2 layernorm + shiftscale + quant fusion by @o-stoner in #15762
  • [None][test] Stabilize scaffolding OpenAI worker tests by @Mgluhovskoi in #16853
  • [None][fix] Don't re-run the SLURM monitor on a terminal job failure by @dpitman-nvda in #15669
  • [https://nvbugs/6484986][test] add UCX cancellation regression coverage by @chienchunhung in #16688
  • [https://nvbugs/6507955][fix] use net_max_seq_len for request admisson instead of inflated one by @bo-nv in #16897
  • [https://nvbugs/6226016][fix] Avoid trusting request-controlled router tokenizers by @yibinl-nvidia in #16697
  • [TRTLLM-12341][feat] Add Whisper support to the PyTorch backend by @pranav-nvidia in #16141
  • [TRTLLM-14135][feat] Add Qwen-Image-Edit-2511 support by @yibinl-nvidia in #16095
  • [TRTLLM-14609][chore] Remove legacy MoE path in TRTLLMGenFusedMoE by @xxi-nv in #16862
  • [None][perf] Avoid Index-K cache materialization for MSA by @peihu-nv in #16856
  • [None][chore] add user by @tburt-nv in #16754
  • [None][feat] Update CuTeDSL MegaMoE kernels by @Barry-Delaney in #16190
  • [TRTLLM-14609][chore] Remove legacy MoE path in DenseGEMMFusedMoE by @xxi-nv in #16865
  • [None][feat] Improve cute dsl radix top-k by @limin2021 in #15756
  • [#15327][feat] Add per-request priority support to OpenAI chat/completions by @sopwg612 in #15329
  • [TRTLLM-13409][test] fail fast + surface server logs when a perf-sanity server dies or never becomes healthy by @JunyiXu-nv in #16403
  • [TRTLLM-14541][fix] VisualGen: deterministic autotuner tactics across runs and ranks by @luyiyun1021 in #16782
  • [https://nvbugs/6503299][fix] Default fabric memory KV pool for Python cache transceiver by @chuangz0 in #16832
  • [None][perf] Preserve default V2 KV cache pool sizing by @2ez4bz in #16783
  • [None][fix] enable static EPLB for the one-model DSpark drafter by @longlee0622 in #16938
  • [https://nvbugs/6255417][fix] Unwaive qwen3next ci test by @JadoTu in #16924
  • [None][feat] cache transceiver test in Perf sanity by @chuangz0 in #16674
  • [https://nvbugs/6490033][fix] Relaxed the assertion to accept both is_eagle3() and… by @trtllm-agent in #16793
  • [None][fix] Keep MRoPE delta read slots dense across mixed batches by @yechank-nvidia in #16944
  • [https://nvbugs/6210714][test] Unwaive TestQwen3_5_35B_A3B fp8 block reuse test by @VALLIS-NERIA in #16997
  • [https://nvbugs/6465993][fix] unwaive mamba tests by @bo-nv in #16941
  • [TRTLLM-14571][infra] Enable container-local AutoTuner cache in CI by @YihuiLu512 in #16794
  • [None][fix] Fix Qwen3.5 weight-load memory growth and MTP CUTLASS fallback by @Wanli-Jiang in #16936
  • [TRTLLM-14551][perf] avoid GDN state reset host synchronization by @liji-nv in #16716
  • [https://nvbugs/6487040][test] Wait for gen-log end-of-write sentinel before parsing per-iter step time by @chenfeiz0326 in #16717
  • [https://nvbugs/6337224][fix] Update PERF_SANITY_DIR to include aggregated/; in recipe_to_server_config… by @tensorrt-cicd in #15484
  • [TRTLLM-14736][chore] Split the sampler package into per-feature modules by @zhaoyangwang-nvidia in #16981
  • [#8384][fix] use dict.get() instead of getattr() for rope_scaling dict access by @wojciech-wais in #11575
  • [None][feat] Multimodal encoder cache: per-item partial hits by @aswinvisva in #16817
  • [https://nvbugs/6163690][fix] Use PreTrainedTokenizerFast in trtllm-bench prepare_dataset to avoid NemotronH config parsing error by @pamelap-nvidia in #16820
  • [None][perf] Fuse index-q/index-k projections in MinimaxM3 by @brb-nv in #16904
  • [None][perf] Fuse MiniMax-M3 MoE routing by @peihu-nv in #16859
  • [https://nvbugs/6529626][fix] Pin mcp<2.0.0 and unwaive the scaffolding tests by @JunyiXu-nv in #16980
  • [None][perf] Add Qwen Image VisualGen perf fastpaths by @pst2154 in #16142
  • [None][fix] Increase max top logprobs limit by @yibinl-nvidia in #16851
  • [https://nvbugs/6424956][fix] Support large FP8 quantization grids by @lfr-0531 in #16933
  • [https://nvbugs/6523880][fix] Restored the legacy-files.txt entry and regenerated the three derived… by @trtllm-agent in #16943
  • [None][chore] update DeepGEMM to 2.6.1 by @Barry-Delaney in #16673
  • [None][test] add tests to qa list by @xinhe-nv in #16988
  • [None][fix] cascade: wire workspace regardless of multi_block_mode by @Nic-bit in #16929
  • [None][infra] Remove unused GitLab token env from pytest by @mzweilz in #16946
  • [None][feat] Support the DMD2-distilled Cosmos3-Super-Text2Image-4Step checkpoint by @ishovkun in #16563
  • [None][feat] MTP one-model advanced_sampling_mode: skip redundant top-k / top-p filter kernels with additional config enum by @jhaotingc in #16561
  • [https://nvbugs/6517842][fix] Handle mutable tensor lists in remove c… by @liji-nv in #16926
  • [https://nvbugs/6276841][fix] When torch_compile=True, pass kv_cache_config=KvCacheConfig(free_gpu_memory_fra… by @tensorrt-cicd in #15095
  • [TRTLLMINF-250][infra] Change jnlp image from urm.nvidia.com to artifactory.pdx.nvidia.com by @yiqingy0 in #16996
  • [None][infra] Select UCX env for perf sanity by cluster name by @chuangz0 in #16725
  • [None][fix] SA spec dec: promote accepted hybrid recurrent states in-worker by @brnguyen2 in #16759
  • [None][infra] Container vulnerability fix by @yuanjingx87 in #16694
  • [None][perf] Fused SwiGLU-OAI by @brb-nv in #16905
  • [None][fix] Add mutex to avoid potentially concurrent modifications to std::unordered_map by @yihwang-nv in #16439
  • [TRTLLM-13229][feat] implement repetition / frequency / presence penalties for TorchSampler by @lori-ren in #16485
  • [https://nvbugs/6463829][fix] Fix fp8 MoE test by @brb-nv in #16514
  • [https://nvbugs/6487836][chore] unwaive test_performance_alignment[1] by @tburt-nv in #17066
  • [None][test] Stabilize scaffolding LLM tests by @Mgluhovskoi in #16966
  • [None][test] Stabilize TRTLLM scaffolding worker test by @Mgluhovskoi in #16964
  • [None][fix] SpecDecOneEngineForCausalLM: accept optional hidden_size/vocab_size for composite configs by @brnguyen2 in #16762
  • [TRTLLM-14287][feat] Qwen Image CFG parallelism support by @yibinl-nvidia in #16384
  • [None][feat] KVCacheManagerV2 C++ translation by @lowsfer in #14047
  • [https://nvbugs/6305404][chore] Unwaive DeepSeek V3 Lite L0 test by @Mgluhovskoi in #16975
  • [TRTLLM-13230][feat] support min_p sampling for TorchSampler by @lori-ren in #16590
  • [TRTLLM-14609][chore] Remove ENABLE_CONFIGURABLE_MOE escape hatch and remaining MoE legacy relics by @xxi-nv in #17009
  • [None][refactor] Clean up model paths and remove deprecated configurations in performance tests by @yufeiwu-nv in #17057
  • [None][feat] Enable MM encoder cache on Qwen3.x and Gemma4 VLMs by @2ez4bz in #16662
  • [TRTLLM-14304][feat] Integrate embeddings cache with encoder side-stream by @2ez4bz in #16554
  • [https://nvbugs/6523767][fix] Size MPI worker-identity barrier timeout to cover worker bootstrap by @trtllm-agent in #16971
  • [https://nvbugs/6287561][fix] Add get_sm_version() < 90 check at the top of run_MTP() in… by @tensorrt-cicd in #15343
  • [None][infra] Add blossom-ci authorized users by @yiqingy0 in #17109
  • [TRTLLM-14779][fix] Clear capture-only sampling override from cached CUDA graph metadata by @xwang233 in #17033
  • [None][feat] Log running metric estimates during long lm-eval runs by @brnguyen2 in #16752
  • [None][fix] Fix Qwen3Next MoE expert-quant probe and GDN verify tensor alignment by @Wanli-Jiang in #17051
  • [TRTLLM-14010][feat] report KV cache transfer state on executor hangs by @bo-nv in #16300
  • [TRTLLM-14511][feat] BREAKING: refactor per-request perf metrics for multi-process trtllm-server by @reasonsolo in #16668
  • [None][infra] Bump version to 1.3.0rc24 by @mikeiovine in #17070
  • [TRTLLM-14177][feat] support reference images in FLUX.2 by @karljang in #16644
  • [TRTLLM-14709][infra] Require packaging>=24.2 for FlashInfer source builds by @brnguyen2 in #17075
  • [https://nvbugs/6451425][fix] Remove llama3 eagle test waive by @mikeiovine in #17128
  • [None][feat] Support the DMD2-distilled Cosmos3 4-step image-to-video checkpoint by @ishovkun in #16690
  • [None][perf] AllReduce + ResidualAdd + RMSNorm by @brb-nv in #17091
  • [None][fix] Bind explicit DP rank for new conversations by @krishung5 in #16815
  • [https://nvbugs/6537081][fix] Fix import error by @2ez4bz in #17112
  • [TRTLLM-14497][feat] Add BF16/FP8 refit for qwen3.5_397b by @hchings in #16642
  • [https://nvbugs/5708901][perf] avoid logits copies when computing logprobs by @ixlmar in #16958
  • [TRTLLM-12352][feat] complete MX post-transform qualification foundation by @chienchunhung in #16458
  • [https://nvbugs/6478692][fix] Pass max_workers=16 to _build in EPDVariant.nano_omni_fp8 (mirroring… by @trtllm-agent in #16672
  • [https://nvbugs/6510284][fix] Clamp benchmark fill target in PyExecutor by @chienchunhung in #16961
  • [None][fix] reserve draft KV at disagg transition for gen-only-dspark case by @reasonsolo in #16990
  • [https://nvbugs/6441022][fix] Enable CUDA graph for final context token computation by @SimengLiu-nv in #16746
  • [None][perf] Fuse QK-norm + RoPE and overlap qkv/idx_qk math by @brb-nv in #16906
  • [None][feat] Support DSA MTP indexer top-k sharing by @nvxuanyuc in #15806
  • [https://nvbugs/6541322][fix] Unwaive fixed test by @2ez4bz in #17131
  • [None][feat] Enable PyTorch profiler traces for VisualGen by @chang-l in #16814
  • [https://nvbugs/6450338][fix] Default to FP32 accumulation in allreduce fusion kernels by @dongfengy in #16753
  • [None][feat] Add duration-based execution to benchmark by @weikuo0506 in #13385
  • [None][infra] Always install latest openssl, libssl3t64 by @yuanjingx87 in #17096
  • [https://nvbugs/6528742][test] Disable V2 in legacy Mamba test by @VALLIS-NERIA in #17049
  • [None][chore] Give MLA model-owned aux streams via aux_stream_dict by @dc3671 in #17118
  • [https://nvbugs/6368562][fix] Reserve fp8 context-MLA attention workspace in KV cache estimation by @eopXD in #16399
  • [None][test] Add back new ctx and gen only case by @fredricz-20070104 in #17101
  • [None][test] Add back 1k1k cases for qa side by @fredricz-20070104 in #17099
  • [TRTLLM-14772][doc] Add sudo to apt prerequisites by @bobboli in #17161
  • [None][feat] Add paged KV cache support to Vanilla attention by @yihwang-nv in #16714
  • [TRTLLM-13948][feat] Set DeepSeekV3 to use Python KV-cache transceiver V2 by default by @asfiyab-nvidia in #16908
  • [None][infra] CBTS coverage data enhancement by @crazydemo in #16835
  • [https://nvbugs/6533916][fix] Make sparse attention example runnable by @lfr-0531 in #17055
  • [None][test] add e2e key model perf test by @ruodil in #17047
  • [None][test] Update performance test configuration for qwen3 model to reduce request count from 256 to 64, optimizing resource usage during benchmarking by @yufeiwu-nv in #17197
  • [None][fix] Remove unexistent tests in waive.txt by @shuyixiong in #17198
  • [TRTLLM-14808][fix] MNNVL partial-allocation cleanup and warmup compile fix by @brnguyen2 in #17077
  • [https://nvbugs/6550708][fix] initialize profiler in Cosmos3 test fixture by @achartier in #17212
  • [None][feat] Cosmos3 video-to-video (V2V) generation by @ishovkun in #16155
  • [https://nvbugs/6517844][fix] Fall back to DeepEP when NCCL-EP lacks shared memory by @nv-lschneider in #16956
  • [https://nvbugs/6550708][infra] Re-enable Cosmos3 distilled tests by @achartier in #17214
  • [None][infra] Migrate GB200 jobs off of aws-dfw by @mlefeb01 in #17133
  • [None][fix] Mark prototype status for API flag index_share_for_mtp_iteration by @nvxuanyuc in #17149
  • [None][feat] Qwen-Image TeaCache/Cache-DiT Support by @o-stoner in #16339
  • [TRTLLM-13694][perf] Refresh recipe configs from latest benchmark data by @Mgluhovskoi in #16809
  • [None][feat] Add TriAttention KV-cache compression method by @Hudayday in #16957
  • [TRTLLM-14554][perf] Batch Wan VAE decode along the temporal axis by @taianz-nv in #17002
  • [None][feat] Add Gemma4 MTP assistant support by @lfr-0531 in #15833
  • [https://nvbugs/6481375][test] Unwaive passing DSV3-Lite tests by @lfr-0531 in #17038
  • [https://nvbugs/6533919][fix] Report unknown stages on stderr with difflib near-miss hints, distinguish… by @trtllm-agent in #17065
  • [TRTLLMINF-257][infra] fix build status miss in gitlab when trigger by timer or by hand by @weiminwang-nv in #17058
  • [None][feat] Enable Marlin NVFP4 on Ada Lovelace by @rmeghwal-nv in #16749
  • [None][perf] Deduplicate plan builds for MinimaxM3 in eager mode by @brb-nv in #17092
  • [None][fix] Add opt-in pinned staging for weight-load H2D copies by @qiaoxj07 in #17234
  • [None][feat] Enable block reuse for flashinfer by @2ez4bz in #17106
  • [https://nvbugs/6003113][fix] BREAKING CHANGE: Authenticate disagg request to verify encoded_opaque_state and ctx_info_endpoint by @yibinl-nvidia in #16002
  • [TRTLLMINF-218][infra] Gate multi-GPU CI stages behind 'ci: full pre-merge approved' label by @ZhanruiSunCh in #16578
  • [None][feat] BREAKING Support saving last N turns in per-conversation policy by @jiaganc in #16883
  • [None][fix] KVCacheManagerV2: keep partial rewind endpoints reusable via per-page token coverage by @lowsfer in #17122
  • [https://nvbugs/6525011][fix] Store the owning output tensor in _graph_output_refs for each graph's lifetime… by @trtllm-agent in #17010
  • [TRTLLM-13696][test] Part2.1: Migrate CPU only tests - runtime by @tongyuantongyu in #16498
  • [TRTLLM-14864][feat] Support per-request seed in TorchSampler by @zhaoyangwang-nvidia in #17172
  • [TRTLLMINF-237][infra] Adopt resourceLedger.withResource for BuildDockerImage by @dpitman-nvda in #17136
  • [https://nvbugs/6533913][docs] Update pinned container for GPT OSS by @2ez4bz in #17228
  • [None][fix] Update VisualGen test CODEOWNERS by @yibinl-nvidia in #16848
  • [https://nvbugs/6426847][fix] Unwaive DeepSeekV3Lite MTP test by @cascade812 in #17217
  • [None][chore] Remove unused server_endpoint field from RankInfo by @Tabrizian in #16693
  • [https://nvbugs/6550126][fix] Renamed the integration test's kwarg and its constant to… by @trtllm-agent in #17207
  • [https://nvbugs/6442073][fix] Enforce a fixed max_seq_len for Qwen's AttentionOp caching purposes by @moraxu in #16568
  • [None][fix] Align perf launcher with pytest shard by @chienchunhung in #17134
  • [None][feat] make disaggregated server keep-alive timeout configurable by @qiaoxj07 in #16430
  • [TRTLLM-14904][fix] Work around flashinfer 0.6.15 autotuner cache-key hash/eq inconsistency by @brnguyen2 in #17165
  • [TRTLLMINF-40][fix] Throw typed InfraFailure when SLURM submission yields no job ID by @brnguyen2 in #17255
  • [None][chore] Fix guardword by @tongyuantongyu in #17250
  • [None][fix] Announce MTP shapes to attention metadata in layer-wise benchmarks by @dc3671 in #17247
  • [https://nvbugs/6550727][fix] Pin _force_non_greedy_for_capture=False on the test double so it models a… by @trtllm-agent in #17205
  • [None][perf] Allocate DSA indexer k-cache only for layers that own an indexer by @Tabrizian in #16558
  • [None][infra] Enable OCI-AGA cluster for GB300 by @yuanjingx87 in #16977
  • [None][feat] Share Expert Fusion with cherry-pick #11143 by @leslie-fang25 in #15297
  • [#17158][fix] Reject NaN top_p, min_p and temperature in SamplingParams by @Yigtwxx in #17159
  • [None][fix] Fix Qwen3.5 MoE fallback, GDN alignment, FP8 activation, and draft KV cache by @Wanli-Jiang in #17120
  • [https://nvbugs/6517834][fix] Fix attn_dense LoRA input dim and Triton MoE padded-view output by @trtllm-agent in #17153
  • [TRTLLM-14138][fix] Pre-allocate CUDA graph padding dummy during warmup by @kaiyux in #16072
  • [https://nvbugs/6528742][test] Unwaive Nemotron V3 Super Mamba tests by @VALLIS-NERIA in #17309
  • [TRTLLM-14702][feat] Integrate Kimi K3 KDA decode kernel by @pengbowang-nv in #17054
  • [TRTLLM-13696][test] Part2.2: Migrate CPU only tests - generic, models, disagg by @tongyuantongyu in #17290
  • [None][feat] add CuteDSL FP8/FP16 MLA decode attention fmha lib by @haow-nv in #15138
  • [https://nvbugs/6379316][fix] Reject MNNVL on split NVLink topology by @karljang in #16603
  • [TRTLLM-14701][feat] Update trtllm-gen batchedGemm kernel drop for Kimi K3 MoE by @rosong11 in #17190
  • [https://nvbugs/6561547][fix] Change both fixtures from scope="module" to scope="class" so the ~84 GB is… by @trtllm-agent in #17302
  • [None][fix] Suppress agent-path junit for monitor-detected SLURM infra retries by @dpitman-nvda in #17130
  • [TRTLLM-14903][fix] Free partially-allocated warmup dummy KV blocks and count spec extra tokens in warmup block estimates by @brnguyen2 in #17162
  • [None][doc] Fix broken figures and math rendering in tech blogs 25 and 26 by @zhenhuaw-me in #17100
  • [TRTLLM-14557][test] Add single-device visual-gen feature accuracy regression tests by @zhaoyuanh-nvidia in #16683
  • [https://nvbugs/5805494][fix] Use int64 indexing in trtllm-gen block-scale MoE kernels by @dbari in #15887
  • [None][feat] Add Kimi K3 KDA prefill/MTP decode CuTe DSL kernels and fused attention-residual kernel by @brnguyen2 in #17225
  • [None][fix] Bound GEN log sentinel wait by @chienchunhung in #17140
  • [None][fix] Stop NVLink probe from throwing on out-of-range link indices by @dc3671 in #17060
  • [https://nvbugs/6327149][fix] Handle EXAONE 4.5 33B memory constraints by @yechank-nvidia in #16992
  • [https://nvbugs/6545424][perf] Enable Qwen3.5 fused ops under torch.c… by @liji-nv in #17243
  • [None][test] Adjust perf test cases to avoid OOM and remove outdated test cases by @yufeiwu-nv in #17303
  • [https://nvbugs/6550127][fix] Support Gemma4 multimodal cache partial hits by @2ez4bz in #17231
  • [https://nvbugs/6428094][chore] Unwaive DeepSeekV3Lite tp2pp2 test by @cascade812 in #17155
  • [https://nvbugs/6265490][fix] Defer LoRA path validation by @yibinl-nvidia in #17138
  • [None][test] default the disagg cache-transceiver precheck to off by @chuangz0 in #17337
  • [TRTLLMINF-240][infra] L0 job enhancement for automatic nightly release by @niukuo in #17248
  • [None][fix] Drain in-flight requests before clearing the KV cache reuse state by @lowsfer in #17163
  • [None][infra] Stop CBTS coverage writes from failing the stage results tar by @crazydemo in #17304
  • [https://nvbugs/5945081][fix] un-waive DeepSeek-V3-Lite NVFP4 pp4 CUTLASS test by @xxi-nv in #17301
  • [None][test] Enable warmup request for gen_only perf sanity lanes by @erictsai-nv in #17098
  • [https://nvbugs/6567403][fix] Waive hanging MoE multi-GPU tests on Hopper by @xxi-nv in #17344
  • [None][test] Fix perf-core timeouts and 96G-part coverage for large-checkpoint cases by @ruodil in #17183
  • [None][fix] BREAKING Block reuse policy rename and add more tests by @jiaganc in #17277
  • [https://nvbugs/6388153][fix] Modify the broadcast of sample state in pp to default to synchronous mode. by @WeiHaocheng in #16170
  • [https://nvbugs/6567403][fix] Revert #17010 to unblock DGX_H100 PyTorch-Others-1 stage timeout by @xxi-nv in #17352
  • [None][ci] Trigger multi-GPU tests for CUDA graph runner changes by @QiJune in #17357
  • [None][chore] Update flashinfer-python from 0.6.15 to 0.6.16 by @nv-guomingz in #17175
  • [https://nvbugs/6564714][fix] Fix the FP8BlockScaleMoERunner get_fallback_tactic by @leslie-fang25 in #17328
  • [None][test] Run thop custom-op schema checks in the CPU pre-merge stage by @brnguyen2 in #17322
  • [None][fix] Fix Qwen3 w4a8 model execution failure and add unit test (re-open of #14527) by @rosenrodt in #17196
  • [#17156][fix] Flush buffered text in DeepSeekR1Parser.finish() by @Yigtwxx in #17157
  • [TRTLLM-14953][feat] Add ability to honour generation_config.json sampling defaults by @DomBrown in #17213
  • [https://nvbugs/6490049][test] Unwaive GB300 Kimi disagg e2e by @chienchunhung in #16920
  • [TRTLLM-14813][feat] Add Kimi K3 (KimiLinear) model by @brnguyen2 in #17269
  • [https://nvbugs/6513132][fix] DSA: rebuild token-to-request map inside the MTP draft loop by @zheyuf in #16925
  • [None][fix] Filter empty aux buffers from NIXL transfers by @chuangz0 in #16993
  • [https://nvbugs/6487038][test] Unwaive GB300 Kimi gen-only by @chienchunhung in #17386
  • [None][infra] Using agent to triage risks detected from PLC pipeline by @yuanjingx87 in #16510
  • [https://nvbugs/6506920][fix] Recover DSpark E2E coverage by @longlee0622 in #17310
  • [None][test] Remove UCX cases on qa side by @fredricz-20070104 in #17389
  • [None][test] Enable overlap schedule on qa side for disagg perf by @fredricz-20070104 in #17390
  • [https://nvbugs/6482589][fix] Make CuError pickle-safe across process boundaries by @lowsfer in #17289
  • [None][infra] Unwaive perf-sanity disagg gen_only cases by @chenfeiz0326 in #17350
  • [TRTLLM-14345][feat] Improve the GDN Replay Kernel Under Low Latency by @JadoTu in #16768
  • [None][fix] Warm up flashinfer sampling module during engine warmup by @qiaoxj07 in #17286
  • [TRTLLMINF-240][infra] stabilize nightly Docker stage selection by @niukuo in #17356
  • [https://nvbugs/6523520][fix] Halve gb200 r1-fp4 128k8k con128 multi_round to fit perf-sanity budget by @chenfeiz0326 in #17397
  • [None][infra] Add blossom-ci authorized users by @yiqingy0 in #17400
  • [None][test] Remove selected GPT-OSS V1 KV cache tests from CI by @yizhang-nv in #17252
  • [TRTLLM-14934][feat] add MoE implementation identity and contract layer by @xxi-nv in #17270
  • [TRTLLM-14822][feat] deprecate WIDEEP MoE backend by @xxi-nv in #17119
  • [https://nvbugs/6555875][fix] refresh OpenAIServer test fake by @reasonsolo in #17401
  • [None][feat] Default Kimi K2.5 to KV cache manager V2 by @liji-nv in #17125
  • [TRTLLM-14831][chore] Consolidate package bootstrap and relocate restricted deserialization by @YihuiLu512 in #17254
  • [None][fix] laguna: honour attention_factor as final YaRN scaling coefficient by @joerowell in #16379
  • [None][fix] Tolerate ADP pad-dummy surplus instead of asserting by @lingjiew in #17278
  • [None][feat] Default MiniMax M2 to KV cache manager V2 by @liji-nv in #17360
  • [None][infra] Enable source code scanning for nightly release by @yuanjingx87 in #17394
  • [None][refactor] Unify sparse attention framework with clean backend interfaces by @lfr-0531 in #12733
  • [None][fix] Set in_mtp_draft_loop on the synthetic DSA metadata stub by @brnguyen2 in #17416
  • [None][fix] tolerate windowed-group speculative spill in disagg KV send by @qiaoxj07 in #17361
  • [TRTLLM-14813][test] Port Kimi K3 unit tests and wire GB300 L0 stages by @brnguyen2 in #17332
  • [https://nvbugs/6523751][fix] Raise per-test timeout for gpt-oss-120b on SM120 by @farazkh80 in #17423
  • [TRTLLM-12720][feat] Support nvfp4 w4a16 on sm120 by @pamelap-nvidia in #16511
  • [https://nvbugs/6533914][docs] Fix trtllm-bench prepare-dataset: --tokenizer is not a valid global option by @FrankD412 in #17076
  • [None][feat] Add FastWan2.2 TI2V-5B DMD pipeline (3-step text-to-video) by @Kambili in #16162
  • [None][perf] Fold q/k/v quantization into qknorm_rope_fused kernel & remove contiguous by @brb-nv in #17093
  • [TRTLLM-14813][doc] Add Kimi K3 examples and deployment guide by @brnguyen2 in #17333
  • [https://nvbugs/6482566][fix] Make MoE all-to-all completion-flag timeout phase-aware by @xwang233 in #17376
  • [https://nvbugs/6428096][fix] Unwaive DeepSeekV3Lite compiled tp2pp2 by @cascade812 in #17375
  • [TRTLLM-14708][fix] Populate speculative-decoding request perf metrics in the PyTorch one-engine flow by @brnguyen2 in #17082
  • [TRTLLM-14814][feat] Kimi K3 serving parsers, chat template, and speculative decoding (suffix automaton + DFlash scaffold) by @brnguyen2 in #17327
  • [TRTLLM-14815][feat] Enable disaggregated serving for Kimi K3 by @brnguyen2 in #17334
  • [None][perf] optimize encoder-decoder PyTorch performance by @cascade812 in #16706
  • [https://nvbugs/6388363][test] Unwaive the test of TestDeepSeekV3Lite… by @WeiHaocheng in #17398
  • [None][fix] fix disagg overlap slot headroom without MTP by @reasonsolo in #17282

New Contributors

  • @MengmSun made their first contribution in #16799
  • @hNSBQZ made their first contribution in #15976
  • @sopwg612 made their first contribution in #15329
  • @Nic-bit made their first contribution in #16929
  • @krishung5 made their first contribution in #16815
  • @weikuo0506 made their first contribution in #13385
  • @rmeghwal-nv made their first contribution in #16749
  • @Yigtwxx made their first contribution in #17159
  • @haow-nv made their first contribution in #15138
  • @zhaoyuanh-nvidia made their first contribution in #16683
  • @lingjiew made their first contribution in #17278
  • @Kambili made their first contribution in #16162

Full Changelog: v1.3.0rc23...v1.3.0rc24

Check out latest releases or
releases around NVIDIA/TensorRT-LLM v1.3.0rc24

Don't miss a new TensorRT-LLM release

NewReleases is sending notifications on new releases.

Get notifications