github NVIDIA/TensorRT-LLM v1.4.0rc3
on GitHub

pre-releaseone hour ago

Highlights

  • Known Issues

    • DSA models, including GLM-5.2, may produce incorrect output when using an NVFP4 KV cache.
    • Kimi K3 DSpark with MegaMoE CuTe DSL TEP and CUDA graphs may hang during GB300 warmup.
    • DeepSeek-V4-Pro initialization may run out of GPU memory when MPI workers are reused after large MoE workloads.
    • Tensor-parallel serving with RPC orchestration may hang during model initialization when reusing an MPI worker pool across threads.

  • Model Support

    • Add Muse Spark Open model and configuration support #21042, #20643
    • Add LTX-2.3 text-to-video and retake pipelines #18111
    • Add GLM-5.2 NVFP4 Modeling V2 targets for GB300 and SM103 #20524, #20662
    • Add MiniMax-M3 NVFP4 low-latency decode for Blackwell and MiniMax-M3.1-Flash DSpark serving #20306, #20922
    • Consolidate Kimi K3 native modeling and support NVFP4 KV caching #20893, #20888
    • Add MiniMax-H3 CPU offloading and tensor parallelism for its text encoder and DiT #21301, #19807, #20176
    • Qualify the Qwen3.5 dense vision-language model for ModelExpress #20764
    • Enable DeepSeek-V4.1-Flash on SM120 #20769

  • API

    • Remove unused LlmRequest fields (BREAKING) #21036
    • Remove obsolete TensorRT-backend compatibility surfaces (BREAKING) #21041
    • Default to eight serving frontends with a single-frontend fallback (BREAKING) #19379
    • Add image, video, and audio inputs to OpenEngine gRPC #20446, #20654, #20655
    • Expose OpenEngine Prometheus metrics and standard gRPC health checks #21034, #20437
    • Add keepalive events for Responses and Anthropic Messages streams #21061, #21030
    • Accept Chat Completions service tier, client metadata, and store fields #21154, #20864
    • Allow Responses web-search requests by default when the server cannot execute them #21065

  • Feature

    • Add unified KV cache manager foundations, sparse attention arenas, and host-backed pools #20178, #21129
    • Add aggregated Helix KV sizing, request preemption, and phase-dependent MLA heads #20759, #21122, #20833
    • Support attention DP connectors, per-view KV-head transfers, and owner-only Helix MLA writes #19369, #20709, #21245
    • Protect BF16 context-first KV transfers and attribute transfer timeouts to their owners #21272, #20708
    • Add trusted KV hints, immutable request metadata, and OpenEngine image routing metadata #20269, #21125
    • Add paged block-causal KV caching and cached attention for VisualGen #19938
    • Add native sparse MLA attention for DeepSeek-V4 generation and DSA Vanilla sparse attention #20209, #16309
    • Support native FP8 MoE LoRA weights and DynamicEPLB runtime integration #18555, #19646
    • Add DSpark ragged graphs and FlashInfer ring-buffer replay for GDN MTP #20072, #20008
    • Add layer and buffer reuse policies for attention and SSM #21088
    • Add blob backend batch registration, bounded queueing, and a CPU example #20884
    • Add TTFT-percentile admission and cancel paired generation when context processing fails #20736, #20491
    • Optimize Helix prefill, Kimi K3 attention-DP execution, and KDA prefill #20939, #20597, #20941
    • Optimize one-model speculative decoding, including vocabulary-parallel selection and DSpark verification #20940
    • Accelerate Qwen3.5, Qwen3.8-Flash-Next, and GLM-5.3-Flash serving on Blackwell #20815, #20937, #20755
    • Add opt-in tiled QKV preprocessing and FP8 block-scale low-M GEMM #21198, #20942
    • Enable FlashInfer GDN chunked prefill on SM107 and fused AdaLN on SM103 #21210, #21396
    • Improve GVR V2 sampling, SID-GR beam search, and GDN execution #21271, #20998, #21268, #21126, #20830, #21132
    • Optimize OpenEngine image preparation, gRPC result delivery, and Qwen vision RoPE #21037, #19760, #20932
    • Reduce sampler logprob memory and parallelize KV host-tier initialization #20803, #20790
    • Add optional Morton token ordering for Wan and fuse MiniMax-H3 QK normalization and RoPE #20787, #19645

  • Fix

    • Remove the direct Git dependency that prevented PyPI wheel publication #21502
    • Preserve Kimi K3 thinking settings and fix Modeling V2 checkpoint loading #21391, #21261
    • Correct MiniMax-M3 video timestamps, index-K cache layers, and NVFP4 weight loading #20695, #20693, #20694
    • Fix DeepSeek tool rendering and parsing, including namespaced Responses calls #21051, #21055, #21056, #20873, #21025
    • Preserve Responses stream status, reasoning settings, and parser detokenization #20868, #21053, #21072
    • Correct Anthropic thinking blocks, JSON schema naming, stream metrics, and empty text handling #20866, #20881, #21106, #20867, #20874
    • Stream MiniMax-M2 tool arguments as deltas and correct GLM reasoning parsing #19757, #21109
    • Restrict local request media paths to allowed directories #21183
    • Reject encoder inputs that exceed a single encoder iteration #21199
    • Preserve multimodal UUIDs and allocate global OpenEngine transfer IDs #19529, #21043
    • Preserve per-rank attention-DP KV metrics and coordinate Helix transfer consensus #21131, #19575
    • Fix KV cache sizing, context-only warmup, and confidential-computing page tables #20772, #21343, #17658
    • Keep encoder-decoder cross-attention KV caches in the model dtype #19864
    • Fix MiniCPM-V 4.6 vision attention metadata and Nemotron-H-Puzzle configuration #21251, #20285
    • Bootstrap Ray NCCL expert parallelism from the correct process group #21347
    • Avoid iteration-log and ZeroMQ shutdown hangs and reduce MPI shutdown imports #19764, #21266, #20434
    • Correct MoE deterministic dispatch, weight mapping, and autotuning behavior #20227, #20228, #21339, #21121
    • Preserve quantization exclusion order and validate B12x MoE inputs #20763, #21057
    • Fix request cancellation, child request creation, and cache reset event delivery #20702, #20905, #21142
    • Preserve serving metrics, profiling, liveness, and performance failure logs #19394, #21182, #21148
    • Give logits processors each beam's hypothesis and correct decoder mapping for beam-history checks #20915, #21027
    • Enable NIXL LIBFABRIC and update EFA in container images #19783
    • Exclude vulnerable filelock releases and keep Inductor's async compilation pool alive #21071, #21007
    • Start trtllm-render without a GPU driver and preserve serving schema updates #20842, #20839

  • Documentation

    • Rewrite Modeling V2 developer guides and update supported-model references #21390, #21353, #20747
    • Clarify KV cache managers, sliding-window compatibility, and disaggregated beam search #21437, #21436, #20474
    • Correct parallelism examples, scheduler guidance, and KV cache V2 defaults #21435, #20352, #20447
    • Fix VisualGen examples and refresh the MoE performance dashboard #21171, #21224, #20898

  • Test & Infra

    • Download Boost from AWS Artifactory and migrate build artifacts to AWS Artifactory #21133, #20494
    • Separate nSpect registration from launch checks and update the PLC pipeline #20913, #19557
    • Upgrade FlashInfer Python and XGrammar dependencies #20911, #20624
    • Add Modeling V2 workflows, family-specific libraries, and route registration #20907, #20896, #21200, #20789
    • Expand CBTS coverage for closures, MPI workers, formatting, and VisualGen changes #21160, #20574, #21326
    • Add model and performance coverage for GLM, Muse, Qwen, Kimi, and MiniMax #20999, #21426, #21257, #21256, #21015, #21080
    • Add GPU-free engine initialization, session reuse, and serving frontend CPU tests #21231, #21349, #21413
    • Improve DeepSeek-V4-Pro test startup and device-memory handling #21225, #21402
    • Reorganize executor, decoder, attention, and component-local C++ tests #20669, #21158, #17558, #21022, #20894
    • Retry CI build stages and retain Slurm output with a retention policy #21127, #20398
    • Pre-bake CI images, split CPU stages, and move test-list checks earlier #15398, #20810, #20584
    • Pull CUDA 13 PyTorch wheels from the cu132 mirror and increase dependency installation timeout #21068, #21430
    • Update performance sanity warmup, metric direction, and model GPU selection #21123, #21197, #20466
    • Support upstream vendor promotion and promote native sparse MLA sources #20744, #21006
    • Modernize engine construction, Mamba cache managers, and multimodal request metadata #20851, #18843, #20391
    • Remove legacy renderer, executor, and CI paths #20775, #21066, #20661, #20743

What's Changed

  • [TRTLLMINF-133][infra] Move check testlist stage earlier in CI by @EmmaQiaoCh in #20584
  • [TRTLLM-16711][fix] Remove the legacy renderer paths by @hchings in #20775
  • [None][fix] Honor deterministic mode in fused mHC dispatch by @shikicloud in #20228
  • [TRTLLM-17175][fix] Fix the post-merge leftovers of the kernel and thop test relayout by @YihuiLu512 in #20681
  • [None][fix] MiniMax-M3: resolve an unprefilled prompt as outside reasoning by @pranav-nvidia in #20837
  • [https://nvbugs/6926284][fix] Align qwen3_coder parameter typing with vLLM by @JunyiXu-nv in #20279
  • [None][fix] Chat Completions: accept the OpenAI store field by @JunyiXu-nv in #20864
  • [None][fix] Anthropic Messages: read the chat stream to its end so its metrics are recorded by @JunyiXu-nv in #20867
  • [None][fix] Anthropic Messages: no whitespace-only text blocks by @JunyiXu-nv in #20874
  • [None][test] Remove unused model-path fixtures from integration conft… by @xinhe-nv in #20892
  • [https://nvbugs/6971880][fix] Sync serving schemas on version bumps by @Mgluhovskoi in #20839
  • [https://nvbugs/6937268][fix] Ignore recoverable pip download retry warnings by @BowenFu in #20748
  • [None][feat] Unified kv cache manager foundation by @allisonlim-nv in #20178
  • [None][feat] TTFT-percentile GEN admission for disaggregated serving, and a fix for a context-chunk replay after a skipped forward by @reasonsolo in #20736
  • [None][doc] Fix documentation errors by @nv-guomingz in #20898
  • [None][refactor] Register modeling_v2 routes with a routes decorator by @WeiHaocheng in #20789
  • [None][feat] Add the PrimTS native sparse MLA FMHA for DeepSeek-V4 generation and enable it by default on SM100 by @yunruis in #20209
  • [https://nvbugs/6826000][test] Remove the AgentX cases from the perf-sanity test lists by @chenfeiz0326 in #20985
  • [TRTLLM-14477][refactor] unify multimodal request data and item metadata by @yechank-nvidia in #20391
  • [#19902][fix] Give logits processors each beam's hypothesis under beam search by @lminer in #20915
  • [None][chore] promote flashinfer-prims-ts source from #20209 by @yuxianq in #21006
  • [TRTLLMINF-471][infra] use Austin model scratch for RTX5080, RTXPro6000D and GH200 by @ZhanruiSunCh in #20666
  • [None][build] Improve cpp/tests/CMakeLists.txt by @Funatiq in #20832
  • [https://nvbugs/6920457][fix] Document True as the use_kv_cache_manager_v2 default in the KV cache guide… by @trtllm-agent in #20447
  • [TRTLLM-15764][refactor] Split PyTorchModelEngine construction into named phases by @lori-ren in #20851
  • [TRTLLM-16303][refactor] modularize Mamba cache managers by @VALLIS-NERIA in #18843
  • [None][fix] EagerMoeWorkspace test and LoRA tests by @Funatiq in #20947
  • [https://nvbugs/6920455][fix] Correct the PyTorch scheduler page's PP and inflight_request_ids claims by @trtllm-agent in #20352
  • [None][fix] Use decoder mapping for beam-history rank checks by @chzblych in #21027
  • [None][perf] use native protobuf and streamline gRPC result delivery by @jthomson04 in #19760
  • [TRTLLM-17241][fix] Start trtllm-render on a host with no GPU driver by @hchings in #20842
  • [TRTLLM-14961][refactor] Give the CuteDSL grouped-GEMM MoE implementations canonical identities by @xxi-nv in #20929
  • [TRTLLM-17144][feat] Load the Muse Spark Open (muse_spark) config by @moraxu in #20643
  • [None][perf] Speed up the post-warmup KV cache NaN/Inf scan by @thorjohnsen in #20928
  • [https://nvbugs/6781271][fix] Responses API: agent-client fixes and disaggregated serving by @JunyiXu-nv in #19883
  • [None][infra] PLC Pipeline update by @yuanjingx87 in #19557
  • [None][feat] Expose standard gRPC health in OpenEngine by @tanmayv25 in #20437
  • [TRTLLM-16810][perf] SID-GR 3/N: Reduce per-step host overhead for wide beam search by @mikeiovine in #20830
  • [https://nvbugs/6900756][perf] KVCMv2: fill the initial host tier in parallel, populate in 4 MiB calls by @thorjohnsen in #20790
  • [https://nvbugs/6820152][fix] Fix Wan I2V example test requiring unavailable NVFP4 checkpoint by @o-stoner in #20845
  • [TRTLLM-16599][perf] Fuse MiniMax-H3 BF16 QK-norm and rotary embeddings by @karljang in #19645
  • [None][fix] DeepSeek DSML tool parsers: keep string parameter values as written by @JunyiXu-nv in #21025
  • [None][fix] Anthropic Messages: accept the thinking display setting by @JunyiXu-nv in #20881
  • [None][fix] Anthropic Messages: sign the thinking blocks the server generates by @JunyiXu-nv in #20866
  • [None][fix] DeepSeek DSML tool parsers: the layout newlines around tool calls are not content by @JunyiXu-nv in #20873
  • [None][fix] Cherry-pick #20393: save logs for stress tests by @bo-nv in #20979
  • [None][fix] Python transceiver: run gen-side KV transfer consensus across helix CP ranks under attention-DP by @chuangz0 in #19575
  • [TRTLLM-17226][perf] Reduce sampler logprobs temporary memory by @zhaoyangwang-nvidia in #20803
  • [None][perf] V2 scheduler: demote long uncached prefills behind short reqs by @lancelly in #20600
  • [https://nvbugs/6949270][fix] Size the Wan2.2-I2V serve test start budget to a cold checkpoint read by @trtllm-agent in #20827
  • [TRTLLM-16320][feat] Add FlashInfer ring-buffer replay for GDN MTP by @asfiyab-nvidia in #20008
  • [TRTLLM-17086][feat] Agg Helix KV cache, part 1: prefill allocation and sizing by @asfiyab-nvidia in #20759
  • [https://nvbugs/6913452][fix] Fix flaky SF check in CuTe DSL gather act-fusion relu2 test by @leslie-fang25 in #20799
  • [None][feat] Add the blob backend with batch registration, bounded queueing and a CPU end-to-end example by @nv-xtf in #20884
  • [None][fix] DeepSeek renderers: tools join the leading system message; V4.1 renders developer as system by @JunyiXu-nv in #21051
  • [https://nvbugs/6964707][fix] Let comm op-matrix ranks import the installed tensorrt_llm by @erictsai-nv in #20814
  • [None][fix] Propagate deterministic flags to MoE finalize selection by @shikicloud in #20227
  • [None][infra] Select modeling_v2 CI tests by the node a change lands in by @WeiHaocheng in #20829
  • [https://nvbugs/6920454][fix] Document disaggregated beam search in the Sampling guide by @trtllm-agent in #20474
  • [None][infra] Refine CodeRabbit test review guidance by @xinhe-nv in #21091
  • [https://nvbugs/6932260][fix] Ensure one-task-per-rank for MpiSession.submit by @tongyuantongyu in #20467
  • [None][fix] Explain K/V pool coalescing failures; test Qwen3.8-Flash-Next QSA layouts and FP8/NVFP4 KV by @Wanli-Jiang in #20904
  • [TRTLLM-17005][refactor] Component-local C++ Tests (Part 4) by @Funatiq in #20894
  • [TRTLLMINF-471][fix] restore default model storage for GH200 by @ZhanruiSunCh in #21099
  • [TRTLLM-15765][refactor] Remove the ModelEngine ABC and settle the executor interface by @lori-ren in #20996
  • [None][fix] DeepSeek-V3.2 encoder: restore the blank lines of the tools preamble by @JunyiXu-nv in #21055
  • [None][fix] allocate global transfer IDs for OpenEngine prefill by @tanmayv25 in #21043
  • [TRTLLM-17175][test] Add a CPU test for the CuTe DSL offline loader and fix two drivers by @YihuiLu512 in #21090
  • [https://nvbugs/6999645][fix] Fix VisualGen MGMN example script by @yingguo-trt in #21097
  • [None][fix] Responses: resolve namespace::tool calls and restore a dropped namespace on replay by @JunyiXu-nv in #21056
  • [None][perf] FP8 block-scale low-M GEMM for decode projections (SM100/103, SM107 tactic tables) by @longlee0622 in #20942
  • [None][perf] Qwen: one-launch vision RoPE and batched mRoPE prep by @yechank-nvidia in #20932
  • [None][fix] Count retained snapshots in hybrid cache sizing by @2ez4bz in #20772
  • [None][perf] One-model spec decode: vocab-parallel token selection, fused DSpark Markov chain and spec tail, drafter TP fusion by @longlee0622 in #20940
  • [None][feat] Responses: keepalive events while a stream is silent by @JunyiXu-nv in #21061
  • [https://nvbugs/6913451][fix] Avoid heavy imports in MPI shutdown by @chienchunhung in #20434
  • [TRTLLM-12891][feat] Support attention DP in V2 KV cache connectors by @SimengLiu-nv in #19369
  • [None][feat] Support image inputs in OpenEngine gRPC by @krishung5 in #20446
  • [https://nvbugs/6248648][fix] Materialize KVCM V2 page tables under CC by @dhansen-nvidia in #17658
  • [None][refactor] Improve child request creation and preserve request inputs by @Funatiq in #20905
  • [None][feat] Kimi K3: helix_a2a token-sliced MoE under helix + attention DP by @dc3671 in #20597
  • [TRTLLM-17383][infra] Download Boost from AWS Artifactory by @sharabeshj-nvda in #21133
  • [https://nvbugs/5961912][test] Re-enable CuteDSL NVFP4 MoE backend tests with large intermediate_size by @xxi-nv in #21101
  • [None][perf] GVR K=2048 reg_clus with block64 sampling and parallel rescue by @longcheng-nv in #20998
  • [#19913][chore] Upgrade XGrammar from 0.1.32 to 0.2.8 by @indrajit96 in #20624
  • [TRTLLM-16976][feat] Enable import-time coverage collection for MPI workers by @tongyuantongyu in #20574
  • [None][fix] install CBTS CLI dependencies before selection by @crazydemo in #20976
  • [https://nvbugs/6165866][fix] Re-enable the TRTLLM-Gen MoE cases skipped for the fixed route-map over-read by @xxi-nv in #21100
  • [TRTLLM-16963][feat] Enable MiniMax-H3 DiT tensor parallelism by @yibinl-nvidia in #20176
  • [https://nvbugs/6971730][fix] Keep Inductor's async-compile pool alive by @yuxianq in #21007
  • [None][chore] Split CPU stage by @tongyuantongyu in #20810
  • [TRTLLMINF-550][infra] Pull CUDA 13 PyTorch wheels from the cu132 mirror by @sharabeshj-nvda in #21068
  • [None][feat] Anthropic Messages: keepalive pings while a stream is silent by @JunyiXu-nv in #21030
  • [None][perf] Helix CP: MLA V up-projection before the all-to-all, O(prompt/cp) prompt partition, FIFO all-to-all sender fence, Helix test harness fix by @longlee0622 in #20939
  • [None][infra] temporarily pause L40S pre-merge stage 2 by @hanjingtian in #21173
  • [https://nvbugs/7004398][doc] Fix VisualGen example commands and feature matrix by @yingguo-trt in #21171
  • [None][feat] Support video inputs in OpenEngine gRPC by @krishung5 in #20654
  • [None][test] Pick DeepSeek-V4 Flash/Pro/DSpark TP/EP by GPU count by @xinhe-nv in #20466
  • [None][test] Add DeepSeek-V4.1-Flash, GLM-5.3-Flash, Qwen3.8-Flash-Next, Kimi-K3 and MiniMax-M3 perf-sanity cases by @chenfeiz0326 in #21080
  • [None][perf] Speed up GLM-5.3-Flash serving on Blackwell by @ruocheng-nv in #20755
  • [None][fix] Skip unclassifiable entries in QA GPU split by @crazydemo in #21028
  • [None][perf] Avoid output concatenation in eager GDN by @2ez4bz in #21132
  • [None][fix] Kimi K3 attn_res persistent fused kernel: 64-bit candidate offsets by @zongfeijing in #21011
  • [None][fix] serve: make multi-frontend metrics, profiling and liveness launcher-owned by @lancelly in #19394
  • [None][test] Add coverage for Qwen4ExpForConditionalGeneration by @StanleySun639 in #21015
  • [https://nvbugs/6958731][chore] Report the reason for retryable readiness failures by @chenfeiz0326 in #20863
  • [TRTLLM-17341][refactor] Remove dead code and constant flags left by the runner migration by @lori-ren in #21066
  • [None][feat] modeling_v2: build target C++/CUDA as per-family libraries by @Wanli-Jiang in #20896
  • [None][test] Add a warmup pass before perf-sanity measured clients by @chenfeiz0326 in #21123
  • [None][perf] Serving runtime: disagg/KV-transfer/ADP fixes, KV cache manager V2 prefix wait, ConversationRouter prefix placement, KV-balanced ADP admission by @longlee0622 in #20938
  • [TRTLLM-17217][fix] Use the registered input processor for MX checkpoints by @moraxu in #20762
  • [None][feat] Consolidate Kimi K3 native modeling work into main (#19990, #19991) by @vsabavat in #20893
  • [None][fix] Load compressed-tensors NVFP4 weights in MiniMax-M3 by @yifjiang in #20694
  • [None][perf] SID-GR 4/N: Skip beam search temperature pass when temperature is 1 by @mikeiovine in #21126
  • [https://nvbugs/6986044][fix] Re-enable GLM modeling_v2 routing tests by @dongfengy in #21196
  • [https://nvbugs/6973693][fix] Disable KV block reuse in test_llmapi_backend by @trtllm-agent in #20912
  • [None][perf] Kimi-K3 KDA prefill grid sizing and MTP-verify kernels; DWDP cross-step expert prefetch with configurable slots by @longlee0622 in #20941
  • [TRTLLM-14620][feat] Add DSpark ragged graph and runtime integration by @zhaoyuanh-nvidia in #20072
  • [TRTLLM-16980][feat] modeling_v2: GLM-5.2-NVFP4 sm_103 tep4 target by @erictsai-nv in #20662
  • [None][fix] Derive MiniMax-M3 index-K cache layers from the model config by @yifjiang in #20693
  • [None][fix] MiniMax-M3 VL: forward video metadata so videos get timestamps by @yifjiang in #20695
  • [TRTLLM-14973][fix] Refuse or honour every input on the B12x MoE path by @xxi-nv in #21057
  • [TRTLLM-17121][feat] Support phase-dependent MLA heads for Helix prefill by @zhaoyuanh-nvidia in #20833
  • [TRTLLM-15886][infra] Remove redundant attention unit-test runs from L0 lists by @YihuiLu512 in #20683
  • [https://nvbugs/6926622][doc] Consolidate K-EXAONE supported-model entries by @BowenFu in #20747
  • [TRTLLM-16749][feat] Add MiniMax-H3 text encoder TP support by @yibinl-nvidia in #19807
  • [None][perf] Attention DP: KV-balanced admission of generation-only requests by @lancelly in #21073
  • [None][doc] Update MoE Perf Dashboard data to 1.4.0rc2 by @leslie-fang25 in #21224
  • [None][infra] Add blossom-ci authorized users by @trtllm-agent in #21243
  • [None][fix] Keep target page size for standalone DSpark/DFlash draft KV pool when drafter kernels accept it by @vsabavat in #21046
  • [None][fix] Exclude vulnerable filelock releases by @yuxianq in #21071
  • [None][infra] restore L40S pre-merge stage 2 by @hanjingtian in #21241
  • [None][chore] Clean up GPU architecture handling by @tongyuantongyu in #20137
  • [https://nvbugs/6991642][fix] fix test_startup_metrics_by_server_info by @hchings in #21237
  • [TRTLLM-17005][refactor] Component-local C++ Tests (Part 3a) by @Funatiq in #21022
  • [TRTLLM-17348][refactor] Reorganize engine and decoder tests around component ownership by @lori-ren in #21158
  • [None][infra] Keep CPU test pods off ipp6/ipp9 DC and clean old node blocklists by @weiminwang-nv in #21098
  • [TRTLLM-17448][feat] Support upstream vendor promotion and discoverable bot feedback by @yuxianq in #20744
  • [None][feat] Paged block-causal K/V cache and cached attention for VisualGen by @ishovkun in #19938
  • [https://nvbugs/6890768][fix] Turn TF32 off (allow_tf32=False) around the stub's attention math with a… by @trtllm-agent in #20091
  • [https://nvbugs/6982380][fix] Open the Harmony final channel in JSON mode eval prompts by @trtllm-agent in #21048
  • [TRTLLM-17342][refactor] Remove model_engine.py from legacy formatting and modernize type annotations by @lori-ren in #21159
  • [TRTLLM-17096][feat] runtime hooks: model opt-in engine fast paths by @xwang233 in #20797
  • [None][feat] Add LTX-2.3 t2v and retake pipeline by @yumin066 in #18111
  • [#21032][feat] Expose OpenEngine Prometheus metrics by @tanmayv25 in #21034
  • [https://nvbugs/6758946][fix] Compare teacher-forced logits in the Nemotron VL image batch-equivalence test by @aswinvisva in #20330
  • [None][fix] Correct TPOT metric direction in perf sanity by @dhansen-nvidia in #21197
  • [None][feat] KVCM2 core: sparse attention arena, host-backed pools and Batch by @lowsfer in #21129
  • [https://nvbugs/6970940][fix] Re-read rank-striped read-ahead extents after the page cache drops them by @trtllm-agent in #21094
  • [TRTLLM-17389][feat] Serve MiniMax-M3.1-Flash with its bundled DSpark drafter by @pranav-nvidia in #20922
  • [https://nvbugs/6700875][fix] Drain cached test MPI pools before a test starts a child process by @aswinvisva in #20331
  • [TRTLLM-14892][feat] Qualify the Qwen3.5 dense vision-language root for MX by @moraxu in #20764
  • [https://nvbugs/7004364][fix] Extend GPU lifecycle startup budget by @trtllm-agent in #21219
  • [None][fix] GLM reasoning parser: drop the owed after a tool call; read the thinking mode off the prompt by @JunyiXu-nv in #21109
  • [None][infra] preserve auto-triggered stages in CBTS coverage tier by @crazydemo in #21093
  • [None][feat] Kimi K3: support NVFP4 KV cache (FP4 MLA) with the hybrid KV cache manager by @reasonsolo in #20888
  • [None][fix] Apply ruff-format to model_engine.py DSpark properties by @EmmaQiaoCh in #21307
  • [https://nvbugs/6567554][fix] Default the layer-wise benchmark prefill MoE backend to AUTO by @trtllm-agent in #17373
  • [None][fix] preserve multimodal UUID in V2 KV events by @GuanLuo in #19529
  • [None][feat] Support audio inputs in OpenEngine gRPC by @krishung5 in #20655
  • [TRTLLM-16976][feat] Handle closures in CBTS by @tongyuantongyu in #21160
  • [TRTLLM-14510][feat] BREAKING: serve: default num_serve_frontends to 8 with single-frontend fallback by @lancelly in #19379
  • [TRTLLM-15314][feat] Support native FP8 MoE LoRA weights by @achartier in #18555
  • [https://nvbugs/6907190][test] Prefetch FlashInfer TRT-LLM-Gen artifacts before Ray workers start by @hchings in #21286
  • [None][infra] Add retry for Build stage by @yuanjingx87 in #21127
  • [None][refactor] Split thop.attention into phased fmha by @yihwang-nv in #17558
  • [None][fix] Kimi K3 modeling_v2: fix checkpoint load and DSpark decode against current shared code by @vsabavat in #21261
  • [https://nvbugs/6944355][fix] Enable the KV bounce buffer for the Qwen3-235B TP1-to-TEP4 con64 disagg perf case by @trtllm-agent in #21254
  • [None][test] Add coverage for Glm4MoeForCausalLM by @StanleySun639 in #21256
  • [None][test] Add coverage for GlmMoeDsaForCausalLM by @StanleySun639 in #21257
  • [TRTLLM-17054][refactor] Reuse PyExecutor across profiling resource transitions by @zhaoyangwang-nvidia in #20359
  • [TRTLLM-17275][fix] Bound TriAttention eviction workspace with a resident fallback by @zhaoyangwang-nvidia in #20871
  • [None][infra] Move the pre-merge perf-sanity cases back to post-merge by @chenfeiz0326 in #21320
  • [TRTLLM-17350][perf] Avoid whole-pool SSM copies in custom Mamba2 MTP updates by @zhaoyangwang-nvidia in #21105
  • [None][perf] SID-GR 5/N: Reduce client side overhead in beam search by @mikeiovine in #21268
  • [None][infra] Remove Verl CI stage by @shuyixiong in #20743
  • [TRTLLM-16618][chore] BREAKING: Remove obsolete TensorRT-backend compatibility surfaces by @Funatiq in #21041
  • [None][feat] Add DSA Vanilla sparse attention by @yihwang-nv in #16309
  • [None][test] Add coverage for Gemma4UnifiedForConditionalGeneration by @StanleySun639 in #21255
  • [None][fix] Chat Completions: accept service_tier and client_metadata by @JunyiXu-nv in #21154
  • [None][fix] Perf metrics: the JSONL writer survives unexpected errors by @JunyiXu-nv in #21148
  • [None][fix] Preserve serve performance failure logs by @yufeiwu-nv in #21182
  • [https://nvbugs/6899677][fix] Default NCCL_GRAPH_REGISTER=0 before creating the pipeline-parallel NCCL communicator by @trtllm-agent in #20805
  • [None][fix] Responses: a reasoning object turns thinking on; DeepSeek-V4.1 takes OpenAI effort levels by @JunyiXu-nv in #21053
  • [None][feat] Support layer and buffer reuse policies for attention and SSM by @jiaganc in #21088
  • [TRTLLM-17359][fix] Preserve per-rank attention-DP KV metrics by @chienchunhung in #21131
  • [None][fix] Responses: detokenize for the parsers as chat completions does by @JunyiXu-nv in #21072
  • [None][refactor] Adopt per-family native libraries for K3 modeling v2 by @litaotju in #21200
  • [https://nvbugs/6997729][fix] Keep UCX on loopback in the disagg perf-metrics test by @trtllm-agent in #21236
  • [#20404][perf] Refresh GLM single-node core coverage by @ruodil in #20406
  • [TRTLLM-17145][feat] Add the Muse Spark Open model by @moraxu in #21042
  • [None][fix] DeepSeek-V4.1: namespaced Responses tools and calls in the checkpoint's form by @JunyiXu-nv in #21104
  • [TRTLLM-14849][chore] Rename tests/unittest/_torch/executor to pyexecutor and split out its connector tests by @lori-ren in #20669
  • [None][feat] Add DynamicEPLB kernels and runtime integration by @JintaoPengCS in #19646
  • [None][fix] Progress idle disaggregated worker cancellation by @chienchunhung in #20702
  • [None][doc] Point to model side branches from the supported models page by @brnguyen2 in #21353
  • [TRTLLM-17333][ci] Replace 8-GPU Ray pre-merge stage with single-GPU GDN decode alignment test by @hchings in #21050
  • [None][fix] Run DeepSeek-V4.1-Flash on SM120 by @nvchenghaoz in #20769
  • [None][feat] Add image routing metadata to OpenEngine KV events by @krishung5 in #21125
  • [TRTLLM-17083][feat] Cancel paired GEN on CTX failure or cancellation by @chienchunhung in #20491
  • [None][chore] Update flashinfer-python from 0.6.18 to 0.7.2 by @farazkh80 in #20911
  • [TRTLLM-17500][fix] Publish cache reset events without waiting for inference by @tanmayv25 in #21142
  • [TRTLLM-16598][feat] Sol-Attn: optional Morton (Z-order) token ordering for Wan by @karljang in #20787
  • [TRTLLM-16628][refactor] BREAKING: Remove unused LlmRequest fields by @Funatiq in #21036
  • [None][feat] MiniMax-M3 NVFP4 low-latency decode megakernel for modeling_v2 for Blackwell by @peihu-nv in #20306
  • [None][chore] Hard-link build outputs into the Python package by @yijingl-nvidia in #21292
  • [None][fix] Isolate modeling_v2 tests and scope shared test state by @litaotju in #21345
  • [None][chore] Update Modeling V2 code and test ownership by @litaotju in #21175
  • [#19943][fix] Return an owning fragment from util_convert_type by @peter941221 in #20017
  • [TRTLLM-17014][infra] Migrate build artifacts from On-prem to AWS Artifactory by @sharabeshj-nvda in #20494
  • [https://nvbugs/6858828][fix] Check the FORCE_CHUNK context limit when creating the micro-batch scheduler by @trtllm-agent in #20900
  • [None][perf] Qwen3.5 GB300: upstream the AgentX side-branch optimizations by @dc3671 in #20815
  • [None][perf] Bound OpenEngine image preparation and reuse prefill state by @krishung5 in #21037
  • [#19861][fix] Keep encoder-decoder cross KV caches in model dtype by @iamAbhi-916 in #19864
  • [https://nvbugs/6770977][test] Fix qwen3_32b_fp8 disagg output check and startup budget by @chuangz0 in #21002
  • [https://nvbugs/6858845][fix] Let no-KV-cache runners use FlashInfer attention by @trtllm-agent in #20901
  • [None][docs] Rewrite Modeling V2 developer guides by @litaotju in #21390
  • [TRTLLMINF-132][infra] Pre-bake fat sqsh to eliminate GPU idle time during CI pip installs by @EmmaQiaoCh in #15398
  • [https://nvbugs/6787519][fix] Initialize Nemotron-H-Puzzle configuration correctly by @yt0428 in #20285
  • [None][chore] Remove the Python KV bounce buffer; kv_cache_bounce_size_mb selects the C++ agent bounce by @chuangz0 in #20049
  • [https://nvbugs/6308608][fix] Read request media from local paths only under an allowed directory by @BoyueZ in #21183
  • [None][test] Improve store driver conformance for fixed pools and shutdown by @nv-xtf in #21314
  • [None][fix] Anthropic Messages: name the json_schema built from output_config.format by @JunyiXu-nv in #21106
  • [https://nvbugs/6842275][fix] Normalize unittest parallel-config keys by @trtllm-agent in #19907
  • [https://nvbugs/7023113][test] Guard DSpark DEP4 CuTe DSL test by device memory by @xinhe-nv in #21402
  • [None][test] Add DGX Spark agentic and real-text speculative decoding perf cases by @JennyLiu-nv in #21331
  • [None][fix] Warmup: no generation shapes on a context-only KV cache by @longlee0622 in #21343
  • [None][test] Add GLM-5.3 core performance coverage by @yufeiwu-nv in #20999
  • [None][perf] GVR V2 reg_clus hint-free path: prefix sample, suffix guard and a bounded exact rescue by @longcheng-nv in #21271
  • [None][test] Guard lazy engine initialization and GPU-free imports by @ruodil in #21231
  • [https://nvbugs/7016340][test] Remove the MLA mixed-precision KV numeric test by @Hudayday in #21371
  • [None][fix] KDA Small-BH prefill: 64-bit state-pool row stride by @longlee0622 in #21342
  • [TRTLLM-17557][test] Run the disaggregated launcher telemetry tests in the CPU stage by @JunyiXu-nv in #21412
  • [https://nvbugs/7016691][fix] Bound ZeroMqQueue close linger so teardown cannot hang by @chenfeiz0326 in #21266
  • [None][chore] Remove legacy alternativeTRT CI paths by @chzblych in #20661
  • [https://nvbugs/6997728][fix] Fix DeepSeek-V4-Pro DEP4 MegaMoE CuTe DSL test: skip HF weight cache for oversized checkpoints, drop hard GSM8K floor, fix bootstrap hang by @trtllm-agent in #21225
  • [None][chore] Reject per-layer KV-head configs in the generic HF weight mapper by @lfr-0531 in #21339
  • [TRTLLM-17557][test] Run the SMG single-frontend guard test in the CPU stage by @JunyiXu-nv in #21413
  • [TRTLLM-17556][test] Run every harmony parsing unit test in the CPU stage by @JunyiXu-nv in #21411
  • [None][fix] Correctly select zsh in devcontainer by @Funatiq in #21384
  • [TRTLLM-17125][feat] Preempt requests on KV pressure under aggregated Helix by @asfiyab-nvidia in #21122
  • [None][perf] Enable FlashInfer GDN chunked prefill on SM107 by @aswinvisva in #21210
  • [None][test] Run session-reuse unit tests in CPU CI by @aswinvisva in #21349
  • [TRTLLM-17467][chore] Remove phantom dspark_confidence import hook from accept_stats by @brnguyen2 in #21201
  • [https://nvbugs/6993575][fix] Prime MegaMoE tile variants during autotuning by @trtllm-agent in #21121
  • [https://nvbugs/6908161][fix] Bootstrap NCCL EP from the torch EP group under Ray by @nv-lschneider in #21347
  • [#18297][fix] Prevent iteration log shutdown hangs by @nvegesna-netizen in #19764
  • [None][fix] Enable the fused per-token AdaLN on SM103 (B300/GB300) by @0z5a in #21396
  • [TRTLLM-16750][feat] Support CPU offloading for MiniMax-H3 by @yibinl-nvidia in #21301
  • [None][fix] Fix NCCL scope overhead, benchmark graph defaults and KV bounce streams by @ruodil in #21095
  • [TRTLLM-17218][fix] Keep merged quantization exclude_modules in checkpoint order by @moraxu in #20763
  • [None][feat] Responses: serve a web_search tool the server cannot honour by default; TRTLLM_WEB_SEARCH_UNAVAILABLE=reject keeps the 400 by @JunyiXu-nv in #21065
  • [https://nvbugs/6944606][fix] Recognize Rubin GPUs in VisualGen performance tests by @daichu-nv in #20501
  • [TRTLLM-16460][perf] Qwen3.8-Flash-Next NVFP4 modeling_v2 target: GB300 Pareto optimizations for ISL 8K / OSL 1K by @Wanli-Jiang in #20937
  • [TRTLLM-16460][feat] agent-flow: unified trtllm_modeling_v2 workflow for gated ModelingV2 onboarding and perf tuning by @Wanli-Jiang in #20907
  • [https://nvbugs/7029760][docs] Split parallelism YAML into independent examples by @ruodil in #21435
  • [https://nvbugs/6888945][fix] Use TRTLLM-generated FastWan LPIPS golden by @yibinl-nvidia in #21355
  • [https://nvbugs/6949274][fix] Give five QA list cases bounds that cover a cold checkpoint read by @trtllm-agent in #21302
  • [None][fix] Enable NIXL LIBFABRIC and update EFA in container images by @erezzarum in #19783
  • [None][fix] revert git pin in 8f04d3a by @niukuo in #21502
  • [https://nvbugs/7007647][test] Relax MXFP8 refit logit-overlap threshold to 0.85 by @hchings in #21357
  • [TRTLLMINF-577][infra] increase Python dependency installation timeout to 60 minutes by @hanjingtian in #21430
  • [TRTLLMINF-543][infra] Separate nSpect registration from launch checks and share the release scripts repo with the Kitmaker dry run by @niukuo in #20913
  • [https://nvbugs/7029761][docs] Qualify dynamic-tree sliding-window compatibility by @ruodil in #21436
  • [TRTLLM-17273][chore] Remove dead code in trtllm-serve ahead of the module moves by @JunyiXu-nv in #21176
  • [None][test] extend Qwen3 NVFP4 CUTLASS RTX test timeout by @jieli-matrix in #21421
  • [TRTLLM-17122][feat] Support owner-only MLA KV writes for Helix prefill by @zhaoyuanh-nvidia in #21245
  • [None][feat] Per View KV head transfer by @allisonlim-nv in #20709
  • [None][feat] Protect direct BF16 context-first KV transfers by @chienchunhung in #21272
  • [TRTLLM-16186][fix] Rows name their slots and regions in the backend contract by @Shixiaowei02 in #20975
  • [None][fix] Responses: final snapshot keeps the status the stream reported by @JunyiXu-nv in #20868
  • [None][fix] Kimi-K3: honour the thinking and effort settings of Anthropic and chat requests by @JunyiXu-nv in #21391
  • [None][chore] Refine infra ownership for tests, vendoring, and Jenkins maintenance by @chzblych in #21521
  • [None][fix] Visual-Gen Rubin Bringup - Extend three MiniMax-H3 E2E timeouts to 60 minutes by @daichu-nv in #21487
  • [None][infra] improve CBTS hits for formatting and VisualGen changes by @crazydemo in #21326
  • [https://nvbugs/7011503][fix] Pass max_seq_len to MiniCPM-V 4.6 vision attention metadata by @StanleySun639 in #21251
  • [None][test] Add Muse Spark Open NVFP4 core performance coverage by @yufeiwu-nv in #21426
  • [TRTLLM-17096][feat] modeling_v2: GLM-5.2 NVFP4 low-latency decode target for GB300 by @xwang233 in #20524
  • [#18151][feat] Admit trusted KV hints and carry immutable request metadata by @chienchunhung in #20269
  • [#19755][fix] Stream MiniMax-M2 tool-call arguments as deltas by @MohammadHijjawi97 in #19757
  • [https://nvbugs/7029762][docs] Link current PyTorch KV cache managers by @ruodil in #21437
  • [https://nvbugs/6858505][fix] Reject an encoder input longer than one encoder iteration at admission by @trtllm-agent in #21199
  • [None][feat] Attribute KV transfer timeouts to deadline owners and observed progress by @chienchunhung in #20708
  • [TRTLLMINF-490][infra] Keep SLURM job output and prune it after a retention pe… by @EmmaQiaoCh in #20398
  • [https://nvbugs/7013279][fix] Set host_cache_size 0 for the GB300 Nemotron-Ultra-V3 TEP4 MTP6 perf case by @trtllm-agent in #21550
  • [None][perf] Add opt-in tiled QKV preprocessing by @2ez4bz in #21198

New Contributors

  • @iamAbhi-916 made their first contribution in #19864
  • @nvegesna-netizen made their first contribution in #19764
  • @0z5a made their first contribution in #21396
  • @erezzarum made their first contribution in #19783

Full Changelog: v1.4.0rc2...v1.4.0rc3

Check out latest releases or
releases around NVIDIA/TensorRT-LLM v1.4.0rc3

Don't miss a new TensorRT-LLM release

NewReleases is sending notifications on new releases.

Get notifications