github NVIDIA/TensorRT-LLM v1.4.0rc0
on GitHub

pre-release7 hours ago

Highlights

  • Known Issues

    • Multi-GPU (TP2) streaming or RPC executor workloads may hang during worker initialization when MPI worker pools are reused across LLM instances (observed on H100).
    • DeepSeek MLA models on SM120/SM121 GPUs (for example RTX PRO 6000 Blackwell) may crash with an illegal memory access when CUDA graphs are captured with BF16 BMM.
    • GPT-OSS models on SM120 GPUs may fail with attention data parallelism when automatic communication selects NCCL EP for unsupported hidden sizes.
    • Qwen3 models with tied embeddings (for example Qwen3-0.6B) may fail at model construction when attention data parallelism is enabled with TP greater than 1.
    • OpenTelemetry tracing may export additional FastAPI spans when FastAPI 0.142 or newer is installed.

  • Model Support

    • Add GLM-5.3-Flash (glm5_next) support #19136
    • Support K-EXAONE-2 in the PyTorch backend #17067
    • Add Qwen-Image-Layered with FP8 and Cache-DiT support #16889
    • Load static FP8 Cosmos3 Nano/Super checkpoints without re-quantization #17476
    • Enable EVS for Nemotron Super 3.5 VL and EVS handoff for Nemotron-H in EPD #19648, #19702
    • Load quantized Nemotron-H MTP heads #19558, #19597
    • Add MiniMax-M3 NVFP4 KV cache support with query-length buffer reuse and a native draft KV view #19422, #19768, #19731
    • Add MiniMax H3 ref2va mode and Turbo LoRA tensor-name mapping #19657, #19701
    • Route Qwen3.6/3.8 vision-language checkpoints to the generic Qwen3.5 VLM wrapper #19892

  • API

    • Remove the Python backend of KVCacheManagerV2 (BREAKING) #19154
    • Migrate LlmRequest fields to Python (BREAKING) #19484
    • Add command-line configuration overrides #19555
    • Report per-request speculative-decoding acceptance in responses #19311
    • Return OpenAI-style errors when a request exceeds the context length #19457
    • Add a per-model serving-extension registry to the OpenAI server #19499
    • Ship the OpenEngine gRPC bindings in the wheel, so no separate package install is needed #19371

  • Feature

    • Add a Mooncake store pool, CLI, and V2 scheduler preemption (part 1) #19235
    • Add sparse KV cache configuration, tier-aware locking, and offload with GPU metadata publication; prepare DeepSeek V4 attention for sparse KV offload #19580, #19821, #19502
    • Add VisualGen VSA and SOL sparse attention on PrimTS block-sparse FMHA and integrate updated SM90 SageAttention kernels #18079, #18496
    • Support beam search with the C++ KVCacheManagerV2 #17306
    • Stream C++ KV events with multimodal payloads #19359
    • Add model startup stage timing metrics, sampled iteration statistics, and partial telemetry context for startup failures #18060, #19717, #19598
    • Support model_kwargs with encoder CUDA graphs #19439
    • Speed up MoE routing for batches of up to eight tokens and cache MegaMoE kernel descriptors across forwards #19682, #19901
    • Enable tiled FP8 context FMHA for head dimension 256 on SM120 #19825
    • Add a static tile scheduler override for context FMHA #19873
    • Include KV connector prefixes in V2 scheduler budgeting and add configurable sliding-window attention endpoint retention priority #17974, #19670
    • Weight the speculative draft KV budget split by per-request spend #19744
    • Support post-SiLU clamp in CUTLASS MoE and add an optional finalize-before-combine path to the MoE scheduler #19217, #19749
    • Accept flat int32 arrays as GenerationRequest prompts without converting them to lists #19658
    • Allow the prefix-tokenization cache for MiniMax-M3 text-only prompts #19613
    • Reduce MTP draft-loop and multimodal fusion overhead #19785, #18636

  • Fix

    • Fix host out-of-memory failures by releasing asynchronous IPC payloads when garbage collection is disabled #19869
    • Fix dummy-slot handling for Kimi K2.5 disaggregated EAGLE3 inference with attention data parallelism #19891
    • Harden disaggregated KV transfer: retire ownership after late completion, require verified write coverage before block reuse, and bound KV retirement #19378, #19545, #19673, #19674
    • Report cached_tokens from the prefix-reuse decision rather than the decode position #19743
    • Make seeded requests reproducible with speculative decoding #19601
    • Prevent multimodal scheduling stalls on the KV cache V2 scheduler #19801
    • Fix parsing of hyphenated Kimi K2 tool names, keep Markdown fences in DeepSeek V3 streamed content, and validate unused chat-template controls #19758, #19226, #19351
    • Serialize Hugging Face remote-code loading when building tokenizers #19553
    • Keep MPI progress alive on the pipeline-parallel sample-state relay thread #19643
    • Fix the torch.compile EAGLE3 acceptance length #19899
    • Guard the MiniMax-M3 FP8 indexer against padded cache slots and honor selector strides in MSA short-query attention #19113, #19709
    • Preserve OpenEngine conversation context and prefill handoffs, and release completed OpenEngine results when server garbage collection is disabled #19750, #19858
    • Tear down cleanly when multi-rank warmup fails and scope KV cache warmup capacity constraints to DeepSeek V4 #19456, #19213
    • Set NCCL environment variables before creating the NCCL unique ID #19780
    • Include the SM107 trtllm-gen context FMHA kernels #19626
    • Fix Kimi KDA prefill and decode synchronization and cap the Kimi K3 KDA prefill scratch caches #19581, #19830, #19829, #19885
    • Exclude padded Cosmos3 text from sequence-parallel attention and pin the Cosmos3 guardrail package #18705, #19504
    • Align LTX-2 scale factors with tensor axes #19500
    • Realign misaligned int32 index slices in FlashInfer GDN decode #17028
    • Preserve dense tensor layouts during offloading #18539
    • Skip unused multimodal encoder IPC handles #19791
    • Disable FA4 2CTA by default for VisualGen #19292
    • Prevent unintended dynamic NVRTC linkage through pg_utils #19156
    • Keep telemetry heartbeats running for long-lived sessions and report public runtime architectures #19620, #19704, #19595

  • Documentation

    • Update documentation for NGC PyTorch 26.08 and PyTorch 2.14.0 #19019
    • Remove the outdated Kimi K3 limitation for supported SA speculation #19784

  • Test & Infra

    • Ship a BOLT-optimized SBSA release wheel, and publish and test the post-merge BOLT build alongside the canonical build #19429, #19431, #19603, #19602
    • Remove Conan from the C++ build and improve the build structure #19552, #19445
    • Remove remaining TensorRT-backend leftovers and obsolete TensorRT telemetry compatibility paths #19583, #19585
    • Establish model runner contracts and migrate decoder execution into the runner family #19592, #19696, #19896
    • Update trtllm-gen FMHA artifacts #19591
    • Add VR200 4-GPU serving performance cases for DeepSeek V4 Pro and Kimi K3 #19260
    • Add KV transfer lifecycle and sliding-window attention eviction regression coverage #19663, #19754
    • Replace TinyLlama-1.1B-Chat-v1.0 with Qwen3-0.6B in tests #19178
    • Migrate disaggregated end-to-end tests to C++ bounce buffers #19588
    • Clean up orphaned performance test workers and isolate serving performance test logs #19697, #19930
    • Route documentation changes to docs and CPU validation #19409

What's Changed

  • [None][fix] Fix quantized MTP head loading for Nemotron H by @mikeiovine in #19558
  • [None][perf] Add MiniMax-M3 NVFP4 KV cache support by @peihu-nv in #19422
  • [TRTLLM-16625][fix] Fix seeded request reproducibility on spec dec by @mikeiovine in #19601
  • [TRTLLM-16304][feat] In-tree implementation of staircase by @WeiHaocheng in #19056
  • [https://nvbugs/6801229][fix] Fix Rubin CuTe DSL MoE unit tests by @zhangcl in #19628
  • [https://nvbugs/6786712][fix] Re-landed the previously-verified 3-defect payload tree-identically… by @trtllm-agent in #19427
  • [None][chore] Remove dead code in model engine by @mikeiovine in #19513
  • [TRTLLM-11412][fix] Preserve dense tensor layouts during offloading by @rahul-steiger-nv in #18539
  • [None][feat] Add GLM-5.3-Flash (glm5_next) support by @ruocheng-nv in #19136
  • [None][fix] Make integrator gain gate authoritative by @GuanhuaWang2001 in #19577
  • [None][infra] Add scheduled CodeRabbit semantic checks by @chzblych in #19621
  • [None][fix] Publish semantic reviews with per-PR commit statuses by @chzblych in #19649
  • [None][doc] Update the docs for the NGC PyTorch 26.08 and torch 2.14.0 upgrades by @EmmaQiaoCh in #19019
  • [#17027][fix] Realign misaligned int32 index slices in FlashInfer GDN decode by @Navjot10 in #17028
  • [TRTLLM-12891][feat] Include KV connector prefixes in V2 scheduler budgeting by @eopXD in #17974
  • [None][fix] Require integration evidence in semantic review prompts by @chzblych in #19651
  • [None][feat] Add Qwen-Image-Layered FP8 and Cache-DiT support by @yumin066 in #16889
  • [https://nvbugs/6835139][chore] Unwaive test_compiled_mxfp8_warmup_backend_selection by @mikeiovine in #19656
  • [None][infra] Waive 1 failed cases for main in pre-merge 62247 by @trtllm-agent in #19662
  • [https://nvbugs/6842831][fix] Restore FlashInfer GDN decode test collection by @pranav-nvidia in #19667
  • [https://nvbugs/6820152][test] Enforce VisualGen test requirements by @zhenhuaw-me in #19543
  • [https://nvbugs/6831694][fix] Unwaive MSA cache-view and warmup policy tests by @peihu-nv in #19655
  • [None][fix] Disable FA4 2CTA by default for VisualGen by @o-stoner in #19292
  • [None][fix] Report public runtime architectures in telemetry by @Mgluhovskoi in #19595
  • [None][chore] Waive failing tests by @brb-nv in #19669
  • [None][test] Waive 6 failed cases for main in QA CI by @trtllm-agent in #19671
  • [None][test] Waive 5 failed cases for main in QA CI by @trtllm-agent in #19672
  • [None][chore] Update trtllm-gen FMHA artifacts by @pengbowang-nv in #19591
  • [None][test] Waive 4 failed cases for main in QA CI by @trtllm-agent in #19677
  • [None][fix] Gate Flash-Next FP8 concurrency-16 perf on B300/GB300 by @yufeiwu-nv in #19654
  • [None][test] Waive 5 failed cases for main in QA CI by @trtllm-agent in #19684
  • [None][test] Waive 1 failed cases for main in QA CI by @trtllm-agent in #19686
  • [None][test] Waive 2 failed cases for main in QA CI by @trtllm-agent in #19687
  • [None][infra] Add blossom-ci authorized users by @trtllm-agent in #19689
  • [None][fix] wait for K1 readers before reusing KDA prefill stages by @longlee0622 in #19581
  • [TRTLLM-15824][feat] Enable EVS for Nemotron Super 3.5 VL by @eopXD in #19648
  • [None][test] Waive 10 failed cases for main in QA CI by @trtllm-agent in #19695
  • [None][fix] Guard MiniMax-M3 FP8 indexer against padded -1 cache slots by @brb-nv in #19113
  • [None][feat] agent flow: add per-role backend routing & casebook switch by @GuanhuaWang2001 in #19522
  • [None][chore] Bump version to 1.4.0rc0 by @ZhanruiSunCh in #19616
  • [None][fix] Validate unused chat template controls and normalize router tool arguments by @brnguyen2 in #19351
  • [None][fix] Retire KV transfer ownership after late backend completion by @chienchunhung in #19378
  • [https://nvbugs/6801102][fix] Registered the measured GSM8K reference (89.083) for MXFP8 + FP8 KV cache… by @trtllm-agent in #19633
  • [None][fix] Tear down cleanly when multi-rank warmup fails by @brnguyen2 in #19456
  • [None][fix] Keep telemetry heartbeats running for long-lived sessions by @Mgluhovskoi in #19620
  • [None][fix] Prevent unintended dynamic NVRTC linkage through pg_utils by @yihwang-nv in #19156
  • [None][test] Add coverage for TritonMXFP4LinearMethod.apply by @StanleySun639 in #19675
  • [None][infra] Waive 3 failed cases for main in post-merge 2990 by @trtllm-agent in #19691
  • [https://nvbugs/6848775][waive] Waive test_minimax_m3_piecewise_empty_adp_rank_preserves_caches by @peaceh-nv in #19711
  • [TRTLLM-15760][refactor] Establish model runner contracts and shared model invocation by @lori-ren in #19592
  • [None][fix] Increase DeepSeek V4 FP8 perf server startup timeout by @yufeiwu-nv in #19692
  • [None][perf] MoE routing: small-token single-block kernel for <= 8 tokens in the trtllm-gen custom routing by @Tabrizian in #19682
  • [None][feat] agent-flow: snapshot the composed prompts into the workspace by @kaiyux in #19715
  • [None][fix] Gate disaggregated KV block reuse admission on verified write coverage by @brnguyen2 in #19545
  • [None][infra] Waive 1 failed cases for main in pre-merge 62506 by @trtllm-agent in #19725
  • [None][chore] agent-flow: bump claude-agent-sdk to 0.2.162 and default model to claude-opus-5-5 by @kaiyux in #19720
  • [None][refactor] BREAKING: Remove the Python backend of KVCacheManagerV2 by @lowsfer in #19154
  • [None][test] Remove 32 closed-bug waive entries for main by @trtllm-agent in #19642
  • [None][infra] Waive 2 failed cases for main in pre-merge 62511 by @trtllm-agent in #19728
  • [None][feat] add VisualGen VSA and SOL sparse attention on PrimTS block-sparse FMHA by @heyuhhh in #18079
  • [None][feat] Add a per-model serving-extension registry to the OpenAI server by @brnguyen2 in #19499
  • [None][infra] Waive 3 failed cases for main in pre-merge 62514 by @trtllm-agent in #19730
  • [TRTLLM-16404][fix] support post-SiLU clamp in CUTLASS MoE by @achartier in #19217
  • [TRTLLMINF-336][infra] Apply BOLT to the released SBSA wheel by @mlefeb01 in #19429
  • [None][fix] Move the CuteDSL locality domain weight split into staged hooks by @zhangcl in #19644
  • [None][infra] Waive 1 failed cases for main in pre-merge 62544 by @trtllm-agent in #19742
  • [https://nvbugs/6812347][fix] Include the SM107 trtllm-gen context FMHA kernels by @zhangcl in #19626
  • [TRTLLM-16654][feat] vendor OpenEngine Python bindings by @tanmayv25 in #19371
  • [TRTLLM-16283][feat] Prepare DeepSeek V4 attention for sparse KV offload by @cascade812 in #19502
  • [None][perf] Accept a flat int32 array as GenerationRequest prompt without materializing a list by @peilii in #19658
  • [None][fix] Bound unproven KV retirement with non-resettable deadlines by @chienchunhung in #19673
  • [None][fix] Cosmos3: pin cosmos_guardrail 0.3.2, fetch only the guardrail files it loads, fix the install hint by @ishovkun in #19504
  • [None][test] Waive 1 failed cases for main in QA CI by @trtllm-agent in #19763
  • [https://nvbugs/6843918][fix] Fix gen_only perf-sanity metric bucket, gonc gen-only env, and unwaive fixed cases by @chenfeiz0326 in #19700
  • [https://nvbugs/6625695][fix] Compare teacher-forced logits in the Nemotron VL video batch-equivalence test by @aswinvisva in #19741
  • [None][infra] Reduce semantic review PR timeline noise by @brnguyen2 in #19752
  • [None][fix] Report context-length-exceeded with OpenAI-style message and error code by @brnguyen2 in #19457
  • [TRTLLM-16630][feat] Add CLI config overrides by @Mgluhovskoi in #19555
  • [None][chore] Waive flaky test on main by @brb-nv in #19781
  • [None][fix] Keep exit telemetry independent of blocked heartbeats by @Mgluhovskoi in #19704
  • [None][feat] Mooncake store part 1: pool, CLI, and V2 scheduler preemption by @brb-nv in #19235
  • [TRTLLM-16282][feat] Add sparse KV cache configuration and tier-aware locking by @cascade812 in #19580
  • [None][fix] Latch cached_tokens from the prefix reuse decision, not the decode position by @Tabrizian in #19743
  • [None][feat] Add configurable SWA endpoint retention priority by @xwang233 in #19670
  • [TRTLLM-15824][feat] Enable EVS handoff for Nemotron-H in EPD by @eopXD in #19702
  • [None][feat] Load quantized Nemotron-H MTP replacements by @eopXD in #19597
  • [https://nvbugs/6720547][fix] Exclude padded Cosmos3 text from sequence-parallel attention by @ishovkun in #18705
  • [https://nvbugs/6801108][fix] Set NCCL env vars before ncclGetUniqueId in getComm by @SimengLiu-nv in #19780
  • [None][perf] MTP draft loop: select rows with a prefix slice from draft step 1 by @Tabrizian in #19785
  • [https://nvbugs/6862662][chore] waive test_attention_mla by @tburt-nv in #19808
  • [None][ci] waive pre-existing test failures on main by @brnguyen2 in #19778
  • [None][fix] Disable interactions between automated PR reviewers by @dpitman-nvda in #19805
  • [https://nvbugs/6783973][doc] Stop listing supported SA speculation as a Kimi K3 limitation by @brnguyen2 in #19784
  • [TRTLLMINF-336][infra] Pin one BOLT profile bundle per pipeline by @mlefeb01 in #19602
  • [TRTLLM-16707][feat] Support model_kwargs with encoder CUDA graphs by @tingyangk in #19439
  • [None][fix] Fail stop executor worlds on unproven KV retirement by @chienchunhung in #19674
  • [TRTLLM-15447][feat] Add model startup stage timing metrics by @yijingl-nvidia in #18060
  • [None][fix] Preserve OpenEngine conversation context and prefill handoffs by @tanmayv25 in #19750
  • [None][feat] Report per-request speculative-decoding acceptance on the response by @debermudez in #19311
  • [None][feat] Add iter_perf_stats_interval to sample iteration stats by @longlee0622 in #19717
  • [None][Perf] Resolve the runtime JIT issue by @ziyixiong-nv in #18813
  • [None][feat] Load static FP8 Cosmos3 Nano/Super checkpoints without re-quantizing them by @ishovkun in #17476
  • [https://nvbugs/6809357][fix] Drain incompatible cached test worker pools by @dongfengy in #19770
  • [https://nvbugs/6763479][chore] Update GLM-Image goldens by @jloftin-nv in #19661
  • [TRTLLMINF-15][build] Remove Conan from the C++ build by @Funatiq in #19552
  • [TRTLLM-16672][feat] MiniMax H3 Turbo LoRA Tensor Name Mapping by @yibinl-nvidia in #19701
  • [TRTLLM-16757][fix] Forward-port configuration model-list telemetry capture to main by @Mgluhovskoi in #19811
  • [None][perf] reuse MiniMax-M3 NVFP4 query-length buffer by @pcicotti in #19768
  • [None][build] Improve C++ build structure by @Funatiq in #19445
  • [TRTLLMINF-336][infra] Activate BOLT optimization of the SBSA release image wheel by @mlefeb01 in #19431
  • [None][fix] Honor selector strides in MSA short-query attention by @peihu-nv in #19709
  • [https://nvbugs/6616036][test] Unwaive test by @dominicshanshan in #19637
  • [None][perf] MiniMax-M3: native P128 draft KV view for the NVFP4 hybrid cache by @zheyuf in #19731
  • [None][fix] Support beam search with C++ KVCacheManagerV2 by @yizhang-nv in #17306
  • [None][infra] Add blossom-ci authorized users by @trtllm-agent in #19878
  • [None][infra] Waive 1 failed cases for main in post-merge 3003 by @trtllm-agent in #19879
  • [None][infra] Waive 3 failed cases for main in post-merge 3003 by @trtllm-agent in #19875
  • [None][fix] Scope KVCM warmup capacity constraints to DeepSeek V4 by @yizhang-nv in #19213
  • [None][fix] kdaDecode legacy: __syncwarp before the lane-0 block-reduce stores by @vsabavat in #19830
  • [https://nvbugs/6840916][fix] Clean up orphaned perf test workers on failure by @yufeiwu-nv in #19697
  • [TRTLLM-15885][infra] validate CBTS coverage DB compatibility by @crazydemo in #18802
  • [None][perf] use index copy for multimodal fusion by @yechank-nvidia in #18636
  • [TRTLLM-16773][fix] fix dummy slot issue by @bo-nv in #19891
  • [None][perf] Allow prefix-tokenization cache for MiniMax-M3 text-only prompts by @zheyuf in #19613
  • [TRTLLM-15761][refactor] Parameterize decoder invocation state and separate decoder-only engine state by @lori-ren in #19696
  • [None][test] Exercise SWA endpoint retention against real page eviction by @brnguyen2 in #19754
  • [None][feat] support C++ streaming KV events with multimodal payloads by @GuanLuo in #19359
  • [TRTLLMINF-336][infra] Publish and test the post-merge BOLTed build beside canonical by @mlefeb01 in #19603
  • [TRTLLM-16670][feat] Add MiniMax H3 ref2va mode support by @yibinl-nvidia in #19657
  • [TRTLLMINF-336][infra] Enable BOLT premerge consume by @mlefeb01 in #18677
  • [#19495][fix] Align LTX-2 scale factors with tensor axes by @adenzhou1350 in #19500
  • [TRTLLM-16761][feat] Capture partial telemetry context for LLM startup failures by @Mgluhovskoi in #19598
  • [None][chore] CUTEDSL_FC12 phase-specific smem AB stages with FC2 epilogue overlay by @leslie-fang25 in #19539
  • [None][test] Add KV transfer lifecycle acceptance regressions by @chienchunhung in #19663
  • [None][feat] Weight the draft KV budget split by per-request spend by @brnguyen2 in #19744
  • [None][fix] Fix torch.compile eagle3 AL by @mikeiovine in #19899
  • [TRTLLM-16040][feat] Refresh SM90 SageAttention onto the per-thread-quantized kernel, Integrate to VisualGen by @xrq-phys in #18496
  • [None][test] Replace TinyLlama-1.1B-Chat-v1.0 with Qwen3-0.6B in tests by @xinhe-nv in #19178
  • [None][infra] Waive 1 failed cases for main in post-merge 3005 by @trtllm-agent in #19919
  • [#19756][fix] Parse hyphenated function names in the Kimi K2 tool parser by @MohammadHijjawi97 in #19758
  • [TRTLLM-16614][chore] Remove more TRT leftovers by @Funatiq in #19583
  • [TRTLLM-16217][test] Add VR200 4-GPU serve perf cases for DeepSeek V4 Pro and Kimi K3 by @ruodil in #19260
  • [None][chore] Remove unused negative-test helpers by @chzblych in #19927
  • [TRTLLM-16569][fix] keep tests running on S3 spool errors by @niukuo in #19622
  • [#19225][fix] Keep markdown fences in DeepSeek V3 streamed content by @Yigtwxx in #19226
  • [None][ci] waive pre-existing test failures on main by @brnguyen2 in #19931
  • [TRTLLM-15261][feat] Test stale pointers held by captured graph in metadata by @asfiyab-nvidia in #19306
  • [TRTLLM-16772][fix] Release async IPC payloads with garbage collection disabled by @chienchunhung in #19869
  • [None][fix] OpenEngine: release completed results and honor TRTLLM_SERVER_DISABLE_GC by @nv-yna in #19858
  • [None][fix] Cache resolved MegaMoE kernel descriptors across forwards by @dongfengy in #19901
  • [https://nvbugs/6838021][test] Unwaive MiniMax-M3 MXFP8 piecewise CUDA graph test on B300 by @peihu-nv in #19810
  • [None][fix] Kimi K3 KDA prefill: cap the scratch caches at 2 entries by @longlee0622 in #19885
  • [None][fix] prevent multimodal scheduling stalls on the KV cache V2 scheduler by @yechank-nvidia in #19801
  • [https://nvbugs/6697099][fix] Keep MPI progress alive on the PP sample-state relay thread by @YihuiLu512 in #19643
  • [None][chore] Waive failing test 10/07 by @brb-nv in #19945
  • [None][infra] Waive 3 failed cases for main in post-merge 3007 by @trtllm-agent in #19946
  • [https://nvbugs/6810945][fix] Remove stale Kimi FP8 tests and gate Rubin coverage by @jiaganc in #19918
  • [None][fix] Add a static tile scheduler override for context FMHA by @longlee0622 in #19873
  • [None][fix] skip unused multimodal encoder IPC handles by @yechank-nvidia in #19791
  • [None][feat] Support K-EXAONE-2 in the PyTorch backend by @eagle705 in #17067
  • [None][fix] Kimi K3 attn_res kernels: order the ws_stats reads and fence the slot releases by @vsabavat in #19829
  • [None][refactor] BREAKING: Migrate LlmRequest fields to Python by @Funatiq in #19484
  • [None][fix] route Qwen3.6/3.8 VLM checkpoints to the generic Qwen3.5 VLM wrapper by @yechank-nvidia in #19892
  • [TRTLLM-16185][infra] route documentation changes to docs and CPU validation by @crazydemo in #19409
  • [https://nvbugs/6782263][fix] Serialize HF remote-code loading when building tokenizers by @tianyuxbear in #19553
  • [None][fix] Keep FetchContent patch command a single shell argument by @lori-ren in #19957
  • [None][feat] MoE scheduler: optional finalize-before-combine and FP4 scale-layout contract fix by @brnguyen2 in #19749
  • [TRTLLM-16617][chore] remove obsolete TensorRT telemetry compatibility paths by @Funatiq in #19585
  • [None][fix] agent-flow: spill oversized system prompts to a file and forward Claude CLI stderr by @brnguyen2 in #19766
  • [None][infra] Disable BOLT premerge consume by @yiqingy0 in #19963
  • [None][test] Migrate disaggregated E2E tests to C++ bounce buffers by @chuangz0 in #19588
  • [https://nvbugs/6729495][fix] Restore Nemotron-H weight-update coverage by @shikicloud in #19600
  • [TRTLLM-15762][refactor] Migrate decoder execution and private resources into the runner family by @lori-ren in #19896
  • [None][perf] Enable tiled FP8 context FMHA for head_dim 256 on SM120 by @amukkara in #19825
  • [TRTLLM-16282][feat] Add sparse KV offload and GPU metadata publication by @cascade812 in #19821
  • [None][fix] Isolate serve performance test logs across concurrent launches by @yufeiwu-nv in #19930
  • [TRTLLM-14620][feat] Add ragged sparse-attention row contracts by @zhaoyuanh-nvidia in #18485
  • [https://nvbugs/6345827][test] Unwaive DeepSeekV3Lite disagg MTP test_auto_dtype on H20 and H100 by @moraxu in #19958

New Contributors

Full Changelog: v1.3.0rc29...v1.4.0rc0

Check out latest releases or
releases around NVIDIA/TensorRT-LLM v1.4.0rc0

Don't miss a new TensorRT-LLM release

NewReleases is sending notifications on new releases.

Get notifications