Highlights
-
Known Issues
- DeepSeek-R1 FP4 disaggregated serving on B300 may produce incorrect output under CUDA graphs.
- GLM-5.2 NVFP4 SM103 TP8 may fail to initialize when Modeling V2 is required with FlashInfer 0.7.3.
- Kimi K3 NVFP4 KV may fail to initialize on B300 due to FP4 MLA quantization eligibility.
- Kimi K3 DSpark with MegaMoE CuTe DSL TEP and CUDA graphs may hang during GB300 warmup.
- Qwen3.8-Flash-Next FP8 with PLE offloading, attention DP and MTP may fail during initialization.
-
Model Support
- Add Apertus and Apertus 1.5 text models to the PyTorch backend #19735
- Add GLM-5.2 NVFP4 high-throughput and low-latency Modeling V2 targets for Rubin #21585, #21380, #21590
- Add GLM-5.3-Flash NVFP4 Modeling V2 targets for SM100 and SM120 sparse attention #21787, #20639
- Add MiniMax-M3 NVFP4 low-latency decode Modeling V2 targets for Rubin #20623
- Add Qwen3.8-Flash-Next NVFP4 Modeling V2 targets for GB300 parallelism configurations #21372
- Add native Qwen3 reranking and Muse Spark Open serving support #19550, #21049
- Load Cosmos3-Edge-FP8 checkpoints with their W8A8/W8A16 step policy #21572
- Add compressed-tensors NVFP4 MoE packaging support for MiniMax-M3 #20847
- Qualify Qwen3 dense FP8, tied embeddings, and Qwen3 MoE BF16 for ModelExpress #20766, #20767, #20765
- Add FastH3 support with VSA and MiniMax-H3 Ulysses attention #20648, #21222
-
API
- Use string enums for checkpoint I/O policy and checkpoint formats (BREAKING) #20498, #20703
- Deprecate KV cache manager V1 and the C++ cache transceiver #21525
- Default the Modeling V2 LLM argument to auto #21712
- Add MoeConfig.tactic to select a MoE implementation by ID #21778
- Add opt-in disaggregated transfer diagnostics and startup failure telemetry #18913, #20853
- Support etcd credentials, TLS, and signed records for disaggregated cluster storage #21187, #21716
- Preserve request IDs in Anthropic message IDs and send streaming finish reasons separately #20936, #21149
- Require tagged-union discriminators in serving configuration schemas #21416
- Add keyed log-once support to the C++ logger #21775
-
Feature
- Add sparse KV offload for DeepSeek-V4 Flash and GLM-5.2 using KVCM V2 #21350
- Extend mixed-precision KV caching to speculative decoding, sparse MLA, and cold-page compression #21021, #20197
- Add host-tier suspension and cached MLA gathering for aggregated Helix prefill #21455, #21527
- Add shared drafter KV pools, FP8 MLA inputs, and unified DSpark/DFlash KV caches #21972, #20664, #20486
- Add generation-slot preemption and SINK_REQUIRED_ONLY attention reuse in KV cache V2 #21334, #21432, #21793
- Optimize KVCM V2 SSD offloading #21822
- Add native DSA prefill indexing and exact top-k for SM100 and SM103 #21892
- Add Qwen3.8 QSA sparse attention using the PrimTS backend #20714
- Support FP8 KV cache in non-MLA PrimTS attention and Gemma4 variable-window attention #19396, #18912
- Add dense DWDP for Kimi K3 context servers and balanced expert partitioning #21012
- Add NVLinkOneSided All2All sentinel transport and Rubin MLA, DSpark, DWDP and Helix changes #21668, #21067
- Add NVFP4_PLUS CuTe DSL MoE for Rubin and native BF16 TRTLLM-Gen MoE #21242, #21718
- Add Beam StairCase components, DSpark confidence scheduling, and device-side reranking #21471, #20078, #21596
- Add Gumbel-max text watermarking with speculative decoding #21047
- Add Cosmos3 SeaCache step caching and per-request sigma grids #21288, #20984
- Add VisualGen combined Omni references, token-sharded TP, and Rubin FP8 attention #21226, #20483, #20005
- Support optional MiniMax-H3 host-backup release on wake #19612
- Add FP32, BF16, and FP16 unified GVR self-sampling templates #19285
- Add ModelExpress qualification to the modeling bringup workflow #19996
- Improve executor and hybrid GDN/Mamba performance for RL workloads #20916
- Optimize DeepSeek-V4.1 Flash NVFP4 and GLM-5.2 SM103 prefill and decode #21438, #21695
- Add Muse Spark decoder fast paths and reduce Qwen4-Exp PLE workspace #21356, #21841
- Optimize Kimi K3 MoE routing and decode and add an SM107 MegaMoE tactic #21400, #21399, #21960, #21977
- Reduce executor, sampler, KV scheduler, and long-prompt host overhead #21974, #21308, #20848
- Optimize speculative draft rows, KDA replay, and beam-history copying #22008, #22009, #21035
- Improve DeepGEMM and CuTe DSL warmup and share autotuning resources #21791, #21610, #21615, #21419
- Accelerate FP8 MoE row expansion, linear quantization, and Qwen3.5 GDN projections #21178, #21208, #21287
- Fuse NVFP4 activation quantization and enable tiled MoE quantization #21273, #21239
- Optimize QSA and DSA decode metadata and target MSA JIT kernels to visible GPUs #21409, #19786, #21681
- Support FP8 wide-head MLA verification on SM107 and add PrimTS attention recipes #21423, #20068
- Stream multimodal hashing and move vision pixel normalization to GPU #18637, #20973
- Extend all-reduce tuning and enable NCCL window registration during CUDA graph capture #20241, #20908
- Add PPLX decider support and reduce speculative strict-accept compilation variants #20238, #21424
-
Fix
- Restore auxiliary KV pool selection for DSA FP4 MLA #21547
- Release cached MoE workspaces at executor shutdown and make KVCM V2 teardown safe without CUDA context #20840, #21795
- Release profiling KV pools before NIXL deregistration and unblock timed-out Python transfers #21784, #20943
- Reclaim KV resources after failed transfers and complete cancelled disaggregated requests #21909, #21508, #21482
- Resolve mixed-precision KV-head counts in disaggregated transfers and publish paired draft KV blocks #21597, #19303
- Correct FP8 KV cache for cross attention and size Whisper cross-KV pools from encoder length #20958, #20918
- Fix draft KV over-allocation in chunked prefill and size separate draft mirrors from request history #21546, #21281
- Fix GLM-5.2 Modeling V2 test environment overrides and MiniMax-M3 Eagle3 draft-loop API #22032, #21815
- Correct Qwen3.8 FP8 PLE row masking and SM120 attention-DP padding #21346, #21316
- Fix Qwen3.8 serving, Mistral Large 4 runtime, and Qwen3-VL disaggregated video layout #21930, #21788, #20497
- Correct MXFP4 FP8 MoE SwiGLU and input-quantization scaling #22108, #22106
- Fix Qwen MoE FP8 LoRA test configuration and MiniMax subprocess imports #21680, #21671
- Correct MNNVL fused RMSNorm shared memory and KDA Small-BH TMA-store fencing #21785, #21908
- Support non-power-of-two expert counts and hidden sizes in MoE routing #21923, #21862
- Fix Helix MLA decode selection, contiguous NCCL inputs, and breakable-prefill NCCL hangs #21582, #21213, #21799
- Initialize FlashInfer FMHA on all pipeline ranks before warmup #21203
- Enforce encoder token and batch budgets and reject unsupported recompute configurations #21933, #22103
- Preserve MTP quantization exclusions and prevent stale multimodal item lifetimes #21840, #21986
- Fix stop and bad-string tokenization and remove streamed stop-word IDs from final text #21614, #21341
- Correct GLM and Kimi tool parsing and preserve final streamed tool calls #21798, #21108, #21503, #21625, #21975
- Cap decoded audio samples and keep coordinator sockets in owner-only directories #20987, #21204
- Release NIXL transfer status handles without holding the GIL #21612
- Correct GQA workspace sizing, GVR row-window bounds, and non-power-of-two MoE routing #21863, #22184, #21923
- Preserve standalone VisualGen teardown and FastWan warmup sampling defaults #21417, #21029
- Pin sentencepiece below 0.2.3 and preserve DeepSeek-V4 metadata behavior #21790, #21319
- Stabilize MPI startup and readiness and free executors during MPI worker teardown #22055, #21777
- Correct attention-DP token embeddings and Qwen4 kernel synchronization #20089, #21853
- Fix DeepGEMM MoE memory bounds, KDA update synchronization, and GDN workspace selection #21401, #22179, #21332
- Correct Lamport all-reduce clear sizes and use 64-bit convolution-state offsets #21283, #21684
- Correct speculative-decoding kernel specialization and DFlash logit and window settings #21666, #21367
- Separate generation cancellation from receive retirement and report native KV transfer results #21491, #21538
- Correct n-gram pool pruning and fail requests cleanly when grammar creation fails #20487, #20903
- Align PrimTS decode with its runtime ABI and correct MoE autotuning keys #22050, #21190
- Default dense Qwen3.5 and Qwen3.8 NVFP4 checkpoints to Marlin on Ada and Hopper #21434
- Bound context-logits memory with a CPU fallback and validate RADIO video weights at load #21315, #21631
- Preserve Kimi K3 Responses usage accounting and correct JSON-schema example inputs #21489, #21026
- Handle unavailable benchmark latency and NVML memory results #22178, #21340
- Fail unsupported non-generation requests and executor loops cleanly #21584
- Support optional native RankInfo authentication #21486
- Correct Helix context-parallel validation and avoid PyTorch mimalloc huge pages #21192, #21463
- Correct CGA reduction split caps and KV-window logging #21767, #20650
- Accept supplied token IDs in the MoE benchmark top-k hook #21961
-
Documentation
- Document runtime speculation, DSpark confidence, and Modeling V2 routing #21895, #21601, #21910
- Clarify KV-cache reuse, deployment memory settings, and speculative decoding configuration #21896, #21893, #21894, #22084
- Add dense DWDP and dynamic expert replication technical blogs #21807, #21492
- Refresh model references and the MoE performance dashboard #21737, #21749, #21757
-
Test & Infra
- Upgrade Mooncake to 0.3.14 and FlashInfer Python to 0.7.3 #22191, #21824
- Reject direct dependencies in root requirement files and update release version tooling #21817, #21813
- Prebuild x86 C++ tests and split GB300 and B200 pre-merge stages #22124, #22307
- Migrate test results to Artifactory and upload checkpoints from a separate watcher #21223, #21942
- Capture setup timeouts, classify pytest timeouts, and configure per-stage rerun policies #19333, #17808, #21189
- Expand CBTS selection and coverage for model aliases, Python changes, and CPU tests #21323, #21418, #21782, #21717
- Add model coverage for DeepSeek-V4.1, Kimi K3/K2.5, GLM5 and Gemma4 #21063, #21504, #21500, #21505, #21766
- Add GLM-5.3-Flash core performance coverage and refresh model performance-sanity lists #22163, #21832
- Run Kimi K3 MMMU accuracy with KV cache manager V2 #20038
- Move pyexecutor, attention, disaggregation, and host-only MoE tests to CPU stages #21700, #21703
- Improve Slurm runner isolation, network setup, and test result logging #22101, #21727, #21794
- Add Pyrefly type checking and advisory test coverage #21325, #22006
- Send PLC reports through Cora and update PLC repository URL handling #21970, #21360, #22060
- Synchronize release/1.3 changes to main and refresh PrimTS vendor sources #20843, #21354, #21670
- Update platform context and GPU-count markers in QA tests #21534, #21541, #19212
- Consolidate encoder runners and refactor request handoff and hybrid speculative cache state #21428, #21768, #21324
- Remove obsolete TensorRT backend kernels and legacy CI performance paths #21543, #21524
- Pause and restore B200 CI stages during an outage and adjust Rocky Linux 8 image repositories #22165, #22212, #22244
- Add Modeling V2 regression jobs, routing gates, and test selection by change scope #21629, #21603
- Improve serving route coverage, scheduler and benchmark graph tests, and disaggregated stress tests #21406, #21232, #21234, #21397
- Add EFA functional CI coverage and improve distributed initialization #21516, #20704
- Record checkpoint policy and runtime image in startup telemetry #19174
- Define multinode test launch parameters in YAML and configure evaluation MPI ranks #21704, #21764
- Track unfinished MoE tests, use node IPs for disaggregated performance tests, and refine CI stage naming #21545, #20556, #21497
- Enable GCP IAD CI and localize build arguments and sanity-pod configuration #18914, #22102
- Reorganize component-local C++ tests and encoder and multimodal runner setup #21024, #21410, #21914
- Remove retired semantic review automation and unscheduled or retired performance tests #22187, #22217, #21929, #21990, #21710, #21991
- Resume modeling bringup from checkpoints and refine the GLM Rubin target #21677, #21812
- Reuse warmup synchronization, transport typed request inputs, and adopt canonical modules #20813, #21375, #22086
- Refine CBTS acceptance references and test scopes #21691, #21776
- Correct Qwen3.8 prompts, MoE and all-reduce reference checks, and Kimi and Qwen test setup #22180, #22076, #20110, #21874, #21876
- Repair executor, disaggregation, ADP connector, and configuration CLI test fixtures #22201, #22014, #21994, #21672, #21465, #21686, #21661, #21441
- Update Modeling V2 test selection, test placement, and architecture checks #22085, #21911, #21949
- Improve disaggregated test ports and multi-frontend performance-sanity reservations #21249, #21606
- Update Gemma4 and GLM test cache configuration and serve constructor baseline #21993, #21692, #22174
- Build FlashMLA without relocatable device code and update FlashInfer tile validation #22028, #22004
- Update code ownership, platform-neutral performance recipes, and Rubin integration follow-ups #21514, #21488, #21976
- Remove retired FlashInfer MoE and All2All paths and unscheduled Nemotron tests #21013, #22081
What's Changed
- [TRTLLM-17557][fix] Make the configuration-database CLI test runnable by @JunyiXu-nv in #21441
- [TRTLLM-16725][refactor] Centralize PyTorch distributed initialization by @yijingl-nvidia in #20704
- [None][feat] Run GLM-5.3-Flash sparse attention on SM120 through FlashInfer by @nvchenghaoz in #20639
- [TRTLLM-17505][feat] Add the SINK_REQUIRED_ONLY attention reuse policy to KV Cache Manager v2 by @Hudayday in #21432
- [None][perf] DSA: skip the indexer schedule rebuild for index-sharing draft steps and the duplicated kv-lens hook by @Tabrizian in #19786
- [https://nvbugs/7015107][fix] Keep the Lamport all-reduce clear size in bytes by @pranav-nvidia in #21283
- [TRTLLM-16722][feat] Dflash Unified KVCache by @allisonlim-nv in #20486
- [None][perf] Enable tiled Triton NVFP4 quantization for CuteDSL MoE by @2ez4bz in #21239
- [TRTLLM-14620][feat] Enable opt-in DSpark decode confidence scheduling by @zhaoyuanh-nvidia in #20078
- [None][infra] Enabling GCP_IAD in CI by @yuanjingx87 in #18914
- [None][fix] Kimi-K3: count Responses usage as Chat Completions does by @JunyiXu-nv in #21489
- [https://nvbugs/7003540][ci] Unwaive fixed bugs. by @SimengLiu-nv in #21464
- [TRTLLM-17229][feat] Extend the mixed-precision KV cache to speculative decoding and the sparse MLA backends by @Hudayday in #21021
- [https://nvbugs/7023194][fix] Restore auxiliary KV pool selection for DSA FMHA by @Hudayday in #21547
- [https://nvbugs/7003528][fix] Align ADP connector fixtures with executor contracts by @trtllm-agent in #21465
- [None][perf] autotuner: share one CUDA graph pool per tuning pass by @xguannv in #21419
- [None][feat] Support FP32, BF16 and FP16 with unified self-sampling GVR templates by @longcheng-nv in #19285
- [TRTLLM-17390][fix] Size the separate draft-KV mirror from request history by @pranav-nvidia in #21281
- [None][feat] Add NVFP4_PLUS (E2M1 + UE5M3 per 32) CuteDSL MoE on Rubin by @Tracin in #21242
- [TRTLLMINF-586][infra] Add platform context to LLMAPI test stages by @mzweilz in #21534
- [None][fix] GLM tool parsers: a fresh call in an open call's arguments abandons it by @JunyiXu-nv in #21108
- [None][test] Add coverage for KimiK3ForConditionalGeneration by @StanleySun639 in #21505
- [None][test] Add coverage for Glm5NextForConditionalGeneration by @StanleySun639 in #21500
- [None][test] Add coverage for KimiK25ForConditionalGeneration by @StanleySun639 in #21504
- [None][test] Add coverage for DeepseekV41ForConditionalGeneration by @StanleySun639 in #21063
- [https://nvbugs/6953985][fix] Check the KV window crossing before the logger level by @trtllm-agent in #20650
- [None][fix] Kimi K2 tool parser: release the arguments of a call that ends the stream by @JunyiXu-nv in #21503
- [None][test] State the GPU count of multi-GPU QA core tests with markers by @jieli-matrix in #21541
- [TRTLLM-14978][perf] Kimi-K3 decode attention-residual kernel: CuTe DSL PDL mix with the MoE residual folded in, MXFP8 hand-off to KDA, gated/sigmoid low-M GEMM variants by @longlee0622 in #21399
- [None][perf] Move vision pixel normalization from CPU to GPU by @yechank-nvidia in #20973
- [TRTLLM-17603][feat] Fail unsupported non-generation requests and executor loops cleanly instead of mid-run by @lori-ren in #21584
- [TRTLLM-16807][feat] Support combined Omni references in video serving by @yibinl-nvidia in #21226
- [None][feat] MiniMax-M3 NVFP4 low latency decode megakernel for Rubin (sm_107) TP2 / TP4 / TP8 in modeling_v2 by @peihu-nv in #20623
- [TRTLLM-17151][fix] Apply the logit transform and sliding window DFlash drafters declare by @moraxu in #21367
- [None][fix] Complete queued context cancellations before activation by @chienchunhung in #21482
- [None][feat] Add opt-in disaggregated transfer diagnostics by @chienchunhung in #18913
- [None][feat] VC-Attention-QK(16,8) recipes for PrimsTS FMHA context by @harrisonzhy in #20068
- [TRTLLM-17615][fix] Initialize ownership in peer rejection test by @allisonlim-nv in #21661
- [None][chore] Mass merge release/1.3 branch back to main (batch 2) by @SimengLiu-nv in #20843
- [None][feat] Visual Gen token-sharded TP: gather FP8 block-scale inputs by @chang-l in #20483
- [TRTLLM-17518][refactor] Build EncoderExecutor from shared setup components instead of ModelEngine by @lori-ren in #21410
- [None][fix] perf-sanity: let the multi-frontend port reservation pass the trtllm-serve launcher probe by @JunyiXu-nv in #21606
- [None][perf] strict_accept: one compiled variant for every batch by @xguannv in #21424
- [https://nvbugs/6813945][fix] Use stored metadata server URLs as is and support etcd credentials and signed server records by @BoyueZ in #21187
- [TRTLLM-16460][perf] One-launch decode metadata for QSA rows and GDN replay work items by @Wanli-Jiang in #21409
- [#19902][perf] Size the beam-history copy by generated tokens, not prompt-length spread by @lminer in #21035
- [None][chore] promote flashinfer-prims-ts source from #20068 by @yuxianq in #21670
- [https://nvbugs/7028059][fix] Fix Qwen MoE native FP8 LoRA reference test configuration by @lori-ren in #21680
- [None][refactor] Use a platform-neutral LTX2 perf recipe filename by @daichu-nv in #21488
- [None][test] Fix two ModelEngine tests broken by concurrent merges by @lori-ren in #21686
- [#21318][fix] Drop streamed stop-word ids from the final streamed text by @spped2000 in #21341
- [None][fix] causal_conv1d: form conv-state slot offsets in 64 bits by @Wanli-Jiang in #21684
- [TRTLLM-17612][refactor] Remove unused kernels after TensorRT backend removal by @Funatiq in #21543
- [https://nvbugs/6858547][fix] Ignore cp_type with one CP rank and require HELIX with more by @trtllm-agent in #21192
- [TRTLLM-17229][fix] Resolve the KV-head count of mixed-precision high layers in disaggregated transfer by @Hudayday in #21597
- [https://nvbugs/6993575][fix] Remove the DeepSeek-V4 metadata recompile and nondeterministic mHC split-K by @chenfeiz0326 in #21319
- [https://nvbugs/6910975][fix] Keep GDN decodes out of a bounded prefill state workspace by @chenfeiz0326 in #21332
- [None][fix] Chat streaming: send finish_reason on its own chunk by @JunyiXu-nv in #21149
- [None][fix] Wait for the rank-0 VisualGen worker teardown at external-launch shutdown by @luyiyun1021 in #21417
- [TRTLLM-17123][feat] Gather cached MLA KV for Helix prefill by @zhaoyuanh-nvidia in #21527
- [None][chore] Deprecate KV cache manager V1 and the C++ cache transceiver by @Shixiaowei02 in #21525
- [TRTLLM-16725][refactor] BREAKING: Use a string enum for checkpoint I/O policy by @yijingl-nvidia in #20498
- [None][infra] remove DeepSeek R1 multi-node perf tests by @fredricz-20070104 in #21710
- [TRTLLM-14620][feat] Add device re-ranking for DSpark confidence scheduling by @zhaoyuanh-nvidia in #21596
- [None][fix] DenseGEMM MoE: keep FC1/FC2 global accesses inside the problem by @ziyixiong-nv in #21401
- [https://nvbugs/7049655][test] Unwaive test_init_from_args_forwards_vision_gpu_normalize by @lori-ren in #21761
- [TRTLLMINF-579][infra] Rename Check Test List stage to Check-Test-List and mark it skipped when not run by @EmmaQiaoCh in #21497
- [https://nvbugs/6682814][fix] Cherry-pick #20230: Use node IPs for perf-sanity disagg servers and report unresolvable bind hosts by @JunyiXu-nv in #20556
- [None][doc] Update MoE Perf Dashboard data to 1.4.0rc3 by @leslie-fang25 in #21757
- [None][fix] Isolate fallback hwloc and add EFA functional CI coverage by @chzblych in #21516
- [None][infra] Add blossom-ci authorized users by @trtllm-agent in #21783
- [None][feat] Staircase: Rubin GLM Optimized Low Latency (Conc up to 56) by @dongfengy in #21380
- [None][infra] Retire legacy L0 perf testing and protect QA benchmarks by @chzblych in #21524
- [TRTLLM-17005][refactor] Component-local C++ Tests (Part 5) by @Funatiq in #21024
- [None][fix] Declare 16 MPI ranks for multi-node evaluation tests by @crazydemo in #21764
- [TRTLLMINF-587][infra] Record MoE sub-test unfinished tests in a separate file by @yiqingy0 in #21545
- [https://nvbugs/7003528][fix] Restore ADP connector CPU test coverage by @allisonlim-nv in #21672
- [None][perf] Build MSA fmha_sm100 JIT kernels for the visible GPUs' target only by @chenfeiz0326 in #21681
- [TRTLLM-14883][feat] Qualify Qwen3 MoE (CUTLASS BF16) for MX by @moraxu in #20765
- [None][infra] Let CBTS rules own CPU test selection by @crazydemo in #21782
- [None][feat] Dense DWDP for Kimi-K3 context servers and a balanced DWDP expert partition by @zongfeijing in #21012
- [None][fix] Narrow CBTS test scopes using verified pre-images by @crazydemo in #21776
- [https://nvbugs/6953985][fix] Keep PyTorch's mimalloc arenas off transparent huge pages by @trtllm-agent in #21463
- [https://nvbugs/6921831][test] Add Gemma4 multi-image encoder budget test by @yingguo-trt in #21766
- [https://nvbugs/7044831][test] Unwaive the mixed-precision disaggregation tests fixed by #21597 by @Hudayday in #21774
- [TRTLLM-17677][perf] Split the low-M GEMM autotuning race across the tuning group by @chenfeiz0326 in #21615
- [None][feat] DSpark Unified KVCache standalone by @allisonlim-nv in #20664
- [None][feat] trtllm_modeling_v2: scoped regression jobs, gate routing and tune-stage robustness by @Wanli-Jiang in #21629
- [TRTLLM-17238][feat] Add PrimTS sparse core backend for Qwen3.8 QSA attention by @farazkh80 in #20714
- [TRTLLM-17527][refactor] transport typed Python request inputs by @Funatiq in #21375
- [TRTLLM-17519][refactor] Merge PoolingRunner into EncoderRunner with packed and scheduled entry points by @lori-ren in #21428
- [None][feat] Qwen3.8-Flash-Next NVFP4 modeling_v2: one target for TP1/TP4/TEP4/DEP4/DEP2 on GB300 by @Wanli-Jiang in #21372
- [TRTLLM-14978][perf] Kimi-K3 MoE decode front: routing + moe_sort in one launch, CuTe router logits, router-first order, NVFP4 latent_down epilogue by @longlee0622 in #21400
- [TRTLLM-17088][feat] Suspend to the host tier under aggregated Helix by @asfiyab-nvidia in #21455
- [None][doc] Point MiniMax-M3.1-Flash to its latest release container by @pranav-nvidia in #21737
- [#19820][feat] Add Apertus and Apertus 1.5 (text) to the PyTorch backend by @maxwbuckley in #19735
- [TRTLLM-14620][doc] Add DSpark confidence reproduction guide and benchmark kit by @zhaoyuanh-nvidia in #21601
- [None][feat] Optionally release MiniMax-H3 host backups on wake by @rystewart-nvidia in #19612
- [None][feat] VisualGen: dense FP8 attention and fused QK norm/RoPE/FP8 ops for Rubin by @wu6u3tw in #20005
- [TRTLLM-16725][refactor] BREAKING: Use a string enum for checkpoint formats and simplify checkpoint I/O helpers by @yijingl-nvidia in #20703
- [https://nvbugs/7028062][fix] Fix MiniMax test subprocess imports by @peihu-nv in #21671
- [None][feat] Add Gemma4 variable-window attention by @achartier in #18912
- [https://nvbugs/6907083][test] Unwaive MiniMax H3 two-GPU Ulysses LPIPS test by @yibinl-nvidia in #21823
- [None][perf] Fuse NVFP4 activation quant into SwiGLU (SM100+) and add+RMSNorm (SM12x) by @amukkara in #21273
- [#20969][fix] Qwen3.5/3.8: default dense NVFP4 checkpoints to Marlin on Ada/Hopper by @Smallfu666 in #21434
- [https://nvbugs/7043301][test] Turn off the HF weight cache in the GLM-5.2 SM103 TEP4 GSM8K test by @trtllm-agent in #21692
- [TRTLLM-17054][refactor] Reuse sampling synchronization and sequence slots across warmup phases by @zhaoyangwang-nvidia in #20813
- [None][fix] Port native disaggregation KV result report token - #20723 by @BoyueZ in #21538
- [None][refactor] Make GLM Rubin modeling v2 target self-contained by @dongfengy in #21812
- [None][perf] Staircase: Rubin GLM C1 Megakernel by @dongfengy in #21590
- [None][fix] Fix Slurm NCCL and pytest failures, continue nightly scans and trim CI logs by @chzblych in #21727
- [TRTLLM-17328][chore] Remove quantized FlashInfer TRTLLM-Gen MoE leaves and FlashInfer two-sided alltoall by @rosong11 in #21013
- [https://nvbugs/7047852][fix] Use the default dtype in the Kimi K3 modeling_v2 GSM8K tests by @trtllm-agent in #21874
- [TRTLLM-16806][feat] Enable TRTLLM Ulysses attention for MiniMax-H3 by @yibinl-nvidia in #21222
- [https://nvbugs/7042835][test] Unwaive unittest/disaggregated fixed by #21597 by @Hudayday in #21810
- [TRTLLMINF-472][infra] Migrate test results to artifactory.nvidia.com by @sharabeshj-nvda in #21223
- [None][fix] Narrow CBTS acceptance reference selection to consumers by @crazydemo in #21691
- [None][infra] Narrow CBTS perf selection by changed model aliases by @crazydemo in #21323
- [None][infra] select covered Python files across repository directories by @crazydemo in #21418
- [TRTLLMINF-225][infra] Check the root requirement files in pre-commit and reject direct dependencies by @niukuo in #21817
- [https://nvbugs/7061740][fix] Update Qwen3.5 checkpoint loader test API by @fredricz-20070104 in #21876
- [https://nvbugs/6858527][fix] Check RADIO video_embedder weights when loading, not per video by @trtllm-agent in #21631
- [https://nvbugs/6858496][fix] Fail disagg gen requests whose grammar matcher cannot be created by @trtllm-agent in #20903
- [TRTLLMINF-207][infra] classify pytest timeout results by @hanjingtian in #17808
- [None][feat] Anthropic Messages: carry the request id in the message id by @JunyiXu-nv in #20936
- [https://nvbugs/6891165][fix] Pass chat_template_kwargs in the request body of the JSON-schema serve example by @JunyiXu-nv in #21026
- [TRTLLMINF-451][infra] Capture setup timeouts and defer unfinished cleanup until JUnit save by @yiqingy0 in #19333
- [None][infra] Introduce Pyrefly to do type check by @HuiGao-NV in #21325
- [None][test] expose subprocess output in serving e2e tests by @xinhe-nv in #21794
- [https://nvbugs/6964706][fix] Make KVCacheManagerV2 teardown safe without a CUDA context by @karljang in #21795
- [https://nvbugs/6988867][fix] Bind the disagg test server to a kernel-assigned port by @trtllm-agent in #21249
- [None][chore] Relax Modeling V2 naming and select Modeling V2 tests by change scope by @erictsai-nv in #21603
- [None][docs] Clarify Modeling V2 rules for Modeling V1 code, native code and routing by @erictsai-nv in #21910
- [None][fix] GLM tool parsers: deliver an argument the schema does not type as written by @JunyiXu-nv in #21798
- [TRTLLM-17163][test] Muse Spark Open stress settings and disagg cancellation marathon attach mode by @moraxu in #21397
- [None][chore] Refresh PrimTS source onto FlashInfer main (2026-11-07) by @yuxianq in #21354
- [TRTLLMINF-607][infra] Set the version with scripts/bump_version.py and check every version record by @niukuo in #21813
- [None][feat] Add Cosmos3 per-request sigma grid selection by @skasetty-Siri in #20984
- [None][fix] Require the tag of tagged unions in the serve config schemas by @skasetty-Siri in #21416
- [TRTLLM-17209][feat] Qualify FP8 block-scale weights for MX (Qwen3 dense) by @moraxu in #20766
- [None][fix] Keep compiled Helix NCCL all-to-all inputs contiguous by @yijingl-nvidia in #21213
- [None][fix] Stop scheduling cancelled disaggregated requests by @chienchunhung in #21508
- [None][perf] Support FP8 rowwise GDN projections in Qwen3.5 and fuse their norm + per-token FP8 quantize by @amukkara in #21287
- [https://nvbugs/7049768][fix] Clamp CGA reduction split cap silently by @yuxianq in #21767
- [https://nvbugs/7054722][fix] Pin sentencepiece below 0.2.3 by @weiminwang-nv in #21790
- [None][feat] Rubin merge back: MLA, DSpark, DWDP, Helix by @mingyangHao in #21067
- [https://nvbugs/7066079][doc] Use nested KV cache memory settings in deployment YAML by @ruodil in #21893
- [TRTLLM-16673][feat] Add FastH3 support with VSA by @yibinl-nvidia in #20648
- [https://nvbugs/7066119][fix] Make the low-M GEMM max_m_for test SM107-aware by @longlee0622 in #21949
- [https://nvbugs/7012145][fix] Support non-power-of-2 expert counts in Triton FP8 MoE align by @xxi-nv in #21923
- [https://nvbugs/7049820][fix] Fence KDA Small-BH K1 tiles before their TMA store by @trtllm-agent in #21908
- [None][perf] CuTe DSL MLA: admit FP8 wide-head verify groups on SM107 by @xguannv in #21423
- [None][feat] GEN slot preemption: pause decode requests in place to admit at-risk waiters by @reasonsolo in #21334
- [None][feat] Add keyed log-once support to the C++ logger by @yuxianq in #21775
- [TRTLLM-17683][refactor] Prepare only multimodal inputs in the standalone multimodal encoder runner by @lori-ren in #21914
- [None][doc] List Kimi K3.1 Pro feature-branch support by @arysef in #21749
- [None][test] Guard benchmark CUDA graph configuration by @ruodil in #21234
- [None][test] Guard scheduler admission and CUDA graph eligibility by @ruodil in #21232
- [None][perf] stream multimodal content hashing by @yechank-nvidia in #18637
- [None][feat] modeling_v2: default the LLM arg to auto by @WeiHaocheng in #21712
- [None][fix] CuteDSL MoE: keep inplace FC1 outputs out of the Rubin autotune key by @Barry-Delaney in #21190
- [None][test] Place the Kimi K3 unit tests by scope by @erictsai-nv in #21911
- [None][fix] Fix draft KV over-allocation during chunked prefill by @pengbowang-nv in #21546
- [TRTLLM-17505][feat] KVCM2: pool SINK_REQUIRED_ONLY lifecycles like REQUIRED ones by @Hudayday in #21793
- [None][feat] Classify startup failures, surface their root cause and report them in telemetry by @zhenhuaw-me in #20853
- [TRTLLM-17516][fix] Bound context logits memory with CPU fallback by @zhaoyangwang-nvidia in #21315
- [TRTLLM-14853][chore] Update CODEOWNERS after the code-layout moves by @YihuiLu512 in #21514
- [https://nvbugs/7001101][fix] Put the coordinator Unix socket in an owner-only directory by @trtllm-agent in #21204
- [None][infra] Skip retired H100 perf stage through maintenance config by @chzblych in #21991
- [#19173][fix] Record checkpoint policy and runtime image in startup telemetry by @chienchunhung in #19174
- [None][perf] autotuner: drive DeepGEMM JIT warmup on a thread pool and stop re-walking excluded-op sweeps by @hyukn in #21610
- [None][feat] Support FP8 KV cache for non-MLA PrimsTS attention by @yuxianq in #19396
- [None][perf] Quantize large bf16 activations of static-FP8 linears with a Triton kernel by @aswinvisva in #21208
- [#19861][fix] Fix FP8 KV cache for cross attention on the PyTorch backend by @lminer in #20958
- [TRTLLM-17393][perf] MiniMax-M3 sparse attention: trim per-step host work by @pranav-nvidia in #20848
- [None][perf] Reduce Qwen4-Exp PLE padding and workspace by @2ez4bz in #21841
- [None][chore] Allow NCCL window registration during CUDA graph capture by @nv-lschneider in #20908
- [None][feat] Enable PPLX decider support by @anurags25 in #20238
- [None][fix] Tool parsers: report the calls a stream's last increment brings by @JunyiXu-nv in #21975
- [None][fix] Separate generation cancellation from receive retirement by @chienchunhung in #21491
- [TRTLLM-17146][feat] Add the Muse Spark Open decoder fast paths by @moraxu in #21356
- [https://nvbugs/6227108][perf] extend AllReduce tuning to the resolved token budget by @dhansen-nvidia in #20241
- [None][test] Remove GLM-5, Qwen3 and DSV4-Pro gen_only perf-sanity cases; add MiniMax-M3, Kimi-K3 and GLM-5.3-Flash cases by @chenfeiz0326 in #21832
- [TRTLLM-16624][fix] Fix Helix MLA decode kernel selection by @zhaoyuanh-nvidia in #21582
- [None][feat] Add Modeling V2 GLM-5.3-Flash NVFP4 targets on SM100 with fused MTP-Eagle by @zhaoyangwang-nvidia in #21787
- [https://nvbugs/7071872][fix] Align disaggregation cancellation fixtures with batch state by @yuxianq in #21994
- [None][fix] Acquire at cluster scope on the mailbox waits of the Qwen4 mid-rows GEMM and HC decode kernels by @bastianhagedorn in #21853
- [TRTLLMINF-572][infra] Support setting separate rerun policies for each stages by @yiqingy0 in https://github.com/NVIDIA/TensorRT-LLM/pull/21189
- [https://nvbugs/7066081][doc] Correct the draft-target decoding_type value by @ruodil in https://github.com/NVIDIA/TensorRT-LLM/pull/21894
- [https://nvbugs/7066083][doc] Distinguish V1 and V2 partial-reuse copying by @ruodil in https://github.com/NVIDIA/TensorRT-LLM/pull/21896
- [None][fix] Collect SLURM CBTS coverage before uploading results by @crazydemo in https://github.com/NVIDIA/TensorRT-LLM/pull/21717
- [https://nvbugs/6607482][fix] Skip the KV host tier in the Gemma4 dummy E2E tests by @sunnyqgg in https://github.com/NVIDIA/TensorRT-LLM/pull/21993
- [TRTLLM-17628][test] Define multinode test launch parameters in YAML by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/21704
- [TRTLLM-17599][test] Run the host-only MoE unit tests in the CPU stage by @rosong11 in https://github.com/NVIDIA/TensorRT-LLM/pull/21703
- [TRTLLM-17388][feat] MiniMax-M3: load the compressed-tensors NVFP4 MoE packaging by @pranav-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/20847
- [TRTLLM-16624][fix] Do not specialize spec-decode Triton kernels on batch-dependent arguments by @zhaoyuanh-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/21666
- [https://nvbugs/7075896][fix] Fix pyexecutor request retirement unit test failure by @lori-ren in https://github.com/NVIDIA/TensorRT-LLM/pull/22014
- [None][doc] Add a tech blog on dense DWDP by @zongfeijing in https://github.com/NVIDIA/TensorRT-LLM/pull/21807
- [TRTLLM-17210][feat] Qualify tied word embeddings for the MX Qwen3 dense profile by @moraxu in https://github.com/NVIDIA/TensorRT-LLM/pull/20767
- [None][test] remove DSpark AgentX perf cases by @fredricz-20070104 in https://github.com/NVIDIA/TensorRT-LLM/pull/21990
- [TRTLLM-16288][feat] Integrate sparse KV offload for DeepSeek-V4 Flash and GLM5.2 on KVCM2 by @cascade812 in https://github.com/NVIDIA/TensorRT-LLM/pull/21350
- [TRTLLM-17261][test] Add route, middleware-order and shim-identity snapshots and a CPU test seam for trtllm-serve by @JunyiXu-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/21406
- [TRTLLM-17659][refactor] Pin down the encoder-to-decoder handoff in PyExecutor by @lori-ren in https://github.com/NVIDIA/TensorRT-LLM/pull/21768
- [TRTLLM-424][feat] Add Gumbel-max text watermarking with spec dec support by @hchings in https://github.com/NVIDIA/TensorRT-LLM/pull/21047
- [None][test] Remove unscheduled Nemotron V3 Super parallelism entries from l0_dgx_b200 by @Wanli-Jiang in https://github.com/NVIDIA/TensorRT-LLM/pull/22081
- [None][fix] Keep Slurm runners local to each parallel branch by @ZhanruiSunCh in https://github.com/NVIDIA/TensorRT-LLM/pull/22101
- [https://nvbugs/7074476][fix] Align full-context decode with the PrimTS runtime ABI by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/22050
- [None][feat] Fused DSA prefill indexer + exact top-k (throwaway top-k) on SM100/SM103 by @limin2021 in https://github.com/NVIDIA/TensorRT-LLM/pull/21892
- [None][doc] Clarify fused-target naming and target order in the Modeling V2 speculative guide by @zhaoyangwang-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/22084
- [None][fix] Reclaim KV cache resources after failed or timed-out disaggregated transfers by @reasonsolo in https://github.com/NVIDIA/TensorRT-LLM/pull/21909
- [https://nvbugs/7024242][fix] Scope the GLM-5.2 sm_103 tp8 accuracy engine's environment overrides by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/22032
- [None][fix] Localize build arguments and sanity pod configurations by @chzblych in https://github.com/NVIDIA/TensorRT-LLM/pull/22102
- [None][chore] Remove Wan2.1 from VisualGen perf sanity by @taianz-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/21929
- [None][doc] Add ISER-MoE Dynamic In-Switch Replication of Hot Experts for Load Balancing in Expert-Parallel tech blog by @JintaoPengCS in https://github.com/NVIDIA/TensorRT-LLM/pull/21492
- [#20485][fix] Fix NGramPoolManager candidate pruning and single-match invariant by @kunalsanga in https://github.com/NVIDIA/TensorRT-LLM/pull/20487
- [None][fix] Accept input_ids in the bench_moe supplied top-k hook by @Barry-Delaney in https://github.com/NVIDIA/TensorRT-LLM/pull/21961
- [https://nvbugs/6949277][fix] Give the Nemotron-3-Ultra tests a timeout that covers a cold checkpoint read by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/22052
- [https://nvbugs/6487918][test] Unwaive Qwen3.5-4B BF16 accuracy on GB300 by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/22126
- [TRTLLM-17136][feat] Serve Muse Spark Open: reasoning and tool parsers, effort scale, native tools by @moraxu in https://github.com/NVIDIA/TensorRT-LLM/pull/21049
- [https://nvbugs/7034340][fix] Free the executor in the MPI worker teardown by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/21777
- [None][fix] Move two Qwen4 unit tests under ops/ and read YAML QA lists in the Modeling V2 CI selection check by @erictsai-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22085
- [https://nvbugs/5651865][test] Unwaive one GPT-OSS TP4 CUTLASS FP8 accuracy test by @leslie-fang25 in https://github.com/NVIDIA/TensorRT-LLM/pull/21968
- [None][perf] modeling_v2 GLM-5.2 NVFP4 sm_103 tep4: faster prefill and decode, MTP3 / MTP5, long-context decode by @lijingticy22 in https://github.com/NVIDIA/TensorRT-LLM/pull/21695
- [https://nvbugs/6621362][fix] Cherry-pick #20459: Unblock Python disagg transfers stalled by a context-side timeout by @bo-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/20943
- [None][infra] unwaive test case by @HuiGao-NV in https://github.com/NVIDIA/TensorRT-LLM/pull/22116
- [TRTLLMINF-472][fix] Upload checkpoints from a watcher beside the tests by @sharabeshj-nvda in https://github.com/NVIDIA/TensorRT-LLM/pull/21942
- [https://nvbugs/7066082][doc] Document runtime speculation controls by @ruodil in https://github.com/NVIDIA/TensorRT-LLM/pull/21895
- [https://nvbugs/7066117][test] Unwaive Qwen3.5 VL and modeling unit tests fixed by #21876 by @ruocheng-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22087
- [None][perf] Build FlashMLA without relocatable device code by @ruocheng-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22028
- [https://nvbugs/6908275][fix] Correct HF reference precision in BF16 all-reduce accuracy test by @yizhang-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/20110
- [TRTLLM-17230][feat] Support First N/Last N Mix precision for cold-page KV Cache Compression by @Hudayday in https://github.com/NVIDIA/TensorRT-LLM/pull/20197
- [None][perf] Build each DeepGEMM and CuTe DSL warmup kernel once across the ranks of a launch by @chenfeiz0326 in https://github.com/NVIDIA/TensorRT-LLM/pull/21791
- [None][fix] NIXL: release transfer status handles without holding the GIL by @xguannv in https://github.com/NVIDIA/TensorRT-LLM/pull/21612
- [TRTLLM-17676][fix] Reject encoder-decoder models in recompute_active_requests by @lori-ren in https://github.com/NVIDIA/TensorRT-LLM/pull/22103
- [https://nvbugs/7046451][fix] Size the GQA segmented-decode FMHA workspace for FlashInfer's LSE softmax stats by @dc3671 in https://github.com/NVIDIA/TensorRT-LLM/pull/21863
- [None][perf] Cut per-turn prompt handling cost for long multi-turn prompts by @WeiHaocheng in https://github.com/NVIDIA/TensorRT-LLM/pull/21308
- [None][fix] Align the serve constructor ratchet with its landing baseline by @JunyiXu-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22174
- [TRTLLM-16416][test] Select QA tests by target machine during collection by @jieli-matrix in https://github.com/NVIDIA/TensorRT-LLM/pull/19212
- [None][infra] pause DGX B200 stages for nsc-svg outage by @hanjingtian in https://github.com/NVIDIA/TensorRT-LLM/pull/22165
- [None][chore] Follow-ups for #21067 review comments by @xguannv in https://github.com/NVIDIA/TensorRT-LLM/pull/21976
- [None][feat] agent-flow: opt-in MX qualification for modeling-bringup by @moraxu in https://github.com/NVIDIA/TensorRT-LLM/pull/19996
- [None][fix] Qwen3.8-Flash-Next: mask FP8 PLE rows by bytes under attention DP by @Muchuan-Jin in https://github.com/NVIDIA/TensorRT-LLM/pull/21346
- [None][fix] Runtime fixes found while bringing up Mistral Large 4 by @yizhang-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/21788
- [https://nvbugs/6891163][fix] Build embed_tokens as a full copy (no mapping) when enable_attention_dp and… by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/20089
- [TRTLLM-16303][refact] Keep hybrid speculative replay state in KV cache regions by @VALLIS-NERIA in https://github.com/NVIDIA/TensorRT-LLM/pull/21324
- [https://nvbugs/7028063][fix] Correct MNNVL fused RMSNorm shared-memory allocation by @dc3671 in https://github.com/NVIDIA/TensorRT-LLM/pull/21785
- [https://nvbugs/6858551][fix] Keep each encoder step within encoder_max_num_tokens and encoder_max_batch_size by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/21933
- [https://nvbugs/6178915][fix] Use raw FC1 scale for MXFP4 FP8 MoE input quantization by @xxi-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22106
- [None][test] Run Kimi K3 MMMU accuracy test on KV cache manager V2 by @yizhang-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/20038
- [None][fix] Respect FastWan sampling defaults during startup warmup by @taianz-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/21029
- [https://nvbugs/7023112][fix] Release profiling KV pools before NIXL deregistration by @zhaoyangwang-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/21784
- [None][infra] Sending PLC slack report with CORA bot by @yuanjingx87 in https://github.com/NVIDIA/TensorRT-LLM/pull/21970
- [https://nvbugs/6939213][fix] Release cached MoE runner workspaces at executor shutdown by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/20840
- [https://nvbugs/7028060][fix] Stop shimming FlashInfer's W4A16 tile validator once it takes enforce_search_floor by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/22004
- [TRTLLM-17360][feat] Add Beam StairCase operators and model components by @Mgluhovskoi in https://github.com/NVIDIA/TensorRT-LLM/pull/21471
- [https://nvbugs/6813945][fix] Support etcd credentials, TLS and signed records for the disagg cluster storage by @BoyueZ in https://github.com/NVIDIA/TensorRT-LLM/pull/21716
- [https://nvbugs/7028061][fix] Align MiniMax M3 Eagle3 draft-loop API by @Mgluhovskoi in https://github.com/NVIDIA/TensorRT-LLM/pull/21815
- [None][infra] Update getPLCRepoUrlKey by @yuanjingx87 in https://github.com/NVIDIA/TensorRT-LLM/pull/21360
- [None][test] Give the gen-preemption fake executor the encoder scheduling methods by @karljang in https://github.com/NVIDIA/TensorRT-LLM/pull/22201
- [None][perf] Record KDA replay acceptance in one Triton launch by @zhaoyangwang-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/22009
- [None][feat] MegaMoE: sync the vendored CuTe DSL kernels from upstream by @Barry-Delaney in https://github.com/NVIDIA/TensorRT-LLM/pull/21977
- [None][perf] Add the Kimi K3 SM107 MegaMoE generation-phase tactic at 512 tokens per rank by @Barry-Delaney in https://github.com/NVIDIA/TensorRT-LLM/pull/21960
- [None][feat] Add SeaCache step caching for Cosmos3 by @skasetty-Siri in https://github.com/NVIDIA/TensorRT-LLM/pull/21288
- [None][test] Add GLM-5.3-Flash NVFP4 core performance coverage by @yufeiwu-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22163
- [None][infra] Extend advisory Pyrefly coverage to tests by @HuiGao-NV in https://github.com/NVIDIA/TensorRT-LLM/pull/22006
- [None][fix] Keep 16-bit GVR register plans within their row window by @longcheng-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22184
- [TRTLLM-14855][chore] Use the canonical modules instead of the layout compatibility shims by @YihuiLu512 in https://github.com/NVIDIA/TensorRT-LLM/pull/22086
- [None][chore] Update flashinfer-python from 0.7.2 to 0.7.3 by @farazkh80 in https://github.com/NVIDIA/TensorRT-LLM/pull/21824
- [https://nvbugs/7046452][fix] Keep non-power-of-two hidden sizes off the MoE decode front's Triton router GEMM by @dc3671 in https://github.com/NVIDIA/TensorRT-LLM/pull/21862
- [None][fix] Kimi-K3 tool parser: warn only when a call names no declared tool by @JunyiXu-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/21625
- [None][fix] Fix the breakable-prefill NCCL window hang behind the Qwen3.5 NCCL_SYMMETRIC exclusion by @dc3671 in https://github.com/NVIDIA/TensorRT-LLM/pull/21799
- [https://nvbugs/7082290][test] Unwaive the Modeling V2 CI selection checks fixed by #22085 by @erictsai-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22215
- [https://nvbugs/7074936][test] Align the TP2 MoE RMS acceptance criterion by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/22076
- [None][test] Use real text for Qwen3.8 and fix shared prompt counts by @yufeiwu-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22180
- [None][infra] Add blossom-ci authorized users by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/22235
- [#21317][fix] trtllm-bench: fall back to system memory when NVML memory query is not supported by @spped2000 in https://github.com/NVIDIA/TensorRT-LLM/pull/21340
- [#21613][fix] Tokenize stop and bad strings when end_id is given explicitly by @Anshtyagi1729 in https://github.com/NVIDIA/TensorRT-LLM/pull/21614
- [#20986][fix] Cap decoded audio samples in AudioMediaIO by @LeizhenZhang in https://github.com/NVIDIA/TensorRT-LLM/pull/20987
- [None][fix] Double-buffer update_ready in the KDA decomposed chain kernel by @ruocheng-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22179
- [None][chore] Remove unscheduled Wan 2.2 B200 test entries by @zhenhuaw-me in https://github.com/NVIDIA/TensorRT-LLM/pull/22217
- [TRTLLM-17327][feat] Run unquantized TRTLLM-Gen MoE on native BF16 cubins; fix TRTLLM-Gen routing tiers past 128 experts by @rosong11 in https://github.com/NVIDIA/TensorRT-LLM/pull/21718
- [None][fix] Cherry-pick optional native RankInfo authentication by @yibinl-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/21486
- [None][feat] Shared drafter KV pools, FP8 MLA context inputs and KV cache V2 copy/memory fixes for hybrid speculative models by @xguannv in https://github.com/NVIDIA/TensorRT-LLM/pull/21972
- [None][fix] Runtime fixes and small kernel changes from serving Qwen3.8-Flash-Next-FP8 by @Wanli-Jiang in https://github.com/NVIDIA/TensorRT-LLM/pull/21930
- [#20138][fix] Preserve Qwen3-VL video layout across E/PD by @maheshrbapatu in https://github.com/NVIDIA/TensorRT-LLM/pull/20497
- [TRTLLM-16958][feat] Executor host path and hybrid GDN/Mamba perf improvements for RL by @hchings in https://github.com/NVIDIA/TensorRT-LLM/pull/20916
- [None][chore] Remove retired semantic review automation by @chzblych in https://github.com/NVIDIA/TensorRT-LLM/pull/22187
- [https://nvbugs/6178914][fix] Correct MXFP4 FP8 MoE SwiGLU scaling by @xxi-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22108
- [None][fix] report unavailable benchmark latencies as N/A by @yufeiwu-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22178
- [None][infra] restore DGX B200 stages after nsc-svg maintenance by @hanjingtian in https://github.com/NVIDIA/TensorRT-LLM/pull/22212
- [None][chore] Upgrade Mooncake version to 0.3.14 by @SimengLiu-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22191
- [None][feat] Load nvidia/Cosmos3-Edge-FP8 with its W8A8/W8A16 step policy by @ConstBob in https://github.com/NVIDIA/TensorRT-LLM/pull/21572
- [None][infra] Adjust PLC pipeline by @yuanjingx87 in https://github.com/NVIDIA/TensorRT-LLM/pull/22060
- [TRTLLM-17384][perf] Cut per-step host work of the executor, sampler and KV cache scheduler by @hchings in https://github.com/NVIDIA/TensorRT-LLM/pull/21974
- [None][fix] agent-flow: modeling-bringup resumes with the checkpointed task.yaml by @moraxu in https://github.com/NVIDIA/TensorRT-LLM/pull/21677
- [None][fix] Correct Qwen3.8-Flash-Next attention-DP padding on SM120 by @pamelap-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/21316
- [TRTLLM-17767][refactor] Pin down multimodal item lifetimes across inline, scheduled and prefetched encoding by @lori-ren in https://github.com/NVIDIA/TensorRT-LLM/pull/21986
- [None][perf] Run the draft layer after attention only on drafting rows by @zhaoyangwang-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/22008
- [None][feat] Add sentinel transport to NVLinkOneSided All2All; Retire CFT counted-write by @zhangcl in https://github.com/NVIDIA/TensorRT-LLM/pull/21668
- [https://nvbugs/6982378][fix] Build the FlashInfer FMHA module on all PP ranks before warmup by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/21203
- [https://nvbugs/6943029][fix] Size Whisper's cross-KV pool from its encoder length, not max_input_len by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/20918
- [None][perf] Speed up CUTLASS MoE FP8 block-scale row expansion and activation launch by @ruocheng-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/21178
- [TRTLLM-15161][test] Mark pyexecutor, attention and disaggregation unit tests cpu_only (batches #2, #10, #13, #14) by @karljang in https://github.com/NVIDIA/TensorRT-LLM/pull/21700
- [https://nvbugs/7050010][fix] Preserve MTP quantization exclusions by @trtllm-agent in https://github.com/NVIDIA/TensorRT-LLM/pull/21840
- [TRTLLM-16622][feat] DeepSeek-V4.1: Perf optimizations for DS-V4.1-Flash-NVFP4 by @ziyixiong-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/21438
- [None][fix] Let a paired draft KV pool publish its blocks by @aswinvisva in https://github.com/NVIDIA/TensorRT-LLM/pull/19303
- [None][feat] KVCacheManagerV2: 3 SSD offloading optimizations by @zhangcl in https://github.com/NVIDIA/TensorRT-LLM/pull/21822
- [#19510][feat] Add native Qwen3 reranking support by @anurags25 in https://github.com/NVIDIA/TensorRT-LLM/pull/19550
- [None][feat] StairCase: GLM-5.2 NVFP4 Rubin high-throughput target by @dongfengy in https://github.com/NVIDIA/TensorRT-LLM/pull/21585
- [TRTLLM-14954][feat] Add MoeConfig.tactic to pin a MoE implementation by its id by @xxi-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/21778
- [None][infra] Prebuild x86 C++ tests and consolidate GPU CI stages by @chzblych in https://github.com/NVIDIA/TensorRT-LLM/pull/22124
- [None][fix] Stabilize MPI startup, readiness and CI tests by @chzblych in https://github.com/NVIDIA/TensorRT-LLM/pull/22055
- [None][infra] Disable UBI repos in the Rocky Linux 8 CI image build by @weiminwang-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22244
- [None][infra] Split GB300 and DGX B200 pre-merge tests by @chzblych in https://github.com/NVIDIA/TensorRT-LLM/pull/22307
- [https://nvbugs/5989912][test] Unwaive Relu2 NVFP4 MoE backend test by @xxi-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/22104
New Contributors
- @spped2000 made their first contribution in #21341
- @maxwbuckley made their first contribution in #19735
- @Smallfu666 made their first contribution in #21434
- @skasetty-Siri made their first contribution in #20984
- @arysef made their first contribution in #21749
- @bastianhagedorn made their first contribution in #21853
- @kunalsanga made their first contribution in https://github.com/NVIDIA/TensorRT-LLM/pull/20487
- @lijingticy22 made their first contribution in https://github.com/NVIDIA/TensorRT-LLM/pull/21695
- @Muchuan-Jin made their first contribution in https://github.com/NVIDIA/TensorRT-LLM/pull/21346
- @Anshtyagi1729 made their first contribution in https://github.com/NVIDIA/TensorRT-LLM/pull/21614
- @LeizhenZhang made their first contribution in https://github.com/NVIDIA/TensorRT-LLM/pull/20987
- @maheshrbapatu made their first contribution in https://github.com/NVIDIA/TensorRT-LLM/pull/20497
Full Changelog: v1.4.0rc3...v1.4.0rc4