github NVIDIA/TensorRT-LLM v1.4.0rc4

pre-release58 minutes ago

Highlights

  • Known Issues

    • DeepSeek-R1 FP4 disaggregated serving on B300 may produce incorrect output under CUDA graphs.
    • GLM-5.2 NVFP4 SM103 TP8 may fail to initialize when Modeling V2 is required with FlashInfer 0.7.3.
    • Kimi K3 NVFP4 KV may fail to initialize on B300 due to FP4 MLA quantization eligibility.
    • Kimi K3 DSpark with MegaMoE CuTe DSL TEP and CUDA graphs may hang during GB300 warmup.
    • Qwen3.8-Flash-Next FP8 with PLE offloading, attention DP and MTP may fail during initialization.
  • Model Support

    • Add Apertus and Apertus 1.5 text models to the PyTorch backend #19735
    • Add GLM-5.2 NVFP4 high-throughput and low-latency Modeling V2 targets for Rubin #21585, #21380, #21590
    • Add GLM-5.3-Flash NVFP4 Modeling V2 targets for SM100 and SM120 sparse attention #21787, #20639
    • Add MiniMax-M3 NVFP4 low-latency decode Modeling V2 targets for Rubin #20623
    • Add Qwen3.8-Flash-Next NVFP4 Modeling V2 targets for GB300 parallelism configurations #21372
    • Add native Qwen3 reranking and Muse Spark Open serving support #19550, #21049
    • Load Cosmos3-Edge-FP8 checkpoints with their W8A8/W8A16 step policy #21572
    • Add compressed-tensors NVFP4 MoE packaging support for MiniMax-M3 #20847
    • Qualify Qwen3 dense FP8, tied embeddings, and Qwen3 MoE BF16 for ModelExpress #20766, #20767, #20765
    • Add FastH3 support with VSA and MiniMax-H3 Ulysses attention #20648, #21222
  • API

    • Use string enums for checkpoint I/O policy and checkpoint formats (BREAKING) #20498, #20703
    • Deprecate KV cache manager V1 and the C++ cache transceiver #21525
    • Default the Modeling V2 LLM argument to auto #21712
    • Add MoeConfig.tactic to select a MoE implementation by ID #21778
    • Add opt-in disaggregated transfer diagnostics and startup failure telemetry #18913, #20853
    • Support etcd credentials, TLS, and signed records for disaggregated cluster storage #21187, #21716
    • Preserve request IDs in Anthropic message IDs and send streaming finish reasons separately #20936, #21149
    • Require tagged-union discriminators in serving configuration schemas #21416
    • Add keyed log-once support to the C++ logger #21775
  • Feature

    • Add sparse KV offload for DeepSeek-V4 Flash and GLM-5.2 using KVCM V2 #21350
    • Extend mixed-precision KV caching to speculative decoding, sparse MLA, and cold-page compression #21021, #20197
    • Add host-tier suspension and cached MLA gathering for aggregated Helix prefill #21455, #21527
    • Add shared drafter KV pools, FP8 MLA inputs, and unified DSpark/DFlash KV caches #21972, #20664, #20486
    • Add generation-slot preemption and SINK_REQUIRED_ONLY attention reuse in KV cache V2 #21334, #21432, #21793
    • Optimize KVCM V2 SSD offloading #21822
    • Add native DSA prefill indexing and exact top-k for SM100 and SM103 #21892
    • Add Qwen3.8 QSA sparse attention using the PrimTS backend #20714
    • Support FP8 KV cache in non-MLA PrimTS attention and Gemma4 variable-window attention #19396, #18912
    • Add dense DWDP for Kimi K3 context servers and balanced expert partitioning #21012
    • Add NVLinkOneSided All2All sentinel transport and Rubin MLA, DSpark, DWDP and Helix changes #21668, #21067
    • Add NVFP4_PLUS CuTe DSL MoE for Rubin and native BF16 TRTLLM-Gen MoE #21242, #21718
    • Add Beam StairCase components, DSpark confidence scheduling, and device-side reranking #21471, #20078, #21596
    • Add Gumbel-max text watermarking with speculative decoding #21047
    • Add Cosmos3 SeaCache step caching and per-request sigma grids #21288, #20984
    • Add VisualGen combined Omni references, token-sharded TP, and Rubin FP8 attention #21226, #20483, #20005
    • Support optional MiniMax-H3 host-backup release on wake #19612
    • Add FP32, BF16, and FP16 unified GVR self-sampling templates #19285
    • Add ModelExpress qualification to the modeling bringup workflow #19996
    • Improve executor and hybrid GDN/Mamba performance for RL workloads #20916
    • Optimize DeepSeek-V4.1 Flash NVFP4 and GLM-5.2 SM103 prefill and decode #21438, #21695
    • Add Muse Spark decoder fast paths and reduce Qwen4-Exp PLE workspace #21356, #21841
    • Optimize Kimi K3 MoE routing and decode and add an SM107 MegaMoE tactic #21400, #21399, #21960, #21977
    • Reduce executor, sampler, KV scheduler, and long-prompt host overhead #21974, #21308, #20848
    • Optimize speculative draft rows, KDA replay, and beam-history copying #22008, #22009, #21035
    • Improve DeepGEMM and CuTe DSL warmup and share autotuning resources #21791, #21610, #21615, #21419
    • Accelerate FP8 MoE row expansion, linear quantization, and Qwen3.5 GDN projections #21178, #21208, #21287
    • Fuse NVFP4 activation quantization and enable tiled MoE quantization #21273, #21239
    • Optimize QSA and DSA decode metadata and target MSA JIT kernels to visible GPUs #21409, #19786, #21681
    • Support FP8 wide-head MLA verification on SM107 and add PrimTS attention recipes #21423, #20068
    • Stream multimodal hashing and move vision pixel normalization to GPU #18637, #20973
    • Extend all-reduce tuning and enable NCCL window registration during CUDA graph capture #20241, #20908
    • Add PPLX decider support and reduce speculative strict-accept compilation variants #20238, #21424
  • Fix

    • Restore auxiliary KV pool selection for DSA FP4 MLA #21547
    • Release cached MoE workspaces at executor shutdown and make KVCM V2 teardown safe without CUDA context #20840, #21795
    • Release profiling KV pools before NIXL deregistration and unblock timed-out Python transfers #21784, #20943
    • Reclaim KV resources after failed transfers and complete cancelled disaggregated requests #21909, #21508, #21482
    • Resolve mixed-precision KV-head counts in disaggregated transfers and publish paired draft KV blocks #21597, #19303
    • Correct FP8 KV cache for cross attention and size Whisper cross-KV pools from encoder length #20958, #20918
    • Fix draft KV over-allocation in chunked prefill and size separate draft mirrors from request history #21546, #21281
    • Fix GLM-5.2 Modeling V2 test environment overrides and MiniMax-M3 Eagle3 draft-loop API #22032, #21815
    • Correct Qwen3.8 FP8 PLE row masking and SM120 attention-DP padding #21346, #21316
    • Fix Qwen3.8 serving, Mistral Large 4 runtime, and Qwen3-VL disaggregated video layout #21930, #21788, #20497
    • Correct MXFP4 FP8 MoE SwiGLU and input-quantization scaling #22108, #22106
    • Fix Qwen MoE FP8 LoRA test configuration and MiniMax subprocess imports #21680, #21671
    • Correct MNNVL fused RMSNorm shared memory and KDA Small-BH TMA-store fencing #21785, #21908
    • Support non-power-of-two expert counts and hidden sizes in MoE routing #21923, #21862
    • Fix Helix MLA decode selection, contiguous NCCL inputs, and breakable-prefill NCCL hangs #21582, #21213, #21799
    • Initialize FlashInfer FMHA on all pipeline ranks before warmup #21203
    • Enforce encoder token and batch budgets and reject unsupported recompute configurations #21933, #22103
    • Preserve MTP quantization exclusions and prevent stale multimodal item lifetimes #21840, #21986
    • Fix stop and bad-string tokenization and remove streamed stop-word IDs from final text #21614, #21341
    • Correct GLM and Kimi tool parsing and preserve final streamed tool calls #21798, #21108, #21503, #21625, #21975
    • Cap decoded audio samples and keep coordinator sockets in owner-only directories #20987, #21204
    • Release NIXL transfer status handles without holding the GIL #21612
    • Correct GQA workspace sizing, GVR row-window bounds, and non-power-of-two MoE routing #21863, #22184, #21923
    • Preserve standalone VisualGen teardown and FastWan warmup sampling defaults #21417, #21029
    • Pin sentencepiece below 0.2.3 and preserve DeepSeek-V4 metadata behavior #21790, #21319
    • Stabilize MPI startup and readiness and free executors during MPI worker teardown #22055, #21777
    • Correct attention-DP token embeddings and Qwen4 kernel synchronization #20089, #21853
    • Fix DeepGEMM MoE memory bounds, KDA update synchronization, and GDN workspace selection #21401, #22179, #21332
    • Correct Lamport all-reduce clear sizes and use 64-bit convolution-state offsets #21283, #21684
    • Correct speculative-decoding kernel specialization and DFlash logit and window settings #21666, #21367
    • Separate generation cancellation from receive retirement and report native KV transfer results #21491, #21538
    • Correct n-gram pool pruning and fail requests cleanly when grammar creation fails #20487, #20903
    • Align PrimTS decode with its runtime ABI and correct MoE autotuning keys #22050, #21190
    • Default dense Qwen3.5 and Qwen3.8 NVFP4 checkpoints to Marlin on Ada and Hopper #21434
    • Bound context-logits memory with a CPU fallback and validate RADIO video weights at load #21315, #21631
    • Preserve Kimi K3 Responses usage accounting and correct JSON-schema example inputs #21489, #21026
    • Handle unavailable benchmark latency and NVML memory results #22178, #21340
    • Fail unsupported non-generation requests and executor loops cleanly #21584
    • Support optional native RankInfo authentication #21486
    • Correct Helix context-parallel validation and avoid PyTorch mimalloc huge pages #21192, #21463
    • Correct CGA reduction split caps and KV-window logging #21767, #20650
    • Accept supplied token IDs in the MoE benchmark top-k hook #21961
  • Documentation

    • Document runtime speculation, DSpark confidence, and Modeling V2 routing #21895, #21601, #21910
    • Clarify KV-cache reuse, deployment memory settings, and speculative decoding configuration #21896, #21893, #21894, #22084
    • Add dense DWDP and dynamic expert replication technical blogs #21807, #21492
    • Refresh model references and the MoE performance dashboard #21737, #21749, #21757
  • Test & Infra

    • Upgrade Mooncake to 0.3.14 and FlashInfer Python to 0.7.3 #22191, #21824
    • Reject direct dependencies in root requirement files and update release version tooling #21817, #21813
    • Prebuild x86 C++ tests and split GB300 and B200 pre-merge stages #22124, #22307
    • Migrate test results to Artifactory and upload checkpoints from a separate watcher #21223, #21942
    • Capture setup timeouts, classify pytest timeouts, and configure per-stage rerun policies #19333, #17808, #21189
    • Expand CBTS selection and coverage for model aliases, Python changes, and CPU tests #21323, #21418, #21782, #21717
    • Add model coverage for DeepSeek-V4.1, Kimi K3/K2.5, GLM5 and Gemma4 #21063, #21504, #21500, #21505, #21766
    • Add GLM-5.3-Flash core performance coverage and refresh model performance-sanity lists #22163, #21832
    • Run Kimi K3 MMMU accuracy with KV cache manager V2 #20038
    • Move pyexecutor, attention, disaggregation, and host-only MoE tests to CPU stages #21700, #21703
    • Improve Slurm runner isolation, network setup, and test result logging #22101, #21727, #21794
    • Add Pyrefly type checking and advisory test coverage #21325, #22006
    • Send PLC reports through Cora and update PLC repository URL handling #21970, #21360, #22060
    • Synchronize release/1.3 changes to main and refresh PrimTS vendor sources #20843, #21354, #21670
    • Update platform context and GPU-count markers in QA tests #21534, #21541, #19212
    • Consolidate encoder runners and refactor request handoff and hybrid speculative cache state #21428, #21768, #21324
    • Remove obsolete TensorRT backend kernels and legacy CI performance paths #21543, #21524
    • Pause and restore B200 CI stages during an outage and adjust Rocky Linux 8 image repositories #22165, #22212, #22244
    • Add Modeling V2 regression jobs, routing gates, and test selection by change scope #21629, #21603
    • Improve serving route coverage, scheduler and benchmark graph tests, and disaggregated stress tests #21406, #21232, #21234, #21397
    • Add EFA functional CI coverage and improve distributed initialization #21516, #20704
    • Record checkpoint policy and runtime image in startup telemetry #19174
    • Define multinode test launch parameters in YAML and configure evaluation MPI ranks #21704, #21764
    • Track unfinished MoE tests, use node IPs for disaggregated performance tests, and refine CI stage naming #21545, #20556, #21497
    • Enable GCP IAD CI and localize build arguments and sanity-pod configuration #18914, #22102
    • Reorganize component-local C++ tests and encoder and multimodal runner setup #21024, #21410, #21914
    • Remove retired semantic review automation and unscheduled or retired performance tests #22187, #22217, #21929, #21990, #21710, #21991
    • Resume modeling bringup from checkpoints and refine the GLM Rubin target #21677, #21812
    • Reuse warmup synchronization, transport typed request inputs, and adopt canonical modules #20813, #21375, #22086
    • Refine CBTS acceptance references and test scopes #21691, #21776
    • Correct Qwen3.8 prompts, MoE and all-reduce reference checks, and Kimi and Qwen test setup #22180, #22076, #20110, #21874, #21876
    • Repair executor, disaggregation, ADP connector, and configuration CLI test fixtures #22201, #22014, #21994, #21672, #21465, #21686, #21661, #21441
    • Update Modeling V2 test selection, test placement, and architecture checks #22085, #21911, #21949
    • Improve disaggregated test ports and multi-frontend performance-sanity reservations #21249, #21606
    • Update Gemma4 and GLM test cache configuration and serve constructor baseline #21993, #21692, #22174
    • Build FlashMLA without relocatable device code and update FlashInfer tile validation #22028, #22004
    • Update code ownership, platform-neutral performance recipes, and Rubin integration follow-ups #21514, #21488, #21976
    • Remove retired FlashInfer MoE and All2All paths and unscheduled Nemotron tests #21013, #22081

What's Changed

New Contributors

Full Changelog: v1.4.0rc3...v1.4.0rc4

Don't miss a new TensorRT-LLM release

NewReleases is sending notifications on new releases.