Key Features and Enhancements
-
Rubin Support (Preview)
- Added preview support for NVIDIA Rubin (SM107), validated on Vera Rubin.
- Added DeepSeek-V4 and Kimi K3 support on Vera Rubin.
-
Model Support
- Added DeepSeek-V4 support on Blackwell and Rubin (preview).
- Added Kimi K3 support on Blackwell and Rubin (preview), including multimodal and multi-node serving. Attention data parallelism (DEP16) requires GB300-class GPU memory.
- Added Qwen3.5 support on Hopper, Blackwell and Rubin (preview), including MTP and NVFP4 on Blackwell, and Qwen3.6 and Qwen3.8-Flash-Next support on Blackwell.
- Added MiniMax-M3 support on Blackwell and Rubin (preview).
- Added GLM-5, GLM-5.2, Kimi-K2.5 and Step-3.7-Flash support on Blackwell.
- Added Nemotron Nano Omni support on Hopper and Blackwell.
- Added Gemma 4 support on Blackwell, including multimodal.
- Added Laguna support on Hopper and Blackwell.
- Added prefix caching for Mamba hybrid models, such as Qwen3.5 and Nemotron Super V3.
-
Features
- Performance:
- Optimized DeepSeek-V4, DeepSeek-V3.2, Qwen3.5, Gemma 4, MiniMax-M3 and Kimi K3 with model-specific kernel fusions.
- Added locality domain (uGPU) localization for NVFP4 MoE on Rubin, which keeps memory traffic local to each GPU die. It is enabled by default; set
DISABLE_LOCALITY_DOMAINS=1to disable it. - Reduced host overhead with async preprocessing and sync-free beam search, and reduced startup time with checkpoint read-ahead.
- Added cascade attention and low-M BF16 GEMM dispatch for SM10x.
- Disaggregated Serving:
- Added gen-first scheduling, conversation affinity routing, beam search and hybrid model support.
- Added disaggregated serving for Qwen3.8-Flash-Next.
- Speculative Decoding:
- Added DSpark, DFlash, Suffix Automaton and PARD drafters.
- Added EAGLE3 dynamic trees, Eagle 3.1 draft models and speculative decoding with LoRA.
- Added rejection sampling for one-model paths and acceptance-rate-based auto-disable for dynamic speculation.
- Added loading MTP weights from standalone checkpoints.
- Parallelism and MoE:
- Added DWDP (Distributed Weight Data Parallelism), extended Helix context parallelism, and a KV cache-aware attention-DP router.
- Added the
DENSEGEMM,MEGAMOE_DEEPGEMM,MEGAMOE_CUTEDSL,CUTEDSL_FC12(Rubin only) andMARLINMoE backends. - Added an NCCL-EP backend and support for up to 2048 experts.
- Added Router Replay (
return_routed_experts) to return per-token MoE routing for RL training.
- KV Cache:
- KV Cache Manager V2 is selected per model and enabled by default for most model families.
- Added KV cache compression (TriAttention and NVFP4 cold-page compression), NVFP4 KV cache and disk cache.
- Added per-conversation block reuse and exact multimodal KV block hashing.
- Added FlexKV and LMCache connectors.
- Quantization:
- Added MXFP8 (W8A8) linear and MoE support, including MXFP8 checkpoints such as MiniMax-M3.
- Added W4A16 NVFP4 weight-only quantization through Marlin, so NVFP4 checkpoints run on Hopper and Ada Lovelace.
- Added tunable NVFP4 quantization.
- Added support for both legacy and ModelOpt 1.x quantization config formats.
- Sparse Attention: Added DSA FP4/FP8 decode kernels, GVR Top-K, Skip Softmax Attention (with optional 2:4 activation sparsity on Rubin) and MiniMax-M3 MSA.
- Serving and APIs:
- Added the Anthropic Messages API (
/v1/messages) and extended the Responses API. - Added a gRPC server and multi-process HTTP frontends.
- Added per-request
priorityand named-functiontool_choiceto the OpenAI-compatible APIs. - Added
autotool and reasoning parser selection, and GLM-4.7, GLM-5 and Laguna tool parsers. - Added Prometheus metrics and
/start_profileand/stop_profileprofiling endpoints. - Added
trtllm-serveoptions--served_model_name,--kv_cache_dtypeand--middleware. - Added Completions API logprobs and acceptance of
parallel_tool_calls.
- Added the Anthropic Messages API (
- Sampling and LoRA:
- Added
min_p,bad_words, repetition/frequency/presence penalties, full beam search options and processed logprobs to TorchSampler. - Added an experimental
thinking_token_budgetfor reasoning models. - Added multi-LoRA serving with CUDA graphs, per-expert MoE LoRA and dense FP8 LoRA.
- Added
- Others:
- Added prefix-aware scheduling, enabled by default, to avoid duplicate prefill of shared prefixes.
- Added prefill and breakable CUDA Graph support, and custom attention masks in the TRTLLM attention backend.
- Added a multimodal encoder cache and CUDA graphs for multimodal encoders.
- Added inflight weight updates for RL workflows.
- Added
/v1/embeddingsand anllm.encode()fast path for encoder-only models. - Added prototype ModelExpress checkpoint loading.
- Performance:
-
Documentation
- Migration Guide: Added the TensorRT Backend Removed migration guide.
- Deployment Guides: Added deployment guides for GLM-5, Kimi K3, MiniMax-M3, Nemotron 3, Qwen3.8-Flash-Next and Qwen3.8/Qwen3.5.
- Feature Documentation: Added guides for KV cache compression, embeddings, ModelExpress, sub-agent routing and encoder-decoder models, and restructured the installation guide.
- Developer Guides: Added guides for sparse attention, KV cache compression, telemetry and custom kernels.
- Tech Blogs: Published blogs on:
- "Accelerating Long-Context Inference with Skip Softmax Attention"
- "Sparse Attention in TensorRT LLM"
- "Optimizing MoE Communication with One-Sided AlltoAll Over NVLink"
- "DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72"
- "Tuning CUDA Graph Batch Sizes for Higher Output Throughput"
- "Temporal Correlation Meets Sparse Attention: Guess-Verify-Refine Top-K for Blackwell"
- "Helix Parallelism in TensorRT-LLM: Scaling Multi-Million-Token Decoding with KV Cache Sharding"
- "Joint Optimization of Agent Applications and TensorRT-LLM"
- "MoE as Dense GEMM: Optimizing Low-Latency MoE Inference on NVIDIA Blackwell"
- "DeepSeek-V4 on NVIDIA Blackwell: Model-Specific and Agentic-Workload Optimizations in TensorRT LLM"
- "Evaluating Agentic Serving with Trace Replay and Job-Level Metrics"
- "GVR V2: Self-Sampling and Multi-Thresholding for Faster Exact Top-K"
- Tech blogs 1–9 were renamed to
blog01–blog09, and links to the old file names were updated.
VisualGen
VisualGen is a new vertical solution in TensorRT-LLM for image and video generation models. It is served through trtllm-serve alongside LLM inference. See the VisualGen documentation for details.
- Models
- Added image generation models: FLUX.1, FLUX.2, GLM-Image and Qwen-Image, including Qwen-Image-Edit and Qwen-Image-Layered.
- Added video generation models: Wan 2.2 (including 5B TI2V and FastWan), LTX-2, HunyuanVideo 1.5, Cosmos3 and MiniMax-H3.
- Features
- Added image generation, image editing and video generation endpoints in
trtllm-serve. - Added tensor, context and CFG parallelism, NVFP4 and FP8 quantization, CUDA graphs and runtime LoRA.
- Added TeaCache and Cache-DiT caching, sparse attention, and FlashAttention 4, FlashInfer and cuDNN attention backends.
- Added a
trtllm-evalpipeline for VisualGen models.
- Added image generation, image editing and video generation endpoints in
- Documentation
- VisualGen examples
- Published tech blogs on:
- "Scaling Video Generation Across NVL72 rack with TensorRT-LLM"
- "Accelerating Video Generation with GEMM Quantization, Attention Quantization and Skip Softmax Attention in TensorRT-LLM"
- Known Issues
- FLUX.1 checkpoints statically quantized and exported by NVIDIA Model Optimizer may fail to initialize because the exported transformer config sets
out_channelsto null.
- FLUX.1 checkpoints statically quantized and exported by NVIDIA Model Optimizer may fail to initialize because the exported transformer config sets
Infrastructure Changes
- The base Docker image for TensorRT-LLM is updated to
nvcr.io/nvidia/pytorch:26.09-py3. - The base Docker image for TensorRT-LLM Backend is updated to
nvcr.io/nvidia/tritonserver:26.09-py3. - The dependent CUDA Toolkit version is updated to 13.4.1.
- The dependent public PyTorch version is updated to 2.14.0.
- Python 3.10 and 3.12 are supported.
- TensorRT LLM now provides two sets of pre-built Python wheels, each compatible with a different PyTorch build:
- Upstream public PyTorch: wheels are hosted on PyPI.
- NGC PyTorch container: wheels are released through https://pypi.nvidia.com/trtllm_ngcpytorch/.
- TensorRT, NVIDIA ModelOpt and ONNX are no longer dependencies of TensorRT-LLM.
- The dependent transformers version is updated to 5.5.4.
- The dependent flashinfer-python version is updated to 0.6.18.
- The dependent triton version is updated to 3.8.0.
- The dependent NCCL version is updated to 2.31.2.
- The dependent NIXL version is updated to 1.4.0.
- The dependent UCX version is updated to 1.22.
- The dependent CUTLASS version is updated to 4.4.2.
- The dependent xgrammar version is updated to 0.1.32.
- The C++ standard is updated to C++20.
API Changes
- [BREAKING CHANGE] TensorRT backend removed. PyTorch is the only backend.
trtllm-build,TrtLlmArgs,BuildConfig,MedusaDecodingConfig,LookaheadDecodingConfig,trtllm-bench buildandtrtllm-bench --engine_dirare removed. See the migration guide. - [BREAKING CHANGE] Legacy C++ Executor and runtime APIs removed. The C++
ExecutorAPI, the legacy runtime and batch-manager APIs, and thecpp_onlybuild option are removed. - [BREAKING CHANGE] TensorRT-engine Triton backend models removed. Only the
llmapiTriton model remains. - [BREAKING CHANGE] TRTLLMSampler removed.
TorchSampler, which requires FlashInfer, is the only sampler. Thesampler_typeargument is ignored with a warning, anddisable_flashinfer_samplingis removed. - [BREAKING CHANGE] Two-model speculative decoding removed. The
eagle3_one_modelandmtp_eagle_one_modelfields are removed; use the one-model implementations. - [BREAKING CHANGE] AutoDeploy, Star Attention and WIDEEP removed. AutoDeploy and Star Attention are removed, and the WIDEEP MoE backend can no longer be selected.
- [BREAKING CHANGE] CLI flags override YAML config. Explicit
trtllm-serve,trtllm-evalandtrtllm-benchflags now take precedence over YAML config values. - [BREAKING CHANGE] KV cache transfer timeout defaults to 60 seconds. Previously there was no timeout; set it with
cache_transceiver_config.kv_transfer_timeout_ms. - [BREAKING CHANGE] Per-request chat templates are opt-in. Enable them with
trtllm-serve --allow_request_chat_template. - [BREAKING CHANGE] Logprobs capped at 100. Larger
logprobsorprompt_logprobsvalues raise aValueError. - [BREAKING CHANGE]
/perf_metricsendpoint removed. Use theX-TRTLLM-return-metrics: 1request header instead. - [BREAKING CHANGE] Dynamic speculation fields renamed. Use
acceptance_rate_window_sizeandacceptance_rate_threshold; the threshold is now an acceptance rate in [0, 1]. - [BREAKING CHANGE] Obsolete options removed. This includes Eagle
choicesandtrtllm-serve --fail_fast_on_attention_window_too_large. - NVFP4 cold-page compression remains opt-in and disabled by default. When enabled for DeepSeek-V4, NoPE uses NVFP4 and RoPE now defaults to 2FP4 (
nvfp4_residual_dim=64). Setskip_rope_quantization=Trueto keep RoPE at its original precision, as before, ornvfp4_residual_dim=0to use single NVFP4. The active GPU KV-cache dtype is unchanged. See the NVFP4 cold-page compression example. - [DEPRECATION] The TRITON MoE backend (
TritonFusedMoE,moe_config.backend="TRITON") is deprecated as of TensorRT-LLM 1.3 (2026-09) and will be removed after the 3-month migration period. Its only remaining role is a modest performance edge for GPT-OSS on Hopper withW4A16_MXFP4— the single configurationAUTOresolves to TRITON, and the format an MXFP4 GPT-OSS checkpoint takes on SM90. As the model set and the supported platforms keep growing, a single-scenario MoE path is no longer worth its maintenance cost.moe_config.backend="CUTLASS"replaces it functionally on Hopper: it servesW4A16_MXFP4on SM90 along with the unquantized BF16 and FP8 per-tensor paths, and MoE backend resolution already degrades to it automatically when TRITON declines a layer. During the migration period TRITON keeps working and logs a one-time warning. See the deprecation policy. - [DEPRECATION] The
trtllm-serveoptions--backend,--moe_cluster_parallel_sizeand--metrics-log-interval, and thetrtllm-evaloption--backend, are deprecated. - [DEPRECATION]
TorchCompileConfig.enable_piecewise_cuda_graphandcapture_num_tokens,MTPDecodingConfig.num_nextn_predict_layers, andperf_metrics_max_requestsare deprecated. - [DEPRECATION] Disaggregated serving without
internal_request_auth_keyis deprecated. Set the same key in the disaggregated server config and every worker config. - Usage telemetry is on by default. To opt out, set
TRTLLM_NO_USAGE_STATS=1orDO_NOT_TRACK=1, pass--no-telemetryto the CLI, or settelemetry_config=TelemetryConfig(disabled=True)in the LLM API. See the telemetry guide. - GPT-OSS defaults to the TRTLLM MoE backend on Blackwell.
TorchCompileConfig.max_num_streamsnow defaults to 3 instead of 1.
Fixed Issues
- Disaggregated Serving:
- Fixed a hang with context pipeline parallelism and generation tensor parallelism, a known issue in 1.2.
- Fixed hangs with block reuse after eviction and gen-only asynchronous KV transfer.
- Fixed an error on one rank crashing all generation ranks.
- Attention:
- Fixed an illegal memory access when
max_seq_lenexceedsmax_position_embeddings. - Fixed an XQA race condition, attention sink accuracy and a sliding window illegal memory access.
- Fixed FP8 accuracy with Skip Softmax Attention.
- Fixed a DeepSeek-V3.2 illegal memory access with CUDA graphs and host KV cache offload.
- Fixed an illegal memory access when
- Speculative Decoding: Fixed EAGLE3 crashes on Qwen3-235B-A22B NVFP4 and MTP crashes on Nemotron Super on Hopper.
- KV Cache: Fixed accuracy loss with VSWA and block reuse, and free-block counter corruption with host offload.
- Pipeline Parallelism: Fixed hangs with MTP, guided decoding and executor shutdown.
- MoE: Fixed a DeepEP hang caused by non-deterministic garbage collection.
- Kernels: Fixed Qwen2.5 mixed-precision accuracy, two-shot AllReduce accuracy and FP8 block-scale GEMM NaN outputs.
- Sampling: Fixed
ignore_eossuppressing the wrong token in TorchSampler, and NaN sampling parameters are now rejected. - General: Fixed encoder-decoder beam search corruption and out-of-memory errors with many LoRA adapters or during checkpoint prefetch.
Known Issues
- Kimi-K3 NVFP4 may fail to load its tokenizer with the bundled transformers version.
- In disaggregated serving with DSpark speculative decoding, the draft KV cache is not transferred to the generation worker, which lowers the acceptance rate. Aggregated serving is not affected.
- GLM-5.2 NVFP4 on GB200 (TP=8, attention DP disabled) may segfault during warmup when
max_num_tokensis greater than 4096. Keepmax_num_tokensat 4096 or lower. - Disaggregated serving of Nemotron-3-Super-120B-A12B FP8 on B300 may fail when concurrency exceeds the maximum batch size.
- Disaggregated serving of Qwen3-32B-FP8 on B300 may produce incorrect output.
- The
/v1/responsesendpoint has compatibility gaps with some input formats, streaming events, conversation continuation and tool calls (Qwen3, GPT-OSS). - Encoder-decoder models with an FP8 KV cache can overrun the cross-attention KV cache pool, causing an illegal memory access or incorrect output.
- Qwen3-32B with TP=4 on AWS p4d (A100) may segfault during autotuner warmup when the
aws-ofi-ncclplugin is enabled.