NVIDIA/Model-Optimizer 0.48.0

ModelOpt 0.48.0 Release

on GitHub

New Features

Sparsity

• Add skip-softmax calibration through vLLM for FlashAttention and FlashInfer, requiring --validation_prompts_file and validating serving sparsity on calibration and held-out prompts before exporting prefill/decode fits and their validation scope as sparse_attention_config. Calibration uses single-sequence, unchunked eager execution; serving preserves the calibrated 128-token KV-tile granularity and 128-row prefill Q tiles, autotunes only its execution schedule, and uses a smaller Q tile for one-token decode.
• Sparse-only vLLM installs now reject unsupported DCP, DBO/ubatching, speculative decoding, and FULL mixed-batch CUDA graphs; calibrated decode also rejects FULL decode graphs.

Quantization

• Add metadata-only export specifications and non-mutating per-weight quantizer-state capture.
• Add functional quantized-weight packing, topology-aligned state merge/select and transport APIs, and canonical Hugging Face quantization-config construction.
• HF export records canonical activation schemes, group/block sizes and weight-only MXFP4 metadata; mixed-format exports retain unquantized-layer exclusions, and non-square FP8 block layouts are rejected.
• Add Q8_0 weight-only quantization with 32-value GGML blocks, packed unified HF and Megatron export, and a built-in q8_0 PTQ recipe.
• Add Hugging Face PTQ calibration and export support for Nemotron-H MTP modules, preserving calibrated expert input scales.
• Add the models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/ptq/nvfp4-mse PTQ recipe, the PTQ stage for NVIDIA Nemotron 3.5 Super VL: weight-MSE calibrated NVFP4 routed and shared MoE experts, FP8 attention projections, Mamba in_proj/out_proj and lm_head, and an FP8 KV cache. The latent MoE, router, embeddings and the vision and audio encoders stay in BF16, and the MTP rules apply only to checkpoints that carry an MTP tail. It reproduces the released precision assignment, not the released weights: the published NVFP4 checkpoint adds quantization-aware distillation after this stage.
• Backfill checkpoint aliases for three more published NVFP4 releases: moonshotai/Kimi-K2.7-Code, google/gemma-4-26B-A4B-it and google/diffusiongemma-26B-A4B-it. Each imports an existing general recipe wholesale rather than copying its body.
• Backfill checkpoint aliases for nine more published NVFP4 releases, so each is reachable from its source model's hub path: zai-org/GLM-5.1 and GLM-5.2, MiniMaxAI/MiniMax-M2.5 and MiniMax-M3, deepseek-ai/DeepSeek-V3.1 and DeepSeek-V3.2, Qwen/Qwen3-235B-A22B-Instruct-2507 and -Thinking-2507, and Qwen/Qwen3.6-27B. Each imports an existing general or architecture recipe wholesale rather than copying its body.
• Add composed Hugging Face AutoQuantize recipes that run fixed PTQ or weight AutoQuantize before
  a separate KV-cache AutoQuantize stage, with independent resumable checkpoints for the weight and
  KV searches. Uniform-weight plus layer-wise mixed-KV exports are marked unsupported for deployment
  until a runtime consumes that metadata combination.
• Add IQ1_S and IQ2_XS weight-only quantization with GGML-compatible 256-value block encoders, built-in iq1_s / iq2_xs PTQ recipes, and unified HF and Megatron export of the packed blocks. Quantized weights must have a final dimension divisible by 256, and Megatron export requires tensor and pipeline parallel sizes of 1.
• Add iq2_xxs weight-only quantization with a CUDA encoder and a general/ptq recipe, at 2.0625 bits per weight between iq1_s and iq2_xs. The same 256-value block constraint applies.
• Add iq2_s weight-only quantization with a CUDA encoder and a general/ptq recipe, at 2.5625 bits per weight above iq2_xs. The same 256-value block constraint applies.
• Add iq1_m weight-only quantization with a CUDA encoder and a general/ptq recipe, at 1.75 bits per weight between iq1_s and iq2_xxs. The same 256-value block constraint applies.
• GPTQ (algorithm: {method: gptq}) now supports the GGML formats (iq1_s, iq1_m, iq2_xxs, iq2_xs, iq2_s, q8_0). Each GGML block is quantized as a whole and its error compensated in the later columns, and export writes exactly the codes GPTQ chose. block_size must be a multiple of the format's block size (256 for the IQ formats). The built-in iq1_s, iq1_m, iq2_xxs, iq2_xs and iq2_s recipes now quantize only the MLP/MoE weights and calibrate them with GPTQ, so they need calibration data.
• The iq1_s, iq1_m, iq2_xxs, iq2_xs and iq2_s encoders accept an optional per-input-column importance that weights each value's squared error in the codebook search, as llama.cpp's imatrix does. GPTQ's new importance_weighted option passes them the square root of its Hessian diagonal, and the built-in IQ1 and IQ2 recipes enable it, so each block's search favours the columns its outputs depend on most.
• A recipe can now delegate its whole body to another recipe with a top-level $import; any top-level key given alongside it overrides the imported one. metadata.recipe_type became optional along with it: a recipe states its kind with a # modelopt-schema: comment, with metadata.recipe_type, or by delegating to a recipe that does, and only a recipe that another file imports has to carry the schema comment. Whatever a recipe does state must be true: a schema comment and a recipe_type must agree, and so must a recipe and the recipe it delegates to. modelopt_recipes/models/ uses this for checkpoint entries that a portable recipe already reproduces: the entry aliases that recipe instead of copying it.
• Backfill the recipes behind NVIDIA's already-published checkpoints under modelopt_recipes/models/, so a released checkpoint's quantization scheme is reachable from its own model-hub path rather than only from the general tier. For example, moonshotai/Kimi-K2.6 (published as nvidia/Kimi-K2.6-NVFP4) and Qwen/Qwen3.5-397B-A17B (published as nvidia/Qwen3.5-397B-A17B-NVFP4-V2) each alias a portable recipe wholesale -- the general expert-only NVFP4 recipe and the qwen3_5_moe architecture recipe respectively -- rather than copying its body; other checkpoints follow in separate changes.
• Add layerwise.export_dir: layerwise calibration writes each decoder layer to its own quantized checkpoint shard as it finishes, so no separate export_hf_checkpoint() pass is needed and, with layerwise.checkpoint_dir, an interrupted run resumes without redoing finished layers. Calibration writes the layer shards; finalize() on the exporter left on the model adds the tail shard, the index and the config artifacts, and the checkpoint does not load until it runs. examples/hf_ptq does this for you. Supports FP8 and NVFP4 on single-process models, resident or offloaded, including multimodal models and models with MTP layers; other formats and placements raise NotImplementedError before calibration starts.
• Add support for quantizing and calibrating enabled operators outside the transformer layers, such as lm_head, when using layerwise calibration.
• Add an end-to-end BEVFormer ONNX PTQ example with temporal calibration data generation, INT8 and FP8 quantization, TensorRT engine building, and nuScenes accuracy evaluation. See examples/onnx_ptq/bevformer/README.md for details.
• Add a reusable local-Hessian NVFP4 PTQ recipe and the quantization recipe used for nvidia/Qwen3.8-27B-NVFP4.
• Add experimental dynamic FP8 fake quantization of GatedDeltaNet chunk-boundary states and WY activations for training through the standard quant_cfg interface. The fused GDN path requires fla-core==0.5.1 and chunk size 64; state emulation requires SM89 or newer.
• Add fake quantization of the sparse-attention indexer key cache and query for DeepSeek-V4 (vLLM and Megatron-Core) and GLM-5.3-Flash (vLLM) through the new indexer_k_quantizer and indexer_q_quantizer. Enable them by importing the configs/ptq/units/indexer_k_nvfp4 and configs/ptq/units/indexer_q_nvfp4 units (NVFP4 with the global scale fixed to 1) into a recipe.
• Add indexer_scorer_kwargs_quantizer to pass keyword arguments, such as the numerics options of a DeepGEMM build, to the DeepGEMM indexer scorer kernel of GLM-5.3-Flash per layer in vLLM fake-quant serving; see examples/vllm_serve/README.md.
• Add the configs/ptq/units/kv_nvfp4_mla recipe unit for fake quantization of the MLA KV cache (DeepSeek-V3, GLM-5.3-Flash, ...) in vLLM fake-quant serving: NVFP4 for the latent and FP8 for the RoPE key. See examples/vllm_serve/README.md for an example recipe.
• Add fake quantization of MoE expert-parallel communication to vLLM fake-quant serving through the new dispatch_quantizer and combine_quantizer on vLLM's routed experts, for MoE layers built on vLLM's fused-MoE layer and any all-to-all backend, and for the DeepGEMM Mega-MoE kernel of DeepSeek-V4 and Kimi-K3 (--moe-backend deep_gemm_mega_moe). Enable them by importing the configs/ptq/units/moe_dispatch_nvfp4 and configs/ptq/units/moe_combine_nvfp4 units into a recipe; see examples/vllm_serve/README.md.
• vLLM fake-quant serving now runs on pre-quantized checkpoints such as FP8 when the recipe leaves those layers unquantized (for example a KV-cache-only recipe), and on MLA models with an FP8 KV cache; both previously failed during quantization.
• KV_QUANT_CFG presets in vLLM fake-quant serving now quantize the MLA KV cache; on vLLM 0.16 and later they silently quantized nothing for MLA models.
• vLLM fake-quant serving of NVFP4 and cast-mode KV-cache configs no longer needs --enforce-eager. examples/vllm_serve/vllm_serve_fakequant.py now sets VLLM_DISABLE_COMPILE_CACHE=1 for fake-quant serves by default, because a cached torch.compile graph of the same model without the fake quant would otherwise be reused.
• examples/deepseek/deepseek_v4/ptq.py now supports DeepSeek-V4.1-Flash, whose reference implementation uses 32x32 FP8 blocks and a tokenizer-aware Transformer; this is the setup used to produce nvidia/DeepSeek-V4.1-Flash-NVFP4. DeepSeek-V4-Pro checkpoints work as before.

Speculative Decoding

• Add the DFlash2 draft variant, selected with dflash_architecture_config.projector_type="dflash2": DFlash's one-pass parallel backbone plus a grouped dynamic convolution around every attention/MLP sublayer (conv_kernel_size / conv_group_size) and a low-rank candidate selector (selector_rank / selector_top_k, weighted by dflash_selector_loss_alpha). Exported checkpoints declare DFlash2DraftModel and load in the SGLang/vLLM DFlash2 serving path.
• LiLiCorr now trains offline and in streaming mode, not only online: the target logits its distractor penalty reads are reconstructed from the captured final hidden state, the same path self-logit distillation uses.
• Add dflash_lilicorr_w_cal (default 0.0, off): an optional LiLiCorr proposal-calibration term, a KL divergence from the target's distribution over each slot's candidates to the reranker's candidate distribution, usable in place of the distractor penalty.
• examples/speculative_decoding/distributed_generate/server_generate_vlm_sglang.py --api_mode openai works against any OpenAI-compatible server, including vLLM, and gains --media_inline to send media as base64 data: URIs. Without --media_url_base or --media_inline the media path is still passed through verbatim, which only servers that read their own filesystem resolve (SGLang native does; vLLM needs --allowed-local-media-path); that case now warns.

Megatron Framework (M-LM / M-Bridge)

• Add chat-data distillation to examples/megatron_bridge/distill.py: --sft --sft_hf_dataset takes a local chat .jsonl or a Hugging Face Hub dataset, renders each conversation with the model's chat template, and --sft_loss_mode selects which turns are trained. Requires a Megatron-Bridge with DirectHFSFTDatasetConfig.
• Add freeze_mtp to the Megatron DistillationConfig (--freeze_mtp in examples/megatron_bridge/distill.py): the student's MTP head keeps its input weights and its loss is skipped, so distillation trains the rest of the model on the distillation loss alone.
• Add output_format: chat_jsonl to prepare_megatron_data_blend: it writes a weighted blend as train.jsonl / validation.jsonl chat data for distill.py --sft --sft_hf_dataset, so the distillation loss can be restricted to assistant tokens. See the Megatron data preparation guide.
• Add mergeable LoRA QAD for fake-quantized dense Megatron/Transformer Engine linear layers. Enable it with --lora_rank in Megatron-Bridge PTQ, train through the normal QAD flow, and export a standalone quantized checkpoint without deployment adapters.
• Add optional Top-P (nucleus) truncation to the Megatron top-k logits KD loss (now TopLogitsKLLoss) via logit_kl_top_p and logit_kl_top_p_min_k in DistillationConfig: after the global Top-K selection, only the smallest prefix whose cumulative teacher probability reaches top_p (with a floor of min_k entries) contributes to the KL, mirroring --logits-save-top-p / --logits-save-top-p-min-k in Megatron-LM's logits saver.
• Add an end-to-end W4A4 NVFP4 PTQ and QAD tutorial for Qwen3.6-35B-A3B also covering evaluation and vLLM throughput benchmarking. See examples/megatron_bridge/tutorials/Qwen3.6-35B-A3B/README.md for details.
• Add --mlflow <tracking-uri> to the examples/megatron_bridge scripts that write a checkpoint -- prune_minitron.py, quantize.py, distill.py, export_quantized_megatron_to_hf.py and export_distilled_megatron_to_hf.py (MLflow's own MLFLOW_TRACKING_URI is honoured too). Each run records the invocation, its arguments as searchable params and its log -- a distillation records its training metrics instead of the rank-0 log -- and writes .experiment.json into the checkpoint it produced, so a pruning, a quantization, the distillation that refines its checkpoint and the export that deploys it can be traced to one another; uploading the checkpoints themselves stays off unless --mlflow_log_checkpoints is passed.
• Add --ep_size to examples/megatron_bridge/export_quantized_megatron_to_hf.py so large MoE models with grouped-GEMM experts can be exported with their experts sharded across GPUs. Checkpoints built with --no_moe_grouped_gemm must still be exported at --ep_size 1.
• Add unified HF export of GLM-5 / GLM-5.2 (glm_moe_dsa) checkpoints quantized with examples/megatron_bridge. The MTP layer, which Megatron-Bridge does not build for these models, is copied through unquantized from the source checkpoint. DSA sparse-attention models also gain FP8 KV-cache quantization on Megatron.

Misc

• Add --modelopt-* options to the examples/vllm_serve launcher for fakequant calibration and checkpoint reload. Point it at an exported fakequant model directory to reload automatically: Hugging Face exports use vllm_fq_modelopt_state.pth, while Megatron-Core exports now include quant_recipe.yaml alongside quantizer_state.pth for reload.

• Add a kd_loss_alpha option to HuggingFace KDTrainer to blend cross-entropy and knowledge distillation losses during training.

• A tracked examples/hf_ptq/hf_ptq.py run now writes .experiment.json into --export_path and uploads the same file with the run, so a checkpoint on disk names the experiment and MLflow run id that produced it. The pointer is written only once the export completes, and an export that is not tracked removes one it would otherwise inherit from a reused --export_path or from a quantized source checkpoint.

• export_hf_checkpoint and the vLLM fake-quant export now copy the source checkpoint's non-model files (tokenizer, processor, remote code, chat templates, ...) into the export unchanged. They read the source from local disk only: get a local copy of a Hugging Face Hub model with modelopt.torch.export.ensure_local_checkpoint and load it from there.

• Add Parallel Decoding Distillation (PDD) to modelopt.torch.fastgen with Qwen-Image training, distributed-checkpoint export, and PDD-2/4/8 inference. AutoModel remains an unmodified pinned runtime dependency.

Backward Breaking Changes

• modelopt.torch.utils.distributed.broadcast/allgather now reject objects of unregistered custom classes. Register them with torch.serialization.add_safe_globals.

• Bump minimum transformers version to 5.5 instead of 4.57; transformers 4.x is no longer supported. Upgrade with pip install -U "nvidia-modelopt[hf]".

• modelopt.torch.distill.plugins.megatron.TopKLogitsKLLoss (logit_kl_topk in DistillationConfig) is renamed to TopLogitsKLLoss, keeping the old name as a deprecated alias. It now normalizes both distributions over the full vocabulary instead of re-normalizing over the Top-K entries, and always appends a "ghost" token holding the probability mass outside the Top-K to both student and teacher (matching Megatron-LM's offline cached-logits KD loss). Loss values change for existing logit_kl_topk runs.

• LogitsAndIntermediatesLossBalancer (Megatron distillation plugin) no longer rescales the distillation loss to the magnitude of the LM loss when the LM loss is included. The total is now the fixed convex combination (1 - alpha) * lm_loss + alpha * kd_loss with DistillationConfig.kd_loss_alpha (in [0, 1]), matching Megatron-LM's offline cached-logits KD. The default kd_loss_alpha=1.0 skips the LM loss, as before. DistillationConfig.skip_lm_loss and kd_loss_scale are removed and raise ValueError if passed; set kd_loss_alpha instead. In examples/megatron_bridge/distill.py, --kd_loss_alpha replaces --no_skip_lm_loss and --kd_loss_scale.

• The examples/vllm_serve fakequant launcher no longer supports vLLM 0.9.0. Upgrade to a version listed as tested in the example README.

• The Megatron-Core DeepSeek-V4 indexer (CSAIndexer) is now a quantization module and persists its quantizer state in the checkpoint as indexer._extra_state. A DeepSeek-V4 model quantized with an earlier release resumes from its torch_dist checkpoint only with a non-strict load (--dist-ckpt-strictness log_unexpected in Megatron-LM) until it is saved again.

• examples/hf_ptq no longer detects MTP layers by name. Checkpoint weights the model has no parameter for -- an MTP head, an auxiliary tower -- are worked out once, when the model loads, and carried into the export unchanged. Load with the new modelopt.torch.utils.plugins.model_load_utils.load_pretrained, as hf_ptq does, to record them then; export_hf_checkpoint works them out itself for a model loaded another way. Everything the loader did place goes through the normal export path. This removes load_mtp_weights, mtp_layer_prefixes_from_checkpoint and their support matrix of MTP storage conventions, along with _add_mtp_exclusions and the pre-quantization enable: False entries hf_ptq appended to the recipe's quant_cfg. Two consequences: MTP layers now follow the recipe like any other module instead of being force-excluded by the script -- matching examples/megatron_bridge, which has no MTP-specific code at all -- and quantization_config.ignore can no longer claim a layer is unquantized that the export in fact quantized. Recipes importing configs/ptq/units/default_disabled_quantizers still disable mtp.*, so their behaviour is unchanged; a recipe omitting that unit will now quantize an MTP the model actually built.

• examples/hf_ptq --vllm_fakequant_export now raises NotImplementedError when the checkpoint holds weights the model has no parameter for and a shard actually provides them (an MTP head, an auxiliary tower). The fake-quant exporter writes only model-backed state, so it would otherwise drop those weights silently -- and a fake-quant checkpoint is evaluated, where a missing head changes the score rather than failing loudly. Use the unified HF export, which carries them through. Buffers Transformers recomputes are not weights to lose: *.inv_freq is skipped even when a shard provides it, since older Llama/Mistral-lineage conversions do list it in the index and refusing an export over it would reject checkpoints that export correctly today. The check runs immediately after the model loads, not at export time, so an incompatible run fails before calibration rather than after it.

• The modelopt.onnx.quantization.ort_patching module has been removed with no
  compatibility shim; update direct imports to use ort_session for model loading
  and session setup, ort_calibration or ort_calibration_per_node for calibration,
  ort_quantization for static Q/DQ quantization, and ort_patches for patch composition.

• The modelopt.onnx.quantization.graph_utils module has been removed with no
  compatibility shim; update direct imports using this migration map:

  • modelopt.onnx.quantization.graph_indexing: expand_node_names_from_patterns,
    find_mha_partitions, get_fusible_backbone,
    get_tensor_consumer_node_indices, get_tensor_consumer_nodes,
    get_tensor_from_name, get_tensor_producer_nodes, has_const_input,
    has_path_type, is_const_input, and match_fp8_mha_pattern.
  • modelopt.onnx.quantization.graph_selection: find_nodes_from_convs_to_exclude,
    find_nodes_from_matmul_to_exclude, find_nodes_from_mha_to_exclude,
    find_nodes_to_exclude, get_extended_model_outputs, get_input_shapes,
    and validate_op_types_spelling.
  • modelopt.onnx.quantization.graph_rewrites: cast_custom_ops,
    convert_fp16_io, insert_fp8_mha_casts, insert_matmul_casts,
    remove_output_initializers, and remove_redundant_cast_nodes.
  • modelopt.onnx.quantization.qdq_graph: build_non_residual_input_map,
    classify_partially_quantized_weighted_ops, classify_partition_nodes,
    filter_quantizable_kgen_heads, find_conv_to_layernorm_nodes,
    get_concat_eliminated_tensors, get_layer_info,
    get_layer_precision_mapping, get_resize_scales, print_stat,
    remove_partial_input_qdq, should_quantize_to_8bit, and
    validate_8bit_layers.

• Layerwise calibration now uses prior-layer QDQ activations by default
  (layerwise.get_qdq_activations_from_prev_layer=True). Set it to False to
  preserve full-precision activations for subsequent layers (the default behavior for
  max calibration without layerwise calibration).

• get_te_hybrid_stack_spec was removed from modelopt.torch.nas.plugins.megatron; it had no use outside tests. Use modelopt.torch.utils.plugins.megatron_layer_specs.te_hybrid_stack_spec_sequential_mlp for the SequentialMLP layout, or megatron.core.models.hybrid.hybrid_layer_specs.hybrid_stack_spec for grouped GEMM.

• Unified HuggingFace export now fails with NotImplementedError when it meets an MoE block whose expert projection names it does not know, instead of assuming Mixtral's w1/w2/w3. If you hit this, register a ModelSpec for the model under modelopt/torch/models/. Every MoE architecture ModelOpt exported correctly before this change is registered, so no supported model regresses.

• --recipe (and modelopt.recipe.load_recipe) now resolve a recipe path filesystem-first: a recipe of the same relative path in the current working directory takes precedence over the shipped built-in of that name, matching how recipe $import paths already resolve. Previously the built-in won.

Deprecations

• The Diffusers ONNX export and TensorRT engine deployment workflow is deprecated and will be removed no earlier than 0.49.0, after the one-release migration period. Export unified Hugging Face checkpoints with --hf-ckpt-dir and follow the diffusion deployment guide to select a compatible backend.
• The examples/vllm_serve QUANT_CFG / KV_QUANT_CFG environment variables and their
  --modelopt-quant-cfg / --modelopt-kv-quant-cfg flags will be deprecated in a future release.
  For new runs, use a PTQ recipe via RECIPE_PATH or --modelopt-recipe-path; unset the old
  variables when switching because the launcher rejects mixing them with a recipe.
• KV-domain searches in examples/hf_ptq now use --kv_auto_quantize_checkpoint;
  --auto_quantize_checkpoint remains a deprecated fallback for a KV-primary recipe for one
  release.
• Rename the architecture-specific recipe tier from modelopt_recipes/huggingface/ to modelopt_recipes/model_type/ to clarify that it holds recipes shared across every checkpoint of a Hugging Face model_type. Saved --recipe huggingface/<model_type>/... paths still resolve via a backward-compatibility alias but now emit a FutureWarning, so update them to model_type/<model_type>/... as the huggingface/ prefix is deprecated.
• The single-format quantization CLI flags are deprecated in favour of --recipe and will be removed in a future release; passing one now emits a FutureWarning. examples/hf_ptq: --qformat and --kv_cache_qformat. examples/megatron_bridge/quantize.py: --quant_cfg, --kv_cache_quant and --weight_only. examples/torch_onnx/torch_quant_to_onnx.py: --qformat. A recipe carries the quantization config, the calibration algorithm and the KV-cache setting in one file, so they cannot drift apart the way separate flags can -- and --recipe already took precedence over all six, silently on hf_ptq and with a warning on megatron_bridge -- with one gap the recipe closes rather than inherits: a weight AutoQuantize recipe that omits kv_cache still falls back to --kv_cache_qformat, so set kv_cache in the recipe when migrating. Use a recipe from modelopt_recipes/general/ptq/, an architecture-specific one under modelopt_recipes/model_type/<model_type>/, or a checkpoint-specific one under modelopt_recipes/models/. The warning fires only when a flag is passed explicitly: --qformat defaults to fp8 and --kv_cache_qformat to fp8_cast, so warning on the defaults would fire on every run, including runs that correctly use --recipe. examples/speculative_decoding/scripts/quantize_drafter.py keeps --qformat undeprecated: it has no --recipe alternative yet.
• The TensorRT-LLM checkpoint export format is deprecated and will be removed in 0.49.0: export_tensorrt_llm_checkpoint and torch_to_tensorrt_llm_checkpoint now emit a DeprecationWarning on use. Use export_hf_checkpoint, which exports a unified Hugging Face checkpoint deployable on TensorRT-LLM, vLLM and SGLang. Its implementation moved to modelopt.torch.export.trtllm, so import those two functions from there and the ModelConfig dataclasses from modelopt.torch.export.trtllm.model_config; both functions remain importable from modelopt.torch.export for this release only.
• modelopt.torch.export.get_model_type and modelopt.torch.export.model_utils.MODEL_NAME_TO_TYPE return TensorRT-LLM model names and are deprecated for removal in 0.49.0 (they now emit a DeprecationWarning); use the Hugging Face model type, model.config.model_type, instead. export_tensorrt_llm_checkpoint now detects decoder_type when it is omitted, and examples/hf_ptq uses Hugging Face model types throughout.
• The evaluation agent skill no longer supports GDPVal: its task recipe, example config and Apptainer SIF helper are removed, and the shared NeMo Gym machinery it carried now lives at references/gym.md with the launcher wrapper renamed scripts/nel-gym.sh. GDPVal is an AA-suite member, so an "AA" request now generates the aa/ tasks only -- report per-task scores rather than an aggregate compared against a published AA Index.
• DBRX model support is removed: ModelOpt no longer quantizes or exports DBRX (DbrxForCausalLM) checkpoints. Use ModelOpt 0.47 to quantize DBRX, or write your own plugin as described in the customized-model quantization guide.
• Deprecate metadata.recipe_type in recipe YAML, for both single-file recipes and a directory recipe's metadata.yml. A recipe now says what kind it is with a # modelopt-schema: comment naming its schema class, or by delegating to a recipe that does (single-file recipes only -- a directory recipe has no body to delegate through); recipe_type is still read and still honoured, so a recipe outside this repo keeps working unchanged, but new recipes should leave it out and every recipe shipped here has been converted. Where both are present they must agree, and so must a recipe and the recipe it delegates to -- a disagreement is an error rather than a silent preference.

Bug Fixes

• Fix corrupted ONNX external tensor data when exporting models on Windows by upgrading ONNX. Regenerate affected models from clean weights.
• Fix import modelopt.torch failing with ModuleNotFoundError for requests or huggingface_hub when nvidia-modelopt is installed without the [hf] extra.
• Harden the distributed object broadcast/allgather helpers and the Megatron expert-parallel export gather so a compromised peer rank can no longer execute code through a crafted pickle payload: payloads load with weights_only=True and the export gathers raw bytes instead of pickled objects.
• Fix LSQ clamping small, representable FP8 block scales too aggressively, which could zero quantized values and suppress scale gradients.
• Fix ONNX AutoTune benchmark initialization with TensorRT 11 while preserving strongly typed networks on TensorRT 10.
• Keep vision-block Linears in high precision in the qwen3_vl and qwen3_5 FP8 vision recipes so exported checkpoints avoid a vLLM vision-attention shape failure. The recipes still quantize primary and deepstack merger Linears; re-quantize affected checkpoints before deployment.
• Fix unified HF export writing a weight name the source checkpoint does not use when a model declares a backwards-compatibility load rename as a reversible WeightRenaming. Reversing such a rule rewrote the canonical name to a legacy spelling, and a deployment runtime keyed on the canonical name then failed to load the quantized checkpoint. Each exported name is now resolved against the names the source checkpoint actually carries, for both the exported tensors and the quantization config's module references.
• Fix exports using the built-in Transformers NemotronH implementation retaining incompatible legacy remote-code config and model mappings. Re-export affected checkpoints before deployment.
• Fix export_hf_checkpoint dropping Diffusers pipeline settings, such as Wan 2.2 boundary_ratio, from model_index.json when components is given. Re-export affected checkpoints to a new directory, since component exports do not overwrite an existing model_index.json.
• Fix INT8 real quantization (mtq.compress) of FP16 weights zeroing or clipping channels with a small amax: their scales underflowed in FP16. Scales of newly compressed INT8 weights are now stored in FP32.
• Fix Hugging Face exports classifying registered Qwen VLM parameters as unplaced source weights, which overwrote quantized tensors with original weights and incorrectly excluded their modules from quantization. Re-export affected checkpoints.
• Right-pad labeled calibration samples to avoid fully masked causal rows that can produce non-finite cuDNN attention gradients during AutoQuantize. Generation previews remove padding before decoding.
• Fix ONNX quantization reusing existing tensor names for scale and zero-point parameters while preserving INT8 calibration-cache lookup. The boolean --target_dla / target_dla=True now preserves scalar multiplication/division operands and padding fill values as constants; use Q/DQ Translator for compatible TensorRT 10.7-and-earlier DLA deployments, or direct Q/DQ with strongly typed DLA enabled in a compatible TensorRT 11.4+ build.
• Fix calibration silently dropping samples when an out-of-memory error forces the batch to be
  split and a smaller working batch size is already known: the slices stopped tiling the batch,
  so the rows between the old and the new width never reached the model and the collected
  statistics covered fewer samples than requested. Splitting a batch that holds a None entry
  no longer raises TypeError either, and a batch holding a value under
  allowed_non_tensor_keys now reports that its rows cannot be split instead.
• Fix Megatron unified HF export of MoE models with grouped-GEMM experts when only the experts are quantized (e.g. nvfp4_experts_only-* recipes): hf_quant_config.json and the quantization_config in config.json were not written, so the quantized experts were served as unquantized weights. Re-export such checkpoints.
• Fix DFlash conversion on NoPE targets whose config leaves rope_theta unset.
• Fix offline DFlash training failing to reconstruct the target logits when the captured hidden states are stored in a different dtype than the target's weights.
• Fix hf_ptq overwriting a model's existing pad_token with eos_token when the model already has a valid padding token. The exported tokenizer now preserves the source model's padding configuration.
• Fix Hugging Face exports dropping off-index safetensors such as GLM-4.7's mtp.safetensors.
• Fix Megatron-Core checkpoint saving for quantized grouped MoE experts when tensor and expert parallelism are both enabled.
• Fix FP8 fake quantization treating a positive amax at or below 2**-24 as zero, which silently zeroed such tensors and channels and every block of a static NVFP4 weight (NVFP4StaticQuantizer, including LSQ) with a global amax below ~3.6e-7. Fake quantization now matches the exported checkpoint.
• Fix megatron_preprocess_data with workers > 1 segfaulting when run in a process that has already initialized CUDA or other native threads (for example in-process after GPU work): the tokenization worker pool now uses the forkserver start method instead of fork.
• Fix unified HuggingFace export of RADIO-based VLMs retaining post-conversion vision and
  projector names instead of restoring the hub layout; deployment loaders could skip those weights.
  Streaming, offload, FSDP2, and layerwise export now reject reverse conversions that regroup
  tensors; use resident export for those models.
• Fix that same reverse rename double-prefixing tensors copied from the source checkpoint rather
  than loaded by the model, when a conversion mapping renames a module to a suffix of its original
  name. Those tensors already carry the hub name, so C-RADIO towers exported
  vision_model.radio_model.radio_model.input_conditioner.norm_mean. Re-export affected
  checkpoints.
• Fix shared ONNX export metadata and Diffusers attention policy: every NVFP4QuantExporter post-process now upgrades the default-domain opset to at least 23, all FP8 custom-op exports re-run ONNX shape/type inference after setting output metadata, and quantized SDPA derives FP8 MHA enablement from the live Q/K/V quantizers instead of honoring a caller-set _disable_fp8_mha attribute.
• Fix ONNX FP16 conversion failing to preserve public output types when type inference changes a graph output declaration before output casts are inserted.
• Fix examples/hf_ptq/hf_ptq.py discarding a completed PTQ run (no checkpoint exported) when the optional post-quantization sanity-check generate() call raised, for example because device_map="auto" placed part of the model on CPU. That failure is now caught and only skips the sanity check; export proceeds regardless.
• Fix examples/megatron_bridge/export_quantized_megatron_to_hf.py storing the MoE router at Megatron's moe_router_dtype, which is a routing compute dtype, not a storage one. The router now exports at the export dtype like every other unquantized weight, matching what hf_ptq.py and the released NVFP4 checkpoints contain; pass moe_router_dtype to export_mcore_gpt_to_hf explicitly if you want the old fp32 storage.
• Fix unified Megatron export writing a second, unreferenced copy of the vocab embedding when a model with MTP layers is exported with pipeline parallelism. The duplicate was never loaded but inflated the checkpoint by the size of the embedding (about 1 GB for Qwen3.6-35B-A3B); re-export to reclaim the space.
• Fail fast on non-finite AutoQuantize output gradients with an actionable error before accumulating sensitivity scores, without changing attention backend settings.
• Fix ONNX INT8 entropy calibration failing or producing invalid quantization parameters for FP16 activations.
• Fix HuggingFace checkpoint export failing with activation scaling factor 0.0 not positive when a dynamic-block quantizer (such as an NVFP4 input quantizer) ends calibration with an amax of zero because the calibration data never activated that layer or expert. Such a quantizer now exports a positive fallback scale and warns instead of crashing, matching what static quantizers already did; if you see the warning, check whether the layer is expected to be inactive and consider a larger calibration size.
• Fix --use_fsdp2 HuggingFace checkpoint export gathering the whole model onto rank 0, which made export the dominant phase of a PTQ run and could exhaust host memory on large models. The model is now split into per-decoder-layer units dealt round-robin across ranks; each rank gathers every unit but keeps, packs, and writes only the ones it owns, so a rank buffers roughly model / world_size instead of the whole checkpoint, and rank 0 writes the combined index. Export configurations that cannot be split this way now raise instead of producing a mismatched checkpoint: FSDP2 combined with another DTensor parallelism (for example FSDP2 + tensor parallel on a 2-D mesh; HSDP is supported), models whose decoder layers cannot be discovered, a decoder layer object reused across layers, and a module that holds the decoder layers while owning parameters of its own.
• Speed up mtq.quantize on FSDP2-sharded fused-MoE models. Promoting static-block weight quantizers gathered each expert's slice of the fused weight across ranks even though only quantizer state is read, adding a collective per expert to calibration.
• Add FP8 and INT8 recipes that quantize timm ResNet shortcut inputs immediately before residual adds. The torch ONNX example now accepts PTQ and AutoQuantize recipes through --recipe and uses --qformat when no recipe is provided. ResNet supports only FP8 and INT8 because TensorRT has limited convolution kernel support; AutoQuantize and other quantization formats are no longer supported for ResNet.
• Fix dflash_fp32_master_weights being inert outside HF Trainer and lossy across a resume, by moving the extra precision from the model into the optimizer. The draft now always matches the frozen base's dtype, and MasterWeightAdamW holds an fp32 master copy plus fp32 Adam moments, applies the update in fp32 and writes the result back at the parameter's dtype -- the standard mixed-precision placement. Nothing has to reconcile dtypes at forward time any more, so evaluation, pseudo_speculative_generate (which AR validation calls outside the Trainer's autocast) and a plain mtsp.convert() followed by a forward all behave the same as training; gradients stay in the base dtype, so the DDP gradient all-reduce is no longer doubled; and the exported drafter is unchanged. The training loop has to build the optimizer -- examples/speculative_decoding does -- and VerifyMasterWeightsCallback raises after the first step if it did not, rather than letting the flag be silently absent for a whole job. dflash_fp32_master_weights now defaults to True: with the draft no longer promoted, turning it on changes optimizer memory and optimizer arithmetic but nothing about the model, so every DFlash-family recipe gets it. Set it to False to reclaim the memory.
• Fix a DDP hang in DFlash training at scale where a rank whose batch contained no valid anchor skipped the draft forward, leaving its rotary buffer list shorter than other ranks' and causing broadcast_buffers to hang. The buffer is now created during modify() before training begins.
• Fix megatron_generate dropping the VLM vision inputs (pixel_values / image_grid_thw / image_sizes) after the first generated token when KV-cache decoding is off, including the automatic fallback under sequence parallelism, which made generation silently ignore the image. No other ModelOpt feature is affected.
• Fix two issues in the vLLM offline hidden-state dump (examples/speculative_decoding/collect_hidden_states/compute_hidden_states_vllm.py) that only surface on large runs. Resume: the filter that skips conversations whose .pt already exists now runs with load_from_cache_file=False. It depends on on-disk state, which is not part of the fingerprint datasets computes from the function and the dataset, so with a persistent HF cache reused across a resumed or requeued run the cached "keep everything" result from an earlier run was replayed and the dump re-generated and overwrote conversations it had already finished (observed: tens of thousands of .pt rewritten while the output count stayed flat). Staging: generation is now chunked (--save-chunk-size, default 256), so each chunk is saved and its staged hidden states freed before the next chunk is generated. Previously the whole dataset was generated before anything was saved, which kept every conversation staged in the connector's shared_storage_path (/dev/shm, i.e. RAM, by default) at once and exhausted it partway through large dumps. Chunking also makes the dump incrementally durable, so an interrupted run keeps its finished conversations and resumes from them. The save path now also frees each conversation's staged hidden states in a finally, so a conversation skipped mid-loop (e.g. a short loss_mask) can no longer leak its staging file, and conversation ids are validated as plain filenames before being used to build output paths.
• Fix YAML config I/O decoding with the locale codepage instead of UTF-8, which made a config containing any non-ASCII byte fail to load on a machine whose locale is not UTF-8 (notably Windows, where the default is cp1252). modelopt/recipe/loader.py, the two ONNX autotune state files, the two transformers config readers, the distill config and the puzzletron profile now pass encoding="utf-8" explicitly. Only the YAML config paths are covered: these are the files most likely to carry non-ASCII text in comments, model names or paths, and the only ones read inside a user's process.
• Hybrid (e.g. Nemotron-H) checkpoints saved by the examples/megatron_bridge scripts now record their layer spec in run_config.yaml in a form that reloads, so they can be converted to HuggingFace; a checkpoint saved by an earlier release still needs its model.hybrid_stack_spec block replaced by hand.
• Fix ONNX export of quantized models failing with Unsupported op_type for real weight quantization: Transpose when a weight reaches its consumer transposed, as in MaxViT's attention blocks. The same export now also quantizes each weight along the axis its Q/DQ pair declares, correcting silently wrong dq_only output when that axis differed from the consumer's default.
• Fix the evaluation agent skill's LiveCodeBench recipe scoring correct solutions that use binary standard I/O (sys.stdin.buffer / sys.stdout.buffer) as failures: the recipe now pins a fixed LiveCodeBench judge in the NeMo-Skills container. LiveCodeBench scores change as a result, so re-run earlier baselines before comparing.
• Fix GPTQ with Megatron data parallelism: each data-parallel replica of a module now builds its Hessian from the whole calibration set instead of only its own shard, so all replicas reach the same quantized weights.
• Fix examples/llm_sparsity/weight_sparsity/finetune.py saving a sharded state dict under FSDP2, the default FSDP version in Transformers 5, which made export_hf_ckpt.py fail to restore it. Re-run finetuning to produce a loadable finetuned_modelopt_state.pth.