NVIDIA/Model-Optimizer 0.45.0

ModelOpt 0.45.0 Release

on GitHub

New Features

Quantization

• Add NVFP4 W4A16 weight-only quantization (w4a16_nvfp4): FP4 weights with group_size=16, BF16 activations, no calibration forward pass required. Use mtq.W4A16_NVFP4_CFG or --qformat w4a16_nvfp4 in hf_ptq.py. vLLM deployment support is in progress.
• Add --cast_mxfp4_to_nvfp4 flag to examples/llm_ptq/hf_ptq.py for closed-form, bit-exact MXFP4 → NVFP4 weight conversion. Supports the GPT-OSS family (openai/gpt-oss-20b, openai/gpt-oss-120b). See examples/llm_ptq/README.md for usage.
• Add --cast_mxfp4_to_nvfp4 flag to examples/deepseek/deepseek_v4/quantize_to_nvfp4.py for closed-form, bit-exact MXFP4 → NVFP4 conversion of DeepSeek V4 routed-expert weights (mirrors the GPT-OSS cast; w1/w3 share one per-tensor scale_2 for the fused GEMM1). Activation input_scale still comes from --amax_path calibration.
• DeepSeek PTQ (examples/deepseek/ptq.py) now defaults to native top-k calibration with post-hoc per-layer peer-max sync of expert input_quantizer.amax; the all-experts path is preserved behind --calib_all_experts.
• Add active-MoE cost accounting for mtq.auto_quantize effective-bits search. Set constraints={"effective_bits": ..., "cost_model": "active_moe", "cost": {"active_moe_expert_ratio": ...}} to weight routed MoE expert costs by active experts per token while keeping shared experts fully counted. The hf_ptq.py AutoQuant path exposes this via --auto_quantize_cost_model active_moe and --auto_quantize_active_moe_expert_ratio.
• Add quantized nn.Embedding support. nn.Embedding is now registered in QuantModuleRegistry and exposes weight_quantizer (embedding table), output_quantizer (lookup activations), and a permanently disabled input_quantizer placeholder — embedding inputs are integer indices and cannot be fake-quantized, so direct enable*() calls raise. export_hf_checkpoint packs quantized embedding weights alongside Linear layers. Embedding quantizers are opt-in (parent_class: nn.Embedding disabled by default).
• Add composable $import system for recipe YAML configs, enabling reusable config snippets referenced via {$import: name} markers. All built-in PTQ recipes converted to use imports with shared snippets under modelopt_recipes/configs/ (numeric formats, quant_cfg building blocks, presets). See composable-imports docs.
• The PTQ example scripts examples/llm_ptq/hf_ptq.py, examples/llm_ptq/multinode_ptq.py and examples/megatron_bridge/quantize.py now derive their --qformat / --kv_cache_qformat (--quant_cfg / --kv_cache_quant for Megatron-Bridge) CLI vocabularies by discovering the YAML presets under modelopt_recipes/configs/ptq/presets/{model,kv}/ rather than carrying hardcoded QUANT_CFG_CHOICES / KV_QUANT_CFG_CHOICES tables. The discovery helper, alias table and ready-built QUANT_CFG_CHOICES / KV_QUANT_CFG_CHOICES mappings now live in modelopt.recipe.presets and are shared by all three scripts. Presets are loaded eagerly into a plain dict at import. Adding a new preset YAML makes it available on the CLI of all three with no script change — note this means each script now accepts every preset under those directories, not just a previously curated subset. All previously-supported short names (int8_sq, nvfp4_awq, fp8_pb_wo, nvfp4_mse, w4a8_awq, nvfp4_local_hessian, fp8_pc_pt, int8_wo) keep working via a small deprecation alias table; new formats should be exposed as preset YAMLs (or, longer term, as full --recipe recipes).
• Add configs/ptq/presets/kv/fp8_cast.yaml and configs/ptq/presets/kv/nvfp4_cast.yaml, promoting fp8_cast / nvfp4_cast to first-class KV presets composed from the existing kv_fp8_cast / kv_nvfp4_cast unit fragments. The previous runtime use_constant_amax post-edit in hf_ptq.py is removed; use_constant_amax: true now lives in the YAML and is therefore authoritative. Custom (out-of-tree) recipes that target a cast KV format must set use_constant_amax: true themselves on the [kv]_bmm_quantizer config — in-tree recipes already do via the kv_*_cast units.
• Add FP8 KV-cache cast variants for the partial-NVFP4 and weight-only general PTQ recipes: general/ptq/nvfp4_mlp_only-kv_fp8_cast, general/ptq/nvfp4_experts_only-kv_fp8_cast, general/ptq/nvfp4_omlp_only-kv_fp8_cast, and general/ptq/nvfp4_weight_only-kv_fp8_cast. These compose the same model-quant configs as their -kv_fp8 siblings with the kv_fp8_cast unit (constant-amax FP8 KV cache, no KV calibration forward pass).
• Add Nemotron-3-Super-120B-A12B PTQ recipes modelopt_recipes/models/Nemotron-3-Super-120B-A12B/super-nvfp4.yaml (MSE-mixed) and super-nvfp4-max-calib.yaml (max-calib mixed): NVFP4 W4A4 routed experts + FP8 per-tensor shared experts / Mamba in/out_proj + FP8 KV cache.
• Group layerwise calibration options under a nested LayerwiseConfig and add two knobs: get_qdq_activations_from_prev_layer (correct GPTQ-Hessian vs max-calib activation semantics — defaults to True for GPTQ, False for max/mse/local_hessian) and save_every (gate per-window next_inputs.pt activation-cache writes). Legacy bool layerwise and flat layerwise_checkpoint_dir keys still work; the bool form emits a DeprecationWarning.
• Add examples/alpamayo showing FP8, NVFP4, and AutoQuantize (mixed-precision) quantization of the Alpamayo (formerly Alpamayo-R1) ~10B vision-language-action model, with a joint VLM + diffusion calibration loop and both fake-quant and --real-quant packed-checkpoint export. See examples/alpamayo/README.md for details.
• Refactor llm_qat example with unified YAML-based configuration and flexible dataset blending. ModelOptArgParser adds --config YAML support with CLI overrides and auto-generates ARGUMENTS.md from dataclass definitions. Dataset blending (configs/dataset/blend.yaml) supports HuggingFace datasets, local JSON/JSONL/Parquet files, and weighted multi-source blends. The legacy FSDP1 accelerate config is removed; llm_qat now documents FSDP2, DeepSpeed, and DDP backends.

Megatron Framework (M-LM / M-Bridge)

• Add quantization examples for the Megatron-Bridge framework (examples/megatron_bridge/): post-training quantization (quantize.py calibrates an HF model via --quant_cfg alias / full config name or a --recipe YAML, with optional KV-cache quant, weight-only, compression, and MoE expert-ratio calibration, and saves a Megatron checkpoint with tensor / pipeline / expert parallelism), export to a deployable HuggingFace (unified) checkpoint for TensorRT-LLM / vLLM / SGLang (export.py), and Quantization Aware Distillation (extend existing distill.py). See examples/megatron_bridge/README.md for details.
• Add Megatron Core export/import mapping for Qwen3-VL (Qwen3VLForConditionalGeneration) vision-language models. The mapping handles the model.language_model. weight prefix used by Qwen3-VL.
• Add shared Megatron-Core calibration forward loop: modelopt.torch.utils.plugins.megatron_calibration.get_megatron_calibration_forward_loop produces the forward_loop callable expected by mtq.quantize / mtp.prune. Replaces the bespoke calibration loops in Megatron-LM and Megatron-Bridge for quantization and pruning with a single canonical implementation.
• Support Megatron-Core checkpoint restore and export for MSE NVFP4StaticQuantizer.
• Add mixed-precision FP8 + NVFP4 export for Megatron-Core: per-layer quant_algo recorded under quantized_layers in hf_quant_config.json, PP-aware kv_cache_dtype gather, fused-QKV exclude split into per-HF-name q/k/v_proj entries.
• Add support for active_params (for MoE models) and memory_mb constraints in Minitron pruning on top of existing params constraint. You can also provide multiple constraints. See examples/pruning/README.md for more details. The underlying utility functions mcore_param_count, mcore_memory_footprint_mb, and print_mcore_model_stats in modelopt.torch.nas.plugins.megatron_model_stats are also available for standalone use to compute parameter counts and memory footprints (weights + KV-cache + Mamba state) for any Megatron-Core model.
• Add Minitron pruning support for Megatron-Bridge Gemma3 models.
• Add end-to-end optimization tutorial for Minitron pruning + two-phase distillation (80B @ 8K + 20B @ 32K long-context = 100B tokens) + FP8 PTQ + vLLM deployment for Nemotron-3-Nano-30B-A3B-BF16 (MoE + Mamba-Transformer hybrid) → Pruned 22B/A3.0B active params, along with data blend preparation steps (with tool-calling data) and detailed pruning / data-blend / long-context ablations. See examples/megatron_bridge/tutorials/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16/README.md for details.

Datasets & Calibration

• Add DATASET_COMBOS to modelopt.torch.utils.dataset_utils — single --dataset tokens that fan out to multiple registered datasets; per-entry num_samples is split evenly across the members. Initial combos: cnn_nemotron_v2_mix (cnn_dailymail + nemotron-post-training-dataset-v2, used by hf_ptq.py when no --dataset is provided) and nemotron-post-training-v3 (the seven nvidia/Nemotron-* SFT datasets added in #1498, mirroring the nemotron-post-training-v3 collection). Combo names are listed by get_supported_datasets() and surfaced in --dataset help. get_dataset_dataloader rejects inputs that mix a combo with one of its member datasets (e.g. cnn_dailymail,cnn_nemotron_v2_mix) to avoid double-sampling, and get_dataset_samples rejects combo names so callers route through the dataloader. hf_ptq.py default --calib_size is bumped from 512 to 1024 so the total calibration sample count under the new default combo matches the previous two-dataset fallback.
• The nemotron-sft-agentic-v2 registered dataset (added in #1498) now uses only the search split. The previously configured interactive_agent and tool_calling splits contain content-level defects (heterogeneous schema and a malformed JSON row, respectively) that cause pyarrow's streaming JSON reader to fail deterministically.
• Add pack=True mode to get_dataset_dataloader (Megatron-LM pretraining-style global-stream document packing): all raw samples concatenated EOS-separated into one token stream, sliced into uniform max_sample_length rows. Used by the shared megatron calibration loop.

Misc

• Add offline DFlash speculative decoding training. Train the draft module from pre-computed base-model hidden states dumped by examples/speculative_decoding/collect_hidden_states/compute_hidden_states_hf.py; base-model transformer layers are deleted after conversion to save memory. Controlled by the auto-derived dflash_offline flag on DFlashConfig (derived from data_args.offline_data_path). The dump scripts now share collect_hidden_states/common.py for aux-layer selection (--aux-layers eagle|dflash|<list>) and optional assistant-token loss_mask for answer-only-loss training.
• Add mtsa.config.SKIP_SOFTMAX_TRITON_CALIB for skip-softmax attention-sparsity calibration through the fused Triton attention_calibrate kernel (HF modelopt_triton backend), measuring multi-threshold tile-skip statistics the way the Triton inference kernel actually skips tiles for both prefill and decode. Exposed as --sparse_attn_cfg skip_softmax_triton_calib in examples/llm_sparsity/attention_sparsity/hf_sa.py (with a new --calib_data_dir flag for RULER calibration data).
• Add DMD2 distillation for few-step diffusion models in examples/diffusers/fastgen/: distill Qwen-Image into a 4/8-step student via Distribution Matching Distillation. See examples/diffusers/fastgen/README.md for details.
• Make .agents/skills/ the canonical location for agent skills; agent-specific directories (.claude/skills/, etc.) are now relative symlinks into .agents/, so one skill suite serves multiple coding agents (Claude Code, Codex). See .agents/README.md.
• Extend Claude Code agent skills for PTQ, deployment, evaluation, monitoring, and baseline-vs-quantized result comparison. Adds evaluation task references for additional benchmarks, stronger PTQ checkpoint validation gates, and session-scoped workspace/job tracking.
• Add SLURM Quality of Service (QoS) support to the ModelOpt launcher. Users can set QoS via slurm_config.qos or SLURM_QOS and the value is forwarded to nemo_run.SlurmExecutor.

Backward Breaking Changes

• KDTrainer / QADTrainer evaluation now reports KD as the primary eval_loss and CE as eval_ce_loss; the previous secondary eval_kd_loss metric is removed.
• Reorganize custom CUDA / Triton kernels under modelopt.torch.kernels into common/attention, quantization/{conv,gemm}, and sparsity/attention. High-level APIs (mtq.quantize, mtsa.sparsify, etc.) are unchanged, but any code importing directly from the kernel subpackages must be updated: there is no backwards-compatibility shim; the old import paths will raise ImportError / ModuleNotFoundError. Migration table:
  • from modelopt.torch.kernels import IS_AVAILABLE, attention, attention_calibrate, register_triton_attention → from modelopt.torch.kernels.common.attention import ...
  • from modelopt.torch.kernels.triton_fa import ... → from modelopt.torch.kernels.common.attention.triton_fa import ...
  • from modelopt.torch.kernels.hf_triton_attention import ... → from modelopt.torch.kernels.common.attention.hf_triton_attention import ...
  • from modelopt.torch.quantization.triton import ... → from modelopt.torch.kernels.quantization.gemm import ...
  • from modelopt.torch.quantization.src.conv.implicit_gemm_cuda import ... → from modelopt.torch.kernels.quantization.conv.implicit_gemm_cuda import ...
  • from modelopt.torch.sparsity.attention_sparsity.kernels import ... → from modelopt.torch.kernels.sparsity.attention import ...
• Deprecated GradNAS pruning algorithm as it is not actively maintained and supports very limited and old models. It is recommended to use Minitron or Puzzletron pruning for LLM models. Also deprecates related examples/chained_optimizations directory.
• Model-specific PTQ quant_cfg adjustments previously hardcoded in examples/llm_ptq/ (build_quant_cfg / mono_quantize) for gemma, mpt, phi4mm, and Nemotron VL are now opt-in model-specific recipes under modelopt_recipes/huggingface/<model_type>/ptq/. Any adjustment specific to a model type or instance must live in that model's recipe; the bare --qformat path produces only the generic numerics. Pass --recipe huggingface/<model_type>/ptq/<recipe> to apply the model's recipe. Covers gemma/mpt w4a8_awq (awq_lite alpha_step=1), gemma int8_sq (SmoothQuant alpha=0.5), phi4mm speech/audio/image/vision exclusions, and Nemotron VL vision-branch exclusions. All shipped recipes also enable FP8 KV-cache cast. MTP dynamic layer exclusion and is_nemotron_vl detection remain in Python.
• The Step3.5-Flash recipe moved from modelopt_recipes/models/Step3.5-Flash/nvfp4-mlp-only.yaml (0.44) to modelopt_recipes/huggingface/step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml to match the huggingface/<model_type>/ptq/ layout convention. Update --recipe paths accordingly.

Deprecations

• Deprecate the public QuantizationArgumentsWithConfig name in modelopt.torch.quantization.plugins.transformers_trainer; it now aliases QuantizationArguments and will be removed in a future release.
• Deprecate the examples/diffusers/eval image-quality evaluation example (ImageReward / CLIP-IQA / CLIP metrics); it is no longer maintained and will be removed in the next release.
• Deprecate examples/llm_autodeploy. The AutoQuant + TensorRT-LLM AutoDeploy workflow it demonstrates will be removed in a future release; use TensorRT-LLM's AutoDeploy directly together with ModelOpt PTQ in examples/llm_ptq.

Bug Fixes

• Fix ShapeInferenceError during ONNX INT8 + FP16 quantization (--high_precision_dtype fp16) of weakly-typed models (e.g. TensorFlow exports) that carry stale rank-0 graph.output shapes or ops such as TopK that ONNX's static shape inference cannot resolve. clear_stale_value_info now reconciles stale output shapes via symbolic shape inference (keeping every output's shape field populated), and AutoCast runs ONNX shape inference in strict mode and falls back to schema-based standalone type inference when it fails, so unresolved ops no longer leave tensors untyped.
• Always list unquantized MoE routers/gates in the exported exclude_modules (NVBug 5718750). get_quant_config only recorded modules that carry a quantizer, but on transformers>=5.0 MoE routers are no longer nn.Linear (e.g. TopKRouter) and never receive one, so the BF16 router weight was written to the checkpoint yet omitted from exclude_modules. vLLM / SGLang then treated it as quantized and failed to load (e.g. Qwen3-30B-A3B NVFP4: AssertionError: Tried to load weights of size [128, 2048] to a parameter of size [128, 1024]). Routers are now detected structurally (an MoE block with an experts container plus a weight-bearing gate / router / shared_expert_gate submodule) and recorded as unquantized regardless of quantizer attachment.
• In Megatron-Core only do EP amax sync for routed expert weights if sync_expert_weight_amax=True. Previously EP amax sync would sync routed expert weights across EP ranks even when sync_expert_weight_amax was False.
• Fix Megatron-Core HF importer to load fused TELayerNormColumnParallelLinear.layer_norm_weight from HF for GPT-family models (Qwen3 etc.) under --export-default-te-spec. Importer now prefers per-context keys fused_input_layernorm / fused_pre_mlp_layernorm (fallback fused_norm for Nemotron-H backward compatibility); mcore_qwen.py provides the new rules. Without this fix, post-prune MMLU sat at chance.
• Fix ONNX AutoCast keep_io_types=True sanity-check failure (Unexpected type in I/O tensor ...) when a network input/output is an empty tensor (a dimension of size 0). Such tensors were "fake-cast" (retyped in place) to the low precision type; because the value-info map aliases the graph.input/graph.output ValueInfoProto, this silently changed the model's I/O type. AutoCast now inserts a real Cast for protected I/O tensors instead.
• Fix INT8 entropy calibration of fp16 ONNX models raising ValueError: Too many bins for data range on numpy >= 2.0. _collect_value in modelopt.onnx.quantization.ort_patching now casts the histogram range endpoints to Python float so bin edges are computed in float64, instead of inheriting the fp16 dtype of an activation tensor with a small range (which collapsed the 128-bin linspace under NEP-50 promotion).
• Fix the GPT-OSS MXFP4 → NVFP4 PTQ path in examples/llm_ptq/hf_ptq.py (used with --cast_mxfp4_to_nvfp4). get_model now loads native MXFP4 checkpoints (openai/gpt-oss-*) dequantized to BF16 GptOssExperts via Mxfp4Config(dequantize=True) on a sequential device map. This fixes a CUDA illegal-memory access during the multi-GPU dequant load and the NotImplementedError for experts type Mxfp4GptOssExperts during unified HF export (the packed-kernel experts wrapper, used when the optional kernels package is installed, is unsupported by export); kernels is no longer required. The --cast_mxfp4_to_nvfp4 step now also resolves a HF Hub ID --pyt_ckpt_path to its local snapshot directory instead of failing with FileNotFoundError.
• Fix _QuantGptOssExperts / _QuantLlama4TextExperts static-block NVFP4 weight calibration raising ValueError: Input shape has changed during the calibration forward. These experts quantize their weights transposed (_transposed_quantize); iter_weights_for_calibration now yields the same transposed view so weight-only calibration and the forward agree on the block-quant shape (and the export _amax orientation).
• Fix unified HF checkpoint export for Llama4 MoE models. The uncalibrated-experts input-quantizer amax fallback in _export_transformers_checkpoint special-cased only QuantGptOssExperts; QuantLlama4TextExperts uses the same fused gate_up_proj / down_proj layout and is now handled by the same branch, fixing the export failure.
• Fix NotImplementedError: "max_all_cuda" not implemented for 'Float8_e4m3fn' during quantization calibration of models with natively FP8 (float8_e4m3fn / float8_e5m2) weights, such as DeepSeek-V3. FP8 dtypes implement no reduction (max/amax), abs, or elementwise maximum kernels, so reduce_amax now upcasts FP8 inputs to the default float dtype before reducing; the upcast is lossless and only affects the FP8 path.