NVIDIA/Model-Optimizer 0.46.0

ModelOpt 0.46.0 Release

on GitHub

New Features

Quantization

• Add NVFP4 and FP8 PTQ recipes with projection-output quantizers for Llama-Nemotron embedding and reranking models (modelopt_recipes/huggingface/nemotron_llama/) and an end-to-end HF embedding/reranking quantize-to-ONNX example (examples/torch_onnx/hf_embedding_quant_to_onnx.py). Quantizing the projection-Linear outputs keeps TensorRT inter-layer activations in FP4, roughly halving engine activation memory versus the plain nvfp4 preset. NVFP4/MXFP8 output quantizers now export through the dynamic quantize path. examples/torch_onnx/torch_quant_to_onnx.py also gains a --recipe flag to load quantization configs from YAML recipes instead of the removed mtq.*_CFG module-constant table.
• Add an end-to-end FAR3D ONNX PTQ example with calibration data generation, INT8 and FP8 quantization, TensorRT engine building, and Argoverse 2 accuracy evaluation. See examples/onnx_ptq/far3d/README.md for details.
• Add Learned Scale Quantization (LSQ) and Dual-LSQ support for quantization-aware distillation, including learnable amax parameters, tied-scale and pre-scale options, focused NVFP4 recipes, and scale-only training.
• Add a fused Triton fast path for the local_hessian NVFP4 weight-scale search, roughly 34x faster than the Python reference sweep on a single 8192x4096 weight and bit-exact with it for fp32/fp16 weights. Used automatically during local_hessian calibration for both dense and fused-MoE expert weights; falls back to the reference sweep on CPU, when Triton is unavailable, or via MODELOPT_NVFP4_TRITON_SWEEP=0.
• Add NVFP4 Four-Over-Six (4/6) weight quantization (mtq.NVFP4_FOUR_OVER_SIX_CFG): MSE weight calibration picks, per block, between an M=6 and an M=4 dynamic range (the choice is folded into the FP8 per-block scales), with the four_over_six: true flag normalizing those scales by 256 (vs 448) for M=4 headroom. Supported via mtq.quantize and HF / Megatron export only -- not mtq.compress, which does not preserve the per-block M=4/M=6 choice.
• Add dLLM (tied-weight PTQ and HF-checkpoint export) support for diffusion-based encoder-decoder LLMs (e.g. DiffusionGemma) whose encoder/decoder stacks share parameters via HF _tied_weights_keys. Modules sharing a source weight are deduplicated at export (~42% storage reduction on nvfp4_experts_only for tied 26B MoE checkpoints), a new sync_tied_input_amax helper max-merges per-side input_quantizer.amax across tied modules so single-backbone consumers don't clip either side, and the exported state dict is reordered so the canonical-side keys win the dedup. Ships a DiffusionGemma recipe under modelopt_recipes/huggingface/diffusion_gemma/ptq/. Non-tied models see no behavioral change.
• Add Torch-TensorRT FP8 deployment example for HuggingFace ViT (examples/torch_trt/): torch_tensorrt_ptq.py covers mtq.quantize → torch_tensorrt.compile(ir="dynamo"), and torch_tensorrt_accuracy.py reports the compiled model's ImageNet-1k top-1/top-5 accuracy (the unquantized baseline is Torch-TensorRT-compiled too, for an apples-to-apples comparison). Ships a ViT-tuned FP8 PTQ recipe under modelopt_recipes/huggingface/vit/ptq/fp8.yaml that quantizes the encoder Linears, patch-embed nn.Conv2d, classifier, per-block LayerNorm inputs, and the attention Q/K/V BMMs and softmax. Verified on google/vit-base-patch16-224: FP8 stays within 0.13 pp Top-1 of the FP16 baseline.
• Add AutoQuantize recipe support: mtq.auto_quantize can be driven declaratively from a YAML recipe (RecipeType.AUTO_QUANTIZE / AutoQuantizeConfig) specifying candidate formats, the effective_bits target, cost model (incl. active_moe and excluded_module_name_patterns), scoring method, and disabled layers. Adds an effective_bits cost-model override on QuantizeConfig / QuantizerAttributeConfig (block-scale-accurate NVFP4 = 4.5 via configs/numerics/nvfp4). Shipped recipes live under modelopt_recipes/general/auto_quantize/ and model-specific ones under modelopt_recipes/huggingface/<model>/auto_quantize/.
• Add module-specific AutoQuantize search spaces through mtq.auto_quantize(..., module_search_spaces=...) and recipe-level auto_quantize.module_search_spaces. Glob-matched decision groups can override the global candidate formats and control whether BF16/no-quant is solver-selectable with allow_no_quant. A recipe can instead reuse a normal PTQ quantize config as the fixed baseline and list only the genuinely searched modules; fixed and searched groups stay in one calibration, scoring, effective-bits, checkpoint, and export flow.
• Add rotate.mode to torch quantizer configs. The default "rotate" keeps the existing rotate-before-quantize behavior; "rotate_back" enables fake-quant rotate → quantize → rotate-back for TensorQuantizer.
• Add a constant_amax QuantizerAttributeConfig field that pins a quantizer's amax to a fixed value and skips activation calibration. Unlike use_constant_amax (which hardcodes 448.0 for KV-cache cast math and registers no buffer), constant_amax stores the constant on the _amax buffer so it is used by both the fake-quant forward and the exported scaling factor — for NVFP4 activations, constant_amax: 2688.0 yields input_scale == 1.0. Ships modelopt_recipes/general/ptq/nvfp4_experts_only_input_scale1-kv_fp8_cast.yaml, which applies this to the MoE expert activation quantizers.
• Add MaxCalibConfig.skip_forward_without_activation_calib (opt-in, default False): max calibration skips the forward_loop when no enabled quantizer needs data-driven activation statistics — e.g. an experts-only recipe using constant_amax / use_constant_amax, or dynamic / MX quantization. Weight calibration still runs on the weight tensors directly, so quantized weights are unchanged. It is opt-in because the forward_loop can carry caller-side effects (notably materializing sharded parameters under DeepSpeed ZeRO-3). Enabled by the nvfp4_experts_only_input_scale1-kv_fp8_cast recipe.
• Add examples/minimax_m3/hf_ptq_mixed_mxfp8_nvfp4.py for streaming MiniMax-M3 export and a model-specific hf_ptq.py recipe that produces an MXFP8 language-model base with MSE-calibrated NVFP4 routed experts directly from BF16. The NVFP4 expert input_scale is fixed to 1.0.

Speculative Decoding

• Add the D-PACE loss objective for DFlash speculative-decoding training (arXiv:2605.18810) and make it the default (dflash_loss_objective: dpace). It replaces the static exponential position decay with dynamic, confidence-derived per-position weights that adapt to whichever block positions currently limit acceptance. Smoothing is controlled by dflash_dpace_alpha (default 0.5); set dflash_loss_objective: decay to restore the previous static schedule. Training-only and detached from the gradient (no architecture or inference change).
• Add streaming speculative-decoding training (EAGLE3 / DFlash): the draft trains on base-model hidden states produced on the fly by a co-located vllm serve (no disk dump), moved trainer-side over NIXL RDMA, scaling to multi-node (dedicated serve replicas + DDP trainers). New launcher examples for NVFP4 Kimi-K2.5 / K2.6 on GB200/aarch64 under tools/launcher/examples/moonshotai/.
• Add Domino speculative-decoding training: the parallel DFlash draft backbone plus a lightweight GRU causal correction head, selected via dflash_architecture_config.projector_type=domino. Trained with a base/final dual loss whose dflash_lambda_base_start/dflash_lambda_base_decay_ratio curriculum decays the base-loss weight 1→0. Exports in the z-lab drafter format; recipe at modelopt_recipes/general/speculative_decoding/domino.yaml. Training only — the inference path is not wired up yet.

Megatron Framework (M-LM / M-Bridge)

• Add Minitron pruning support for Megatron-Core models with the following new attention and MoE variants. For these, only hidden_size is pruned (alongside the usual ffn_hidden_size / num_layers / MoE dimensions); the variant-internal dimensions noted below are not pruned:

  • GatedDeltaNet (linear attention) and gated attention (attention_output_gate), such as Qwen3.5 (hybrid GatedDeltaNet + gated-attention) language models, including MoE variants — attention / linear-attention heads are not pruned.
  • Multi-Latent Attention (MLA), such as DeepSeek — MLA latent ranks are not pruned.
  • Latent MoE, such as Nemotron-3-Super — hidden_size pruning resizes the latent projections while the MoE latent dim itself is not pruned.

• Optimize Minitron pruning support for MoE models using the fused grouped GEMM experts (TEGroupedMLP) in addition to the existing SequentialMLP path. examples/megatron_bridge/prune_minitron.py now uses grouped GEMM by default (pass --no_moe_grouped_gemm to fall back to TESequentialMLP).

• Add Minitron pruning support for the language model part of vision-language models (e.g. Qwen3.5-VL, Gemma3-VL) via examples/megatron_bridge/prune_minitron.py. The language model is pruned while the vision tower is left intact and the full VLM is saved back; hidden_size is not pruned if it is shared with the vision projector. Pruning importance is estimated from image-text calibration (the full VLM forward over vision-conditioned activations) by default, or from a text dataset for text-only ablations.

• Add PTQ support for the language model part of vision-language models (e.g. Qwen3.5-VL, Gemma3-VL) via examples/megatron_bridge/quantize.py. Only the language model is quantized (vision tower + projector left in full precision) and the full VLM is saved as a Megatron checkpoint. The calibration modality is inferred from --calib_dataset_name: an image-text dataset drives the full VLM forward (vision-conditioned activations), while a text dataset runs text-only calibration of the language model. Image-text calibration shards across data-parallel ranks (context parallelism is supported only for text-only calibration). HuggingFace unified export of a quantized VLM is not yet supported.

• Add Megatron-Bridge distillation and Quantization-Aware Distillation (QAD) support for the language model part of vision-language models (e.g. Qwen3.5-VL, Gemma3-VL) via examples/megatron_bridge/distill.py.

• Add context-parallel (CP) and data-parallel (DP) support to the shared Megatron-Core inference/calibration utilities. Under CP, get_megatron_calibration_forward_loop and megatron_mmlu partition each sequence across CP ranks (zigzag load-balanced) and MMLU gathers per-rank logits back to the full sequence for last-token scoring. Under DP, calibration shards the dataset across data-parallel ranks (amax is max-reduced across the DP group inside mtq) and megatron_mmlu shards whole batches and all-reduces the per-subject counts. DP is implicit (world_size / (tp * pp * cp)); examples/megatron_bridge/quantize.py gains a --cp_size flag.

• Add support for retaining all Megatron-Bridge distillation checkpoints via distill.py --checkpoint_keep_last -1 and exporting all or selected iterations with export_distilled_megatron_to_hf.py --export_iterations.

• Add the prepare_megatron_data_blend utility to prepare weighted Megatron data blends from YAML configs, including optional token-budgeted subsets for distillation workflows. See the Megatron data preparation guide.

Misc

• Add the day0-release agent skill, a deterministic end-to-end driver that chains the PTQ → evaluation → comparison skills with an enforced gate after each stage (validating checkpoint coverage, evaluation-run completeness, and the baseline-vs-candidate accuracy threshold) and returns a publish decision (ACCEPT / REGRESSION / ANOMALOUS / INFEASIBLE). v1 reports and stops on regression; the recipe-search loop is deferred.
• Add support for ONNX Q/DQ node placement for DLA via the new flag --target_dla.
• (Experimental) Add pruning examples for Qwen3.5-9B and Nemotron3-Nano using the new experimental puzzletron branch, this branch uses AutoModel for better parallelization and efficiency.

Backward Breaking Changes

• Remove the examples/diffusers/eval image-quality evaluation example (ImageReward / CLIP-IQA / CLIP metrics) and its references in examples/diffusers/README.md. The example was deprecated in 0.45 and is no longer maintained.
• Remove the deprecated examples/llm_autodeploy example (deprecated in 0.45). Use TensorRT-LLM's AutoDeploy directly together with ModelOpt PTQ in examples/hf_ptq.
• Remove the deprecated examples/llm_qad Megatron-LM QAD example (deprecated in 0.45). Use the megatron_bridge QAD example instead, which provides a simpler Python-based interface and better model coverage.
• Dropped VILA / NVILA vision-language model support in examples/hf_ptq. VILA's modeling code requires [transformers](=4.50.0, which conflicts with ModelOpt's minimum supported transformers version. The VILA-specific bootstrap (repo clone, requirements-vila.txt) and loading paths in example_utils.py have been removed.
• Dropped Phi-4-multimodal and Phi-3-vision PTQ support in examples/hf_ptq. Phi-4-multimodal's bundled remote code needs transformers<4.52, below ModelOpt's minimum of 4.57; Phi-3-vision, the superseded predecessor in the same family, is dropped alongside it and is likewise broken on Transformers 5.x. The support-matrix row, the phi4mm model type, the multimodal-detection heuristics that only ever matched these two, the Phi3Image / PhiImage embedding-export exclusions, and the modelopt_recipes/huggingface/phi4mm/ recipes have been removed. Text-only Phi-3/Phi-4 and Phi-3.5-MoE are unaffected.

Deprecations

• examples/hf_ptq AutoQuantize is now driven by an AutoQuantize recipe (--recipe). The --auto_quantize_bits, --auto_quantize_method, --auto_quantize_score_size, --auto_quantize_cost_model, and --auto_quantize_active_moe_expert_ratio flags are deprecated but still work: they are converted into an AutoQuantizeConfig on the fly (emitting a DeprecationWarning) and will be removed in a future release. Prefer a recipe under modelopt_recipes/general/auto_quantize/. See examples/hf_ptq/README.md.
• Renamed examples/llm_ptq to examples/hf_ptq to reflect that it covers Hugging Face LLM and VLM PTQ. A relative symlink examples/llm_ptq to hf_ptq keeps existing paths and commands working; it will be removed in a future release. Please update references to the new examples/hf_ptq path.
• Consolidated examples/vlm_ptq into examples/hf_ptq. Vision-language model PTQ now shares the hf_ptq.py entry point and scripts/huggingface_example.sh; pass --vlm to run the TensorRT-LLM multimodal quickstart smoke test. The examples/vlm_ptq/scripts/huggingface_example.sh entry point is deprecated: it now prints a warning and forwards to the hf_ptq script with --vlm, and will be removed in a future release. See `examples/hf_ptq/README.md <https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/hf_ptq#vlm-quantization).
• Bump minimum transformers version to 4.57 instead of 4.56. Transformers 4.x support will be dropped in a future release.
• Bump minimum nemo container requirement to nemo:26.04 (recommended nemo:26.06) for Megatron-Bridge / Megatron-LM optimization features.
• Python 3.10 support will be dropped in the next release as it is reaching EOL.

Bug Fixes

• Fix NemotronH dense MLP quantization with the nvfp4_mlp_only and nvfp4_omlp_only recipe families. NemotronH registers these projections as mixer.up_proj / mixer.down_proj, which the previous *mlp* selector missed, producing checkpoints with a null quant_algo.
• Fix ShapeInferenceError during ONNX INT8 + FP16 quantization (--high_precision_dtype fp16) of weakly-typed models (e.g. TensorFlow exports) that carry stale rank-0 graph.output shapes or ops such as TopK that ONNX's static shape inference cannot resolve. Stale output shapes are now reconciled via symbolic shape inference, and AutoCast falls back to schema-based type inference so unresolved ops no longer leave tensors untyped.
• Fix fused MoE expert auto-detection (register_fused_experts_on_the_fly) skipping modules without an act_fn attribute. Modules applying a custom gated activation between the two F.linear calls (e.g. MiniMaxM3VLExperts) were silently skipped, leaving routed experts unquantized and failing HF export. Enables NVFP4/FP8 quantization and export for MiniMax-M2 / MiniMax-M3.
• Fix unified HF export emitting transformers' in-memory (post-conversion_mapping) tensor names instead of the original model-hub names, breaking the unified-checkpoint contract (observed on MiniMax-M3). A new quant-aware reverse conversion derives the rename/split rules from the model's conversion mapping and carries each weight's companion scale tensors through the renames and un-fusions, so quantized exports round-trip to the hub names. Mapping ops that cannot be reversed quant-aware yet (e.g. still-stacked fused experts) fall back to the in-memory names instead of aborting the export.
• Fix unified HF export of already-compressed NVFP4 weights, i.e. mtq.compress and hf_ptq.py --low_memory_mode, writing a weight_scale of half the required size with meaningless values (the per-block scale cannot be recomputed from packed nibbles). The export now reuses the per-block scale captured at compression time. The internal _scale / _double_scale quantizer buffers are also removed after use; they previously leaked into the checkpoint and made downstream loaders (vLLM, TensorRT-LLM PyTorch backend) fail with KeyError.
• Fix ONNX FP16/BF16 conversion (--high_precision_dtype fp16) producing inconsistent tensor types on models with control-flow subgraphs. Subgraph nodes now only run in low precision when all their float inputs are subgraph initializers, outer-scope captures and precision boundaries are reconciled with Cast nodes, and Constant folding refreshes value_info so strongly-typed parsers (TensorRT) no longer reject the model. Behavioral change: a weight inside a branch that also reads an outer-scope FP32 activation now stays FP32 instead of being converted.
• Nested submodel reverse mappings are now scoped against registered model namespaces, preventing text-only mappings from capturing an already nested VLM's model.visual.* namespace or double-prefixing model.language_model.* (observed on Qwen3.5).
• Fix vLLM 0.24+ support, where FusedMoE became a factory function and the expert weights moved onto a RoutedExperts submodule, breaking every QuantModuleRegistry lookup with TypeError: issubclass() arg 2 must be a class. The plugin now registers whichever fused-MoE module class the installed release provides. The registry key moves vllm_FusedMoE to vllm_RoutedExperts and quantizer paths gain .routed_experts, so an older modelopt_state does not restore onto 0.24+ as-is.
• Fix examples/vllm_serve serving shared experts uncalibrated: their gate_proj/up_proj quantizer keys were not merged into gate_up_proj on reload, so they matched no module and were dropped.
• Fix Qwen3-VL MoE PTQ failing on transformers>=5.12 with AttributeError: 'QuantQwen3VLMoeTextExperts' object has no attribute 'hidden_size'. transformers 5.12 moved Qwen3VLMoeTextExperts onto the standard fused-experts layout, but the legacy static wrapper shadowed on-the-fly detection. The new layout is now claimed by register_fused_experts_on_the_fly; the legacy wrapper is still registered on transformers<5.12, whose torch.bmm-based forward the generic wrapper cannot intercept.
• Fix examples/hf_ptq multi-node FSDP2 export (--use_fsdp2) failing with RuntimeError: Cannot set version_counter for inference tensor. export_quantized now runs under torch.no_grad() instead of torch.inference_mode(), so the gathered full params stay normal tensors.
• Fix HF checkpoint export failing with AttributeError: 'list' object has no attribute 'keys' for models whose modeling code still declares tied weights in the transformers<5 list format (common among trust_remote_code checkpoints, e.g. stepfun-ai/Step-3.7-Flash). Such models load fine but died at the end of PTQ, after calibration. ModelOpt's save_pretrained patch now normalizes a list-style declaration to the equivalent dict for the duration of the save.
• Fix unified HF export of multimodal models whose vision tower carries its own PrefixChange conversion (LlavaForConditionalGeneration and Gemma3ForConditionalGeneration on transformers>=5.12). The quant-aware reverse conversion ignored transformers' scope_prefix, so the vision tower's prefix rule was applied to every key in the state dict and vLLM rejected the checkpoint with ValueError: There is no module or parameter named 'vision_model'. Reverse rename rules now carry their scope and are applied only to keys under it.
• Fix QLoRA export in examples/llm_qat/export.py failing with AssertionError: Model already has modelopt state!: enable_huggingface_checkpointing already restores the quantized base model's state, so the export now restores only when the model is not already converted. Two further breakages on the same path are also fixed: _restore_qtensor_wrappers matched no modules because PEFT re-parents the quantized linear as <name>.base_layer, and postprocess_state_dict silently dropped every base_layer.* key missing from a hand-maintained rename map (losing the NVFP4 weight_scale_2 global scale and any linear bias) — the rename is now a generic .base_layer. strip.
• Fix --use_fsdp2 PTQ (examples/hf_ptq) failing on models that hold a few parameters in a dtype other than the model's own, with AssertionError: FSDP expects uniform original parameter dtype on the first calibration forward. Nemotron-3-Nano is one such model: its MoE router gates are declared float32 while the rest of the checkpoint is bfloat16, so each decoder layer's FSDP2 shard group mixed dtypes. fsdp2_wrap now passes those off-dtype parameters to fully_shard(ignored_params=...), leaving them replicated in their original dtype instead of casting them, and warns with their names and their share of the model.
• Fix --use_fsdp2 HF export making no progress for hours on large MoE checkpoints. create_fsdp_param_mapping resolved each FSDPParam's module by scanning every model.named_parameters(), and export calls it once per quantized module, so the cost was quadratic in (parameters x modules): harmless for dense models, intractable for a MoE with many experts. Exporting Nemotron-3-Nano-30B-A3B (6,243 parameter tensors, 6,004 quantized modules) spent an estimated 1.9 hours there with every GPU idle. The parameter index is now built once per mapping instead of once per FSDPParam (1151 ms -> 5.1 ms per call), preserving the previous `named_parameters()``-order resolution for tied weights.