NVIDIA/Model-Optimizer 0.47.0

ModelOpt 0.47.0 Release

on GitHub

New Features

Quantization

• ONNX quantization with Autotune now benchmarks placements in the requested runtime precision and retains calibrated INT8/FP8 Q/DQ only when it meets the configured TensorRT speedup threshold (1.02x by default); otherwise it saves the high-precision no-Q/DQ model.

• Add a Muse Glimmer AutoQuantize recipe that searches language-model MLP projections, self-attention projections, and lm_head over W4A16 NVFP4 Four-Over-Six, FP8, and BF16 fallback at 5.5 effective bits while leaving the vision tower unquantized.

• Add examples/alpamayo/qad.py, which runs quantization-aware distillation on the quantized Alpamayo checkpoint produced by examples/alpamayo/quantize.py. It distills the quantized VLM against the original FP16 VLM with QADTrainer, supports FSDP2 for multi-GPU runs, and --export reassembles the trained VLM into a full AlpamayoR1 checkpoint that AlpamayoR1.from_pretrained can reload.

• Add a calibration-free streaming Kimi-K3 converter and checkpoint-mirror recipe for NVFP4 routed experts with input_scale=1.0 and 128x128 block-FP8 KDA/MLA attention weights. The converter operates shard-by-shard on the source checkpoint's packed MXFP4 experts instead of loading the 2.8T model through the in-memory hf_ptq.py path.

• Add end-to-end PETRv1 and PETRv2 ONNX PTQ examples covering calibration, INT8 and FP8 VoVNet backbone quantization, TensorRT deployment, and accuracy evaluation.

• Add opt-in FP8 Vision Encoder recipes under the qwen3_vl and qwen3_5 model types. The vision-only recipe keeps the language model and KV cache in high precision; the joint recipe quantizes Vision Encoder and language-model Linears and uses FP8 KV-cache cast. Both quantize primary and deepstack merger Linears where present, while leaving patch embedding and vision-attention BMMs in high precision. Exported checkpoints require an inference runtime that supports quantized Vision Encoder Linears.

• Add mtq.temporarily_fold_weights for repeated frozen-weight inference and mtq.preserve_quantizer_attributes_context for restoring temporary quantizer property and type changes. Temporary folding snapshots affected fake-quant weights on a configurable device and restores them with their quantizer state; retained pre-quant scales are inactive, while shared weights, shared quantizers, and SequentialQuantizer weights are unsupported.

• Add the nvfp4_act_headroom calibration algorithm for NVFP4 activation global scales. Instead of setting the global scale from the largest per-block amax seen during calibration (plain max, which leaves no room above it so any larger activation saturates), it anchors the scale to a low percentile of the per-block amax distribution, leaving the rest of the FP8 block-scale range as headroom: amax = max(rho * anchor, upper), where anchor and upper are the per-block amaxes at anchor_percentile (default 1) and upper_percentile (default 99.99; set to 100 to never clip calibration data), and rho (default 16384) is the headroom factor. Applies only to NVFP4 dynamic-block input quantizers; SequentialQuantizer activation quantizers raise. Weight scales are an orthogonal axis selected by a nested weight_scale_algorithm (max by default, or mse / local_hessian), so one recipe can combine a weight calibration with this activation policy in a single pass. Ships modelopt_recipes/general/ptq/nvfp4_act_headroom-kv_fp8_cast.yaml, which mirrors nvfp4_default-kv_fp8_cast with only the calibration algorithm swapped and exports a standard NVFP4 checkpoint.

• Add PTQ support for Step-3.7 (stepfun-ai/Step-3.7-Flash), whose routed experts were previously left unquantized. Quantize with the new huggingface/step3p7/ptq/nvfp4_experts_only-kv_fp8_cast or huggingface/step3p7/ptq/nvfp4_mlp_only-kv_fp8 recipes rather than the general ones, which select experts by module names Step does not use.

Megatron Framework (M-LM / M-Bridge)

• Add clamp_kv_cache_scales to export_mcore_gpt_to_hf. Set it to False when exporting a QAT Megatron-Core model to preserve its learned FP8 KV-cache scales; the default retains the existing minimum scale of 1.0.
• Add SFT-masked data support to examples/megatron_bridge/distill.py: --sft --sft_dataset_root <dir> distills on raw prompt-completion JSONL ({"input", "output"} records) with the loss masked to the response tokens, using Megatron-Bridge's FinetuningDatasetConfig and the model's own HuggingFace tokenizer instead of the pretraining GPTDataset and NullTokenizer.
• Add per-expert weight quantization for Transformer Engine TEGroupedLinear (fused MoE experts): each expert now has its own weight_quantizer (a GroupedQuantizer holding one TensorQuantizer per expert) with an independent amax, instead of a single shared amax across all experts. Applies to mtq.quantize calibration, HF / Megatron export, and QAD.
• Add opt-in torch.compile execution for Transformer Engine grouped-linear per-expert weight quantizers while preserving their native checkpoint amax shapes. Set MODELOPT_TEGROUPED_COMPILE_WEIGHT_LOOP=1 before quantized-module conversion; the default path remains eager.
• Add HuggingFace unified export of quantized Qwen3-VL and Qwen3.5-VL checkpoints (PTQ or QAD) via examples/megatron_bridge/export_quantized_megatron_to_hf.py, Qwen3.5-VL additionally covering GatedDeltaNet linear-attention layers and MoE shared experts. Only the language model is quantized; the vision tower is copied from the source HuggingFace checkpoint.
• Megatron-Bridge scripts now choose the MoE expert layout automatically from the model config: the faster fused TEGroupedMLP (grouped GEMM) unless the architecture cannot export it to HuggingFace, in which case SequentialMLP keeps the checkpoint exportable and --no_moe_grouped_gemm forces it explicitly. For the affected architectures this changes MoE activation scales from one shared scale to per-expert.

Misc

• Add modelopt.torch.utils.mlflow.MlflowRunLogger for recording a script run on an MLflow tracking server: the invocation, the ModelOpt version, the run log (captured by teeing stdout/stderr) and any caller-supplied artifacts, with configuration as searchable params. mlflow is an optional dependency, imported only when tracking is enabled.
• Add --mlflow <tracking-uri> to examples/hf_ptq/hf_ptq.py (MLflow's own MLFLOW_TRACKING_URI is honoured too). A tracked run records the invocation, the resolved recipe ($imports expanded), the run log and the quantization summaries, with every command-line argument as a searchable param; failed runs are recorded with their traceback. The experiment defaults to $USER/hf_ptq/<checkpoint basename>-<recipe name or --qformat> and can be overridden with --mlflow_experiment / --mlflow_run_name.
• Add --mlflow <tracking-uri> to examples/vllm_serve/vllm_serve_fakequant.py (MLflow's own MLFLOW_TRACKING_URI is honoured too), so a fake-quant serve records what it quantized and an evaluation of that endpoint can be traced back to a recipe. A tracked run uploads the launcher command, the resolved RECIPE_PATH (or the merged QUANT_CFG/KV_QUANT_CFG when presets are used), the worker log and the quantizer summary; the experiment defaults to $USER/vllm_serve_fakequant/<model basename>-<recipe name or quantization config> and can be overridden with --mlflow-experiment / --mlflow-run-name.

Backward Breaking Changes

• Migrate the FAR3D ONNX PTQ example to the shared evaluator and ModelOpt containers and quantize_vovnet.py. Only the encoder supports INT8 and FP8; decoder calibration, quantization, and related CLI flags are removed, and the decoder remains in its exported mixed FP16/FP32 precision.
• Image-text calibration with --calib_with_images now forwards multimodal batches through the complete VLM for all VLM families, so existing non-Nemotron commands may produce different language-model activation ranges and output scales. Recipe-based VLM PTQ also targets the complete VLM: vision modules stay in high precision by default and are quantized only when a model-specific recipe enables them, so custom recipes must explicitly exclude vision modules when required.
• Move the checkpoint-mirror recipe tier from huggingface/models/<org>/<checkpoint>/ to the top-level models/<org>/<model_id>/, keyed by each recipe's canonical Hugging Face Hub id — so the Step 3.5 Flash recipe moves to models/stepfun-ai/Step-3.5-Flash/ptq/ and the NVIDIA Nemotron recipes gain the NVIDIA- prefix (e.g. models/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse). Update any saved --recipe paths for these checkpoint recipes accordingly; the per-model_type recipes under huggingface/ are unchanged.
• Move the Mistral Medium 3.5 checkpoint-mirror recipe from huggingface/models/nvidia/Mistral-Medium-3.5-128B-NVFP4/ptq/nvfp4-max-calib to models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib, keying it by the canonical Hugging Face base model. Update any saved --recipe paths to the new location.
• Remove the --auto_quantize_bits, --auto_quantize_method, --auto_quantize_score_size, --auto_quantize_cost_model and --auto_quantize_active_moe_expert_ratio flags from examples/hf_ptq (deprecated in 0.46). Use an AutoQuantize --recipe from modelopt_recipes/general/auto_quantize/ instead. Those recipes now also splice in the shared base cost_excluded_layers unit, which the removed CLI applied unconditionally, so a VL model keeps its vision tower and MTP layers out of the effective-bits denominator. On a VL model this changes the per-layer cost weights, so an existing --auto_quantize_checkpoint from an earlier release is rejected with "Use a different checkpoint path"; delete or repoint it to re-run the search.
• Remove the examples/llm_ptq symlink and the examples/vlm_ptq forwarder (both deprecated in 0.46). Use examples/hf_ptq, passing --vlm for vision-language models.
• Remove the backward-compat --qformat / --quant_cfg short names int8_sq, int8_wo, w4a8_awq, nvfp4_awq, nvfp4_mse, nvfp4_local_hessian, fp8_pb_wo and fp8_pc_pt (deprecated in 0.45). Use the preset basename under modelopt_recipes/configs/ptq/presets/model/ instead: int8_smoothquant, int8_weight_only, w4a8_awq_beta, nvfp4_awq_lite, nvfp4_w4a4_weight_mse_fp8_sweep, nvfp4_w4a4_weight_local_hessian, fp8_2d_blockwise_weight_only and fp8_per_channel_per_token. The modelopt.recipe.presets.QFORMAT_ALIASES table and the aliases argument of load_quant_cfg_choices() are removed along with them.
• Remove the legacy layerwise bool form, its use_sequential alias, and the top-level layerwise_checkpoint_dir key from calibration algorithm configs (deprecated in 0.45). Use the nested form, e.g. layerwise: {enable: true, checkpoint_dir: /path}. A pre-0.45 modelopt_state carrying either legacy key now fails validation on restore instead of being migrated; re-save it with a 0.45/0.46 release first.
• Remove in-trainer quantization via QuantizationArguments.quant_cfg / --quant_cfg (deprecated in 0.45); use --recipe. New recipes general/ptq/mxfp4_mlp_weight_only and general/ptq/nvfp4_mlp_weight_only replace MXFP4_MLP_WEIGHT_ONLY_CFG / NVFP4_MLP_WEIGHT_ONLY_CFG in the examples/gpt-oss QAT flow.
• Remove the QuantizationArgumentsWithConfig alias in modelopt.torch.quantization.plugins.transformers_trainer (deprecated in 0.45). Use QuantizationArguments.
• Transformer Engine TEGroupedLinear (fused MoE experts) now uses per-expert weight quantization (one amax per expert) instead of a single shared amax, so ModelOpt checkpoints containing quantized TEGroupedLinear modules saved before 0.47 are not compatible with 0.47. Re-run PTQ to regenerate compatible checkpoints.
• mtq.quantize now raises when a config asks for weight quantization but none of its weight-quantizer patterns match the model, instead of calibrating and exporting a silently unquantized checkpoint ("quant_algo": null). Configs that quantize activations or the KV cache only are unaffected, as are patterns that match and are then disabled by a later entry. If this fires, use the recipe for that architecture under modelopt_recipes/huggingface/<model_type>/ or fix the module patterns. Set MODELOPT_SKIP_WEIGHT_QUANT_CHECK=1 to disable the check process-wide, e.g. for a pipeline-parallel rank whose local stage legitimately has none of the targeted modules.

Deprecations

• Remove examples/llm_eval/lm_eval_tensorrt_llm.py (the trt-llm model) in favor of the TensorRT-LLM backend shipped by lm-evaluation-harness itself (registered as trtllm, also supports loglikelihood_rolling and pipeline parallelism); lm_eval is pinned to >=0.4.12,<0.5. Replace python lm_eval_tensorrt_llm.py --model trt-llm --model_args tokenizer=<tok>,checkpoint_dir=<ckpt> with python lm_eval_trtllm.py --model trtllm --model_args model=<ckpt>,tokenizer=<tok>, and set tensor_parallel_size and max_input_len explicitly — they default to 1 and 2048, and longer prompts are silently truncated. Use lm_eval_trtllm.py rather than the plain lm_eval CLI: it patches an off-by-one in the backend's _parse_logprobs that otherwise raises KeyError on every loglikelihood task. Loglikelihood tasks additionally require TensorRT-LLM >= 1.3.0rc11; generative tasks are unaffected. examples/hf_ptq/scripts/huggingface_example.sh gains --input (BUILD_MAX_INPUT_LEN, default 4096) to size the evaluation engine's context, and honours a preset LM_EVAL_TP to override the tensor-parallel size.
• Bump minimum container requirement to nemo:26.08 for the Megatron-Bridge / Megatron-LM optimization features. Megatron-LM quantization still works on nemo:26.06 but will be dropped in a future release.

Bug Fixes

• Fix shared ONNX export metadata and Diffusers attention policy: every NVFP4QuantExporter post-process now upgrades the default-domain opset to at least 23, all FP8 custom-op exports re-run ONNX shape/type inference after setting output metadata, and quantized SDPA derives FP8 MHA enablement from the live Q/K/V quantizers instead of honoring a caller-set _disable_fp8_mha attribute.
• Fix ONNX FP16 conversion failing to preserve public output types when type inference changes a graph output declaration before output casts are inserted.
• Fix HuggingFace checkpoint export failing with activation scaling factor 0.0 not positive when a dynamic-block quantizer (such as an NVFP4 input quantizer) ends calibration with an amax of zero because the calibration data never activated that layer or expert. Such a quantizer now exports a positive fallback scale and warns instead of crashing, matching what static quantizers already did; if you see the warning, check whether the layer is expected to be inactive and consider a larger calibration size.
• Fix ONNX AutoCast failing on models with external initializers larger than 2 GiB.
• Avoid querying CUDA/Blackwell capability when NVFP4QTensor.quantize uses its CPU path or has the optional TensorRT-LLM fast path disabled.
• Fix NVFP4 ONNX export to quantize FP4 weights with the published FP8 block scales, matching eager ModelOpt packed weights. Block scales below 2**-9 are now clamped to that minimum, and non-finite or negative scales raise an error.
• Fix FP8 ONNX export of BF16 models during real-weight compression.
• Fix Megatron-Bridge Quantization Aware Distillation of a vision-language model silently discarding the ModelOpt state, so the distilled checkpoint restored no quantizers and exported as an unquantized model. Re-run QAD to regenerate any affected checkpoint.
• Fix Megatron-Core HuggingFace export silently omitting fused (grouped GEMM) MoE experts for architectures without an experts.linear_fc1 rule (e.g. Qwen3MoeForCausalLM), which produced a valid-looking checkpoint containing no expert weights. The exporter now raises instead of writing that checkpoint; the scripts also avoid the situation by selecting SequentialMLP for those architectures.
• Fix GatedDeltaNet (Qwen3.5) quantizer exclusions on Megatron-Core: the recipe patterns name the HuggingFace linear_attn module, so the conv1d was calibrated and the alpha / beta gate projections were exported in FP8. conv1d now has a self_attention alias in the default disabled-quantizer units, and the alpha / beta projections are exported in BF16 (they share Megatron's fused in_proj quantizer and cannot be disabled by name).
• Fix quantizing a Megatron-Core GPT output_layer (HuggingFace lm_head) failing to checkpoint: megatron-core drops output_layer._extra_state, where ModelOpt keeps quantizer state, so saving raised and loading silently restored the layer unquantized. ModelOpt now retains that entry for every Megatron model, and disables the workaround once megatron-core keeps it itself.
• Fix Megatron-Core HuggingFace export writing Qwen3.5 / Qwen3.6 MoE routed experts as one packed tensor per layer, which vLLM cannot load (Layer ... has no parameter 'w2_weight_weight_scale_2'). Quantized exports now write one entry per expert with gate_proj / up_proj split, matching the released NVFP4 checkpoints.
• Megatron-Core HuggingFace export now names unquantized grouped-GEMM MoE experts in exclude_modules instead of omitting them, so a mixed-precision checkpoint tells the runtime what is not quantized. This adds one hf_quant_config.json entry per expert per layer for grouped-expert architectures such as NemotronHForCausalLM; the served numerics are unchanged.
• Remove --moe_calib_experts_ratio from examples/megatron_bridge/quantize.py. It was accepted but had no effect there: the option is implemented only for HuggingFace MoE modules, so Megatron models silently ignored it. It remains available in examples/hf_ptq/hf_ptq.py, where it works.
• Fix image-text (VLM) calibration hanging on multi-GPU: the per-subset sample budget used truncating division, so the stream supplied fewer samples than requested (1023 for --calib_num_samples 1024 over 3 subsets) and the strided data-parallel sharder left the trailing rank one sample short, deadlocking calibration until a 30-minute NCCL/gloo timeout. Subset budgets now sum exactly, and the sharder truncates every rank to the same count.
• Fix Megatron-Core quantized KV-cache scales being dropped when a Qwen checkpoint is exported to HuggingFace: k_scale / v_scale were omitted and kv_cache_quant_algo left unset for every Qwen architecture, so a checkpoint calibrated with an FP8 or NVFP4 KV cache silently served an unquantized one.
• Megatron-Core HuggingFace export now verifies its own output: if the exported checkpoint is missing tensors the source checkpoint has (an architecture whose module has no export rule), it raises instead of writing a valid-looking checkpoint. Depth-pruned models and tied embeddings are accounted for.
• Loading a Megatron checkpoint that holds quantizer tensors but no restorable ModelOpt state now raises instead of silently loading the model unquantized.
• Fix image-text calibration for non-Nemotron VLMs by forwarding multimodal batches through the complete VLM.
• Update HuggingFace checkpoint export to use name-based tied-weight deduplication instead of the previous address-based approach. The address-based deduplication could incorrectly drop an untied weight that happened to share memory with a tied one, producing an incomplete checkpoint (observed as a false positive on MiniMax-M2.7).
• Fix EAGLE-3 training with context parallelism (--cp_size > 1 in examples/speculative_decoding), which failed to start on accelerate >= 1.13 and then raised got mixed torch.Tensor and DTensor.
• Polygraphy minimum dependency upgraded to 0.53.4 to solve ONNX AutoCast failures when marking optional graph outputs.
• Fix --kv_cache_free_gpu_memory_fraction having no effect on the lm_eval task of examples/hf_ptq/scripts/huggingface_example.sh, where the KV cache always took TensorRT-LLM's default 90% of free GPU memory and evaluation could run out of memory. examples/llm_eval/lm_eval_trtllm.py now takes kv_cache_free_gpu_memory_fraction in --model_args, defaulting to 0.8.