NVIDIA/Megatron-LM core_v0.20.0

NVIDIA Megatron Core 0.20.0

on GitHub

MoE

• DeepEP v2 dispatcher: adds an experimental DeepEP v2 backend for the flex token dispatcher (#5153).
• NCCL EP MXFP8 communication (experimental): adds MXFP8 dispatch-forward and combine-backward communication to the NCCL EP backend (#6524).
• NCCL EP eager mode: now NCCL EP supports both eager mode and full-iteration graph mode (#6229).
• ShortcutMoE: adds shortcut-connected routing in HybridStack and overlaps expert all-to-all communication with paired attention or Mamba computation; CUDA graphs are not supported (#6959).

Model Architecture and Support

• DeepSeek V4: fully supported by the next-gen HybridModel class (#6403, #6404, #6405, #7469).
• Manifold-constrained hyper-connections (mHC): adds fused mHC connections to HybridModel, including pipeline and virtual-pipeline parallel support (#6401, #7336).
• Gated Delta Product (GDP): adds GDP layers and model-building support for hybrid architectures (#6074).
• Wide residual streams: adds streamwise residual connections, selective residual-stream recomputation, and integration with MTP and MIMO; currently requires PP=1 and CUDA graphs disabled (#6716, #6805, #6855, #7440).
• MTP-only training: adds a mode that freezes the base model and trains only Multi-Token Prediction heads (#7021).

Parallelism

• Megatron-FSDP v2 integration: adds the experimental Megatron Core adapter and Megatron Bridge, with context, expert, pipeline, and hybrid data-parallel sharding support; the adapter requires TP=1 and expert TP=1 (#5865, #6450, #6486, #7220, #6873).
• Megatron-FSDP v2 checkpointing: adds PyTorch distributed checkpoint save/load for model and optimizer state; optimizer checkpointing currently excludes expert-parallel layouts (#6024, #6215).
• Megatron-FSDP v2 supports hybrid FSDP.
  • The user can specify different ZeRO levels for inter-nvlink domains and intra-nvlink domains.
  • The user can specify different configs for the dense section and the MoE section.
• Generalized Tensor Parallelism (GTP): adds configurable batch/sequence partitioning, opt-in latent-projection sharding, and MTP support (#7748, #6383, #6242).

Performance and Memory

• Determinism: multi-release effort, please read our blog post to learn more
• Megatron-FSDP v2 CUDA graphs: adds full-iteration graph capture and replay for supported dense-model configurations (#7075).
• Fused GDN execution: adds Transformer Engine fused GDN and opt-in preprocessing fusion for convolution, layout transforms, normalization, and gates (#7583, #5532).
• MLA and MoE kernel optimizations: adds fused MLA Q up-projection, RoPE, and MXFP8 quantization for supported SM100 configurations, vectorized MLA projection copies, and fused shared-expert/residual additions after MoE combine (#6213, #7957, #7943).
• Weighted checkpoint merging: adds CPU-only weighted merging of compatible, same-layout torch_dist checkpoints (#5114).
• Training observability: adds optional OpenTelemetry traces and metrics for training, pipeline communication, gradient synchronization, and checkpointing (#6415, #6438).

Precision and Optimizers

• Megatron-FSDP v2 mixed precision: enables MXFP8 compute and quantized parameter gathers through the adapter, alongside mixed-precision optimizer support (#7588, #7114, #6506).
• Selective FP32 execution: adds FP32 handling for selected tensors and their optimizer/checkpoint lifecycle, plus an option to compute language-model output logits in FP32 (#5929, #6252).
• Distributed Muon improvements: adds layer-sharded Muon computation across TP and GTP-rematerialization ranks and an optional Triton SYRK path for Newton-Schulz iterations (#6683, #6381).
• DeepSeek-V3 NVFP4 training: adds NVFP4 support including FP4 quantization in active MTP layers (#7328).

Inference

• GDN and GDP dynamic inference: adds GDN serving and extends GDP inference with CUDA graphs, chunked prefill, prefix caching, and speculative decoding (#6499, #6595, #6711, #6780, #7245).
• Prefill-Decode Disaggregation: adds NCCL and NIXL KV-cache transfer paths and SSM-state handoff primitives for prefill/decode disaggregation (#5861, #6222, #6584).
• Batch-Invariant MoE inference: adds batch-invariant BF16/MXFP8 execution paths, including vLLM fused MoE and MXFP8 expert GEMMs (#6521, #7302, #7562).
• Token checkpointing and staging: adds engine hooks to preprocess requests and stage completed token payloads (#7015), extends staged records with compact prompt IDs and media tensors for multimodal NRL training (#7557), and moves staging off the inference loop so slow checkpoint writes do not stall decoding (#8169).
• Expert Tensor Parallel: #4743 supports ETP in the inference engine.
• ShortcutMoE Inference CG: #7772 adds CG support for shortcut MoE, GDP, and wide residuals.
• Inference Service Features and Resiliency: #7828 makes seeded sampling independent of batching, request movement, and other requests’ RNG consumption, #6916 aborts generation when HTTP clients disconnect, preventing orphaned requests from consuming all serving capacity, and large-scale / concurrency optimizations with frontend-coordinator-engine data transfer and replicas in#6497 and #6223.

Datasets

• DP-balanced sequence packing: adds variable-length dataset scheduling, THD-format batch fetching with padding masks, and training-loop integration to balance packed batches across data-parallel ranks (#5903, #6625, #6684, #6742).

Multimodal

• Multimodal (Vision) Inference: #6260 adds image modality support to the dynamic inference engine, #6809 adds video inference support, vision-embedding caching, media-affinity routing, and media-safe prefix hashes; #7598 adds multi-turn expanded-prefix stitching; #7839 serves MIMO-trained VLM checkpoints.
• Heterogeneous MIMO extensions: adds GTP, non-colocated context parallelism, configurable input-projection placement, and model-declared weight refit (#6570, #7435, #6843, #7367).
• RoPE audio encoder: adds dense and packed execution modes with NeMo checkpoint compatibility (#7066).