NVIDIA/Megatron-LM core_v0.19.0 on GitHub

Changelog Details

Add separate mtp_grad_scale_func for MTP loss scaling by @yfw :: PR: #3459
[training migration] Migrate GPT builder by @maanug-nv :: PR: #4741
Make Mamba conv params direct mixer params by @wujingyue :: PR: #4899
Update oncall reviewer assignment by @Phlip79 :: PR: #5093
Pass explicit process groups to hybrid logging by @yashaswikarnati :: PR: #4781
Clean up top-level repository files by @Phlip79 :: PR: #5097
[main] fix(moe): Fix several bugs for DSA rope and spec. by @yuzhongw-nvidia :: PR: #3026
Move MIMO unit tests into models/mimo by @yashaswikarnati :: PR: #5063
test: update DeepSeek FSDP2 GB200 memory golden by @wujingyue :: PR: #5094
ci: make CI resilient to pip/uv network timeouts by @ko3n1g :: PR: #5118
ci: treat docker container-removal conflict as flaky by @ko3n1g :: PR: #5120
Fix GDN DTensor splitting for FSDP checkpointing by @conver334 :: PR: #4843
Remove DeepEP hardware limit check by @janEbert :: PR: #4846
Update transformer-engine dependency to revision 4220403 by @balasaajay :: PR: #5112
Fix MoE aux_loss / z_loss gradient scaling with TP > 1 by @deepakn94 :: PR: #5047
Update Claude copy workflow to enforce user restrictions and improve error messages by @balasaajay :: PR: #5117
Add advisory process group guidance to Claude reviews by @yashaswikarnati :: PR: #5111
build: cap pydantic<2.14 in transformer-engine dependency metadata by @cuichenx :: PR: #5125
fix(test): skip scalar-less tensorboard event files in resume checks by @ko3n1g :: PR: #5121
docs: fix contributor guide typo by @LeSingh1 :: PR: #4858
ci(unit-tests): split slow unit-test buckets over 15min SLA by @ko3n1g :: PR: #5133
Update MINOR version to 19 by @balasaajay :: PR: #5096
Fix Muon QKV split for gated attention by @Moozy23232 :: PR: #4728
Fix DSA indexer loss not averaged across micro-batches by @kaimo455 :: PR: #4070
Roll input IDs for MTP labels by @yfw :: PR: #3457
Adding blackwell tests by @shanmugamr1992 :: PR: #5113
Refactor: Move paged stashing Triton kernels by @Phlip79 :: PR: #5003
Relax atol for test_router_gating_linear router_dtype=torch.float32 by @adityasingh2400 :: PR: #4915
Fix incorrect inference metadata tensor dtypes by @santhnm2 :: PR: #4855
Disable TE cross entropy loss fusion by @mchrzanowski :: PR: #5115
fix(optimizer): gate ChainedOptimizer MXFP8 defer-sync on DDP-level overlap_param_gather by @ko3n1g :: PR: #4982
Pass TP group to unfused cross entropy by @yashaswikarnati :: PR: #5128
test(elastification): quarantine flaky test_gumbel_determinism as flaky_in_dev by @ko3n1g :: PR: #5156
ci(notify): mention mcore-oncall and philipp on critical CI events by @ko3n1g :: PR: #5152
Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill by @mathemakitten :: PR: #3509
ci: Disable a few gb200 test cases to support 2 branches. by @balasaajay :: PR: #5151
build: Switch DSv3 on H100 to HybridEP by @balasaajay :: PR: #5164
Add MTP acceptance rate metrics by @yfw :: PR: #3458
Nemotron Ultra config for ModelOpt examples by @jenchen13 :: PR: #5159
Make MTP / prefix cache stats persist for engine lifetime by @santhnm2 :: PR: #4101
Restore Greptile configuration by @Phlip79 :: PR: #5166
chore: bump _code_freeze workflow to v1.4.2 by @ko3n1g :: PR: #5132
ci: Remove docs build test in favor of release test by @chtruong814 :: PR: #5182
Move TE cross entropy guard to training args by @yaoyu-33 :: PR: #5162
Fix error in deepseek parser by @tdene :: PR: #5136
Clamp max_new_tokens in MInf to mirror vllm by @tdene :: PR: #5181
Fix logprob slicing for 0 generated token case by @santhnm2 :: PR: #5167
[Perf] Fold frozen linear dgrad matmul by @cuichenx :: PR: #5092
build: add managed = true to [tool.uv] by @kajalj22 :: PR: #5190
Stabilize GB200 inference perf tests against cold-start noise by @shanmugamr1992 :: PR: #5171
nvidia style guide audit for getting started folder by @megnvidia :: PR: #5168
AI aided audit for Nvidia Style guidance by @megnvidia :: PR: #5141
Avoid stat syscall in rerun result validation by @dimapihtar :: PR: #5107
fix(combined-1f1b): release loss-node input storage after combined backward by @Wohox :: PR: #4909
Enable selective recompute for norm_out in GDN layers by @xuantengh :: PR: #4715
fix(elastification): align with get_batch + utils refactors by @balasaajay :: PR: #5194
docs: Update Latest News in README.md by @sbhavani :: PR: #3790
Fix bug with Megatron-FSDP zero counter not working with decoupled gradients. by @cspades :: PR: #4802
ci: add smoke tests by @balasaajay :: PR: #5143
Add mtp_detach_heads config to detach MTP head inputs by @yfw :: PR: #3456
docs: fix install guide NGC container anchor by @Connor-XY :: PR: #5224
Fuse per-sequence AlltoAll into a unified one in GDN forward by @xuantengh :: PR: #4913
Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path by @yashaswikarnati :: PR: #5150
Fix CUDA IMA in fsdp_double_buffer when an FSDP unit's bucket doesn't fit the pool by @wujingyue :: PR: #4810
Add named layouts to HyperCommGrid for heterogeneous parallelism by @yashaswikarnati :: PR: #5148
Fix wgrad race condition when using double buffers. by @cspades :: PR: #5222
Move uneven DTensor distributed fixture to conftest by @wujingyue :: PR: #5237
Route bridge communicator cross-grid P2P through a dedicated process group by @yashaswikarnati :: PR: #5234
Fix test_split_tensor_along_last_dim to actually assert correctness by @lichenlu :: PR: #4710
Add optional group= to common_utils model/data-parallel reduction helpers by @yashaswikarnati :: PR: #5251
ci: Allow DCO check in merge queue and add DCO requirement to Contribution guide by @chtruong814 :: PR: #5278
Add MIMO hetero topology + distributed bootstrap (examples/mimo training-loop folder) by @yashaswikarnati :: PR: #5260
Remove checkpoint-time GPU cache reclaim workaround by @shurkat-nvidia :: PR: #5170
Remove duplicate nccl_allocator import by @returnL :: PR: #5057
fix(ci): resolve t5 dataloader stall + GRPO cudagraph-memory regression (CI-validated) by @Connor-XY :: PR: #5280
Fix Dockerfile warnings by @janEbert :: PR: #4856
Fix fused MLA delayed weight grad hooks by @sraman-rgb :: PR: #5273
ci: limit retries on unsuccessful test launches by @balasaajay :: PR: #5275
Thread pg_collection into get_model DDP bucket sizing by @yashaswikarnati :: PR: #5250
Enable non-deterministic results in model configuration for nemotron tests by @balasaajay :: PR: #5239
Stabilize hybrid nanov3 gb200 perf by @shanmugamr1992 :: PR: #5295
Clip mtp grads separately when mtp_detach_heads=True by @yfw :: PR: #4116
Thread pg_collection into train_step reductions by @yashaswikarnati :: PR: #5259
Offline Logits-Based Knowledge Distillation by @AAnoosheh :: PR: #5019
Handle None values in sampling parameters by @tdene :: PR: #5300
Allow for pre-bound socket to be passed in server by @tdene :: PR: #5301
Add moe loss normalization for RL SFT by @pthombre :: PR: #3956
Add code owners for optimizer-related files by @janEbert :: PR: #5297
Fix EP=1 inference by allocating buffers anyway by @mathemakitten :: PR: #5233
Fix crash due to tool call at sequence length by @tdene :: PR: #5302
Inference: Cudagraph-aware admission gating in prefill scheduler by @mathemakitten :: PR: #4870
Account for reasoning token stripping by @tdene :: PR: #5313
Thread pg_collection through wrap_model_chunks_with_ddp by @yashaswikarnati :: PR: #5328
Fix LatentMoE theoretical memory estimate by @Wong4j :: PR: #5145
Add zstandard package to Docker LTS requirements. Fix nightly failures by @balasaajay :: PR: #5347
Thread MIMO support through the stock training loop (schedule + optimizer) by @yashaswikarnati :: PR: #5333
ci: default functional test time limit to 4h for release/weekly scopes by @ko3n1g :: PR: #5360
Fix memory leak with log_max_attention_logit (#4699) by @asolergi-nv :: PR: #5067
Clean up pretrain_gpt.py and pretrain_hybrid.py formatting and remove module globals by @ilml :: PR: #5351
Add full model cuda graph support for MTP inference by @santhnm2 :: PR: #4950
Expand the Mamba prefix caching memory safety check to include scratch space buffers by @santhnm2 :: PR: #5348
Make Megatron RL only materialize last token logit by @tdene :: PR: #4551
Profiling by @jalbericiola :: PR: #3110
Support fused MLA QKV checkpoint reload by @sraman-rgb :: PR: #5310
Add minimal DBuffer implementation by @wujingyue :: PR: #4835
[split 1/4] Fix packed THD RoPE under CP by @HollowMan6 :: PR: #5243
Document agent PR commit sign-off and signing by @wujingyue :: PR: #5381
Remove unused distributed pytest markers by @wujingyue :: PR: #5380
[feat] Support fine-grained activation offloading in fused group mlp by @lhb8125 :: PR: #5082
Thread tensor-parallel group into the RADIO patch embedder by @yashaswikarnati :: PR: #5371
Add MimoModel.zero_grad_buffer delegating to active DDP submodules by @yashaswikarnati :: PR: #5372
[split 3/4] Refactor absorbed MLA projection handling by @HollowMan6 :: PR: #5245
ci: Remove sync skills workflow by @chtruong814 :: PR: #5091
Add flaky marker to fine-grained activation offloading test (#5350) by @balasaajay :: PR: #5368
Revert "Remove checkpoint-time GPU cache reclaim workaround (#5170)" by @balasaajay :: PR: #5366
Update goldens for weekly tests after pytorch and TE bumps. by @balasaajay :: PR: #5399
Add MIMO runtime setup: per-role RNG seeding and DDP wrapping by @yashaswikarnati :: PR: #5285
Add --mamba-training-ssm-states-dtype argument by @tdene :: PR: #5309
Fix Mamba prefix match for chunked prefill by @lmcafee-nvidia :: PR: #4758
Disag MR2: Refit into multiple destination pools and tied-embedding + UVM fixes by @wdykas :: PR: #5187
Disag MR1: Add inference shard specs and pg-collection building by @wdykas :: PR: #5186
Support the MIMO cross-grid path in training loop by @yashaswikarnati :: PR: #5373
Stabilize hybrid_2b GB200 perf test against run-to-run noise by @shanmugamr1992 :: PR: #5364
Consistent oncall schedule by @Phlip79 :: PR: #5404
Disag MR3: Add heterogeneous KV/Mamba reshard planners by @wdykas :: PR: #5188
Add RADIO vision encoder wrapper for MIMO example by @yashaswikarnati :: PR: #5397
Clean up MTP inference control flow by @santhnm2 :: PR: #5418
Add MIMO dual gradient finalization (colocated + non-colocated) by @yashaswikarnati :: PR: #5286
Add RL rollout submission and consumption granularity controls by @lauradang :: PR: #5306
Add --functional-test-name to trigger_internal_ci by @ko3n1g :: PR: #5449
Rename CP batch helpers to describe balancing granularity by @deepakn94 :: PR: #5403
build: point flash_mla at the nv_dev branch by @ko3n1g :: PR: #5448
Add logprobs_mode (raw/processed) to inference config by @tdene :: PR: #5419
ci: Set test_save_verify_integrity_manifest_directly as flaky by @chtruong814 :: PR: #5468
Remove DBuffer mesh axis validation by @wujingyue :: PR: #5441
Add hetero grid args and MoE process groups for MIMO example by @yashaswikarnati :: PR: #5375
Support SWA and sink attention in dynamic inference by @cuichenx :: PR: #5249
feat(inference): default use_coordinator to True in high-level APIs by @shanmugamr1992 :: PR: #5326
Support HybridModel feature specs in ModelOpt by @Phlip79 :: PR: #5354
Add experimental Megatron-FSDP fully_shard implementation by @wujingyue :: PR: #5387
Add inference functions to support MCore-/MBridge- training refactor and remove legacy modelbuilder functions by @shanmugamr1992 :: PR: #5169
ci: launch GB200 unit tests via launch_on_gb200 marker by @ko3n1g :: PR: #5477
build: install flash_mla from source in the CI image by @ko3n1g :: PR: #5481
[split 2/4] Scale DSA indexer loss in pipeline schedules by @HollowMan6 :: PR: #5244
ci: check megatron.training imports in installation test by @ko3n1g :: PR: #5458
Fix merges_file kwarg name in HuggingFaceTokenizer by @muyihao :: PR: #5406
Clean up training.py module header (dedupe + reorganize imports/globals) by @ilml :: PR: #5469
Automated community request assignment by @Phlip79 :: PR: #5147
Thread process groups through training checkpoint paths by @yashaswikarnati :: PR: #5486
Narrow oncall responsibilities by @Phlip79 :: PR: #5490
Add MIMO forward step and per-token loss for hetero training by @yashaswikarnati :: PR: #5376
Add Nemotron6-MoE VLM model provider for MIMO example by @yashaswikarnati :: PR: #5374
ci: auto-retry test-data download in container-build job by @ko3n1g :: PR: #5498
Force RL inference to CP=1 by @tdene :: PR: #5423
Merge cu_seqlens across micro-batch for THD attention by @deepakn94 :: PR: #5454
[split 4/4] Enable DSA CP and THD hooks by @HollowMan6 :: PR: #5246
Fix fused MLA down projection with tensor parallelism by @sraman-rgb :: PR: #5383
Fix NameError in is_flashinfer_min_version when check_equality=False by @adityasingh2400 :: PR: #4961
Add hybrid FSDP unit module support by @Phlip79 :: PR: #4329
fix: set DATA_PATH for moe-dynamic-inference recipe by @ko3n1g :: PR: #5506
Add --qad-train-target {base|mtp|both} for QAD / MTP QAT (frozen-base, frozen-MTP, or co-train) by @yeyu-nvidia :: PR: #4785
[Main] Generalized fix for mxfp8 param gather by @zhongbozhu :: PR: #5236
test: restore G/G + lag=19 for gpt_grpo_tp4_pp1_dp2_8b throughput tests by @lauradang :: PR: #5514
ci: cache-from a single coherent buildcache donor by @ko3n1g :: PR: #5509
Add CUDA graph training iteration test by @wujingyue :: PR: #5417
ci: Use GB300 for Github CI tests by @chtruong814 :: PR: #5520
ci: pin HF_HUB_CACHE to bind-mounted cache for gpt-oss-20b inference test by @ko3n1g :: PR: #5512
Add inter-document attention masking to GPTDataset by @deepakn94 :: PR: #5298
[CI] Fix gpt_dynamic_inference_tp2_pp2_ep2_gptoss_20b_swa tests by @asolergi-nv :: PR: #5527
Implement async scheduling for dynamic inference by @lmcafee-nvidia :: PR: #5453
build: bump transformer-engine to release_v2.16.post by @ko3n1g :: PR: #5517
Fix isort target Python version by @janEbert :: PR: #5567
Fix PR template typo by @janEbert :: PR: #5566
Deduplicate tensor-splitting utility by @anlthms :: PR: #5545
Add CI duties to oncall by @Phlip79 :: PR: #5510
Thread dp_cp/expt_dp process groups through checkpoint load path by @yashaswikarnati :: PR: #5579
Fix TEGroupedMLP pre-backward unshard in fine-grained FSDP hooks for … by @rapatel :: PR: #4990
[training migration] Finish ModelBuilder integration by @maanug-nv :: PR: #5516
Use NVIDIA inference credentials for Claude actions by @Phlip79 :: PR: #5589
Update PR instructions by @Phlip79 :: PR: #5592
Update mcore skill owners by @Phlip79 :: PR: #5586
Add /claude fix workflow for on-demand PR fixes by @Phlip79 :: PR: #4862
[Megatron-FSDP] MaxPoolAllocator for double-buffering hybrid architectures. by @cspades :: PR: #5462
fix(tensor_parallel): _reduce returns unreduced tensor for non-contig… by @Pearblossom-M :: PR: #5338
Add Auto Quantize in ModelOpt quantize example by @jenchen13 :: PR: #4821
[Main][feat] Support CUDA Graph capture offloading modules by @lhb8125 :: PR: #3697
E2E heterogenous non colocated MiMo training by @yashaswikarnati :: PR: #5602
Optimize memory usage of partial CUDA graphs by @jiemingz :: PR: #5451
Document stacked dependent PR handling in split PR skill by @wujingyue :: PR: #5496
Fix Claude reaction permissions by @Phlip79 :: PR: #5613
Fix smoke BERT/T5 test failures by @balasaajay :: PR: #5629
Add NCCL symmetric-memory staging to experimental FSDP by @wujingyue :: PR: #5440
Add smoke test notification functionality and update notify script by @balasaajay :: PR: #5631
Update golden value files for GPT-3 weekly by @balasaajay :: PR: #5459
Ignore contributor DCO failures in Claude fix by @Phlip79 :: PR: #5625
Pre-size the all-gather buffer for inference to max capacity by @santhnm2 :: PR: #5546
Scatter embeddings for sequence parallelism in standalone LM forwards by @kevalmorabia97 :: PR: #5628
Fix inter-document masking crash and NaNs with TP > 1 and micro_batch_size > 1 by @deepakn94 :: PR: #5635
add safe version of numpy.load by @dimapihtar :: PR: #5500
Fuse shared expert MLP with grouped GEMM by @sraman-rgb :: PR: #5604
MoE routing analysis and metrics capture by @mathemakitten :: PR: #5220
Add cspades to oncall rotation by @Phlip79 :: PR: #5695
Add microbatch context helper by @wujingyue :: PR: #5652
ci: Update test configurations to unify legacy scope names by @balasaajay :: PR: #5316
Fix Torch FSDP2 crash: add force_all_reduce kwarg to base finish_grad_sync by @factnn :: PR: #4953
Separate mFSDP v2 unit tests by @wujingyue :: PR: #5640
test(determinism): add determinism tests by @ZhiyuLi-Nvidia :: PR: #5041
deprecate common strategy by @dimapihtar :: PR: #5160
ci: revert unify legacy scope names (#5316) by @ko3n1g :: PR: #5709
Normalize CRLF in Claude fix commands by @Phlip79 :: PR: #5712
Remove some barriers in save_checkpoint_and_time by @shurkat-nvidia :: PR: #5557
Add FSDP NVTX annotations by @wujingyue :: PR: #5704
NCCL EP support by @YangFei1990 :: PR: #5129
Update base image to nvcr.io/nvidia/pytorch:26.06-py3 by @balasaajay :: PR: #5632
Refactor RL rollout pipeline by @lauradang :: PR: #5491
[2/2] Wiring cuDNN fused DSA kernels support with THD, CP and IndexShare (GLM5.2) by @HollowMan6 :: PR: #5099
Triton kernels - avoid recompilation and autotuning in prod by @sidsingh-nvidia :: PR: #5608
Add NeMo Transformer audio encoder model by @yqwangustc :: PR: #5565
remove deprecated modules from core/dist_checkpointing by @dimapihtar :: PR: #5134
fix(fsdp): import os in safe_get_rank fallback by @fallintoplace :: PR: #4959
Add forward all-gather overlap by @wujingyue :: PR: #5513
Fix seq_load_balancing loss with inter-document masking and MBS > 1 by @deepakn94 :: PR: #5696
Avoid X11 master port default by @guihong-nv :: PR: #5299
Fix infinite recursion in abstract tokenizer special-id property aliases by @asadbekXodjayev :: PR: #5445
Set Bert TE spec q/k_layernorm to None by @bbuschkaemper :: PR: #5687
Set is_first_microbatch when quant_recipe is configured by @yezhengmao1 :: PR: #5642
Assign BERT CODEOWNERS to GPT team by @Phlip79 :: PR: #5746
Remove use of exec_module by @jon-barker :: PR: #5744
Short-circuit condition to avoid copying from GPU memory in ChainedOptimizer by @filaretov :: PR: #5623
Increase Megatron-FSDP overlap test dim to 8192 for reliable overlap by @wujingyue :: PR: #5770
Inference: Add profile endpoints to chat completions. by @sidsingh-nvidia :: PR: #5611
Inference: Do not route pad/dummy tokens to any expert by @sidsingh-nvidia :: PR: #4922
Inference: Add load aware routing to prefix caching. by @sidsingh-nvidia :: PR: #5607
fix(clip_grads): handle empty grads_for_norm in inf-norm and p-norm paths by @Mattral :: PR: #5530
test(gpt): AUT-830 mark tp1_pp4_vp1_resume_torch_decoupled_lr flaky on h100 by @svcnemo-autobot :: PR: #5801
Inference: Bring chat completions API inline with vllm/official openAI spec by @sidsingh-nvidia :: PR: #5276
Various ModelOpt fixes: QAD test for CICD, use model builder config instead of model provider, allow loading teacher ckpt independently of student by @AAnoosheh :: PR: #4520
build: Update Transformer Engine to 2.17 by @balasaajay :: PR: #5680
Mamba prefix caching fixes by @santhnm2 :: PR: #5502
Exercise nested MFSDP CUDA graph capture by @wujingyue :: PR: #5796
Implement Quantile Balancing in MoE by @Mellonta :: PR: #5349
Unset NCCL overrides for MFSDP v2 tests by @wujingyue :: PR: #5794
Fix MegatronFSDP root module hook dispatch by @wujingyue :: PR: #5808
Pin cudnn-fe and cuTeDSL version by @balasaajay :: PR: #5812
Inference: Add the nemotron_v3 reasoning parser by @sidsingh-nvidia :: PR: #5634
Missing moe_router_dtype causes unexpected downcast in ModelOpt example by @jinhangchoi :: PR: #5810
chore(tests): AUT-851 move NCCL defaults from run_ci_test.sh to conftest by @svcnemo-autobot :: PR: #5826
Avoid FSDP unit terminology in MFSDP v2 by @wujingyue :: PR: #5793
[refactor] Common combined-1F1B schedule-plan base (1/4 of #4798) by @Connor-XY :: PR: #4941
Fix configured norm epsilon in MambaLayer by @shanhaoli :: PR: #5750
Set num_splits to 0 for FA4 inference by @santhnm2 :: PR: #5804
ci: Enhance nightly/mr/weekly error reporting by @balasaajay :: PR: #5831
feat(docker): Add NCCL installation script and install NCCL 2.30.4 by @balasaajay :: PR: #5815
IMA fix by making the copy of book keeping buffer to GPU blocking by @shanmugamr1992 :: PR: #5715
Add GPTModel to HybridModel migration guide by @Phlip79 :: PR: #5698
Pair frozen FSDP backward hooks by @wujingyue :: PR: #5710
Clarify NVIDIA email signing guidance by @wujingyue :: PR: #5699
Overlap FSDP communication with compute by @wujingyue :: PR: #5719
chore(skills): add Regent Open Plugin manifest by @ko3n1g :: PR: #5840
chore(skills): remove Open Plugin manifest (superseded) by @ko3n1g :: PR: #5842
return prefix cache hits data from the chat completions api by @sidsingh-nvidia :: PR: #5609
Refactor data parallel coordinator to enable modular handlers by @santhnm2 :: PR: #5550
Pass device IDs to cleanup barrier by @wujingyue :: PR: #5702
Fix averaging for MoE z-loss metric tracking by @Marks101 :: PR: #3199
test(mfsdp): AUT-881 mark test_overlaps_communication_and_compute flaky by @svcnemo-autobot :: PR: #5848
Inference: Extend default cuda-graph coverage to 512 tokens by @sidsingh-nvidia :: PR: #5797
Fix for sequence-level aux MoE loss being dependent on batch size by @OlegSudakov :: PR: #5798
Allow parameterless FSDP root modules by @wujingyue :: PR: #5711
Fix issue where parameter groups with different min/max LRs get overridden at checkpoint load time by @jstjohn :: PR: #4705
fix bug where Gemma4 is not working with recompute_granularity = "full" by @warpuv :: PR: #5324
Avoid extra MFSDP v2 model-weight sync memcpy by @wujingyue :: PR: #5834
[experimental] Add experimental/agent_compose placeholder with preview pointer by @ISEEKYAN :: PR: #5639
Stabilize mfsdp_v2 overlap test by enlarging the model by @wujingyue :: PR: #5846
Print important dependencies by @gautham-kollu :: PR: #5814
Test zero-CTA copy-engine all-gather by @wujingyue :: PR: #5858
Route Lion through DistributedOptimizer and support single-moment checkpointing by @deepakn94 :: PR: #5742
Fix broken remove_sharded_tensors public API and re-enable its unit test by @huthvincent :: PR: #5759
Make the model larger and higher mb size to make reduce flakiness by @gautham-kollu :: PR: #5816
Update active oncall to Phlip79 this week by @Phlip79 :: PR: #5896
Overlap async scheduling phases by @lmcafee-nvidia :: PR: #5549
ci: integrate nemo-ci-triage with linear issues management for gitlab failures by @balasaajay :: PR: #5881
Reduce boilerplate around MultiStorageClient feature checks by @Randl :: PR: #5269
Add NeMo waveform audio processor (data-side feature extractor) by @yqwangustc :: PR: #5570
Support HSDP deferred DP-outer gradient reduction by @Achyuthan-S :: PR: #5743
Add fully_shard_optimizer for mixed-precision FSDP by @wujingyue :: PR: #5411
Stabilize perf warmup by @Phlip79 :: PR: #5913
Add compatibility between training CGs and CP>1 by @tdene :: PR: #5894
Log app_finish_time and app_train_loop_finish_time on early-exit path by @aliardaeker :: PR: #5864
Test mFSDP v2 overlap with default and symmetric memory by @wujingyue :: PR: #5859
fix: Harden Claude GitHub workflows by @chtruong814 :: PR: #5408
Reuse profiler helpers in mFSDP v2 symmetric memory tests by @wujingyue :: PR: #5873
Refresh BERT H100 golden values by @Phlip79 :: PR: #5953
Reduce MimoOptimizer update-success across the world for cross-grid consensus by @yashaswikarnati :: PR: #5331
fix(resharding): stabilize NVSHMEM refit copy service by @wdykas :: PR: #5915
[Main] Numerical fix for FC2 expert bias scales when using use_transformer_engine_op_fuser by @zhongbozhu :: PR: #5850
ci: Integrates the latest config-driven nemo-ci-triage Slack and Linear workflow by @balasaajay :: PR: #5957
Inference: Do not let prompt tokens return from the engine, unless requested by @sidsingh-nvidia :: PR: #5918
[Main] Numerical fix for moe single grouped weight with fp8 fp4 primary weight and grad norm spikes by @zhongbozhu :: PR: #5487
fix(ci): AUT-957 support golden checks in merge queue by @svcnemo-autobot :: PR: #5989
Fix formatting error in qwen3_30b_a3b config by @jon-barker :: PR: #5978
Inference: Reduce mamba scratch space size by an order of magnitude. by @sidsingh-nvidia :: PR: #5863
Make LRU prefix caching eviction policy only evict child blocks by @santhnm2 :: PR: #5822
Fix TE grouped MLP fused main-grad setup by @Wong4j :: PR: #5209
Batch-invariant train/inference logprob parity by @wdykas :: PR: #5897
Move FSDP model weight sync to optimizer post-step by @wujingyue :: PR: #5949
fix: allow mtp_num_layers=0 with overlap_moe_expert_parallel_comm by @cuichenx :: PR: #5912
Port Multi-Latent Attention to HybridModel by @janEbert :: PR: #4452
docs(skills): clarify container::lts is the older LTS PyTorch base by @ko3n1g :: PR: #6008
test(hybrid): AUT-971 quarantine Nemotron QAD functional test by @svcnemo-autobot :: PR: #6013
[main] add thd sequence packing dispatcher support for main by @HaochenYuan :: PR: #5008
Add encoder prefetch for heterogeneous MIMO training by @yashaswikarnati :: PR: #5833
Use explicit process groups for dataloader checkpoints by @yashaswikarnati :: PR: #5988
Refit: local plan building, node-add cache key, and NIXL backend by @wdykas :: PR: #5872
[2/2] Add TileLang fused DSA kernels support with THD and CP & Clean up by @HollowMan6 :: PR: #5049
Fix FSDP2 SwiGLU checkpointing. by @cspades :: PR: #5714
Prevent FlashInfer sampling from running with CUDA graphs by @santhnm2 :: PR: #5791
rl: release G-submission gate slots on consumption instead of assembly by @lauradang :: PR: #5853
Enable DDP communication overlap for MIMO training by @yashaswikarnati :: PR: #5979
[Inference] Set different random seeds for each DP rank for generation. by @cspades :: PR: #5983
Inference: Optimized triton kernels to extract mamba states in prefix caching by @sidsingh-nvidia :: PR: #5866
Ensure Mamba prefix cache snapshots are recorded for multi-chunk prompts by @santhnm2 :: PR: #5952
ci(actions): AUT-977 retry transient log artifact uploads by @svcnemo-autobot :: PR: #6027
fix: add additional error checks for flaky failures by @balasaajay :: PR: #6029
fix(inference): AUT-980 disable fp8 primary weights in graph tests by @svcnemo-autobot :: PR: #6038
chore(deps): AUT-967 stabilize Transformer Engine 2.18 upgrade by @svcnemo-autobot :: PR: #5997
[feat] Generalized Tensor Parallelism (GTP) by @fanshiqing :: PR: #4967
Fix gradient-norm undercounting when using EP and TP by @philipcmonk :: PR: #5916
fix(inference): MCORE-536 report dropped prompt token lengths by @svcnemo-autobot :: PR: #6051
[GTP][Feat] Add one-block-ahead prefetch for GTP grouped-expert weights by @fanshiqing :: PR: #6057
chore(codeowners): AUT-1094 add GTP owners by @svcnemo-autobot :: PR: #6062
Deprecate GPTModel in favor of HybridModel by @Phlip79 :: PR: #5911
fix(dist-ckpt): AUT-1100 restore default strategy factories by @svcnemo-autobot :: PR: #6065
Fix CUDA graph correctness issues due to memory bugs by @jiemingz :: PR: #5975
refactor: extract and split common logic between GDN & GDN2 by @xuantengh :: PR: #5843
dist_ckpt: add --stream-ckpt-dequant to fix OOM on large FP8/MXFP8 loads with --fp8-param-gather by @asolergi-nv :: PR: #4451
Add load-time GPT-to-Hybrid checkpoint translation (#5675) by @guihong-nv :: PR: #5792
Populate dp process group in auto-built ProcessGroupCollection in pipeline schedules by @ilml :: PR: #5901
fix(cuda-graphs): MB-928 align DDP initialization with capture stream by @svcnemo-autobot :: PR: #6021
Enforce that the number of optimizer shards used in layout computation is the same used during the training iteration by @deepakn94 :: PR: #6048
NCCL EP zero copy by @YangFei1990 :: PR: #5735
Optimize unit metadata for fused shared experts by @sraman-rgb :: PR: #6053
build: AUT-1117 serialize uv dependency installation by @svcnemo-autobot :: PR: #6090
Extend dynamic inference asynchronous scheduling support by @lmcafee-nvidia :: PR: #5939
Add support for non-Gym multi-turn environments by @tdene :: PR: #5312
Correct prefix-caching ref-count accounting by @tdene :: PR: #6047
send pg group for distributed checkpoint validation by @wdykas :: PR: #6092
cp: chore: update pytest version to 9.1.1 in pyproject.toml and uv.lock (6170) into core_r0.19.0 by @ko3n1g :: PR: #6181
cp: deps: Update black dependency to version 26.3.0 (6180) into core_r0.19.0 by @ko3n1g :: PR: #6189
cp: deps: Update urllib3 to version 2.7.0 (6182) into core_r0.19.0 by @ko3n1g :: PR: #6188
cp: test(optimizer): MCORE-560 cover MoE gradient zero counts (6050) into core_r0.19.0 by @ko3n1g :: PR: #6126
cp: test(inference): MCORE-561 trust FP8 metadata in DeepSeek checkpoints (6049) into core_r0.19.0 by @ko3n1g :: PR: #6138
chore: Update transformer-engine version to 2.18.0+cfda3834 by @balasaajay :: PR: #6220
Update Nemotron 3 Super GB200 release config by @Phlip79 :: PR: #6239
cp: Update DEEPEP_COMMIT in Dockerfile.ci.dev to de0dd1185142c727b9c118a by @svcnemo-autobot :: PR: #6281
build: AUT-1353 bump transformer-engine to 2.17.1 by @svcnemo-autobot :: PR: #6291
cp: set weights_only=True (6150) into core_r0.19.0 by @ko3n1g :: PR: #6351
docs: Update docs version for 0.19.0 by @chtruong814 :: PR: #6537
cp: Update release notes page (6520) into core_r0.19.0 by @Phlip79 :: PR: #6592
beep boop 🤖: Bumping Megatron Core to v0.19.1 by @nvidia-megatron-lm-release-bot[bot] :: PR: #6670
Add separate mtp_grad_scale_func for MTP loss scaling (#3459) by @yfw
[training migration] Migrate GPT builder (#4741) by @maanug-nv
Make Mamba conv params direct mixer params (#4899) by @wujingyue
Update oncall reviewer assignment (#5093) by @Phlip79
Pass explicit process groups to hybrid logging (#4781) by @yashaswikarnati
Clean up top-level repository files (#5097) by @Phlip79
[main] fix(moe): Fix several bugs for DSA rope and spec. (#3026) by @yuzhongw-nvidia
Move MIMO unit tests into models/mimo (#5063) by @yashaswikarnati
test: update DeepSeek FSDP2 GB200 memory golden (#5094) by @wujingyue
Fix GDN DTensor splitting for FSDP checkpointing (#4843) by @conver334
Remove DeepEP hardware limit check (#4846) by @janEbert
Update transformer-engine dependency to revision 4220403 (#5112) by @balasaajay
ci: make CI resilient to pip/uv network timeouts (#5118) by @ko3n1g
ci: treat docker container-removal conflict as flaky (#5120) by @ko3n1g
Fix MoE aux_loss / z_loss gradient scaling with TP > 1 (#5047) by @deepakn94
Update Claude copy workflow to enforce user restrictions and improve error messages (#5117) by @balasaajay
Add advisory process group guidance to Claude reviews (#5111) by @yashaswikarnati
build: cap pydantic<2.14 in transformer-engine dependency metadata (#5125) by @cuichenx
fix(test): skip scalar-less tensorboard event files in resume checks (#5121) by @ko3n1g
docs: fix contributor guide typo (#4858) by @LeSingh1
ci(unit-tests): split slow unit-test buckets over 15min SLA (#5133) by @ko3n1g
Fix Muon QKV split for gated attention (#4728) by @Moozy23232
Fix DSA indexer loss not averaged across micro-batches (#4070) by @kaimo455
Update MINOR version to 19 (#5096) by @balasaajay
Roll input IDs for MTP labels (#3457) by @yfw
Refactor: Move paged stashing Triton kernels (#5003) by @Phlip79
Adding blackwell tests (#5113) by @shanmugamr1992
Relax atol for test_router_gating_linear router_dtype=torch.float32 (#4915) by @adityasingh2400
Fix incorrect inference metadata tensor dtypes (#4855) by @santhnm2
Disable TE cross entropy loss fusion (#5115) by @mchrzanowski
Pass TP group to unfused cross entropy (#5128) by @yashaswikarnati
fix(optimizer): gate ChainedOptimizer MXFP8 defer-sync on DDP-level overlap_param_gather (#4982) by @ko3n1g
test(elastification): quarantine flaky test_gumbel_determinism as flaky_in_dev (#5156) by @ko3n1g
ci(notify): mention mcore-oncall and philipp on critical CI events (#5152) by @ko3n1g
Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill (#3509) by @mathemakitten
ci: Disable a few gb200 test cases to support 2 branches. (#5151) by @balasaajay
Add MTP acceptance rate metrics (#3458) by @yfw
build: Switch DSv3 on H100 to HybridEP (#5164) by @balasaajay
Nemotron Ultra config for ModelOpt examples (#5159) by @jenchen13
Make MTP / prefix cache stats persist for engine lifetime (#4101) by @santhnm2
Restore Greptile configuration (#5166) by @Phlip79
chore: bump _code_freeze workflow to v1.4.2 (#5132) by @ko3n1g
ci: Remove docs build test in favor of release test (#5182) by @chtruong814
Move TE cross entropy guard to training args (#5162) by @yaoyu-33
Fix error in deepseek parser (#5136) by @tdene
Fix logprob slicing for 0 generated token case (#5167) by @santhnm2
[Perf] Fold frozen linear dgrad matmul (#5092) by @cuichenx
Clamp max_new_tokens in MInf to mirror vllm (#5181) by @tdene
build: add managed = true to [tool.uv] (#5190) by @kajalj22
Stabilize GB200 inference perf tests against cold-start noise (#5171) by @shanmugamr1992
nvidia style guide audit for getting started folder (#5168) by @megnvidia
AI aided audit for Nvidia Style guidance (#5141) by @megnvidia
Avoid stat syscall in rerun result validation (#5107) by @dimapihtar
Enable selective recompute for norm_out in GDN layers (#4715) by @xuantengh
fix(elastification): align with get_batch + utils refactors (#5194) by @balasaajay
fix(combined-1f1b): release loss-node input storage after combined backward (#4909) by @Wohox
chore(beep boop 🤖): Bump (main) (2026-06-08) by @github-actions[bot]
docs: Update Latest News in README.md (#3790) by @sbhavani
Fix bug with Megatron-FSDP zero counter not working with decoupled gradients. (#4802) by @cspades
ci: add smoke tests (#5143) by @balasaajay
Add mtp_detach_heads config to detach MTP head inputs (#3456) by @yfw
docs: fix install guide NGC container anchor (#5224) by @Connor-XY
Fuse per-sequence AlltoAll into a unified one in GDN forward (#4913) by @xuantengh
Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path (#5150) by @yashaswikarnati
Fix CUDA IMA in fsdp_double_buffer when an FSDP unit's bucket doesn't fit the pool (#4810) by @wujingyue
Add named layouts to HyperCommGrid for heterogeneous parallelism (#5148) by @yashaswikarnati
Fix wgrad race condition when using double buffers. (#5222) by @cspades
Move uneven DTensor distributed fixture to conftest (#5237) by @wujingyue
Route bridge communicator cross-grid P2P through a dedicated process group (#5234) by @yashaswikarnati
Fix test_split_tensor_along_last_dim to actually assert correctness (#4710) by @lichenlu
Add optional group= to common_utils model/data-parallel reduction helpers (#5251) by @yashaswikarnati
Add MIMO hetero topology + distributed bootstrap (examples/mimo training-loop folder) (#5260) by @yashaswikarnati
ci: Allow DCO check in merge queue and add DCO requirement to Contribution guide (#5278) by @chtruong814
Remove checkpoint-time GPU cache reclaim workaround (#5170) by @shurkat-nvidia
Remove duplicate nccl_allocator import (#5057) by @returnL
fix(ci): resolve t5 dataloader stall + GRPO cudagraph-memory regression (CI-validated) (#5280) by @Connor-XY
Fix Dockerfile warnings (#4856) by @janEbert
Fix fused MLA delayed weight grad hooks (#5273) by @sraman-rgb
ci: limit retries on unsuccessful test launches (#5275) by @balasaajay
Thread pg_collection into get_model DDP bucket sizing (#5250) by @yashaswikarnati
Enable non-deterministic results in model configuration for nemotron tests (#5239) by @balasaajay
Stabilize hybrid nanov3 gb200 perf (#5295) by @shanmugamr1992
Clip mtp grads separately when mtp_detach_heads=True (#4116) by @yfw
Thread pg_collection into train_step reductions (#5259) by @yashaswikarnati
Handle None values in sampling parameters (#5300) by @tdene
Allow for pre-bound socket to be passed in server (#5301) by @tdene
Offline Logits-Based Knowledge Distillation (#5019) by @AAnoosheh
Add moe loss normalization for RL SFT (#3956) by @pthombre
Add code owners for optimizer-related files (#5297) by @janEbert
Fix EP=1 inference by allocating buffers anyway (#5233) by @mathemakitten
Fix crash due to tool call at sequence length (#5302) by @tdene
Inference: Cudagraph-aware admission gating in prefill scheduler (#4870) by @mathemakitten
Account for reasoning token stripping (#5313) by @tdene
Thread pg_collection through wrap_model_chunks_with_ddp (#5328) by @yashaswikarnati
chore(beep boop 🤖): Bump (main) (2026-06-15) by @github-actions[bot]
Fix LatentMoE theoretical memory estimate (#5145) by @Wong4j
Add zstandard package to Docker LTS requirements. Fix nightly failures (#5347) by @balasaajay
Thread MIMO support through the stock training loop (schedule + optimizer) (#5333) by @yashaswikarnati
ci: default functional test time limit to 4h for release/weekly scopes (#5360) by @ko3n1g
Fix memory leak with log_max_attention_logit (#4699) (#5067) by @asolergi-nv
Clean up pretrain_gpt.py and pretrain_hybrid.py formatting and remove module globals (#5351) by @ilml
Add full model cuda graph support for MTP inference (#4950) by @santhnm2
Expand the Mamba prefix caching memory safety check to include scratch space buffers (#5348) by @santhnm2
Make Megatron RL only materialize last token logit (#4551) by @tdene
Profiling (#3110) by @jalbericiola
Support fused MLA QKV checkpoint reload (#5310) by @sraman-rgb
Document agent PR commit sign-off and signing (#5381) by @wujingyue
Add minimal DBuffer implementation (#4835) by @wujingyue
[split 1/5] Fix packed THD RoPE under CP (#5243) by @HollowMan6
Remove unused distributed pytest markers (#5380) by @wujingyue
[feat] Support fine-grained activation offloading in fused group mlp (#5082) by @lhb8125
Thread tensor-parallel group into the RADIO patch embedder (#5371) by @yashaswikarnati
Add MimoModel.zero_grad_buffer delegating to active DDP submodules (#5372) by @yashaswikarnati
[split 3/5] Refactor absorbed MLA projection handling (#5245) by @HollowMan6
ci: Remove sync skills workflow (#5091) by @chtruong814
Add flaky marker to fine-grained activation offloading test (#5350) (#5368) by @balasaajay
Revert "Remove checkpoint-time GPU cache reclaim workaround (#5170)" (#5366) by @balasaajay
Update goldens for weekly tests after pytorch and TE bumps. (#5399) by @balasaajay
Add MIMO runtime setup: per-role RNG seeding and DDP wrapping (#5285) by @yashaswikarnati
Add --mamba-training-ssm-states-dtype argument (#5309) by @tdene
chore(beep boop 🤖): Bump (main) (2026-06-22) by @github-actions[bot]
Fix Mamba prefix match for chunked prefill (#4758) by @lmcafee-nvidia
Disag MR2: Refit into multiple destination pools and tied-embedding + UVM fixes (#5187) by @wdykas
Disag MR1: Add inference shard specs and pg-collection building (#5186) by @wdykas
Support the MIMO cross-grid path in training loop (#5373) by @yashaswikarnati
Stabilize hybrid_2b GB200 perf test against run-to-run noise (#5364) by @shanmugamr1992
Consistent oncall schedule (#5404) by @Phlip79
Disag MR3: Add heterogeneous KV/Mamba reshard planners (#5188) by @wdykas
Add RADIO vision encoder wrapper for MIMO example (#5397) by @yashaswikarnati
Clean up MTP inference control flow (#5418) by @santhnm2
Add MIMO dual gradient finalization (colocated + non-colocated) (#5286) by @yashaswikarnati
Add RL rollout submission and consumption granularity controls (#5306) by @lauradang
Add --functional-test-name to trigger_internal_ci (#5449) by @ko3n1g
Rename CP batch helpers to describe balancing granularity (#5403) by @deepakn94
build: point flash_mla at the nv_dev branch (#5448) by @ko3n1g
Add logprobs_mode (raw/processed) to inference config (#5419) by @tdene
Remove DBuffer mesh axis validation (#5441) by @wujingyue
Support SWA and sink attention in dynamic inference (#5249) by @cuichenx
Add hetero grid args and MoE process groups for MIMO example (#5375) by @yashaswikarnati
ci: Set test_save_verify_integrity_manifest_directly as flaky (#5468) by @chtruong814
feat(inference): default use_coordinator to True in high-level APIs (#5326) by @shanmugamr1992
Support HybridModel feature specs in ModelOpt (#5354) by @Phlip79
Add experimental Megatron-FSDP fully_shard implementation (#5387) by @wujingyue
Add inference functions to support MCore-/MBridge- training refactor and remove legacy modelbuilder functions (#5169) by @shanmugamr1992
ci: launch GB200 unit tests via launch_on_gb200 marker (#5477) by @ko3n1g
build: install flash_mla from source in the CI image (#5481) by @ko3n1g
[split 2/4] Scale DSA indexer loss in pipeline schedules (#5244) by @HollowMan6
ci: check megatron.training imports in installation test (#5458) by @ko3n1g
Fix merges_file kwarg name in HuggingFaceTokenizer (#5406) by @muyihao
Automated community request assignment (#5147) by @Phlip79
Clean up training.py module header (dedupe + reorganize imports/globals) (#5469) by @ilml
Thread process groups through training checkpoint paths (#5486) by @yashaswikarnati
Narrow oncall responsibilities (#5490) by @Phlip79
Add MIMO forward step and per-token loss for hetero training (#5376) by @yashaswikarnati
Add Nemotron6-MoE VLM model provider for MIMO example (#5374) by @yashaswikarnati
ci: auto-retry test-data download in container-build job (#5498) by @ko3n1g
Force RL inference to CP=1 (#5423) by @tdene
Merge cu_seqlens across micro-batch for THD attention (#5454) by @deepakn94
[split 4/4] Enable DSA CP and THD hooks (#5246) by @HollowMan6
Fix fused MLA down projection with tensor parallelism (#5383) by @sraman-rgb
Fix NameError in is_flashinfer_min_version when check_equality=False (#4961) by @adityasingh2400
Add hybrid FSDP unit module support (#4329) by @Phlip79
fix: set DATA_PATH for moe-dynamic-inference recipe (#5506) by @ko3n1g
Add --qad-train-target {base|mtp|both} for QAD / MTP QAT (frozen-base, frozen-MTP, or co-train) (#4785) by @yeyu-nvidia
[Main] Generalized fix for mxfp8 param gather (#5236) by @zhongbozhu
ci: cache-from a single coherent buildcache donor (#5509) by @ko3n1g
Add CUDA graph training iteration test (#5417) by @wujingyue
test: restore G/G + lag=19 for gpt_grpo_tp4_pp1_dp2_8b throughput tests (#5514) by @lauradang
ci: Use GB300 for Github CI tests (#5520) by @chtruong814
ci: pin HF_HUB_CACHE to bind-mounted cache for gpt-oss-20b inference test (#5512) by @ko3n1g
Add inter-document attention masking to GPTDataset (#5298) by @deepakn94
[CI] Fix gpt_dynamic_inference_tp2_pp2_ep2_gptoss_20b_swa tests (#5527) by @asolergi-nv
Implement async scheduling for dynamic inference (#5453) by @lmcafee-nvidia
build: bump transformer-engine to release_v2.16.post (#5517) by @ko3n1g
Fix isort target Python version (#5567) by @janEbert
Fix PR template typo (#5566) by @janEbert
Deduplicate tensor-splitting utility (#5545) by @anlthms
Add CI duties to oncall (#5510) by @Phlip79
Thread dp_cp/expt_dp process groups through checkpoint load path (#5579) by @yashaswikarnati
Fix TEGroupedMLP pre-backward unshard in fine-grained FSDP hooks for … (#4990) by @rapatel
[training migration] Finish ModelBuilder integration (#5516) by @maanug-nv
Use NVIDIA inference credentials for Claude actions (#5589) by @Phlip79
Update PR instructions (#5592) by @Phlip79
Update mcore skill owners (#5586) by @Phlip79
Add /claude fix workflow for on-demand PR fixes (#4862) by @Phlip79
[Megatron-FSDP] MaxPoolAllocator for double-buffering hybrid architectures. (#5462) by @cspades
fix(tensor_parallel): _reduce returns unreduced tensor for non-contig… (#5338) by @Pearblossom-M
Add Auto Quantize in ModelOpt quantize example (#4821) by @jenchen13
[Main][feat] Support CUDA Graph capture offloading modules (#3697) by @lhb8125
E2E heterogenous non colocated MiMo training (#5602) by @yashaswikarnati
Optimize memory usage of partial CUDA graphs (#5451) by @jiemingz
Document stacked dependent PR handling in split PR skill (#5496) by @wujingyue
Fix Claude reaction permissions (#5613) by @Phlip79
Fix smoke BERT/T5 test failures (#5629) by @balasaajay
Add NCCL symmetric-memory staging to experimental FSDP (#5440) by @wujingyue
Add smoke test notification functionality and update notify script (#5631) by @balasaajay
Update golden value files for GPT-3 weekly (#5459) by @balasaajay
Ignore contributor DCO failures in Claude fix (#5625) by @Phlip79
chore(beep boop 🤖): Bump (main) (2026-07-06) by @github-actions[bot]
Pre-size the all-gather buffer for inference to max capacity (#5546) by @santhnm2
Scatter embeddings for sequence parallelism in standalone LM forwards (#5628) by @kevalmorabia97
Fix inter-document masking crash and NaNs with TP > 1 and micro_batch_size > 1 (#5635) by @deepakn94
add safe version of numpy.load (#5500) by @dimapihtar
Fuse shared expert MLP with grouped GEMM (#5604) by @sraman-rgb
MoE routing analysis and metrics capture (#5220) by @mathemakitten
Add cspades to oncall rotation (#5695) by @Phlip79
Add microbatch context helper (#5652) by @wujingyue
ci: Update test configurations to unify legacy scope names (#5316) by @balasaajay
Fix Torch FSDP2 crash: add force_all_reduce kwarg to base finish_grad_sync (#4953) by @factnn
Separate mFSDP v2 unit tests (#5640) by @wujingyue
test(determinism): add determinism tests (#5041) by @ZhiyuLi-Nvidia
deprecate common strategy (#5160) by @dimapihtar
ci: revert unify legacy scope names (#5316) (#5709) by @ko3n1g
Normalize CRLF in Claude fix commands (#5712) by @Phlip79
Remove some barriers in save_checkpoint_and_time (#5557) by @shurkat-nvidia
Add FSDP NVTX annotations (#5704) by @wujingyue
NCCL EP support (#5129) by @YangFei1990
Update base image to nvcr.io/nvidia/pytorch:26.06-py3 (#5632) by @balasaajay
Refactor RL rollout pipeline (#5491) by @lauradang
[2/2] Wiring cuDNN fused DSA kernels support with THD, CP and IndexShare (GLM5.2) (#5099) by @HollowMan6
Triton kernels - avoid recompilation and autotuning in prod (#5608) by @sidsingh-nvidia
Add NeMo Transformer audio encoder model (#5565) by @yqwangustc
remove deprecated modules from core/dist_checkpointing (#5134) by @dimapihtar
fix(fsdp): import os in safe_get_rank fallback (#4959) by @fallintoplace
Add forward all-gather overlap (#5513) by @wujingyue
Fix seq_load_balancing loss with inter-document masking and MBS > 1 (#5696) by @deepakn94
Avoid X11 master port default (#5299) by @guihong-nv
Fix infinite recursion in abstract tokenizer special-id property aliases (#5445) by @asadbekXodjayev
Set Bert TE spec q/k_layernorm to None (#5687) by @bbuschkaemper
Set is_first_microbatch when quant_recipe is configured (#5642) by @yezhengmao1
Assign BERT CODEOWNERS to GPT team (#5746) by @Phlip79
Remove use of exec_module (#5744) by @jon-barker
Short-circuit condition to avoid copying from GPU memory in ChainedOptimizer (#5623) by @filaretov
Increase Megatron-FSDP overlap test dim to 8192 for reliable overlap (#5770) by @wujingyue
Inference: Add profile endpoints to chat completions. (#5611) by @sidsingh-nvidia
Inference: Do not route pad/dummy tokens to any expert (#4922) by @sidsingh-nvidia
Inference: Add load aware routing to prefix caching. (#5607) by @sidsingh-nvidia
fix(clip_grads): handle empty grads_for_norm in inf-norm and p-norm paths (#5530) by @Mattral
test(gpt): AUT-830 mark tp1_pp4_vp1_resume_torch_decoupled_lr flaky on h100 (#5801) by @svcnemo-autobot
Delegate reasoning token retention to the chat template in multi-turn conversations (#5276) by @sidsingh-nvidia
Various ModelOpt fixes: QAD test for CICD, use model builder config instead of model provider, allow loading teacher ckpt independently of student (#4520) by @AAnoosheh
build: Update Transformer Engine to 2.17 (#5680) by @balasaajay
Mamba prefix caching fixes (#5502) by @santhnm2
Exercise nested MFSDP CUDA graph capture (#5796) by @wujingyue
Implement Quantile Balancing in MoE (#5349) by @Mellonta
Unset NCCL overrides for MFSDP v2 tests (#5794) by @wujingyue
Fix MegatronFSDP root module hook dispatch (#5808) by @wujingyue
Pin cudnn-fe and cuTeDSL version (#5812) by @balasaajay
Inference: Add the nemotron_v3 reasoning parser (#5634) by @sidsingh-nvidia
Missing moe_router_dtype causes unexpected downcast in ModelOpt example (#5810) by @jinhangchoi
Avoid FSDP unit terminology in MFSDP v2 (#5793) by @wujingyue
Fix configured norm epsilon in MambaLayer (#5750) by @shanhaoli
[refactor] Common combined-1F1B schedule-plan base (1/4 of #4798) (#4941) by @Connor-XY
chore(tests): AUT-851 move NCCL defaults from run_ci_test.sh to conftest (#5826) by @svcnemo-autobot
Set num_splits to 0 for FA4 inference (#5804) by @santhnm2
ci: Enhance nightly/mr/weekly error reporting (#5831) by @balasaajay
feat(docker): Add NCCL installation script and install NCCL 2.30.4 (#5815) by @balasaajay
IMA fix by making the copy of book keeping buffer to GPU blocking (#5715) by @shanmugamr1992
Add GPTModel to HybridModel migration guide (#5698) by @Phlip79
Pair frozen FSDP backward hooks (#5710) by @wujingyue
Clarify NVIDIA email signing guidance (#5699) by @wujingyue
Overlap FSDP communication with compute (#5719) by @wujingyue
chore(skills): add Regent Open Plugin manifest (#5840) by @ko3n1g
return prefix cache hits data from the chat completions api (#5609) by @sidsingh-nvidia
chore(skills): remove Open Plugin manifest (superseded) (#5842) by @ko3n1g
Refactor data parallel coordinator to enable modular handlers (#5550) by @santhnm2
Pass device IDs to cleanup barrier (#5702) by @wujingyue
Fix averaging for MoE z-loss metric tracking (#3199) by @Marks101
Inference: Extend default cuda-graph coverage to 512 tokens (#5797) by @sidsingh-nvidia
test(mfsdp): AUT-881 mark test_overlaps_communication_and_compute flaky (#5848) by @svcnemo-autobot
Fix for sequence-level aux MoE loss being dependent on batch size (#5798) by @OlegSudakov
Allow parameterless FSDP root modules (#5711) by @wujingyue
Fix issue where parameter groups with different min/max LRs get overridden at checkpoint load time (#4705) by @jstjohn
fix bug where Gemma4 is not working with recompute_granularity = "full" (#5324) by @warpuv
Avoid extra MFSDP v2 model-weight sync memcpy (#5834) by @wujingyue
[experimental] Add experimental/agent_compose placeholder with preview pointer (#5639) by @ISEEKYAN
Stabilize mfsdp_v2 overlap test by enlarging the model (#5846) by @wujingyue
Print important dependencies (#5814) by @gautham-kollu
Test zero-CTA copy-engine all-gather (#5858) by @wujingyue
Route Lion through DistributedOptimizer and support single-moment checkpointing (#5742) by @deepakn94
Fix broken remove_sharded_tensors public API and re-enable its unit test (#5759) by @huthvincent
Make the model larger and higher mb size to make reduce flakiness (#5816) by @gautham-kollu
chore(beep boop 🤖): Bump (main) (2026-07-20) by @github-actions[bot]
Update active oncall to Phlip79 this week (#5896) by @Phlip79
Overlap async scheduling phases (#5549) by @lmcafee-nvidia
ci: integrate nemo-ci-triage with linear issues management for gitlab failures (#5881) by @balasaajay
Reduce boilerplate around MultiStorageClient feature checks (#5269) by @Randl
Add NeMo waveform audio processor (data-side feature extractor) (#5570) by @yqwangustc
Support HSDP deferred DP-outer gradient reduction (#5743) by @Achyuthan-S
Add fully_shard_optimizer for mixed-precision FSDP (#5411) by @wujingyue
Stabilize perf warmup (#5913) by @Phlip79
Add compatibility between training CGs and CP>1 (#5894) by @tdene
Log app_finish_time and app_train_loop_finish_time on early-exit path (#5864) by @aliardaeker
Test mFSDP v2 overlap with default and symmetric memory (#5859) by @wujingyue
fix: Harden Claude GitHub workflows (#5408) by @chtruong814
chore: rotate oncall schedule by @github-actions[bot]
Reuse profiler helpers in mFSDP v2 symmetric memory tests (#5873) by @wujingyue
Reduce MimoOptimizer update-success across the world for cross-grid consensus (#5331) by @yashaswikarnati
Refresh BERT H100 golden values (#5953) by @Phlip79
fix(resharding): stabilize NVSHMEM refit copy service (#5915) by @wdykas
[Main] Numerical fix for FC2 expert bias scales when using use_transformer_engine_op_fuser (#5850) by @zhongbozhu
ci: Integrates the latest config-driven nemo-ci-triage Slack and Linear workflow (#5957) by @balasaajay
[Main] Numerical fix for moe single grouped weight with fp8 fp4 primary weight and grad norm spikes (#5487) by @zhongbozhu
Inference: Do not let prompt tokens return from the engine, unless requested (#5918) by @sidsingh-nvidia
Fix formatting error in qwen3_30b_a3b config (#5978) by @jon-barker
Make LRU prefix caching eviction policy only evict child blocks (#5822) by @santhnm2
Inference: Reduce mamba scratch space size by an order of magnitude. (#5863) by @sidsingh-nvidia
Fix TE grouped MLP fused main-grad setup (#5209) by @Wong4j
fix(ci): AUT-957 support golden checks in merge queue (#5989) by @svcnemo-autobot
Batch-invariant train/inference logprob parity (#5897) by @wdykas
Move FSDP model weight sync to optimizer post-step (#5949) by @wujingyue
fix: allow mtp_num_layers=0 with overlap_moe_expert_parallel_comm (#5912) by @cuichenx
Port Multi-Latent Attention to HybridModel (#4452) by @janEbert
[main] add thd sequence packing dispatcher support for main (#5008) by @HaochenYuan
docs(skills): clarify container::lts is the older LTS PyTorch base (#6008) by @ko3n1g
test(hybrid): AUT-971 quarantine Nemotron QAD functional test (#6013) by @svcnemo-autobot
Add encoder prefetch for heterogeneous MIMO training (#5833) by @yashaswikarnati
Use explicit process groups for dataloader checkpoints (#5988) by @yashaswikarnati
[2/2] Add TileLang fused DSA kernels support with THD and CP & Clean up (#5049) by @HollowMan6
Refit: local plan building, node-add cache key, and NIXL backend (#5872) by @wdykas
Fix FSDP2 SwiGLU checkpointing. (#5714) by @cspades
Prevent FlashInfer sampling from running with CUDA graphs (#5791) by @santhnm2
Enable DDP communication overlap for MIMO training (#5979) by @yashaswikarnati
rl: release G-submission gate slots on consumption instead of assembly (#5853) by @lauradang
Inference: Optimized triton kernels to extract mamba states in prefix caching (#5866) by @sidsingh-nvidia
[Inference] Set different random seeds for each DP rank for generation. (#5983) by @cspades
Ensure Mamba prefix cache snapshots are recorded for multi-chunk prompts (#5952) by @santhnm2
ci(actions): AUT-977 retry transient log artifact uploads (#6027) by @svcnemo-autobot
fix: add additional error checks for flaky failures (#6029) by @balasaajay
fix(inference): AUT-980 disable fp8 primary weights in graph tests (#6038) by @svcnemo-autobot
chore(deps): AUT-967 stabilize Transformer Engine 2.18 upgrade (#5997) by @svcnemo-autobot
[feat] Generalized Tensor Parallelism (GTP) (#4967) by @fanshiqing
Fix gradient-norm undercounting when using EP and TP (#5916) by @philipcmonk
fix(inference): MCORE-536 report dropped prompt token lengths (#6051) by @svcnemo-autobot
[GTP][Feat] Add one-block-ahead prefetch for GTP grouped-expert weights (#6057) by @fanshiqing
chore(codeowners): AUT-1094 add GTP owners (#6062) by @svcnemo-autobot
fix(dist-ckpt): AUT-1100 restore default strategy factories (#6065) by @svcnemo-autobot
Deprecate GPTModel in favor of HybridModel (#5911) by @Phlip79
Fix CUDA graph correctness issues due to memory bugs (#5975) by @jiemingz
Update copy-pr-bot.yaml [skip ci] by @github-actions[bot]
refactor: extract and split common logic between GDN & GDN2 (#5843) by @xuantengh
Add load-time GPT-to-Hybrid checkpoint translation (#5675) (#5792) by @guihong-nv
dist_ckpt: add --stream-ckpt-dequant to fix OOM on large FP8/MXFP8 loads with --fp8-param-gather (#4451) by @asolergi-nv
Populate dp process group in auto-built ProcessGroupCollection in pipeline schedules (#5901) by @ilml
fix(cuda-graphs): MB-928 align DDP initialization with capture stream (#6021) by @svcnemo-autobot
Enforce that the number of optimizer shards used in layout computation is the same used during the training iteration (#6048) by @deepakn94
NCCL EP zero copy (#5735) by @YangFei1990
Optimize unit metadata for fused shared experts (#6053) by @sraman-rgb
build: AUT-1117 serialize uv dependency installation (#6090) by @svcnemo-autobot
Extend dynamic inference asynchronous scheduling support (#5939) by @lmcafee-nvidia
Add support for non-Gym multi-turn environments (#5312) by @tdene
Correct prefix-caching ref-count accounting (#6047) by @tdene
send pg group for distributed checkpoint validation (#6092) by @wdykas
cp: chore: update pytest version to 9.1.1 in pyproject.toml and uv.lock (6170) into core_r0.19.0 (#6181) by @ko3n1g
cp: deps: Update black dependency to version 26.3.0 (6180) into core_r0.19.0 (#6189) by @ko3n1g
cp: deps: Update urllib3 to version 2.7.0 (6182) into core_r0.19.0 (#6188) by @ko3n1g
cp: test(optimizer): MCORE-560 cover MoE gradient zero counts (6050) into core_r0.19.0 (#6126) by @ko3n1g
cp: test(inference): MCORE-561 trust FP8 metadata in DeepSeek checkpoints (6049) into core_r0.19.0 (#6138) by @ko3n1g
chore: Update transformer-engine version to 2.18.0+cfda3834 (#6220) by @balasaajay
Update Nemotron 3 Super GB200 release config (#6239) by @Phlip79
cp: Update DEEPEP_COMMIT in Dockerfile.ci.dev to de0dd1185142c727b9c118a (#6281) by @svcnemo-autobot
build: AUT-1353 bump transformer-engine to 2.17.1 (#6291) by @svcnemo-autobot
cp: set weights_only=True (6150) into core_r0.19.0 (#6351) by @ko3n1g
chore(beep boop 🤖): Bump (core_r0.19.0) (2026-08-10) by @github-actions[bot]
docs: Update docs version for 0.19.0 (#6537) by @chtruong814
chore(beep boop 🤖): Bump (core_r0.19.0) (2026-08-17) by @github-actions[bot]
cp: Update release notes page (6520) into core_r0.19.0 (#6592) by @Phlip79
beep boop 🤖: Bumping Megatron Core to v0.19.1 [skip ci] by @github-actions[bot]

NVIDIA/Megatron-LM core_v0.19.0 NVIDIA Megatron Core 0.19.0 on GitHub

NVIDIA/Megatron-LM core_v0.19.0
NVIDIA Megatron Core 0.19.0

on GitHub