What's Changed
- Fix checkpoint rank selection for Ulysses sequence parallelism by @sfc-gh-thonguyen in #8226
- Return a copy from OnebitLamb.get_lamb_coeffs by @vineethsaivs in #8227
- Update version.txt after 0.19.5 release by @loadams in #8243
- Implement the documented per-param-group lists in OneCycle by @vineethsaivs in #8201
- Keep the elastic batch size within max_train_batch_size by @vineethsaivs in #8237
- [DeepCompile] Fix KeyError on frozen parameters in ZeRO-3 by @roycho96 in #8214
- [DeepCompile] Fix staticmethod handling on Python 3.9 by @baremetaldevx86 in #8240
- Upgrade pip in Python install-smoke jobs by @tohtana in #8247
- (1/2) Implementing Compiler Pass for AutoTP by @therealnaveenkamal in #8204
- Fix DeepCompile last use for unconsumed waits by @tohtana in #8254
- Add configurable sum gradient reduction by @zupengwang in #8232
- Add native (DeepNVMe) host-memory pinning backend for accelerators by @sfc-gh-truwase in #8211
- Add compile.offload_activation_pin_memory for DeepCompile activation offload by @sfc-gh-truwase in #8258
- Fix int32 overflow in Triton grouped-GEMM expert offset by @PKUWZP in #8261
- NCCL and other backend PG timeout 30min -> 10min by @stas00 in #8253
- Fix ZeRO-3 crash in AutoTP universal-checkpoint metadata by @PKUWZP in #8270
- Add CUDA graph support for HybridEngine generation by @PKUWZP in #8271
- Fix docstring Args entries that name a parameter the function does not take by @darkdi in #8223
- Keep parameter dtype through ZeRO-3 weight quantization by @adityasingh2400 in #8215
- [AutoTP] Complete uneven sharding and universal checkpoint support by @jinyouzhi in #8185
- Route FPDT and checkpoint writer pins through accelerator pin_memory by @sfc-gh-truwase in #8257
- Add fused triton kernel for swiglu by @hwchen2017 in #8244
- Make pass contracts cover the passes DeepCompile actually schedules; Renaming files by @pengdurice in #8251
- Non-reentrant activation checkpoint CPU offload by @sfc-gh-truwase in #8282
- Enable activation offloading by @pengdurice in #8255
- [typo] Add raise for RuntimeError by @jinyouzhi in #8281
- Fix ZeRO-1/2 with zero-sized parameters by @fwerkor in #8280
- docs: note async cpu_checkpointing perf and expandable_segments by @sfc-gh-truwase in #8287
- Fix ZeroDivisionError in compute_elastic_config return_microbatch on non-0.2 elasticity by @ebarkhordar in #8286
- Fix repeated FlopsProfiler metric accumulation by @baremetaldevx86 in #8246
- fix: stop DeepSpeedConfig writing max_grad_norm back into the caller's config dict by @ebarkhordar in #8289
- Enable DeepSpeed support on Apple Silicon (MPS) with ZeRO Stage 1-3 by @PKUWZP in #8293
- fix(engine): resolve inference workspace attribute lookup by @nathon-lee in #8288
- [AutoTP] Convert HF embedding_rowwise tp_plan entries to SKIP specs by @ebarkhordar in #8294
- Fix zero reduce bucket size validation by @fwerkor in #8266
- Register native pinned host memory with CUDA for GPU DMA by @sfc-gh-truwase in #8283
- Drop the documented grad_hooks ZeRO option, which does not exist by @vineethsaivs in #8242
- Skip zero-sized parameters in HP fragment mapping by @fwerkor in #8298
- perf(rollout): add HybridEngine rollout profiling by @nathon-lee in #8295
- Route isend/irecv to nonblocking backend methods and stage them as async on MPS by @PKUWZP in #8303
- Fix device mismatch in test_gate_up_partition_covers_the_whole_weight by @alanhuangyoo in #8308
- Import print_dist in auto_tp by @alanhuangyoo in #8311
- Reject invalid ZenFlow ratio and update interval boundaries by @tandede in #8274
- Avoid redundant copies in MPS P2P staging by @FU-max-boop in #8314
- Fix AutoTP metadata updates for unsharded modules by @zupengwang in #8299
- [Apple Silicon Support Phase 1] Add Metal FusedAdam kernel and CPU Adam build for Apple Silicon by @PKUWZP in #8300
- Enable optimized Adam backend for MuonWithAuxAdam optimizer by @jinyouzhi in #8278
- feat(opsd): share prompt prefill across rollout samples by @nathon-lee in #8296
- stop allocating per-element temporaries in the overflow check by @stas00 in #8325
- Fix shared loss gradient accumulation by @tohtana in #8245
- Isolate DeepCompile list-schedule test ops by @tohtana in #8319
- Fix ZeRO-3 synchronization during OPSD rollout by @LiRunGuo in #8264
- [AutoTP] Fix training lm_head routing by @gaoxiaomo in #8302
- Route ZeRO/SuperOffload pin sites through accelerator pin_memory by @sfc-gh-truwase in #8256
- Honor adam_w_mode in the CPU multi_tensor_adam binding by @PKUWZP in #8307
New Contributors
- @sfc-gh-thonguyen made their first contribution in #8226
- @baremetaldevx86 made their first contribution in #8240
- @darkdi made their first contribution in #8223
- @fwerkor made their first contribution in #8280
- @alanhuangyoo made their first contribution in #8308
- @tandede made their first contribution in #8274
- @FU-max-boop made their first contribution in #8314
- @LiRunGuo made their first contribution in #8264
- @gaoxiaomo made their first contribution in #8302
Full Changelog: v0.19.5...v0.19.6