New features
- ZeRO++: A leap in speed for LLM and chat model training with 4X less communication[English] [中文] [日本語]
- H100 support and testing w. FP8 using NVIDIA's TransformerEngine
What's Changed
- Documentation for DeepSpeed Accelerator Abstraction Interface by @delock in #3184
- FP8 unittest for H100 by @jomayeri in #3731
- Fix apex install bugs by @loadams in #3741
- Fix Autotuner get_gas_from_user_config by @straywarrior in #3664
- Include cublas error details when getting cublas handle fails by @jli in #3695
- fix hybrid engine mlp module by @tensor-tang in #3736
- Fix output transpose dimension bugs by @loadams in #3747
- remove UtilsBuilder load, use torch (un)flatten ops by @inkcherry in #3728
- add Chinese Zhihu social account by @conglongli in #3755
- Account for expert parameters when calculating the total number of pa… by @alito in #3720
- fix ccl_backend and residual_add problems by @dc3671 in #3642
- Fix url in getting-started guide (docs) by @acforvs in #3768
- Update deepspeed-chat/japanese/README.md by @eltociear in #3765
- Add H100 workflow and status badge. by @loadams in #3754
- Add an api in deepspeed engine for adjusting micro batch size during training by @kisseternity in #3773
- Prevent hangs in CI during parallel run compilation by @mrwyattii in #2844
- Revert "Prevent hangs in CI during parallel run compilation" by @jeffra in #3817
- [Docs]
chrome://tracingis deprecated by @keyboardAnt in #3805 - Support model declaration in zero.Init context by @tohtana in #3592
- Update zeropp.md by @samadejacobs in #3821
- Reduce Unit Test Times (Part 1) by @mrwyattii in #3829
- Re-enable GPT-J unit tests and refactor inference tests by @mrwyattii in #3618
- Fix racing condition in GatheredParameters by @HeyangQin in #3819
- zero/mics.py: use on_accelerator instead of cuda only by @guoyejun in #3806
- Disable AMD test flows in YML by @loadams in #3847
- Reduce Unit Test Time (Part 2) by @mrwyattii in #3838
- [profiling]add show_straggler argument to log_summary() by @delock in #3579
- checking process_group before merging bucket ranges (#3521) by @clumsy in #3577
- scripts/check-torchcuda.py: add checking for tensor.is_cuda by @guoyejun in #3843
- Zero3 Fix allreduce optimization for extra large tensor by @hablb in #3832
- [zero] revert PR #3166, it disabled grad clip for bf16 by @jeffra in #3790
- Fix transpose convolution FLOPS profiler (retrieval of out_channels) by @pinstripe-potoroo in #3834
- Fix LoRA Fuse/Unfuse in Hybrid Engine by @sxjscience in #3563
- Update pytorch-lightning version in CI by @mrwyattii in #3882
- [Docs] MMEngine has integrated deepspeed. by @HAOCHENYE in #3879
- Add FALCON Auto-TP Support by @RezaYazdaniAminabadi in #3640
- Update apex installation to resolve apex's pyproject.toml issues. by @loadams in #3745
- Extend HE-Lora test with Z3 support + Fix/add guard in HE for Z3 by @awan-10 in #3883
- Separate ZeRO3 InflightParamRegistry for train and eval by @HeyangQin in #3884
- Add GPTNeoX AutoTP support by @Yejing-Lai in #3778
- Fix Meta Tensor checkpoint load for BLOOM models by @lekurile in #3885
- fix error :Dictionary expression not allowed in type annotation Pylance by @digger-yu in #3708
- Fix rnn flop profiler to compute flops instead of macs by @pinstripe-potoroo in #3833
- Update workflows for merge queue by @mrwyattii in #3892
- Avoid deprecation warnings in
CHECK_CUDAby @Flamefire in #3854 - Silence comm.py warning by @mrwyattii in #3893
- Fix a typo of global variable in comm.py by @hipudding in #3852
- [ROCm] Enable TestCUDABackward::test_backward unit tests by @rraminen in #3849
- [profiling][mics]Fix some issues for log_summary(). by @ys950902 in #3899
- fix "undefined symbol: curandCreateGenerator" for quantizer op by @jinzhen-lin in #3846
- fix memory leak with zero-3 by @jeffra in #3903
- fix some typo docs/ by @digger-yu in #3917
- fix: change ==NONE to is under deepspeed/ by @digger-yu in #3923
- Del comment deepspeed.zero.Init() can be used as a decorator by @hipudding in #3894
- Remove the param.ds_tensor from print by @HeyangQin in #3928
- Reduce Unit Test Times (Part 3) by @mrwyattii in #3850
- Update zero_to_fp32.py - to support deepspeed_stage_1 by @PicoCreator in #3936
- [docs] add xTrimoPGLM by @jeffra in #3940
- Update Nvidia docker base image by @KaiChen1008 in #3930
- Fix inference tutorial docs for checkpoints by @loadams in #3955
- fix Megatron-DeepSpeed links by @conglongli in #3956
- skip bcast when enable pp but pp_group_size=1 by @inkcherry in #3915
- Use device_name instead of device index to support other device by @hipudding in #3933
- Create accelerator for apple silicon GPU Acceleration by @NripeshN in #3907
- fix(cpu_accelerator): 🐛 Convert LOCAL_SIZE to integer by @javsalgar in #3971
New Contributors
- @straywarrior made their first contribution in #3664
- @alito made their first contribution in #3720
- @acforvs made their first contribution in #3768
- @keyboardAnt made their first contribution in #3805
- @pinstripe-potoroo made their first contribution in #3834
- @HAOCHENYE made their first contribution in #3879
- @Yejing-Lai made their first contribution in #3778
- @Flamefire made their first contribution in #3854
- @hipudding made their first contribution in #3852
- @PicoCreator made their first contribution in #3936
- @KaiChen1008 made their first contribution in #3930
- @NripeshN made their first contribution in #3907
- @javsalgar made their first contribution in #3971
Full Changelog: v0.9.4...v0.10.0