microsoft/DeepSpeed v0.3.0

DeepSpeed v0.3.0

on GitHub

New features

• DeepSpeed: Extreme-scale model training for everyone
  • Powering 10x longer sequences and 6x faster execution through DeepSpeed Sparse Attention
  • Training a trillion parameters with pipeline parallelism
  • Up to 5x less communication and 3.4x faster training through 1-bit Adam
  • 10x bigger model training on a single GPU with ZeRO-Offload

Software improvements

• Refactor codebase to make cleaner distinction between ops/runtime/zero/etc.
• Conditional Op builds
  • Not all users should have to spend time building transformer kernels if they don't want to use them.
  • To ensure DeepSpeed is portable in multiple environments some features require unique dependencies that not everyone will be able to or want to install.
• DeepSpeed launcher supports different backends in additional to pdsh such as Open MPI and MVAPICH.