accelerate 0.16.0

v0.16.0: Improved and Interactive Documentation, DataLoader Improvements

on Python PyPI

New code exploration doc tool

A new interactive tool has been introduced to the documentation to help users quickly learn how to utilize features of the framework before providing more details on them as shown below:

Not only does it provide a code diff, but it also includes an explanation and links to more resources the user should check out to learn more:

Try it out today in the docs

• Add in code exploration tool to docs by @muellerzr in #1014
• Light vs dark theme based on pick by @muellerzr in #1023

Skip batches in dataloaders

When resuming training, you can more efficiently skip batches in your dataloader with the new skip_first_batches function (also available as a method on your Accelerator).

• Efficiently skip batches in a dataloader by @sgugger in #1002

DeepSpeed integration enhancements:

A new ZeRO-3 init context manager is added to provide granular control to users in situations involving nested/multiple models. Refactoring of DeepSpeed Config file support to remove ambiguity between it and Accelerate config.

Adding support for auto entries in the DeeSpeed config file to be filled via the accelerate launch command. Try it out today by referring to the section Things to note when using DeepSpeed Config File

• ds zero-3 init context manager by @pacman100 in #932
• raise error for duplicate accelerate config values when using deepspeed_config_file by @pacman100 in #941

What's new?

• Flag to silence subprocess.CalledProcessError in launch by @Cyberes in #902
• Add usage examples by @muellerzr in #904
• Expand sanity checks by @muellerzr in #905
• Fix conditional by @muellerzr in #907
• fix issue that amp bf16 does not work for cpu in env with cuda. by @sywangyi in #906
• fsdp enhancements by @pacman100 in #911
• Fix typos accelerate -> accelerator by @pcuenca in #915
• 🚨🚨🚨 Act on deprecations 🚨🚨🚨 by @muellerzr in #917
• fix accelerate test failure with cpu config by @sywangyi in #909
• Introduce project_dir and limit the number of saved checkpoints by @muellerzr in #916
• Specify inference by @muellerzr in #921
• Support init_on_device by @thomasw21 in #926
• ds-z3-init and prepending ds env variables with ACCELERATE_ by @pacman100 in #928
• Honor model dtype in load_checkpoint by @sgugger in #920
• ds zero-3 init context manager by @pacman100 in #932
• Fix silly typo by @tornikeo in #939
• add mixed_precision_type property to AcceleratorState by @pacman100 in #935
• fix batch size in prepare_dataloader for iterable datasets by @sanderland in #937
• fix mp related test fails by @pacman100 in #943
• Fix tracker by @muellerzr in #942
• Fix offload when weights are on the GPU by @sgugger in #945
• raise error for duplicate accelerate config values when using deepspeed_config_file by @pacman100 in #941
• Add is_initialized method and refactor by @muellerzr in #949
• Fix DeepSpeed tests by @muellerzr in #950
• Don't automatically offload buffers when loading checkpoints by @sgugger in #951
• Typo fix in src/accelerate/utils/modeling.py by @ryderwishart in #955
• support master port when using ds multi-node launcher by @pacman100 in #959
• Allowing encoded configuration for DeepSpeed by @cli99 in #895
• Update README.md by @Don9wanKim in #968
• Raise minimum version for distrib launch by @muellerzr in #978
• Fix tied parameters test in big model inference by @sgugger in #979
• Fix type error on line 36 by @dhar174 in #981
• Ensure that last batch doesn't get dropped if perfectly even in gather_for_metrics by @muellerzr in #982
• Skip wandb test for now by @muellerzr in #984
• Fix test for converting tensor to proper dtype by @sgugger in #983
• in sync with trfs, removing style_doc utils and using doc-builder instead by @pacman100 in #988
• Add new release_memory util by @muellerzr in #990
• adding support for kwargs in load_state by @pacman100 in #989
• Fix scheduler incorrect steps when gradient accumulation enabled by @markovalexander in #999
• Fix parameters tying in dispatch_model by @sgugger in #1000
• improve deepspeed notes by @stas00 in #1003
• Update toctree by @muellerzr in #1008
• Add styleguide by @muellerzr in #1007
• Maintain accumulation steps by @muellerzr in #1011
• Saving and loading state hooks by @patrickvonplaten in #991
• Fix test introduced in PR and introduce AcceleratorTestCase by @muellerzr in #1016
• Allow the torch device to be set with an env var by @Yard1 in #1009
• Fix import of LrScheduler by @sgugger in #1017
• Don't force mixed precision as no in examples by @sgugger in #1018
• Include steppage in performance docs by @muellerzr in #1013
• Fix env var by @muellerzr in #1024
• Change default for keep_fp32_wrapper by @muellerzr in #1025
• Fix slow test by keeping tied weights on the same GPU by @sgugger in #1026
• Start of adding examples by @muellerzr in #1001
• More improvements to docstrings + examples by @muellerzr in #1010
• With example by @muellerzr in #1027
• sagemaker launcher fixes by @pacman100 in #1031