facebookresearch/xformers v0.0.17

Binaries for PT 2.0, mem-eff with bias & dropout, and varying seqlen

on GitHub

This release brings some improvements to the memory_efficient_attention

Pip wheels now target pytorch 2.0.0 - conda builds are available for PT 2.0.0, 1.13.1 and 1.12.1

Fixed

• fMHA: Fixed BW pass on Sm86/Sm89 GPUs when K > 64 (RTX 3090, RTX 4090, A6000, ..) [#631]

Added

• fMHA/CUTLASS: Added tensor attn bias support [#587] - contribution from @jfc4050
• fMHA/CUTLASS: Added tensor attn bias grad support [#587] - contribution from @jfc4050
• fMHA/CUTLASS: Added dropout support [#587] - contribution from @jfc4050
• fMHA: Added support for varying sequence lengths [#500]