hpcaitech/ColossalAI v0.4.4

Version v0.4.4 Release Today!

on GitHub

What's Changed

Release

• [release] update version (#6062) by Hongxin Liu

Colossaleval

• [ColossalEval] support for vllm (#6056) by Camille Zhong

Moe

• [moe] add parallel strategy for shared_expert && fix test for deepseek (#6063) by botbw

Sp

• Merge pull request #6064 from wangbluo/fix_attn by Wang Binluo
• Merge pull request #6061 from wangbluo/sp_fix by Wang Binluo

Doc

• [doc] FP8 training and communication document (#6050) by Guangyao Zhang
• [doc] update sp doc (#6055) by flybird11111

Fp8

• [fp8] Disable all_gather intranode. Disable Redundant all_gather fp8 (#6059) by Guangyao Zhang
• [fp8] fix missing fp8_comm flag in mixtral (#6057) by botbw
• [fp8] hotfix backward hook (#6053) by Hongxin Liu

Pre-commit.ci

• [pre-commit.ci] auto fixes from pre-commit.com hooks by pre-commit-ci[bot]

Hotfix

• [hotfix] moe hybrid parallelism benchmark & follow-up fix (#6048) by botbw

Feature

• [Feature] Split cross-entropy computation in SP (#5959) by Wenxuan Tan

Full Changelog: v0.4.4...v0.4.3