We fixed a gradient accumulation bug which was actually discovered since 2021 here, and rediscovered here. Read more in our blog post: https://unsloth.ai/blog/gradient
We have a Colab Notebook for Llama 3.2 using the fixed trainer and a Kaggle Notebook as well.
Essentially theoretically bsz * ga should be equivalent to full batch training with no gradient accumulation, but weirdly the training losses do no match up:
We fixed it in Unsloth!
To use Unsloth's fixed trainer with gradient accumulation, use:
from unsloth import unsloth_train
# trainer_stats = trainer.train() << Buggy if using gradient accumulation
trainer_stats = unsloth_train(trainer) # << Fixed gradient accumulationPlease update Unsloth on local machines (no need for Colab / Kaggle) via:
pip uninstall unsloth -y
pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"Read our blog post: https://unsloth.ai/blog/gradient for more details!
What's Changed
- Llama 3.2 by @danielhanchen in #1058
- Fix merges by @danielhanchen in #1079
- Handle absolute paths for save_to_gguf using pathlib by @giuliabaldini in #1120
- Only remove folder in sentencepiece check if it was created by @giuliabaldini in #1121
- Gradient Accumulation Fix by @danielhanchen in #1134
New Contributors
- @giuliabaldini made their first contribution in #1120
Full Changelog: September-2024...October-2024