hiyouga/LlamaFactory v0.9.0

v0.9.0: Qwen2-VL, Liger-Kernel, Adam-mini

on GitHub

Congratulations on 30,000 stars 🎉 Follow us at X (twitter)

New features

• 🔥Support fine-tuning Qwen2-VL model on multi-image datasets by @simonJJJ in #5290
• 🔥Support time&memory-efficient Liger-Kernel via the enable_liger_kernel argument by @hiyouga
• 🔥Support memory-efficient Adam-mini optimizer via the use_adam_mini argument by @relic-yuexi in #5095
• Support fine-tuning Qwen2-VL model on video datasets by @hiyouga in #5365 and @BUAADreamer in #4136 (needs patch huggingface/transformers#33307)
• Support fine-tuning vision language models (VLMs) using RLHF/DPO/ORPO/SimPO approaches by @hiyouga
• Support Unsloth's asynchronous activation offloading method via the use_unsloth_gc argument
• Support vLLM 0.6.0 version
• Support MFU calculation by @yzoaim in #5388

New models

• Base models
  • Qwen2-Math (1.5B/7B/72B) 📄🔢
  • Yi-Coder (1.5B/9B) 📄
  • InternLM2.5 (1.8B/7B/20B) 📄
  • Gemma-2-2B 📄
  • Meta-Llama-3.1 (8B/70B) 📄
• Instruct/Chat models
  • MiniCPM/MiniCPM3 (1B/2B/4B) by @LDLINGLINGLING in #4996 #5372 📄🤖
  • Qwen2-Math-Instruct (1.5B/7B/72B) 📄🤖🔢
  • Yi-Coder-Chat (1.5B/9B) 📄🤖
  • InternLM2.5-Chat (1.8B/7B/20B) 📄🤖
  • Qwen2-VL-Instruct (2B/7B) 📄🤖🖼️
  • Gemma-2-2B-it by @codemayq in #5037 📄🤖
  • Meta-Llama-3.1-Instruct (8B/70B) 📄🤖
  • Mistral-Nemo-Instruct (12B) 📄🤖

New datasets

• Supervised fine-tuning datasets
  • Magpie-ultra-v0.1 (en) 📄
  • Pokemon-gpt4o-captions (en&zh) 📄🖼️
• Preference datasets
  • RLHF-V (en) 📄🖼️
  • VLFeedback (en) 📄🖼️

Changes

• Due to compatibility consideration, fine-tuning vision language models (VLMs) requires transformers>=4.35.0.dev0, try pip install git+https://github.com/huggingface/transformers.git to install it.
• visual_inputs has been deprecated, now you do not need to specify this argument.
• LlamaFactory now adopts lazy loading for multimodal inputs, see #5346 for details. Please use preprocessing_batch_size to restrict the batch size in dataset pre-processing (supported by @naem1023 in #5323 ).
• LlamaFactory now supports lmf (equivalent to llamafactory-cli) as a shortcut command.

Bug fix

• Fix LlamaBoard export by @liuwwang in #4950
• Add ROCm dockerfiles by @HardAndHeavy in #4970
• Fix deepseek template by @piamo in #4892
• Fix pissa savecallback by @codemayq in #4995
• Add Korean display language in LlamaBoard by @Eruly in #5010
• Fix deepseekcoder template by @relic-yuexi in #5072
• Fix examples by @codemayq in #5109
• Fix mask_history truncate from last by @YeQiuO in #5115
• Fix jinja template by @YeQiuO in #5156
• Fix PPO optimizer and lr scheduler by @liu-zichen in #5163
• Add SailorLLM template by @chenhuiyu in #5185
• Fix XPU device count by @Zxilly in #5188
• Fix bf16 check in NPU by @Ricardo-L-C in #5193
• Update NPU docker image by @MengqingCao in #5230
• Fix image input api by @marko1616 in #5237
• Add liger-kernel link by @ByronHsu in #5317
• Fix #4684 #4696 #4917 #4925 #4928 #4944 #4959 #4992 #5035 #5048 #5060 #5092 #5228 #5252 #5292 #5295 #5305 #5307 #5308 #5324 #5331 #5334 #5338 #5344 #5366 #5384