LM-Cocktail

Merge language models (e.g., Llama, bge) to improve the general ability of models.
This method can be used to:

• Mitigate the Problem of Catastrophic Forgetting
• Improve the performance of new tasks without fine-tuning
• Approximate multitask learning or model ensemble

More details please refer to paper and code