Training Compute-Optimal Large Language Models

Summary

This DeepMind paper introduces the 'Chinchilla' model, a large language model trained based on a compute-optimal scaling law. The authors challenge the prevailing scaling laws that prioritize increasing model size over data size, arguing that these models are undertrained given their compute budget. The study empirically demonstrates that for a fixed compute budget, it's more efficient to train smaller models on more data. Chinchilla achieves significantly better performance than larger models (e.g., GPT-3) of comparable compute, and even on some tasks surpasses models trained with much more compute. The paper explores the trade-offs between model size, dataset size, and training compute, providing a novel set of scaling laws and a more efficient approach to training large language models. The results highlight that the existing paradigm of simply scaling model parameters isn't necessarily the best approach for achieving optimal performance with a given compute constraint and introduces guidelines for more efficiently training large language models.

Key Takeaways

1. For a fixed compute budget, increasing the size of the training dataset is more effective than increasing the model size.
2. Chinchilla achieves superior performance to models like GPT-3 of comparable compute budget, demonstrating the efficiency gains of the proposed scaling laws.
3. The paper provides new scaling laws suggesting optimal model size and data size for a given compute budget.
4. The research challenges the practice of simply scaling model parameters without considering data size, emphasizing the critical importance of data in training large language models.