Improving language models by retrieving from trillions of tokens

Summary

This DeepMind paper introduces Retro (Retrieval-augmented Transformer), a language model architecture that improves performance by retrieving information from a massive, trillion-token corpus. The core idea is to augment a Transformer language model with a retrieval component. During training, the model learns to retrieve relevant text passages from the corpus based on the current input. The retrieved passages are then incorporated into the model's context, allowing it to leverage external knowledge. This approach allows the model to learn more efficiently, generalize better, and achieve state-of-the-art results on several downstream tasks, especially in areas requiring factual accuracy and commonsense reasoning. The paper details the architecture, training procedure, and extensive evaluation of the Retro model, comparing it to other language models and ablation studies to highlight the importance of the retrieval component and the massive corpus. The findings indicate a significant improvement over standard transformer models on tasks that demand factual accuracy and reasoning ability, while also maintaining good performance on general language tasks. The ability to retrieve and integrate information allows the model to address the limitations of purely parametric models, particularly on tasks with sparse data or where external knowledge is crucial. The use of trillions of tokens for retrieval allows Retro to access a broad range of information, enhancing its capabilities.

Key Takeaways

1. Retro is a retrieval-augmented Transformer that significantly improves language model performance.
2. The retrieval component allows Retro to access and incorporate external knowledge, enhancing factual accuracy and reasoning.
3. Training Retro on a trillion-token corpus yields substantial improvements in performance.
4. Retro achieves state-of-the-art results on a variety of downstream tasks.
5. The retrieval mechanism makes the model more robust to errors and inconsistencies in its parameters.