How might LLMs store facts | Chapter 7, Deep Learning
Large language models store facts, like "Michael Jordan plays basketball," in multi-layer perceptrons, though understanding this storage remains complex.
MAIN POINTS FROM TRANSCRIPT
- Language models memorize numerous facts, like athlete-sport associations, within their parameters.
- Multi-layer perceptrons (MLPs) are identified as key components storing these facts.
- Understanding MLP computations is challenging, despite their relative simplicity compared to other components.
TAKEAWAYS
- Researchers study where and how language models store factual information.
- MLPs play a significant role in fact storage within language models.
- Simplifying the understanding of MLP computations remains a primary goal.