Birth of a Transformer: A Memory Viewpoint
Open accessAbstract
Large language models based on transformers have achieved great empirical successes. However, as they are deployed more widely, there is a growing need to better understand their internal mechanisms in order to make them more reliable. These models appear to store vast amounts of knowledge from their training data, and to adapt quickly to new information provided in their context or prompt. We study how transformers balance these two types of knowledge by considering a synthetic setup where tokens are generated from either global or context-specific bigram distributions. By a careful empirical analysis of the training process on a simplified two-layer transformer, we illustrate the fast learning of global bigrams and the slower development of an "induction head" mechanism for the in-context bigrams. We highlight the role of weight matrices as associative memories, provide theoretical insights on how gradients enable their learning during training, and study the role of data-distributional properties.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cited by
Sorted by citations
Titans: Learning to Memorize at Test Time
Behrouz, A., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2501.00663 · Cited by 24 · see citing works
“The primary building blocks of Transformers–attention modules—function as associative memory blocks [3], where they learn to store key-value associations and retrieve them by computing pairwise similarity between queries (i.e., search signals) and keys (i.e., contexts).”
JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
Tian, Y., Wang, Y., Zhang, Z., Chen, B., & Du, S. (2023) · arXiv · doi:10.48550/arxiv.2310.00535 · Cited by 2 · see citing works
Zhang, J., Nolte, N., Sadhukhan, R., Chen, B., & Bottou, L. (2024) · arXiv · doi:10.48550/arxiv.2405.06394
“Unlike transformers whose internal mechanism are hard to decipher (Olsson et al., 2022; Bietti et al., 2024), Memory Mosaics achieve these capabilities in comparatively transparent ways.”
Self-citationBehrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2504.13173
ATLAS: Learning to Optimally Memorize the Context at Test Time
Behrouz, A., Li, Z., Kacham, P., Daliri, M., Deng, Y., Zhong, P., Razaviyayn, M., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2505.23735
- Authors
- Bietti, A., Cabannes, V., Bouchacourt, D., Jegou, H., & Bottou, L.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 1560-1588
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2306.00802
- OpenAlex
- W7133187679
- Cited by
- 13
- References
- 0
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 3 · 2023: 1
- FWCI
- 59.93
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Arts and Humanities › Philosophy › Memory, History, Trauma, Identity
- Social Sciences › Psychology › Developmental and Educational Psychology › Identity, Memory, and Therapy
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Historical Studies and Socio-cultural Analysis
- Keywords
- transformer models, memory-based models, sequence modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
