The DOI matches its DataCite record; the references it cites are not checked yet.
Transformer Feed-Forward Layers Are Key-Value Memories
Open accessAbstract
Feed-forward layers constitute two-thirds of a transformer model's parameters, yet their role in the network remains under-explored. We show that feed-forward layers in transformer-based language models operate as key-value memories, where each key correlates with textual patterns in the training examples, and each value induces a distribution over the output vocabulary. Our experiments show that the learned patterns are human-interpretable, and that lower layers tend to capture shallow patterns, while upper layers learn more semantic ones. The values complement the keys' input patterns by inducing output distributions that concentrate probability mass on tokens likely to appear immediately after each pattern, particularly in the upper layers. Finally, we demonstrate that the output of a feed-forward layer is a composition of its memories, which is subsequently refined throughout the model's layers via residual connections to produce the final output distribution.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Middle layers often contain neurons that represent higher-level contextual features.
“research has shown that neurons across different layers tend to specialize, with middle layers often containing neurons that represent higher-level contextual features [26-29].”
From From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Task (2025) · cites this paper as [26]
Section 3.2 (page unavailable) reports that feed-forward memory keys differ by layer: layers 1–9 are dominated by shallow patterns, while layers 10–16 show more semantic patterns from similar contexts. This supports layer specialization and higher-level contextual features, but the paper locates those features in upper layers rather than characterizing middle layers as often containing them. (LLM, conf=0.84)
Experimental — accuracy not yet measuredHigher-level reasoning and conceptual encodings in LLMs can be found in intermediate and deeper layers.
“Notably, these units are mostly located in intermediate and deeper layers, matching results from previous research that show higher-level reasoning and conceptual encodings in LLMs can be found in these layers [26-29].”
From From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Task (2025) · cites this paper as [26]
Section 3.2 Results reports that upper layers (10–16) show more semantic patterns than lower layers, and Table 1 includes a semantic relation detected at layer 10. This supports conceptual encodings in intermediate-to-deeper layers, but the paper does not establish that higher-level reasoning is found there. Page numbers are not available in our copy. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThese 286 units were predominantly located in middle layers, approximately layers 12-22.
“These 286 units were predominantly located in the middle layers of the model (approximately layers 12-22), consistent with the distribution of units encoding the agent's position (Fig. 3) and with previous studies reporting that higher-level conceptual processing in LLMs tends to emerge in intermediate layers [26-29].”
From From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Task (2025) · cites this paper as [26]
The citation is attached to the sentence’s background claim about where semantic processing emerges, not to the citing authors’ count of 286 units. Section 3.2 reports more semantic patterns in the upper layers (10–16) of the source’s 16-layer model, versus shallow patterns in layers 1–9. It does not establish an intermediate-layer range of 12–22 or the distribution of the 286 units. (LLM, conf=0.85)
Experimental — accuracy not yet measuredMLPs are generally used to store factual knowledge learned by the model, while attention heads play consistent linguistically interpretable roles.
“Many studies have investigated that MLPs are generally used to store the factual knowledge learned by the model (Geva et al., 2020; 2022), while attention heads played consistent linguistically interpretable roles (V oita et al., 2019).”
From From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning (2024) · cites this paper as Geva et al., 2020
The Introduction and Discussion and Conclusion (pages unavailable in our copy) describe feed-forward layers as key-value memories whose keys capture textual patterns and whose values help predict the next token. They do not establish the broader claim that MLPs generally store the model’s factual knowledge. The attention-head clause is attributed to Voita et al. (2019), not to this source. (LLM, conf=0.85)
Experimental — accuracy not yet measuredMLPs are key components for memory storage in Transformers.
“Geva et al. (2020) discover that MLPs are the key components for memory storage in Transformers”
From Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models (2024) · cites this paper as Geva et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists haven’t been fetched yet
References 23 · Cited by 353
Research works printed but not linked: 28
- Alexei Baevski and Michael Auli. 2019. Adaptive in- put representations for neural language modeling. In International Conference on Learning Representa- tions (ICLR).
- Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learn- ers. In Proceedings of Neural Information Process- ing Systems (NeurIPS).
- Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? An analysis of BERT's attention. In Black- BoxNLP Workshop at ACL.
- Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. 2019. What is one grain of sand in the desert? analyz- ing individual neurons in deep nlp models. In Pro- ceedings of the AAAI Conference on Artificial Intel- ligence, volume 33, pages 6309-6317.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In North American Association for Com- putational Linguistics (NAACL), pages 4171-4186, Minneapolis, Minnesota. · DOI · Paper page
- Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. 2020. Analyzing individual neu- rons in pre-trained language models. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
- Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visualizing higher-layer fea- tures of a deep network. University of Montreal, 1341(3):1.
- Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. 2020. Explaining black box predictions and unveiling data artifacts through influence func- tions. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 5553-5563, Online. Association for Computa- tional Linguistics. · DOI
- Alon Jacovi, Oren Sar Shalom, and Yoav Goldberg. 2018. Understanding convolutional neural networks for text classification. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and In- terpreting Neural Networks for NLP, pages 56-65, Brussels, Belgium. Association for Computational Linguistics. · DOI
- Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 3651-3657, Florence, Italy. Associa- tion for Computational Linguistics. · DOI
- Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Lin- guistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. International Conference on Learning Repre- sentations (ICLR).
- Jesse Mu and Jacob Andreas. 2020. Compositional ex- planations of neurons. In Proceedings of Neural In- formation Processing Systems (NeurIPS).
- Milad Nasr, Reza Shokri, and Amir Houmansadr. 2019. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE Symposium on Security and Privacy (SP), pages 739-753. · DOI · Paper page
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In North American Chapter of the Asso- ciation for Computational Linguistics (NAACL).
- Ofir Press, Noah A. Smith, and Omer Levy. 2020. Im- proving transformer models by reordering their sub- layers. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 2996-3005, Online. Association for Computa- tional Linguistics. · DOI · Paper page
- Bhargav Pulugundla, Yang Gao, Brian King, Gokce Keskin, Harish Mallidi, Minhua Wu, Jasha Droppo, and Roland Maas. 2021. Attention-based neural beamforming layers for multi-channel speech recog- nition. arXiv preprint arXiv:2105.05920. · arXiv
- Nils Rethmeier, Vageesh Kumar Saxena, and Isabelle Augenstein. 2020. Tx-ray: Quantifying and explain- ing model-knowledge transfer in (un-) supervised nlp. In Conference on Uncertainty in Artificial In- telligence, pages 440-449.
- PMLR.
- S. Sukhbaatar, J. Weston, and R. Fergus. 2015. End- to-end memory networks. In Advances in Neural Information Processing Systems (NIPS).
- Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. 2019. Augmenting self-attention with persistent memory. arXiv preprint arXiv:1907.01470. · arXiv
- Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems (NIPS), pages 5998-6008.
- Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63-76, Florence, Italy. As- sociation for Computational Linguistics. · DOI · Paper page
- Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in Neural Information Processing Systems, 33.
- Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lift- ing, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics.
- Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 11-20, Hong Kong, China. Associ- ation for Computational Linguistics. · DOI · Paper page
- Hongfei Xu, Qiuhui Liu, Deyi Xiong, and Josef van Genabith. 2020. Transformer with depth-wise lstm. arXiv preprint arXiv:2007.06257. · arXiv
Metadata
- Authors
- Geva, M., Schuster, R., Berant, J., & Levy, O.
- Affiliations
- Tel Aviv University (IL)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 5484-5495
- arXiv
- 2012.14913
- OpenAlex
- W3117576675
- Cited by
- 353
- References
- 23
- Citations by year
- 2026: 88 · 2025: 100 · 2024: 63 · 2023: 68 · 2022: 21 · 2021: 11 · 2020: 2
- FWCI
- 18.79
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Funders
- Intel Corporation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- key-value memory, language models, interpretability, skip connections
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
