CoChat
Sign Up
Cited by 4Topic ModelingMemory Processes and InfluencesNatural Language Processing Techniques

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models

Open access
Cao, Boxi, Tang, Qiaoyu, Lin, Hongyu, Jiang, Shanshan, Dong, Bin, Han, Xianpei, Chen, Jiawei, Wang, Tianshu, Sun, Le·2024·Proceedings of the Language Resources and Evaluation Conference
DOI 10.63317/4h8nkuscdqqa
Open this paper in CoChat — free accountOpen PDF

Abstract

Memory is one of the most essential cognitive functions serving as a repository of world knowledge and episodes of activities.In recent years, large-scale pre-trained language models have shown remarkable memorizing ability.On the contrary, vanilla neural networks without pre-training have been long observed suffering from the catastrophic forgetting problem.To investigate such a retentive-forgetful contradiction and understand the memorizing dynamic mechanism of language models, we conduct thorough experiments by controlling the target knowledge types, the learning strategies and the learning schedules.We find that: 1) Vanilla language models without pre-training are forgetful; 2) Pre-training leads to retentive language models; 3) Knowledge relevance and diversification significantly influence the memory formation.These conclusions are useful for understanding the abilities of pre-trained language models and shed light on designing and evaluating new learning and inference algorithms of language models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.63317/4h8nkuscdqqa

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs face significant challenges maintaining long-term memory due to context window limitations and catastrophic forgetting.

    “LLMs face significant challenges maintaining long-term memory due to context window limitations and catastrophic forgetting [114].”

    From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [114]

    Section 3.1 (page unavailable) reports that vanilla language models rapidly forget previously learned factual knowledge when they learn something new, describing this as catastrophic forgetting. The paper does not attribute that problem to context-window limitations; Section 4.2 (page unavailable) also finds that pre-training enables long-term memorization, so the result does not apply equally to all LLMs. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. Richard C Atkinson and Richard M Shiffrin. 1971. The control of short-term memory. Scientific american, 225(2):82-91.
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  3. Boxi Cao, Hongyu Lin, Xianpei Han, Fangchao Liu, and Le Sun. 2022. Can prompt probe pretrained language models? understanding the invisible risks from a causal view. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 5796-5808, Dublin, Ireland. Association for Computational Linguistics. · DOI
  4. Boxi Cao, Hongyu Lin, Xianpei Han, and Le Sun. 2024. The life cycle of knowledge in big lan- guage models: A survey. Machine Intelligence Research, pages 1-22.
  5. Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingy- ong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable or educated guess? revisiting language models as knowledge bases. In Pro- ceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1860-1874, Online. Association for Com- putational Linguistics. · DOI
  6. Nicholas Carlini, Daphne Ippolito, Matthew Jagiel- ski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646. · arXiv
  7. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting train- ing data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650.
  8. John Ceraso. 1967. The interference theory of forgetting. Scientific American, 217(4):117-127.
  9. Zhiyuan Chen and Bing Liu. 2018. Lifelong ma- chine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1-207.
  10. Eugène JFM Custers. 2010. Long-term retention of basic science knowledge: a review study. Ad- vances in Health Sciences Education, 15(1):109- 128.
  11. Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A con- tinual learning survey: Defying forgetting in clas- sification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366- 3385.
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Lin- guistics. · DOI · Paper page
  13. Wlodzislaw Duch, Richard Jayadi Oentaryo, and Michel Pasquier. 2008. Cognitive architectures: Where do we go from here? In Agi, volume 171, pages 122-136.
  14. Hermann Ebbinghaus. 2013. Memory: A contri- bution to experimental psychology. Annals of neurosciences, 20(4):155.
  15. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measur- ing and improving consistency in pretrained lan- guage models. Transactions of the Association for Computational Linguistics, 9:1012-1031. · DOI · Paper page
  16. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Asso- ciation for Computational Linguistics, 8:34-48. · DOI · Paper page
  17. Michael Eysenck. 2012. Attention and arousal: Cognition and performance. Springer Science & Business Media.
  18. Vitaly Feldman. 2020. Does learning require mem- orization? a short tale about a long tail. In Pro- ceedings of the 52nd Annual ACM SIGACT Sym- posium on Theory of Computing, pages 954- 959.
  19. Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do Neural Language Representations Learn Physical Commonsense? ArXiv preprint, abs/1908.02899. · arXiv
  20. Robert M French. 1991. Using semi-distributed representations to overcome catastrophic forget- ting in connectionist networks. In Proceedings of the 13th annual cognitive science society con- ference, volume 1, pages 173-178.
  21. Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135.
  22. Susan E Gathercole. 1998. The development of memory. The Journal of Child Psychology and Psychiatry and Allied Disciplines, 39(1):3-27.
  23. Alexander Gepperth and Cem Karaoguz. 2016. A bio-inspired incremental learning architecture for applied perceptual problems. Cognitive Compu- tation, 8(5):924-934.
  24. Philip Goelet, Vincent F Castellucci, Samuel Schacher, and Eric R Kandel. 1986. The long and the short of long-term memory-a molecu- lar framework. Nature, 322(6078):419-422.
  25. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An em- pirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211. · arXiv
  26. Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stan- ley Jungkyu Choi, and Minjoon Seo. 2022. To- wards continual knowledge learning of language models. In ICLR.
  27. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-FACTR: Multilingual factual knowledge re- trieval from pretrained language models. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 5943-5959, Online. Associa- tion for Computational Linguistics. · DOI · Paper page
  28. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Gra- ham Neubig. 2020b. How can we know what language models know? Transactions of the As- sociation for Computational Linguistics, 8:423- 438. · DOI · Paper page
  29. Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowl- edge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 3250- 3258, Online. Association for Computational Lin- guistics. · DOI
  30. Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935- 2947.
  31. Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside BERT's linguistic knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Inter- preting Neural Networks for NLP, pages 241- 253, Florence, Italy. Association for Computa- tional Linguistics. · DOI
  32. James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. 1995. Why there are comple- mentary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419.
  33. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist net- works: The sequential learning problem. In Psy- chology of learning and motivation, volume 24, pages 109-165. Elsevier.
  34. R Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2021. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. arXiv preprint arXiv:2111.09509. · arXiv
  35. Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. 2021. An em- pirical investigation of the role of pre-training in lifelong learning. CoRR, abs/2112.09153. · arXiv
  36. George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):81.
  37. Jaap MJ Murre and Joeri Dros. 2015. Replica- tion and analysis of ebbinghaus' forgetting curve. PloS one, 10(7):e0120644.
  38. German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71.
  39. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Lin- guistics. · DOI · Paper page
  40. Benedikt Pfülb and Alexander Gepperth. 2019. A comprehensive, application-oriented study of catastrophic forgetting in dnns. arXiv preprint arXiv:1905.08101. · arXiv
  41. Leo Postman. 1961. The present status of inter- ference theory. In Conference on Verbal Learn- ing and Verbal Behavior, 1959, US. McGraw-Hill Book Company.
  42. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  43. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67.
  44. Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. 2021. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations.
  45. Fiona M Richardson and Michael SC Thomas. 2008. Critical periods and catastrophic in- terference effects in the development of self- organizing feature maps. Developmental sci- ence, 11(3):371-389.
  46. Anthony Robins. 1995. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Sci- ence, 7(2):123-146.
  47. Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098. · arXiv
  48. Larry R Squire. 1986. Mechanisms of memory. Science, 232(4758):1612-1619.
  49. Larry R Squire and Stuart M Zola. 1998. Episodic memory, semantic memory, and amnesia. Hip- pocampus, 8(3):205-211.
  50. Michael Tänzer, Sebastian Ruder, and Marek Rei. 2022. Memorisation versus generalisation in pre- trained language models. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 7564-7578, Dublin, Ireland. Association for Computational Linguistics. · DOI
  51. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Di- panjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence struc- ture in contextualized word representations. In 7th International Conference on Learning Repre- sentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  52. Kushal Tirumala, Aram H Markosyan, Luke Zettle- moyer, and Armen Aghajanyan. 2022. Memo- rization without overfitting: Analyzing the train- ing dynamics of large language models. arXiv preprint arXiv:2205.10770. · arXiv
  53. Endel Tulving and Hans J Markowitsch. 1998. Episodic and declarative memory: role of the hippocampus. Hippocampus, 8(3):198-204.
  54. Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: a free collaborative knowledgebase. Com- munications of the ACM, 57(10):78-85.
  55. John T Wixted and Ebbe B Ebbesen. 1991. On the form of forgetting. Psychological science, 2(6):409-415.
  56. Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelli- gence. In International Conference on Machine Learning, pages 3987-3995.
  57. PMLR.
  58. Yu Zhang and Qiang Yang. 2021. A survey on multi- task learning. IEEE Transactions on Knowledge and Data Engineering.
  59. Xuhui Zhou, Yue Zhang, Leyang Cui, and Dan- dan Huang. 2020. Evaluating commonsense in pre-trained language models. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Appli- cations of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9733-9740. AAAI Press.

Cited by

Sorted by citations

Metadata

Authors
Cao, B., Tang, Q., Lin, H., Jiang, S., Dong, B., Han, X., Chen, J., Wang, T., & Sun, L.
Affiliations
  • University of Chinese Academy of Sciences (CN)
  • Chinese Academy of Sciences (CN)
ORCID
Venue
Proceedings of the Language Resources and Evaluation Conference
Publisher
European Language Resources Association (ELRA) and ICCL
Published
2024-05
Pages
14016-14036
ISSN
2522-2686
DOI
10.63317/4h8nkuscdqqa
OpenAlex
W7131886094
Cited by
4
References
1
Citations by year
2025: 1 · 2024: 1
FWCI
0.59
Type
conference paper
Language
en
Open access
Locations
  • Language Resources and Evaluation Conference (LREC) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Natural Science Foundation of China (62106251, 62122077)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
knowledge memorization, language models, knowledge retention, forgetting, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review