CoChat
Sign Up
PreprintCited by 226Topic ModelingPrivacy-Preserving Technologies in DataAdversarial Robustness in Machine Learning

Quantifying Memorization Across Neural Language Models

Open access
Carlini, Nicholas, Ippolito, Daphne, Jagielski, Matthew, Lee, Katherine, Tramer, Florian, Zhang, Chiyuan·2022·arXiv
DOI 10.48550/arxiv.2202.07646arXiv 2202.07646
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated easy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others). We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data. Memorization significantly grows as we increase (1) the capacity of a model, (2) the number of times an example has been duplicated, and (3) the number of tokens of context used to prompt the model. Surprisingly, we find the situation becomes more complicated when generalizing these results across model families. On the whole, we find that memorization in LMs is more prevalent than previously believed and will likely get worse as models continues to scale, at least without active mitigations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2202.07646

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The log-linear relationship between citation and similarity reflects the memorization scaling law.

    “A strong positive correlation ( r = 0 . 75 , p < . 001 ) indicates a clear log-linear relationship, which reflects the memorization scaling law [12].”

    From Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy (2025) · cites this paper as [12]

    Section 5 (p. 7) reports that memorization scales log-linearly with model size, data duplication, and context length. The paper does not examine citation–similarity relationships or establish that the citing document’s reported correlation reflects those memorization trends. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Methods like improving model capacity, increasing frequency of exposure to examples, and enriching prompt content possess potential to alleviate difficulties of long-tail information storage and information lag.

    “Methods like improving the model capacity, increasing the frequency of exposure to examples, and enriching the content of prompt possess the potential to alleviate the difficulties of long-tail information storage and information lag [165].”

    From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [165]

    The Abstract (p. 1) reports that memorization of training data increases with model capacity, example duplication, and prompt context; Section 4.3 (p. 5) shows that longer context makes more memorized sequences extractable. The paper does not establish that these changes alleviate difficulties of long-tail information storage or information lag. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Contrary to the discoverability phenomenon where memorization emerges only when sufficient context is given, highly redundant knowledge can be recalled even with minimal prompting.

    “Contrary to the discoverability phenomenon [12] where memorization emerges only when sufficient context is given, our results suggest the opposite direction: highly redundant knowledge can be recalled even with minimal prompting.”

    From Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy (2025) · cites this paper as [12]

  • The probability of reproducing training data correlates with its frequency of appearance, and information repeated multiple times during pretraining is more likely to be memorized and reproduced verbatim.

    “For example, the probability of reproducing training data has been shown to correlate with its frequency of appearance [12].”

    From Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy (2025) · cites this paper as [12]

  • The memorization tendency of neural networks grows with model size.

    “this memorization tendency grows with model size [34, 54]”

    From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [34]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 26

  1. Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.
  2. Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private BERT. arXiv preprint arXiv:2108.01624 , 2021. · arXiv
  3. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium , 2019.
  4. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv preprint arXiv:2012.07805 , 2020. · arXiv
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021. · arXiv
  6. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.
  7. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 , 2021. · arXiv
  8. Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems , 2020.
  9. Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security , pages 1322-1333, 2015.
  10. Karan Ganju, Qi Wang, Wei Yang, Carl A Gunter, and Nikita Borisov. Property inference attacks on fully connected neural networks using permutation invariant representations. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security , pages 619-633, 2018.
  11. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020. · arXiv
  12. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 123-129, 2018.
  13. Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD? arXiv preprint arXiv:2006.07709 , 2020. · arXiv
  14. Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In 28th { USENIX } Security Symposium ( { USENIX } Security 19) , pages 1895-1912, 2019.
  15. Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.06539 , 2022. · arXiv
  16. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris CallisonBurch, and Nicholas Carlini. Deduplicating training data makes language models better. CoRR , abs/2107.06499, 2021. URL https://arxiv.org/abs/2107.06499 . · arXiv
  17. R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN. CoRR , abs/2111.09509, 2021. URL https://arxiv.org/abs/2111.09509 . · arXiv
  18. Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Adversary instantiation: Lower bounds for differentially private machine learning. arXiv preprint arXiv:2101.04535 , 2021. · arXiv
  19. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020. URL http://jmlr.org/papers/v21/20-074.html .
  20. Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. Training production language models without memorizing user data. arXiv preprint arXiv:2009.10031 , 2020. · arXiv
  21. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP) , pages 3-18. IEEE, 2017.
  22. Aleena Thomas, David Ifeoluwa Adelani, Ali Davody, Aditya Mogadala, and Dietrich Klakow. Investigating the impact of pre-trained word embeddings on memorization in neural networks. In International Conference on Text, Speech, and Dialogue , pages 273-281. Springer, 2020.
  23. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF) , pages 268-282. IEEE, 2018.
  24. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR , 2017.
  25. Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938 , 2021. · arXiv
  26. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. · arXiv

Cited by

Sorted by citations

Related

Authors
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2202.07646
arXiv
2202.07646
OpenAlex
W4221159672
Cited by
204
References
0
Citations by year
2026: 14 · 2025: 46 · 2024: 52 · 2023: 73 · 2022: 19
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
memorization, large language models, training data leakage, log-linear relationship, model capacity, prompt length, privacy, fairness, model scaling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review