Preventing generation of verbatim memorization in language models gives a false sense of privacy
Open accessAbstract
Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette Choo, Nicholas Carlini. Proceedings of the 16th International Natural Language Generation Conference. 2023.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There are limitations with the memorization analysis, e.g., recent work advocates for metrics beyond exact match and alternative prompt search strategies.
“Note there are limitations with our analysis - for example, recent work advocates for metrics beyond exact match (Ippolito et al., 2023) and alternative prompt search strategies (Kassem et al., 2024).”
From The Llama 3 Herd of Models (2024) · cites this paper as Ippolito et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Self-citationSpace/time trade-offs in hash coding with allowable errors
Bloom, B. (1970) · CACM · doi:10.1145/362686.362692 · Cited by 8,424 · see citing works
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578 · see citing works
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Self-citationEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Comparison and evaluation of code clone detection techniques and tools: A qualitative approach
Roy, C. K., Cordy, J. R., & Koschke, R. (2009) · Science of Computer Programming · doi:10.1016/j.scico.2009.02.007 · Cited by 1,002 · see citing works
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · see citing works
Self-citationThe Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629 · see citing works
Self-citationTheory and Practice of Bloom Filters for Distributed Systems
Tarkoma, S., Rothenberg, C. E., & Lagerspetz, E. (2011) · IEEE Communications Surveys & Tutorials · doi:10.1109/surv.2011.031611.00024 · Cited by 525 · see citing works
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks
Zhang, Y., Jia, R., Pei, H., Wang, W., Li, B., & Song, D. X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00033 · Cited by 481 · see citing works
Deduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304 · see citing works
Self-citationAn Evaluation Framework for Plagiarism Detection
Potthast, M., Stein, B., Barrón‐Cedeño, A., & Rosso, P. (2010) · Cited by 290 · see citing works
DART: Open-Domain Structured Data Record to Text GenerationPreprint
Nan, L., Radev, D. R., Zhang, R., Rau, A., Sivaprasad, A., Hsieh, C.-H., Tang, X., Vyas, A., Verma, N., Krishna, P., et al. (2020) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2007.02871 · Cited by 234 · see citing works
Quantifying Memorization Across Neural Language Models
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022) · arXiv · doi:10.48550/arxiv.2202.07646 · Cited by 204 · see citing works
Self-citationWhat Does it Mean for a Language Model to Preserve Privacy?
Brown, H., Lee, K., Mireshghallah, F., Shokri, R., & Tramèr, F. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534642 · Cited by 165 · see citing works
Self-citationPythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01373 · Cited by 160 · see citing works
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159 · see citing works
Differentially Private Fine-tuning of Language Models
Yu, D., Naik, S., Bačkurs, A., Gopi, S., Inan, H. A., Kamath, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L., et al. (2024) · Journal of Privacy and Confidentiality · doi:10.29012/jpc.880 · Cited by 152 · see citing works
Differentially Private Fine-tuning of Language Models
Yu, D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Kamath, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.06500 · Cited by 101 · see citing works
Reconstructing Training Data with Informed Adversaries
Balle, B., Cherubin, G., & Hayes, J. (2022) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46214.2022.9833677 · Cited by 93 · see citing works
Semantic Noise Matters for Neural Natural Language Generation
Dušek, O., Howcroft, D. M., & Rieser, V. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8652 · Cited by 85 · see citing works
Large-Scale Differentially Private BERT
Anil, R., Ghazi, B., Gupta, V., Kumar, R., & Manurangsi, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.484 · Cited by 68 · see citing works
Deduplicating Training Data Mitigates Privacy Risks in Language Models
Kandpal, N., Wallace, E., & Raffel, C. (2022) · arXiv · doi:10.48550/arxiv.2202.06539 · Cited by 68 · see citing works
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
Tirumala, K., Markosyan, A., Zettlemoyer, L., & Aghajanyan, A. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2773 · Cited by 67 · see citing works
Counterfactual Memorization in Neural Language Models
Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tramèr, F., & Carlini, N. (2023) · neural information processing systems · doi:10.52202/075280-1708 · Cited by 54 · see citing works
Self-citationCounterfactual Memorization in Neural Language Models
Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tramèr, F., & Carlini, N. (2021) · arXiv · doi:10.48550/arxiv.2112.12938 · Cited by 54 · see citing works
Self-citationReconstructing Training Data from Trained Neural Networks
Haim, N., Vardi, G., Yehudai, G., Shamir, O., & Irani, M. (2022) · neural information processing systems · doi:10.52202/068431-1665 · Cited by 48 · see citing works
Bloom Filters, Adaptivity, and the Dictionary Problem
Bender, M. A., Farach-Colton, M., Goswami, M., Johnson, R., McCauley, S., & Singh, S. (2018) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2018.00026 · Cited by 48 · see citing works
Understanding Unintended Memorization in Language Models Under Federated Learning
Thakkar, O. D., Ramaswamy, S., Mathews, R., & Beaufays, F. (2021) · Proceedings of the Third Workshop on Privacy in Natural Language Processing · doi:10.18653/v1/2021.privatenlp-1.1 · Cited by 39 · see citing works
Training Production Language Models without Memorizing User Data
Ramaswamy, S., Thakkar, O., Mathews, R., Andrew, G., McMahan, H. B., & Beaufays, F. (2020) · arXiv · doi:10.48550/arxiv.2009.10031 · Cited by 37 · see citing works
Provably Confidential Language Modelling
Zhao, X., Li, L., & Wang, Y.-X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.69 · Cited by 17 · see citing works
Defending against Reconstruction Attacks with Rényi Differential PrivacyPreprint
Stock, P., Shilov, I., Mironov, I., & Sablayrolles, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.07623 · Cited by 6 · see citing works
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Flagged first, then by citations
Research works printed but not linked: 6
- Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyz- ing text with the natural language toolkit. " O'Reilly Media, Inc.".
- Jehoshua Bruck, Jie Gao, and Anxiao Jiang. 2006. Weighted Bloom filter. In 2006 IEEE International Symposium on Information Theory, pages 2304- 2308. IEEE.
- Ondrej Dušek and Filip Jurcıcek. 2016. A context- aware natural language generation dataset for dia- logue systems. In Workshop on collecting and gen- erating resources for chatbots and conversational agents-development and evaluation, pages 6-9.
- Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. Totto: A controlled table-to- text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 1173-1186.
- Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. Mlsum: The multilingual summarization corpus. In 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051-8067. Association for Computational Linguistics.
- Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays. 2021. Under- standing unintended memorization in language mod- els under federated learning. In Proceedings of the Third Workshop on Privacy in Natural Language Processing, pages 1-10.
Cited by
Sorted by citations
AI agents under threat: A survey of key security challenges and future pathways
Deng, Z., Guo, Y., Han, C., Ma, W., Xiong, J., Wen, S., & Xiang, Y. (2025) · ACM Computing Surveys · doi:10.1145/3716628 · Cited by 210 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Privacy preserving large language models: ChatGPT case study based vision and framework
Ullah, I., Hassan, N., Gill, S. S., Suleiman, B., Ahanger, T. A., Shah, Z., Qadir, J., & Kanhere, S. S. (2024) · IET Blockchain · doi:10.1049/blc2.12091 · Cited by 33 · see citing works
Benchmark Data Contamination of Large Language Models: A Survey
Xu, C., Guan, S., Greene, D., & Kechadi, M.-T. (2024) · arXiv · doi:10.48550/arxiv.2406.04244 · Cited by 22 · see citing works
Creative Writers' Attitudes on Writing as Training Data for Large Language Models
Gero, K. I., Desai, M., Schnitzler, C., Eom, N., Cushman, J., & Glassman, E. L. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713287 · Cited by 16 · see citing works
Lee, K., Cooper, A. F., & Grimmelmann, J. (2024) · ACM Symposium on Computer Science and Law (CSLAW) · doi:10.1145/3614407.3643696 · Cited by 83 · see citing works
Self-citationOn the Societal Impact of Open Foundation Models
Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., Hopkins, A., Bankston, K., Biderman, S., Bogen, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.07918 · Cited by 11 · see citing works
“Weights (and associated computational artifacts made available, such as activations and gradients) permit a wide range of adaptation methods for modifying the model, such as quantization (Frantar et al., 2023), fine tuning (Zhang et al., 2023; Dettmers et al., 2023), and pruning (Xia et al., 2023).”
Katzy, J., Popescu, R. M., Arie Van Deursen, A., & Izadi, M. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652298 · Cited by 12 · see citing works
Salerno, F., Al-Kaswan, A., & Izadi, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00080 · Cited by 7 · see citing works
A survey on privacy issues and mitigation strategies for LLMs in healthcare
Alissa, K. A. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-08146-1 · Cited by 6 · see citing works
Privacy-Aware Document Visual Question Answering
Tito, R., Nguyen, K., Tobaben, M., Kerkouche, R., Souibgui, M. A., Jung, K., Jälkö, J., d’Andecy, V. P., Joseph, A., Kang, L., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70552-6_12 · Cited by 8 · see citing works
Stalnaker, T., Wintersgill, N., Chaparro, O., Heymann, L. A., Penta, M. D., Germán, D. M., & Poshyvanyk, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3743133 · Cited by 3 · see citing works
Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?
Panaitescu-Liess, M.-A., Che, Z., An, B., Xu, Y., Pathmanathan, P., Chakraborty, S., Zhu, S., Goldstein, T., & Huang, F. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34684 · Cited by 3 · see citing works
Stalnaker, T., Wintersgill, N., Chaparro, O., Heymann, L. A., Tufo, M., Germán, D. M., & Poshyvanyk, D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5353252 · Cited by 3 · see citing works
Detecting Training Data For Large Language Models: A Survey
Yang, C., Li, J., Shulin, L. A. N., Wang, Y., Du, H., Gong, C., Yao, X., Niyato, D. T., & Zhu, L. (2026) · ACM Computing Surveys · doi:10.1145/3779430 · Cited by 2 · see citing works
Dualmark: A novel dual watermarking approach for large language models
Qiang, Z., Hao, J., Qiang, J., Zhu, Y. Q., Zhang, C., Liu, Y., & Li, W. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104707 · Cited by 2 · see citing works
Self-citationThe mosaic memory of large language models
Shilov, I., Meeus, M., & de Montjoye, Y.-A. (2026) · Nature Communications · doi:10.1038/s41467-026-68603-0 · Cited by 11 · see citing works
Can Personal Health Information Be Secured in LLM? Privacy Attack and Defense in the Medical Domain
Kang, Y., Kim, E., & Cho, Y.-S. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765105 · Cited by 1 · see citing works
Towards More Realistic Extraction Attacks: An Adversarial Perspective
More, Y., Ganesh, P., & Farnadi, G. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.62 · Cited by 1 · see citing works
Interrogating LLM design under copyright law
Wei, J. T.-Z., Wang, M., Godbole, A., Choi, J., & Jia, R. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732193 · Cited by 1 · see citing works
The KL3M Data Project: Copyright-Clean Training Resources for Large Language ModelsPreprint
Bommarito, M. J., Bommarito, J., & Katz, D. M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5211933 · Cited by 1 · see citing works
FBF: A High-Efficient Query Mechanism for Keyword Search in Online Social Networks
Huang, J., Tong, Y., Hang, B., Xu, D., Wang, F., & Yu, J. (2024) · Tehnicki vjesnik - Technical Gazette · doi:10.17559/tv-20230511000622
Unified lookup tables: training foundation models on encoded data
Janakarajan, N., Morales, I., Alberts, M., Giovannini, A. A., Manica, M., & Foncubierta–Rodríguez, A. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ae143c
Utilization of Large Language Models for conformity assessment: Chances, Threats, and Mitigations
Litzinger, J., Peters, D., Thiel, F., & Tschorsch, F. (2025) · Annals of Computer Science and Information Systems · doi:10.15439/2025f7772
Copyright Protection of General Information via Simulation Task Supervision
Hu, C., Hu, Y., Li, H., Hu, Y., & Zhu, N. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3638667
Uncovering Prompt Elements: Cloning System Prompts from Behavioral Traces
Qian, Y., Peng, F., Wu, H., Chen, L., & Mao, B. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00045
Nandy, S., Gupta, K., & Gupta, A. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-44344-2.00006-6
Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs
Liu, R., Evans, D., & Xiong, L. (2026) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp63933.2026.00165
A Unified Framework of Extractability in Large Language Models
Santos, S. M., Carrasco, M., & Yovine, S. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100335
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
- Authors
- Ippolito, D., Tramer, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette Choo, C., & Carlini, N.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the 16th International Natural Language Generation Conference
- Published
- 2023
- Pages
- 28-53
- OpenAlex
- W4389009541
- Cited by
- 28
- References
- 36
- Citations by year
- 2026: 6 · 2025: 15 · 2024: 6 · 2023: 1
- FWCI
- 4.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- International Conference on Natural Language Generation (publishedVersion, cc-by)
- Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
- ETH Zürich Research Collection
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- language models, privacy, adversarial attacks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
