Extracting Training Data from Large Language Models
Open accessAbstract
It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Hallucination can lead to potential privacy violations.
“Hallucination can also lead to potential privacy violations.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [26]
Section 3.3 (pp. 4–5) says language models can disclose memorized training data in inappropriate contexts, creating privacy risks. It does not identify hallucination as the cause; the privacy mechanism it describes is memorization and extraction. (LLM, conf=0.82)
Experimental — accuracy not yet measuredClient-level differential privacy has been proposed for FL.
“In order to prevent any client from trying to reconstruct the private data of another client by exploiting the global model, client-level differential privacy [338] has been proposed for FL.”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [338]
Section 8 (p. 12) discusses differential privacy as a defense against training-data leakage and reports privacy benefits from user-level DP models for language modeling. It does not say that client-level DP has been proposed for federated learning or that it prevents one client from reconstructing another’s data through a global model. (LLM, conf=0.82)
Experimental — accuracy not yet measuredUser-level differential privacy can be used in FL's iterative training process.
“user-level differential privacy can be used in FL's iterative training process [3, 338, 336, 68]”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [338]
Section 8 (p. 12) says user-level differential privacy has shown privacy benefits for language models and describes differentially private training. It does not address federated learning (FL) or its iterative training process, so it supports only the differential-privacy training part of the claim. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe probabilistic nature of LLMs and diffusion models limits their utility due to inherent instabilities and potential biases in training data.
“The probabilistic nature of LLMs and diffusion models further limits their utility due to inherent instabilities and potential biases in their training data [71].”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [71]
Section 4.2 (p. 5) describes language models as probabilistic generative models, supporting that limited part of the claim. The paper discusses privacy leakage from training data (Section 8, p. 12), but does not say that probabilistic behavior limits utility through inherent instability or training-data bias, and it does not address diffusion models. (LLM, conf=0.78)
Experimental — accuracy not yet measuredNeural networks possess an intrinsic tendency to memorize training data.
“Neural networks possess an intrinsic tendency to memorize training data [35]”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [35]
Section 3.1 (p. 3) says memorization is in many ways an essential component of language models, and the Introduction (p. 1) demonstrates that large language models memorize individual training examples. The claim extends that finding to neural networks generally; the cited passages do not establish an intrinsic tendency across all neural networks. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Language models can be prompted to recover and generate sensitive personal information from the training corpus.
“Carlini et al. [26] demonstrate that language models can be prompted to recover and generate sensitive personal information from the training corpus (e.g., email address, phone/fax number, and physical address).”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [26]
Trained ML models might be queried for specific information about individuals.
“Given that trained ML models might be queried for specific information about individuals [26]”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [26]
Previous work [284] has demonstrated that large language models, especially GPT-2 [285], memorize and leak individual training examples.
“Previous work [284] has demonstrated that large language models, especially GPT-2 [285], memorize and leak individual training examples.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [284]
GMAI models will probably use large architectures, and larger models are more prone to memorizing training data and directly repeating it to users.
“Furthermore, GMAI models will probably use large architectures, but larger models are more prone to memorizing training data and directly repeating it to users 47 .”
From Foundation models for generalist medical artificial intelligence (2023) · cites this paper as [47]
Machine learning models usually have different confidence levels toward the model's outputs.
“Machine learning models usually have different confidence levels toward the model's outputs [101].”
From Machine Unlearning: A Survey (2024) · cites this paper as [101]
Many algorithms used in federated learning such as Secure Aggregation and mechanisms of adding noise to achieve differential privacy are not designed to work with compressed or quantized communications.
“Many algorithms used in federated learning such as Secure Aggregation [79] and mechanisms of adding noise to achieve differential privacy [3, 338] are not designed to work with compressed or quantized communications.”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [338]
Section 8 (p. 12) discusses differential privacy for training language models and its privacy–utility tradeoff. It does not address whether differential-privacy mechanisms work with compressed or quantized communications. Searches for those terms and for federated learning and secure aggregation found no passage addressing the claimed incompatibility. (LLM, conf=0.75)
Experimental — accuracy not yet measuredPrior work has explored avenues to mitigate the privacy-accuracy trade-off, including collecting more private data, designing privacy-friendly model architectures, or leveraging priors on the private data domain.
“Prior work has explored a number of avenues to mitigate this trade-off between privacy and accuracy, including collecting more private data [338], designing privacy-friendly model architectures [367], or leveraging priors on the private data domain [449].”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [338]
The citation [338] is attached to “collecting more private data,” not to the other strategies in the sentence. Section 8 (p. 12) discusses the privacy–utility trade-off and recommends filtering sensitive content from training data, but does not describe collecting more private data to mitigate that trade-off. Searches for that strategy and for using larger datasets found no supporting passage. (LLM, conf=0.73)
Experimental — accuracy not yet measuredStructured query interfaces and strict delimitation of untrusted inputs are promising, but they remain difficult to enforce across realistic tool chains.
“Structured query interfaces and strict delimitation of untrusted inputs are promising, but they remain difficult to enforce across realistic tool chains [82], [161], [172].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [172]
The Threat Model (p. 4) discusses black-box queries to a language model, and Mitigating Privacy Leakage in LMs (p. 12) discusses training-data safeguards. Neither addresses structured query interfaces, delimiting untrusted inputs, or enforcement across tool chains; searches for those concepts found no supporting passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Self-citationLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · see citing works
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · see citing works
RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response
Erlingsson, Ú., Pihur, V., & Korolova, A. (2014) · Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2660267.2660348 · Cited by 1,838 · see citing works
Self-citationScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 2× · see citing works
“Moreover, it increases training time, which can further reduce accuracy because current LMs are limited by the cost of training [31, 33, 51].”
BackgroundSelf-citationHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242 · see citing works
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
Yeom, S., Giacomelli, I., Fredrikson, M., & Jha, S. (2018) · 2018 IEEE 31st Computer Security Foundations Symposium (CSF) · doi:10.1109/csf.2018.00027 · Cited by 1,001 · see citing works
Learning Differentially Private Recurrent Language Models
McMahan, H. B., Ramage, D., Talwar, K., & Zhang, L. (2017) · arXiv · doi:10.48550/arxiv.1710.06963 · Cited by 849 · see citing works
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · see citing works
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2019) · arXiv · doi:10.48550/arxiv.1911.00650 · Cited by 264
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.164 · Cited by 264 · see citing works
Privacy Risks of General-Purpose Language Models
Pan, X., Zhang, M., Ji, S., & Yang, M. (2020) · 2020 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40000.2020.00095 · Cited by 188 · see citing works
Flagged first, then by citations
Research works printed but not linked: 44
- [1][1] Martín Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM CCS , 2016. -
- [4][4] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR , 2015. -
- [5][5] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. JMLR , 2003. -
- [6][6] Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? arXiv preprint arXiv:2012.06421 , 2020. - · arXiv
- [7][7] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020. - · arXiv
- [8][8] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium , 2019. -
- [9][9] Kamalika Chaudhuri and Claire Monteleoni. Privacypreserving logistic regression. In NIPS , 2009. -
- [10][10] Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. Gmail smart compose: Real-Time assisted writing. In KDD , 2019. -
- [11][11] Andrea Continella, Yanick Fratantonio, Martina Lindorfer, Alessandro Puccetti, Ali Zand, Christopher Kruegel, and Giovanni Vigna. Obfuscation-Resilient Privacy Leak Detection for Mobile Apps Through Differential Analysis. In NDSS , 2017. -
- [12][12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL , 2019. -
- [20][20] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In NeurIPS , 2020. -
- [21][21] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In ACM CCS , 2015. -
- [22][22] Philip Gage. A new algorithm for data compression. C Users Journal , 12(2):23-38, 1994. -
- [26][26] Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013. - · arXiv
- [27][27] Peter Henderson, Koustuv Sinha, Nicolas AngelardGontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in datadriven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 123-129, 2018. -
- [28][28] Sorami Hisamoto, Matt Post, and Kevin Duh. Membership inference attacks on sequence-to-sequence models: Is my data in your machine translation system? In TACL , 2020. -
- [29][29] Andrew Hoang, Antoine Bosselut, Asli Celikyilmaz, and Yejin Choi. Efficient adaptation of pretrained transformers for abstractive summarization. arXiv preprint arXiv:1906.00138 , 2019. - · arXiv
- [30][30] Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR , 2020. -
- [31][31] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In ACL , 2018. -
- [33][33] Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD? In NeurIPS , 2020. -
- [34][34] Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In USENIX Security Symposium , 2019. -
- [37][37] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In NAACL , 2016. -
- [38][38] Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E Gonzalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers. In ICML , 2020. -
- [40][40] Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. Understanding membership inferences on well-generalized learning models. arXiv preprint arXiv:1802.04889 , 2018. - · arXiv
- [46][46] Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization. In ACM SIGSAC , 2018. -
- [47][47] Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In IEEE S&P , 2019. -
- [48][48] Helen Nissenbaum. Privacy as contextual integrity. Washington Law Review , 2004. -
- [51][51] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? In EMNLP , 2019. -
- [55][55] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR , 2020. -
- [56][56] Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. Training production language models without memorizing user data. arXiv preprint arXiv:2009.10031 , 2020. - · arXiv
- [57][57] Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review , 1990. -
- [59][59] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In EMNLP , 2020. -
- [61][61] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL , 2016. -
- [62][62] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP , 2020. -
- [63][63] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 , 2019. - · arXiv
- [64][64] Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In ACM CCS , 2015. -
- [66][66] Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In ACM CCS , 2020. -
- [67][67] Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. In KDD , 2018. -
- [68][68] Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. Understanding unintended memorization in federated learning. arXiv preprint arXiv:2006.07490 , 2020. - · arXiv
- [70][70] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , 2017. -
- [73][73] Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. Analyzing information leakage of updates to natural language models. In ACM CCS , 2020. -
- [74][74] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In NeurIPS , 2019. -
- [75][75] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR , 2017. -
- [76][76] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. DialoGPT: Large-scale generative pretraining for conversational response generation. In ACL Demo Track , 2020.
Cited by
Sorted by citations
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · see citing works
“Furthermore, GMAI models will probably use large architectures, but larger models are more prone to memorizing training data and directly repeating it to users 47 .”
MethodTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · see citing works
“Observed risks include privacy violations [34].”
BackgroundInfluentialBandi, A., Adapa, P. V. S. R., & Kuchi, Y. E. V. P. K. (2023) · Future Internet · doi:10.3390/fi15080260 · Cited by 638 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
Promises and challenges of generative artificial intelligence for human learning
Yan, L., Greiff, S., Teuber, Z., & Gašević, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02004-5 · Cited by 499 · see citing works
“Studies on human–AI collaboration indicate that cocreating content with GenAI can meet diverse learning needs, pro viding students with relevant and accessible materials to support their individual paths efficiently and creatively 21–23 .”
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514 · see citing works
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424 · see citing works
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · see citing works
“…it is now increasingly understood that large language models can learn undesirable (e.g. biased) behavior from unlabeled training data, e.g., Bender and Koller [2020] and Bender et al. [2021], or can reveal training data, as well as sensitive information in the training data [Carlini et al., 2020].”
BackgroundLarge language models in health care: Development, applications, and challenges
Yang, R., Tan, T. F., Lu, W., Thirunavukarasu, A. J., Ting, D. S. W., & Liu, N. (2023) · Health Care Science · doi:10.1002/hcs2.61 · Cited by 366 · see citing works
Wordcraft: Story Writing With Large Language Models
Yuan, A., Coenen, A., Reif, E., & Ippolito, D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511105 · Cited by 374 · see citing works
When large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367 · see citing works
“Previous work [269] has demonstrated that large language models, especially GPT-2 [270], memorize and leak individual training examples.”
Deduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304 · see citing works
Self-citationDocumenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · see citing works
Artificial intelligence risk management framework :
National Institute of Standards and Technology (US) (2024) · NIST Trustworthy and Responsible AI · doi:10.6028/nist.ai.600-1 · Cited by 274 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
“Neural networks, especially large language models, possess an intrinsic tendency to memorize training data (Carlini et al., 2021).”
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207 · see citing works
Self-citationMultimodal datasets: misogyny, pornography, and malignant stereotypes
Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021) · arXiv · doi:10.48550/arxiv.2110.01963 · Cited by 160 · see citing works
Xu, H., Zhu, T., Zhang, L., Zhou, W., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3603620 · Cited by 164 · see citing works
Societal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131 · see citing works
Liesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111 · see citing works
Han, A., Zhou, X., Cai, Z., Han, S., Ko, R., Corrigan, S., & Peppler, K. A. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642438 · Cited by 114 · see citing works
“AI hallucination occurs when a machine learning model generates content that doesn’t align with the intended output [13, 18].”
History, development, and principles of large language models: an introductory survey
Wang, Z., Chu, Z., Doan, T. V., Ni, S., Yang, M., & Zhang, W. (2025) · AI and Ethics · doi:10.1007/s43681-024-00583-7 · Cited by 109 · see citing works
Differentially Private Fine-tuning of Language Models
Yu, D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Kamath, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.06500 · Cited by 101 · see citing works
Zhang, Z., Jia, M., Lee, H.-P. (., Yao, B., Das, S., Lerner, A., Wang, D., & Li, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642385 · Cited by 95 · see citing works
“These disclosures, in turn, can expose users to a whole suite of emerging privacy and security risks [7, 8, 52, 66].”
BackgroundMethodResultInfluentialDou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N., & Choi, Y. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.501 · Cited by 118 · see citing works
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N. A., & Choi, Y. (2021) · arXiv · doi:10.48550/arxiv.2107.01294 · Cited by 90 · see citing works
Data Contamination: From Memorization to Exploitation
Magar, I., & Schwartz, R. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.18 · Cited by 88 · see citing works
Retrieval-based Language Models and Applications
Asai, A., Min, S., Zhong, Z., & Chen, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts) · doi:10.18653/v1/2023.acl-tutorials.6 · Cited by 81 · see citing works
The Gradient of Generative AI Release: Methods and Considerations
Solaiman, I. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593981 · Cited by 90 · see citing works
Detecting LLM-Generated Text in Computing Education: Comparative Study for ChatGPT Cases
Orenstrakh, M. S., Karnalim, O., Suárez, C. A., & Liut, M. (2024) · 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00027 · Cited by 79 · see citing works
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A., & Viviano, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.24 · Cited by 69 · see citing works
A Systematic Review of Adversarial Machine Learning Attacks, Defensive Controls and Technologies
Malik, J., Muthalagu, R., & Pawar, P. M. (2024) · IEEE Access · doi:10.1109/access.2024.3423323 · Cited by 76 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70 · see citing works
How should the advancement of large language models affect the practice of science?
Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401227121 · Cited by 74 · see citing works
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
Goodman, S. M., Buehler, E., Clary, P., Coenen, A., Donsbach, A., Horne, T. N., Lahav, M., MacDonald, R., Michaels, R. B., Narayanan, A., et al. (2022) · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3517428.3544819 · Cited by 71 · see citing works
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548 · see citing works
EW-Tune: A Framework for Privately Fine-Tuning Large Language Models with Differential Privacy
Behnia, R., Ebrahimi, M. R., Pacheco, J., & Padmanabhan, B. (2022) · 2022 IEEE International Conference on Data Mining Workshops (ICDMW) · doi:10.1109/icdmw58026.2022.00078 · Cited by 60 · see citing works
Identifying and Mitigating the Security Risks of Generative AI
Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. (2023) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000041 · Cited by 57 · see citing works
Self-citationGenerative Ai in EU Law: Liability, Privacy, Intellectual Property, and Cybersecurity
Novelli, C., Casolari, F., Hacker, P., Spedicato, G., & Floridi, L. (2024) · doi:10.2139/ssrn.4821952 · Cited by 57 · see citing works
<scp>PLeak:</scp> Prompt Leaking Attacks against Large Language Model Applications
Hui, B., Yuan, H., Gong, N., Burlina, P., & Cao, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670370 · Cited by 55 · see citing works
Discovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09251 · Cited by 53 · see citing works
Self-citationUnlearn What You Want to Forget: Efficient Unlearning for LLMs
Chen, J., & Yang, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.738 · Cited by 52 · see citing works
Changing the World by Changing the Data
Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51 · see citing works
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 45 · see citing works
Natural Language Processing in Marketing
Hartmann, J., & Netzer, O. (2023) · Artificial Intelligence in Marketing · doi:10.1108/s1548-643520230000020011 · Cited by 38 · see citing works
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 36 · see citing works
“…content have drawn significant attention from the community (Bang et al., 2023; Alkaissi & McFarlane, 2023; Zheng et al., 2023; Shi et al., 2023; Carlini et al., 2021; Huang et al., 2022; Shao et al., 2023; Li et al., 2023; Wei et al., 2023; Zhou et al., 2023b; Zou et al., 2023, inter alia ).”
BackgroundJacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.308 · Cited by 36 · see citing works
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10160 · Cited by 36 · see citing works
Preventing generation of verbatim memorization in language models gives a false sense of privacy
Ippolito, D., Tramer, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette Choo, C., & Carlini, N. (2023) · Proceedings of the 16th International Natural Language Generation Conference · doi:10.18653/v1/2023.inlg-main.3 · Cited by 28 · see citing works
Self-citationAI’s Epistemic Harm: Reinforcement Learning, Collective Bias, and the New AI Culture War
Humphreys, D. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00928-y · Cited by 26 · see citing works
“6 4 There are also malicious ways in which LLMs can give misleading information by being manipulated or hacked (Carlini et al., 2020 ; Humphreys et al., 2024 ).”
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20 · see citing works
Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generation
Li, Y., Liu, G., Wang, C., & Yang, Y. (2024) · arXiv · doi:10.48550/arxiv.2406.19234 · Cited by 17 · see citing works
“• Zlib Entropy Attack [38]: this attack performs MIAs using the zlib entropy of the target samples.”
What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A. S., & Viviano, J. D. (2021) · arXiv · doi:10.48550/arxiv.2105.02732 · Cited by 17 · see citing works
Membership Inference Attacks against GANs by Leveraging Over-representation Regions
Hu, H., & Pang, J. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3485338 · Cited by 16 · see citing works
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17 · see citing works
“In another work, Carlini et al. [258] performed a training data extraction attack on a GPT-2 model and extracted personally identifiable information from the training data.”
MethodLearning to Ask Like a Physician
Lehman, E., Lialin, V., Legaspi, K. E., Sy, A. J., Pile, P. T., Alberto, N. R., Ragasa, R. R., Puyat, C. V., Taliño, M. K., Alberto, I. R., et al. (2022) · Proceedings of the 4th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2022.clinicalnlp-1.8 · Cited by 10 · see citing works
Re-evaluating Theory of Mind evaluation in large language models
Hu, J., Sosa, F., & Ullman, T. (2025) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0499 · Cited by 10 · see citing works
Neural networks memorise personal information from one sample
Hartley, J., Sanchez, P. P., Haider, F., & Tsaftaris, S. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-48034-3 · Cited by 10 · see citing works
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9 · see citing works
Benchmarking Benchmark Leakage in Large Language Models
Xu, R., Wang, Z., Fan, R.-Z., & Liu, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18824 · Cited by 6 · see citing works
Feedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3 · see citing works
“…(Blodgett et al., 2020b;a; Rivera et al., 2024; Xu et al., 2024), output misinformation (Lin et al., 2021) and hallucinations (Ji et al., 2023), expose private training data (Carlini et al., 2020), and act deceptively (OpenAI, 2023a; Pan et al., 2023; Scheurer et al., 2023; Hubinger et al., 2024).”
BackgroundEIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
Liao, Z., Mo, L., Xu, C., Kang, M., Zhang, J., Xiao, C., Tian, Y., Li, B., & Sun, H. (2024) · arXiv · doi:10.48550/arxiv.2409.11295 · Cited by 1 · see citing works
“This finding aligns with conclusions from related studies (Carlini et al., 2021; Mo et al., 2024b), which suggest that more capable models are also more vulnerable to adversarial attacks.”
Do as We Do, Not as You Think: the Conformity of Large Language Models
Weng, Z., Chen, G., & Wang, W. (2025) · arXiv · doi:10.48550/arxiv.2501.13381 · Cited by 1 · see citing works
“However, research suggests that LLMs can only retain a fraction of their pre-training knowledge due to memory constraints [64, 65].”
MRMMIA: Membership Inference Attacks on Memory in Chat Agents
Chen, K., Pang, Y., & Wang, T. (2026) · arXiv
“…infeasible, attacks instead rely on single-model token statistics, including loss-or perplexity-based scores or reference-model calibration [2], paraphrase-based neighborhood comparison [14], low-probability token aggregation [23, 30], and self-calibrated reference models for the…”
Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance
Benhenda, M. (2026) · arXiv
“Carlini et al. (2021; 2022) established theoretical and empirical foundations for understanding memorization in large language models, demonstrating that training data can be extracted from models through carefully designed queries.”
How to Steal Reasoning Without Reasoning Traces
Zhang, T., Morris, J. X., & Shmatikov, V. (2026) · arXiv · doi:10.48550/arxiv.2603.07267
“In this paper, we focus on the setting where the teacher’s reasoning traces are not available .”
HCAST: Human-Calibrated Autonomy Software Tasks
Rein, D., Becker, J., Deng, A., Nix, S., Canal, C., O'Connel, D., Arnott, P., Bloom, R., Broadley, T., Garcia, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.17354
“Large language models and AI agents are capable of memorizing substantial amounts of information (Carlini et al., 2021).”
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075
Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E., Japkowicz, N., & Viktor, H. (2022) · arXiv · doi:10.48550/arxiv.2210.07321 · Cited by 196 · see citing works
In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT
Shen, X., Chen, Z., Backes, M., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.08979 · Cited by 76 · see citing works
“Previous studies have also shown that LLM is vulnerable to various types of attacks, such as adversarial attacks [27, 30, 37, 39], back-door attacks [11, 21], prompt injection [31, 57], obfuscation [40], and data extraction attacks [19].”
- Authors
- Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., & Raffel, C.
- Affiliations
- Google (United States) (US)
- University of California, Berkeley (US)
- Northeastern University (US)
- Harvard University (US)
- OpenAI (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 2633-2650
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2012.07805
- OpenAlex
- W3112689365
- Cited by
- 727
- References
- 15
- Citations by year
- 2026: 17 · 2025: 157 · 2024: 210 · 2023: 208 · 2022: 87 · 2021: 46 · 2020: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- USENIX Security Symposium
- Funders
- National Science Foundation (1804222)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- training data extraction, large language models, GPT-2, data privacy, adversarial attacks, training data leakage, personally identifiable information
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
