CoChat
Sign Up
PreprintCited by 5Privacy-Preserving Technologies in DataAdversarial Robustness in Machine LearningPrivacy, Security, and Data Protection

Knowledge Unlearning for Mitigating Privacy Risks in Language Models

Open access
Jang, Joel, Yoon, Dongkeun, Yang, Sohee, Cha, Sungmin, Lee, Moontae, Logeswaran, Lajanugen, Seo, Minjoon·2022·arXiv
DOI 10.48550/arxiv.2210.01504arXiv 2210.01504
Open this paper in CoChat — free accountOpen PDF

Abstract

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger LMs; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method and a decoding method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. We release the code and dataset needed to replicate our results at https://github.com/joeljang/knowledge-unlearning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2210.01504

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most surveyed papers leverage the predefined performance metric of the forgetting dataset, that is, accuracy in a classification or completion dataset.

    “Most papers (Eldan and Russinovich 2023; Jang et al 2022; Yao et al 2023; Pawelczyk et al 2023; Chen and Yang 2023; Ni et al 2023; Pochinkov and Schoots 2023 ) leverage the predefined performance metric of the forgetting dataset, that is, the accuracy in a classification or completion dataset.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Jang et al 2022

    Section 3.2 (p. 5) says the paper introduces Extraction Likelihood and uses it alongside Memorization Accuracy to determine whether target token sequences have been forgotten. The classification-dataset accuracy reported in Table 2 (Section 4.1, p. 7) measures retained general language-model performance, not forgetting. Thus this paper does not use a predefined classification- or completion-dataset accuracy metric as its forgetting measure. (LLM, conf=0.86)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Jang et al (2022) propose the extraction likelihood metric (n-gram overlaps) and the memorization accuracy metric (matching identical token sequences), and define unlearning as complete when both are below a threshold for the forgetting test set.

    “Jang et al (2022) propose the extraction likelihood metric, which considers overlaps of n -grams, and the memorization accuracy metric, for matching identical token sequences. They define unlearning as complete when both metrics are lower than a threshold for the forgetting test set.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Jang et al (2022)

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • The first evaluation approach applies forgetting techniques to a standard, publicly available, pre-trained LLM with pre-existing biases from its training data.

    “The first, a more realistic scenario contemplated by Jang et al ( 2022); Yao et al (2023); Yu et al (2023 ); Limisiewicz and Mareček ( 2022), involves applying forgetting techniques to a standard, publicly available, pre-trained LLM with pre-existing biases from its training data.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Jang et al 2022

    Section 4.1 (p. 6) says the authors applied unlearning to GPT-Neo models already pretrained on the Pile, and distinguishes those models from others that were not open-sourced. The Abstract (p. 1) describes information memorized during pretraining as a privacy risk, but the cited passages do not establish that the models had pre-existing biases from their training data. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Exact generation aims to prevent the model from generating verbatim copies of the text to be forgotten, and the perplexity metric is often employed for this.

    “Exact generation : The goal is to prevent the model from generating verbatim copies of the text to be forgotten. This is common in scenarios involving copyright. To that end, the perplexity metric is often employed (Jang et al 2022; Yao et al 2023; Wu et al 2023; Pochinkov and Schoots 2023).”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Jang et al 2022

    Section 3.2 (p. 5) defines forgetting in terms of whether target token sequences remain susceptible to extraction, supporting the goal of preventing reproduction of forgotten text. But it uses extraction likelihood and memorization accuracy to assess that goal. Appendix B (p. 14) measures perplexity on separate validation samples, not as a metric for exact generation or forgetting. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Jang et al (2022) found that unlearning many samples at once substantially degrades the performance of LLMs, and unlearning them sequentially can mitigate this degradation.

    “The authors found that unlearning many samples at once substantially degrades the performance of LLMs, and unlearning them sequentially can mitigate this degradation.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Jang et al (2022)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 49

  1. Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pp. 308-318, 2016.
  2. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 2357-2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1245. URL https://aclanthology.org/N19-1245 . · DOI
  3. Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private bert. arXiv preprint arXiv:2108.01624 , 2021. · arXiv
  4. Tuomas Aura, Thomas A Kuhn, and Michael Roe. Scanning electronic documents for personally identifiable information. In Proceedings of the 5th ACM workshop on Privacy in electronic society , pp. 41-50, 2006.
  5. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pp. 7432-7439, 2020.
  6. Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP) , pp. 141-159. IEEE, 2021.
  7. Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tram` er. What does it mean for a language model to preserve privacy? arXiv preprint arXiv:2202.05520 , 2022. · arXiv
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.
  9. Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy , pp. 463-480. IEEE, 2015.
  10. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pp. 2633-2650, 2021.
  11. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 , 2022a. · arXiv
  12. Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems , 2022b. URL https://openreview.net/forum?id=ErUlLrGaVEU .
  13. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv , abs/1803.05457, 2018. · arXiv
  14. Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association , 24(3):596-606, 2017.
  15. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=r1l73iRqKm .
  16. Cynthia Dwork. Differential privacy: A survey of results. In International conference on theory and applications of models of computation , pp. 1-19. Springer, 2008.
  17. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pp. 265-284. Springer, 2006.
  18. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020. · arXiv
  19. Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems , 32, 2019.
  20. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9304-9312, 2020.
  21. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) , pp. 394-398, Montr´ eal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https://aclanthology.org/S12-1052 .
  22. Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pp. 11516-11524, 2021.
  23. Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205.12628 , 2022. · arXiv
  24. Matthew Jagielski, Om Thakkar, Florian Tram` er, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. Measuring forgetting of memorized training examples. arXiv preprint arXiv:2207.00099 , 2022. · arXiv
  25. Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating everevolving language models. arXiv preprint arXiv:2204.14211 , 2022a. · arXiv
  26. Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. In International Conference on Learning Representations , 2022b. URL https: //openreview.net/forum?id=vfsRB5MImo9 .
  27. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 2567-2577, 2019.
  28. Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.06539 , 2022. · arXiv
  29. Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 8460-8478, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.579. URL https://aclanthology.org/ 2022.acl-long.579 . · DOI · Paper page
  30. Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems , 34:29348-29363, 2021.
  31. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris CallisonBurch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 8424-8445, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.577. URL https://aclanthology.org/ 2022.acl-long.577 . · DOI · Paper page
  32. Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wallace. Does bert pretrained on clinical notes reveal sensitive data? In NAACL-HLT , pp. 946-959, 2021. URL https://doi.org/10.18653/v1/2021.naacl-main.73 . · DOI
  33. Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=bVuP3ltATMz .
  34. Pierre Lison, Ildik´ o Pil´ an, David Sanchez, Montserrat Batet, and Lilja Øvrelid. Anonymisation models for text data: State of the art, challenges and future directions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 4188-4203, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.323. URL https://aclanthology.org/2021.acl-long.323 . · DOI · Paper page
  35. Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami Smaili, Rahul Gupta, and Richard Zemel. Differentially private decoding in large language models. arXiv preprint arXiv:2205.13621 , 2022. · arXiv
  36. Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review , 29(3):229-235, 2013.
  37. Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N Ravi. Deep unlearning via randomized conditionally independent hessians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10422-10431, 2022.
  38. Denis Paperno, Germ´ an Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern´ andez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1144. URL https://aclanthology.org/P16-1144 . · DOI · Paper page
  39. Fabio Petroni, Tim Rockt¨ aschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? In EMNLP , 2019.
  40. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic opendomain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 5370-5381, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1534. URL https: //aclanthology.org/P19-1534 . · DOI
  41. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.
  42. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pp. 3-18. IEEE, 2017.
  43. Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. Can you put it all together: Evaluating conversational agents' ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 2021-2030, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.183. URL https://aclanthology.org/2020.acl-main.183 . · DOI
  44. Kushal Tirumala, Aram H Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. arXiv preprint arXiv:2205.10770 , 2022. · arXiv
  45. Eduard Fosch Villaronga, Peter Kieseberg, and Tiffany Li. Humans forget, machines remember: Artificial intelligence and the right to be forgotten. Computer Law & Security Review , 34(2): 304-313, 2018.
  46. Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=Q42f0dfjECO .
  47. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019. · arXiv
  48. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. · arXiv
  49. Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=ei3SY1\_zYsE .

Cited by

Sorted by citations

Authors
Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., & Seo, M.
Affiliations
  • Korea Advanced Institute of Science and Technology (KR)
  • Konkuk University (KR)
  • University of Illinois Chicago (US)
  • LG AI Research (South Korea) (KR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2210.01504
arXiv
2210.01504
OpenAlex
W4302306312
Cited by
4
References
0
Citations by year
2026: 1 · 2025: 1 · 2023: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Korea Advanced Institute of Science and Technology (2019-0-00075)
  • Ministry of Science and ICT, South Korea (2019-0-00075, No.2019-0-00075)
  • Institute for Information and Communications Technology Promotion (2019-0-00075)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
Keywords
data preprocessing, differential privacy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review