The DOI matches its Crossref record; 128 of the 131 references it cites were found in a registry or an index; 62 of the 114 sources its claims cite were checked for support.
Digital forgetting in large language models: a survey of unlearning methods
Open accessAbstract
Abstract Large language models (LLMs) have become the state of the art in natural language processing. The massive adoption of generative LLMs and the capabilities they have shown have prompted public concerns regarding their impact on the labor market, privacy, the use of copyrighted work, and how these models align with human ethics and the rule of law. As a response, new regulations are being pushed, which require developers and service providers to evaluate, monitor, and forestall or at least mitigate the risks posed by their models. One mitigation strategy is digital forgetting: given a model with undesirable knowledge or behavior, the goal is to obtain a new model where the detected issues are no longer present. Digital forgetting is usually enforced via machine unlearning techniques, which modify trained machine learning models for them to behave as models trained on a subset of the original training data. In this work, we describe the motivations and desirable properties of digital forgetting when applied to LLMs, and we survey recent works on machine unlearning. Specifically, we propose a taxonomy of unlearning methods based on the reach and depth of the modifications done on the models, we discuss and compare the effectiveness of machine unlearning methods for LLMs proposed so far, and we survey their evaluation. Finally, we describe open problems of machine unlearning applied to LLMs and we put forward recommendations for developers and practitioners.
Overview
CoChat Summary
## TL;DR
This is a survey of digital forgetting and machine unlearning for large language models (LLMs), with a taxonomy organized by how deeply methods modify the model and a discussion of evaluation and open challenges. It compares 22 unlearning methods, 28 forgetting datasets, 31 retaining datasets, and 15 LLMs (plus size variants), concluding that method choice depends on the forgetting target, guarantees, utility, and runtime.
## Key findings
- The survey analyzes and compares 22 unlearning methods, 28 datasets used for unlearning and evaluation of forgetting, 31 retaining datasets, and 15 models (plus size variations) used in the literature.
- The paper proposes a taxonomy with four main categories of unlearning methods: global weight modification, local weight modification, architecture modification, and input/output modification.
- It reports that runtime for surveyed methods ranges from minutes to hours, and in one case longer than a day; examples given include minutes for Wang et al. (2023), Yao et al. (2023), Limisiewicz and Mareček (2022), Jang et al. (2022), Yu et al. (2023), hours for Chen and Yang (2023), and longer than a day for Lu et al. (2022).
- The paper notes that utility degradation in retained performance usually falls within 1–5%, though it can reach 20% in some studies; it also states that larger models tend to exhibit better utility retention.
## Methods
- Literature survey and synthesis of recent work on digital forgetting and machine unlearning in LLMs, focusing on papers mostly published in 2023 and early 2024.
- The review organizes methods by the depth and reach of the model changes and by their working principles, then compares their forgetting, retaining, and runtime evaluation practices.
- The paper’s evidence base is the set of surveyed works and their reported datasets, models, and metrics; it does not present a new experimental dataset or original model training.
## Limitations
- The paper states that no de facto standard datasets exist to explicitly evaluate unlearning methods, which makes comparisons difficult.
- It notes that most surveyed works provide no provable guarantees, relying instead on empirical evaluation or auditing; this limits ex ante assurance of forgetting.
- It says that many evaluations are dataset-specific, all surveyed methods are evaluated on English-language datasets, and there is no evidence about performance in other languages.
- Evident from the text: this is a survey, so its conclusions depend on the scope and quality of the existing literature rather than new experimental results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
42 Valid · 5 Not valid · 28 Not checked · 127 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Most papers (Eldan and Russinovich 2023; Jang et al 2022; Yao et al 2023; Pawelczyk et al 2023; Chen and Yang 2023; Ni et al 2023; Pochinkov and Schoots 2023 ) leverage the predefined performance metric of the forgetting dataset, that is, the accuracy in a classification or completion dataset.”
What the sources say
- Partly supportedcites Eldan and Russinovich 2023 sourceExperimental — accuracy not yet measured
Section 3.2.1 and Additional information (pages not provided) describe evaluating forgetting through completions of 300 prompts. But the paper uses a custom, weighted familiarity score rather than a predefined accuracy metric for a forgetting dataset. Section 3.1 uses established benchmarks to assess preservation of general capabilities, not forgetting. (LLM, conf=0.86)
- Disputedcites Jang et al 2022 sourceExperimental — accuracy not yet measured
Section 3.2 (p. 5) says the paper introduces Extraction Likelihood and uses it alongside Memorization Accuracy to determine whether target token sequences have been forgotten. The classification-dataset accuracy reported in Table 2 (Section 4.1, p. 7) measures retained general language-model performance, not forgetting. Thus this paper does not use a predefined classification- or completion-dataset accuracy metric as its forgetting measure. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
- Disputedcites Yao et al 2023 sourceExperimental — accuracy not yet measured
Section 5 (Evaluation Design) defines context-dependent unlearning efficacy rather than dataset accuracy. In the harmfulness application (Dataset and Model), the paper measures the rate of outputs flagged as harmful; in the copyrighted-content application (Setting), it measures the rate of completions exceeding a BLEU-based leakage threshold. Neither is the predefined accuracy metric attributed to this paper. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Related surveys on digital forgetting and machine unlearning, such as Qu et al (2023); Xu et al (2023); Nguyen et al (2022); Shaik et al (2023), have focused on the general ML case by considering models used to classify tabular or image data.”
What the sources say
- Not matchedcites Qu et al 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Xu et al 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Nguyen et al 2022 sourceExperimental — accuracy not yet measured
The Introduction (p. 2) describes this as a comprehensive survey of machine unlearning, and the published-datasets table (pp. 14–16) includes tabular and image classification. But the same table also covers text, sequence, and graph data, so the source does not support characterizing its focus solely as classification of tabular or image data. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, GA can cause the model to lose its understanding of the language (Eldan and Russinovich 2023 ).”
What the sources say
- Supportedcites Eldan and Russinovich 2023 source
In Section 2, the paper describes training on text while negating the loss function and explains that this could cause the model to unlearn the meaning of ordinary words rather than just the target material. It identifies the underlying risk as impairing understanding of language in general, which supports the citing sentence's qualified 'can cause' claim. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The authors demonstrate the ability of their method to decrease stereotyping in questionanswering over nine different social groups with a single prompt.”
What the sources say
- Supportedcites Gallegos et al (2024) source
The Introduction explicitly says the authors demonstrate reduced stereotyping in question-answering across nine social groups with a single prompt. The Conclusion also reports bias reductions across nine groups. The evaluation was limited to multiple-choice questions, but the citing sentence does not claim otherwise. (LLM, conf=0.95)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Furthermore, the success of unlearning depends on the specific target data and the domain of the to-be-forgotten data (Smith et al 2023).”
What the sources say
- Partly supportedcites Smith et al 2023 sourceExperimental — accuracy not yet measured
Section 12.2 (Machine Unlearning; page unavailable) says the forgetting threshold depends on which data samples are chosen and reports that larger forget sets degraded model performance. This lends limited support to the importance of target data, but the passage concerns the measurement of forgetting rather than directly establishing that unlearning success depends on the target samples. It does not address whether success depends on the domain of the data to be forgotten. (LLM, conf=0.77)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
47 more claims · 11 with their cited sources checked
Create a free account to see all 202 claims and to verify the 15 references still open.
- Probably not a substantial claim. 96
- It may state two separate claims as one. 86
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“Namely, encoder-only architectures such as BERT (Devlin et al 2018 ) –used mostly for text classification–, decoder-only archi - tectures such as GPT (Radford et al 2018) – for text generation–, and encoder-decoder architectures, such as T5 (Raffel et al 2020) –for text-to-text processing, such as…”
InfluentialAdvances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578 · cited 1×
“Mechanisms such as differentially private stochastic gradient descent (DP-SGD) (Abadi et al 2016) and private aggregation of teacher ensembles (PATE) (Papernot et al 2018) have been used in the last years to train privacy-preserving machine learning models.”
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 4×
“Even when no ex ante guarantees exist, evaluation of unlearning for privacy (which amounts to evaluating whether the model knows a particular information) has been studied for years, and techniques such as MIAs can be used for empirical evaluation (Carlini et al 2021; Nasr et al 2023).”
InfluentialDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Research works printed but not linked: 10
- Carlini N, Tramer F, Wallace E, et al (2021) Extracting training data from large language models. In: 30th USENIX Security Symposium (USENIX Security 21), pp 2633–2650
- Cettolo M, Niehues J, Stüker S, et al (2014) Report on the 11th IWSLT evaluation campaign. In: Federico M, Stüker S, Yvon F (eds) Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign@IWSLT 2014, Lake Tahoe, CA, USA, December 4-5, 2014, https://aclanthology.org/2014.iwslt-evaluation.1
- Commission European (2019) Ethics guidelines for trustworthy AI. Publications Office of the EU, doi/. https://doi.org/10.2759/346720 · doi:10.2759/346720
- Gordon A, Kozareva Z, Roemmele M (2012) SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In: Agirre E, Bos J, Diab M, et al (eds) *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012). Association for Computational Linguistics, Montréal, Canada, pp 394–398, https://aclanthology.org/S12-1052
- Houlsby N, Giurgiu A, Jastrzebski S, et al (2019) Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning, PMLR, pp 2790–2799
- Klimt B, Yang Y (2004) Introducing the enron corpus. In: CEAS 2004 - First Conference on Email and Anti-Spam, July 30-31, 2004, Mountain View, California, USA, http://www.ceas.cc/papers-2004/168.pdf
- Maas AL, Daly RE, Pham PT, et al (2011) Learning word vectors for sentiment analysis. In: Lin D, Matsumoto Y, Mihalcea R (eds) The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA. The Association for Computer Linguistics, pp 142–150, https://aclanthology.org/P11-1015/
- Radford A, Wu J, Child R et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9
- Raffel C, Shazeer N, Roberts A et al (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21(1):5485–5551
- Rowling JK (2000) Harry Potter and the Sorcerer’s Stone. Bloomsbury
Cited by
A Survey of Machine Unlearning
Nguyen, T. T., Huynh, T. T., Ren, Z., Nguyen, P. L., Liew, A. W.-C., Yin, H., & Nguyen, Q. V. H. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3749987 · Cited by 164
A Survey of Machine Unlearning
Nguyen, T. T., Huynh, T. T., Ren, Z., Nguyen, P. L., Liew, A. W.-C., Yin, H., & Nguyen, Q. V. H. (2022) · arXiv · doi:10.48550/arxiv.2209.02299 · Cited by 193
Machine Unlearning: Taxonomy, Metrics, Applications, Challenges, and Prospects
Li, N., Zhou, C., Gao, Y., Chen, H., Zhang, Z., Kuang, B., & Fu, A. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3530988 · Cited by 132
“Studies indicate that LLMs can generate biased output, such as “men are suited to be engineers, women to be nurses” [27].”
Road of Large Language Model: Source, Challenge, and Future Perspectives
Zhao, W., Yang, X. S., Lv, Z., Xu, C. X., & Guan, Z. (2025) · Research · doi:10.34133/research.0655 · Cited by 20
“Therefore, using methods such as unlearning [ 40 ] or contrastive learning [ 41 ], we can guide LLMs to avoid generating negative evaluation outputs, thereby achieving human preference alignment in complex scenarios.”
Position: LLM Unlearning Benchmarks are Weak Measures of Progress
Thaker, P., Hu, S., Kale, N., Maurya, Y., Wu, Z. S., & Smith, V. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00035 · Cited by 77
“We point the reader to a number of comprehensive surveys on unlearning for further background, including [18, 19, 20, 21, 22, 23, 24, 25, 26, 27].”
Metadata
- Authors
- Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K.
- Affiliations
- Universitat Rovira i Virgili (ES)
- Corresponding author
- Sánchez, David
- Venue
- Artificial Intelligence Review
- Published
- 2025-01-13
- Volume
- 58
- Issue
- 3
- ISSN
- 0269-2821 (ISSN-L)
- 1573-7462
- OpenAlex
- W4406299883
- Semantic Scholar
- 41b4cd49570fc4330b600d85da1994b0532a8f95
- Cited by
- 31
- References
- 67
- Citations by year
- 2026: 16 · 2025: 15
- FWCI
- 12.54
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Artificial Intelligence Review (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, other-oa)
- Rovira & Virgili repository (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Universitat Rovira i Virgili
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- machine unlearning, large language models, AI regulation, AI risk mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Digital forgetting in large language models: a survey of unlearning methods. https://app.cochat.ai/papers/10.1007/s10462-024-11078-6
- The paper
- Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K. (2025). Digital forgetting in large language models: a survey of unlearning methods. Artificial Intelligence Review. https://doi.org/10.1007/s10462-024-11078-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.