Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 128 of the 131 references it cites were found in a registry or an index; 62 of the 114 sources its claims cite were checked for support.

Digital forgetting in large language models: a survey of unlearning methods

Open access
Blanco-Justicia, Alberto, Jebreel, Najeeb, Manzanares-Salor, Benet, Sánchez, David, Domingo-Ferrer, Josep, Collell, Guillem, Eeik Tan, Kuan · 2025 · Artificial Intelligence Review

Abstract

Abstract Large language models (LLMs) have become the state of the art in natural language processing. The massive adoption of generative LLMs and the capabilities they have shown have prompted public concerns regarding their impact on the labor market, privacy, the use of copyrighted work, and how these models align with human ethics and the rule of law. As a response, new regulations are being pushed, which require developers and service providers to evaluate, monitor, and forestall or at least mitigate the risks posed by their models. One mitigation strategy is digital forgetting: given a model with undesirable knowledge or behavior, the goal is to obtain a new model where the detected issues are no longer present. Digital forgetting is usually enforced via machine unlearning techniques, which modify trained machine learning models for them to behave as models trained on a subset of the original training data. In this work, we describe the motivations and desirable properties of digital forgetting when applied to LLMs, and we survey recent works on machine unlearning. Specifically, we propose a taxonomy of unlearning methods based on the reach and depth of the modifications done on the models, we discuss and compare the effectiveness of machine unlearning methods for LLMs proposed so far, and we survey their evaluation. Finally, we describe open problems of machine unlearning applied to LLMs and we put forward recommendations for developers and practitioners.

Overview

CoChat Summary

## TL;DR

This is a survey of digital forgetting and machine unlearning for large language models (LLMs), with a taxonomy organized by how deeply methods modify the model and a discussion of evaluation and open challenges. It compares 22 unlearning methods, 28 forgetting datasets, 31 retaining datasets, and 15 LLMs (plus size variants), concluding that method choice depends on the forgetting target, guarantees, utility, and runtime.

## Key findings

  • The survey analyzes and compares 22 unlearning methods, 28 datasets used for unlearning and evaluation of forgetting, 31 retaining datasets, and 15 models (plus size variations) used in the literature.
  • The paper proposes a taxonomy with four main categories of unlearning methods: global weight modification, local weight modification, architecture modification, and input/output modification.
  • It reports that runtime for surveyed methods ranges from minutes to hours, and in one case longer than a day; examples given include minutes for Wang et al. (2023), Yao et al. (2023), Limisiewicz and Mareček (2022), Jang et al. (2022), Yu et al. (2023), hours for Chen and Yang (2023), and longer than a day for Lu et al. (2022).
  • The paper notes that utility degradation in retained performance usually falls within 1–5%, though it can reach 20% in some studies; it also states that larger models tend to exhibit better utility retention.

## Methods

  • Literature survey and synthesis of recent work on digital forgetting and machine unlearning in LLMs, focusing on papers mostly published in 2023 and early 2024.
  • The review organizes methods by the depth and reach of the model changes and by their working principles, then compares their forgetting, retaining, and runtime evaluation practices.
  • The paper’s evidence base is the set of surveyed works and their reported datasets, models, and metrics; it does not present a new experimental dataset or original model training.

## Limitations

  • The paper states that no de facto standard datasets exist to explicitly evaluate unlearning methods, which makes comparisons difficult.
  • It notes that most surveyed works provide no provable guarantees, relying instead on empirical evaluation or auditing; this limits ex ante assurance of forgetting.
  • It says that many evaluations are dataset-specific, all surveyed methods are evaluated on English-language datasets, and there is no evidence about performance in other languages.
  • Evident from the text: this is a survey, so its conclusions depend on the scope and quality of the existing literature rather than new experimental results.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10462-024-11078-6

Cites

128 of 131 references verified1 citation errors · 2 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

42 Valid · 5 Not valid · 28 Not checked · 127 Own results and interpretations

47 more claims · 11 with their cited sources checked

Create a free account to see all 202 claims and to verify the 15 references still open.

See all claims — free account
  • Probably not a substantial claim. 96
  • It may state two separate claims as one. 86

Create a free account to see these claims.

Citations

Cites

Show all 67 references

Research works printed but not linked: 10

  1. Carlini N, Tramer F, Wallace E, et al (2021) Extracting training data from large language models. In: 30th USENIX Security Symposium (USENIX Security 21), pp 2633–2650
  2. Cettolo M, Niehues J, Stüker S, et al (2014) Report on the 11th IWSLT evaluation campaign. In: Federico M, Stüker S, Yvon F (eds) Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign@IWSLT 2014, Lake Tahoe, CA, USA, December 4-5, 2014, https://aclanthology.org/2014.iwslt-evaluation.1
  3. Commission European (2019) Ethics guidelines for trustworthy AI. Publications Office of the EU, doi/. https://doi.org/10.2759/346720 · doi:10.2759/346720
  4. Gordon A, Kozareva Z, Roemmele M (2012) SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In: Agirre E, Bos J, Diab M, et al (eds) *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012). Association for Computational Linguistics, Montréal, Canada, pp 394–398, https://aclanthology.org/S12-1052
  5. Houlsby N, Giurgiu A, Jastrzebski S, et al (2019) Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning, PMLR, pp 2790–2799
  6. Klimt B, Yang Y (2004) Introducing the enron corpus. In: CEAS 2004 - First Conference on Email and Anti-Spam, July 30-31, 2004, Mountain View, California, USA, http://www.ceas.cc/papers-2004/168.pdf
  7. Maas AL, Daly RE, Pham PT, et al (2011) Learning word vectors for sentiment analysis. In: Lin D, Matsumoto Y, Mihalcea R (eds) The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA. The Association for Computer Linguistics, pp 142–150, https://aclanthology.org/P11-1015/
  8. Radford A, Wu J, Child R et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9
  9. Raffel C, Shazeer N, Roberts A et al (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21(1):5485–5551
  10. Rowling JK (2000) Harry Potter and the Sorcerer’s Stone. Bloomsbury

Cited by

See all 31

Metadata

Authors
Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K.
Affiliations
Universitat Rovira i Virgili (ES)
Corresponding author
Sánchez, David
Venue
Artificial Intelligence Review
Published
2025-01-13
Volume
58
Issue
3
ISSN
  • 0269-2821 (ISSN-L)
  • 1573-7462
OpenAlex
W4406299883
Cited by
31
References
67
Citations by year
2026: 16 · 2025: 15
FWCI
12.54
Type
journal article
Language
en
Open access
Licence
Locations
  • Artificial Intelligence Review (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • Rovira & Virgili repository (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
Universitat Rovira i Virgili
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
machine unlearning, large language models, AI regulation, AI risk mitigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Digital forgetting in large language models: a survey of unlearning methods. https://app.cochat.ai/papers/10.1007/s10462-024-11078-6
The paper
Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K. (2025). Digital forgetting in large language models: a survey of unlearning methods. Artificial Intelligence Review. https://doi.org/10.1007/s10462-024-11078-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.