A Survey of Machine Unlearning
Open accessAbstract
Today, computer systems hold large amounts of personal data. Yet while such an abundance of data allows breakthroughs in artificial intelligence, and especially machine learning (ML), its existence can be a threat to user privacy, and it can weaken the bonds of trust between humans and AI. Recent regulations now require that, on request, private information about a user must be removed from both computer systems and from ML models, i.e. ``the right to be forgotten''). While removing data from back-end databases should be straightforward, it is not sufficient in the AI context as ML models often `remember' the old data. Contemporary adversarial attacks on trained models have proven that we can learn whether an instance or an attribute belonged to the training data. This phenomenon calls for a new paradigm, namely machine unlearning, to make ML models forget about particular data. It turns out that recent works on machine unlearning have not been able to completely solve the problem due to the lack of common frameworks and resources. Therefore, this paper aspires to present a comprehensive examination of machine unlearning's concepts, scenarios, methods, and applications. Specifically, as a category collection of cutting-edge studies, the intention behind this article is to serve as a comprehensive resource for researchers and practitioners seeking an introduction to machine unlearning and its formulations, design criteria, removal requests, algorithms, and applications. In addition, we aim to highlight the key findings, current trends, and new research areas that have not yet featured the use of machine unlearning but could benefit greatly from it. We hope this survey serves as a valuable resource for ML researchers and those seeking to innovate privacy technologies. Our resources are publicly available at https://github.com/tamlhp/awesome-machine-unlearning.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Related surveys on digital forgetting and machine unlearning have focused on the general ML case by considering models used to classify tabular or image data.
“Related surveys on digital forgetting and machine unlearning, such as Qu et al (2023); Xu et al (2023); Nguyen et al (2022); Shaik et al (2023), have focused on the general ML case by considering models used to classify tabular or image data.”
From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Nguyen et al 2022
The Introduction (p. 2) describes this as a comprehensive survey of machine unlearning, and the published-datasets table (pp. 14–16) includes tabular and image classification. But the same table also covers text, sequence, and graph data, so the source does not support characterizing its focus solely as classification of tabular or image data. (LLM, conf=0.84)
Experimental — accuracy not yet measuredNguyen et al refer to two levels of forgetting guarantees, exact and approximate, and Xu et al further expand approximate forgetting into strong and weak forgetting, where strong forgetting is equivalent to Nguyen et al's approximate forgetting and weak forgetting only applies to model outputs.
“Nguyen et al (2022) refer to two levels of guarantees, exact and approximate . Xu et al (2023) further expand approximate forgetting into strong and weak forgetting, where strong forgetting is equivalent to the definition of approximate forgetting by Nguyen et al ( 2022), and weak forgetting only applies to the outputs of the model.”
From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Nguyen et al 2022
Definitions of unlearning by Nguyen et al and Xu et al include the original training dataset D as a parameter of the unlearning mechanism, but access to D is not always feasible.
“Definitions of unlearning by Nguyen et al (2022); Xu et al (2023) include D as a parameter of the unlearning mechanism (as in F ( D,D f , A ( D )) ), but access to D is not always feasible.”
From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Nguyen et al 2022
The unlearning guarantees defined by Nguyen et al (2022) and Xu et al (2023) are heavily inspired by differential privacy, and privacy attacks such as MIAs or reconstruction attacks have been used to measure privacy leak risk and evaluate unlearning effectiveness.
“Not surprisingly, the unlearning guarantees defined by Nguyen et al ( 2022) and Xu et al (2023 ) are heavily inspired by differential pri -vacy, and privacy attacks, such as membership inference attacks or reconstruction attacks, have been used to measure the risk of privacy leaks and to evaluate the effectiveness of unlearning.”
From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Nguyen et al (2022)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1× · see citing works
“The MNIST dataset [83] consists of 60,000 training examples and 10,000 test samples that have been standardised to 20x20 pixels, centred by centre of mass in 28x28 pixels, and sheared by horizontally shifting so that the major axis is vertical.”
On Information and Sufficiency
Kullback, S., & Leibler, R. A. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729694 · Cited by 20,193 · cited 1× · see citing works
“5 ∗ KL(Td (x) | |m), whereM is unlearned model,Td is a competent teacher, and KL is The Kullback-Leibler divergence [84],m = M (x)+Td (x) 2 .”
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · see citing works
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300 · cited 1× · see citing works
“wrong output ( ZRF = 0) or random output ( ZRF = ( )”
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · cited 1× · see citing works
“Catastrophic forgetting is a phenomenon where deep neural networks perform badly after learning too many tasks [81].”
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578 · cited 1× · see citing works
“tially private models suffer a significant loss in accuracy even for large ε [1, 18].”
The MovieLens Datasets: History and Context
Harper, F., & Konstan, J. (2016) · TIIS · doi:10.1145/2827872 · Cited by 5,938 · cited 1× · see citing works
“Movielens [63] comprises movie ratings from MovieLens, a website that provides movie recommendations.”
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · see citing works
The Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2014) · Foundations and Trends® in Theoretical Computer Science · doi:10.1561/0400000042 · Cited by 5,557 · cited 1× · see citing works
“( ′ | 𝑤 ) is assumed to be a As a result, a candidate set of model parameters 𝑃𝑟 ( 𝑤 | 𝐷 ) is constructed by simply maximizing the posterior probability 𝑃𝑟 ( 𝑤 | 𝐷 ) , i.e.:”
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 5× · see citing works
“In [135], a set of shallow models were trained on a different training set from that used for the target model.”
InfluentialDifferential Privacy: A Survey of Results
Dwork, C. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-79228-4_1 · Cited by 3,542 · see citing works
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117 · see citing works
Andrzejak, R. G., Lehnertz, K., Mormann, F., Rieke, C., David, P., & Elger, C. E. (2001) · Physical Review E · doi:10.1103/physreve.64.061907 · Cited by 3,032 · see citing works
The EU General Data Protection Regulation (GDPR)
Voigt, P., & von dem Bussche, A. (2017) · doi:10.1007/978-3-319-57959-7 · Cited by 3,001 · cited 1× · see citing works
“However, the latter focuses only on the data perspective following the General Data Protection Regulation (GDPR) [160], while machine unlearning also addresses privacy problems from a model perspective.”
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
The information bottleneck method
Tishby, N., Pereira, F. C., & Bialek, W. (2000) · arXiv · doi:10.48550/arxiv.physics/0004057 · Cited by 2,626 · see citing works
Simplifying Graph Convolutional NetworksPreprint
Wu, F., Zhang, T., Junior, A. H. D. S., Fifty, C., Yu, T., & Weinberger, K. Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07153 · Cited by 2,251 · see citing works
Deep learning and the information bottleneck principle
Tishby, N., & Zaslavsky, N. (2015) · Information Theory Workshop · doi:10.1109/itw.2015.7133169 · Cited by 2,170 · see citing works
The EU General Data Protection Regulation (GDPR)
Russell, S. (2020) · doi:10.1093/oso/9780198826491.001.0001 · Cited by 2,103 · see citing works
Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods
Hüllermeier, E., & Waegeman, W. (2021) · Machine Learning · doi:10.1007/s10994-021-05946-3 · Cited by 1,834 · cited 1× · see citing works
“metric that measures how much we know about the optimal hypothesis in the parameter space [72] That is, are we certain that the current model parameters are optimal for a given dataset?).”
Opening the Black Box of Deep Neural Networks via InformationPreprint
Shwartz-Ziv, R., & Tishby, N. (2017) · arXiv.org · doi:10.48550/arxiv.1703.00810 · Cited by 1,674 · see citing works
Differentially Private Empirical Risk Minimization
Chaudhuri, K., Monteleoni, C., & Sarwate, A. (2009) · Journal of machine learning research · doi:10.5555/1953048.2021036 · Cited by 1,634 · cited 1×
“𝑟 _ 𝑠𝑢𝑏 𝑢 = 𝑈 ( 𝐷,𝐷 𝑓 ,𝐴 ( 𝐷 )) by generating an approximate version of 𝐷 𝑓 .”
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · see citing works
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H.-T., & Zhao, B. Y. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00031 · Cited by 1,486 · see citing works
Open Graph Benchmark: Datasets for Machine Learning on GraphsPreprint
Hu, W., Fey, M., Žitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., & Leskovec, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00687 · Cited by 1,439 · see citing works
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the LoopPreprint
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., & Xiao, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03365 · Cited by 1,339 · cited 1× · see citing works
“The objective of the Large-scale Scene Understanding (LSUN) [170] challenge is to establish a new standard for the categorization and comprehension of large-scale scenes.”
GradientBased Learning Applied to Document Recognition
Haykin, S., & Kosko, B. (2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544976.ch9 · Cited by 1,208 · cited 1× · see citing works
“The MNIST dataset [85] contains over 70,000 samples that have been standardised to 20x20 pixels and sheared by horizontally shifting so that the major axis is vertical.”
Deep face recognition: A survey
Wang, M., & Deng, W. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.081 · Cited by 1,093 · see citing works
New Insights and Perspectives on the Natural Gradient MethodPreprint
Martens, J. (2014) · Journal of machine learning research · doi:10.48550/arxiv.1412.1193 · Cited by 884 · cited 1× · see citing works
“wrong output ( ZRF = 0) or random output ( ZRF = ( )”
Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., & Backes, M. (2019) · Proceedings 2019 Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23119 · Cited by 852 · see citing works
Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2021) · 2021 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40001.2021.00019 · Cited by 810 · cited 2× · see citing works
“) is the parameterized mapping function from the input space Z to the output space (i.e., the machine weak unlearning [4].”
AI can be sexist and racist — it’s time to make it fair
Zou, J., & Schiebinger, L. (2018) · Nature · doi:10.1038/d41586-018-05707-8 · Cited by 765 · cited 1× · see citing works
“more likely to consider African-American offenders to have higher risk scores than Caucasians, even though ethnicity information is not part of the input [181].”
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681 · see citing works
Efficient noise-tolerant learning from statistical queries
Kearns, M. J. (1998) · Journal of the ACM · doi:10.1145/293347.293351 · Cited by 679 · cited 1× · see citing works
“More precisely, statistical query learning assumes that most of the learning algorithms can be represented as a sum of some efficiently computable transformations, called statistical queries [79].”
Adversarial Attacks and Defenses in Deep Learning
Ren, K., Zheng, T., Qin, Z., & Liu, X. (2020) · Engineering · doi:10.1016/j.eng.2019.12.012 · Cited by 678 · cited 2× · see citing works
“Recently, deep learning models have been shown to be vulnerable to external attacks, especially adversarial attacks [121].”
Towards Making Systems Forget with Machine Unlearning
Cao, Y., & Yang, J. (2015) · 2015 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2015.35 · Cited by 622 · see citing works
Bitansky, N., Canetti, R., Chiesa, A., & Tromer, E. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090263 · Cited by 493 · cited 1× · see citing works
“2 implies the zero difference in the distribution of weights, i.e., 𝑃𝑟 ( 𝑤 ) = 𝑃𝑟 ( 𝑤 ) , where the 𝑟 ( 𝑟 ) and 𝑤 𝑢 are the parameters of the models given by 𝑈 ( .”
Boosting methods for multi-class imbalanced data classification: an experimental review
Tanha, J., Abdi, Y., Samadi, N., Razzaghi-Asl, N., & Asadpour, M. (2020) · Journal Of Big Data · doi:10.1186/s40537-020-00349-y · Cited by 457 · cited 1× · see citing works
“However, partitioning data by class itself does not help the model’s training in the first place, as learning the differences between classes is the core of many learning algorithms [146].”
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
Golatkar, A., Achille, A., & Soatto, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00932 · Cited by 394 · cited 10× · see citing works
“overtrained deep neural networks by actively unlearning useless, obsolete, or redundant data samples that could cause catastrophic forgetting [35, 53].”
InfluentialHamilton, W. L. (2020) · Synthesis lectures on artificial intelligence and machine learning · doi:10.2200/s01045ed1v01y202009aim046 · Cited by 329 · cited 1× · see citing works
“Graph representation learning is a well-established research direction in machine learning, specifically in deep learning [20, 64].”
Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher
Chundawat, V. S., Tarun, A., Mandal, M., et al. (2022) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2205.08096 · Cited by 309
Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning
Salem, A., Bhattacharya, A., Backes, M., et al. (2019) · USENIX Security Symposium · doi:10.48550/arxiv.1904.01067 · Cited by 299
Quark: Controllable Text Generation with Reinforced UnlearningPreprint
Lu, X., Welleck, S., Jiang, L., Hessel, J., Qin, L., West, P., Ammanabrolu, P., & Choi, Y. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2205.13636 · Cited by 299 · see citing works
Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation
Wang, Z., Qinami, K., Karakozis, I. C., Genova, K., Nair, P., Hata, K., & Russakovsky, O. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00894 · Cited by 284 · see citing works
DeltaGrad: Rapid retraining of machine learning modelsPreprint
Wu, Y., Dobriban, E., & Davidson, S. (2020) · International Conference on Machine Learning · doi:10.48550/arxiv.2006.14755 · Cited by 278 · cited 3× · see citing works
“15, we only know that the density function of 𝑃𝑟 ( 𝑤 | 𝐷 ) is proportional to a function 𝑓 ( 𝑤 ) = 𝑃𝑟 ( 𝐷 | 𝑤 ) 𝑃𝑟 ( 𝑤 ) , which means 𝑃𝑟 ( 𝑤 | 𝐷 ) cannot be directly sampled.”
Algorithms that remember: model inversion attacks and data protection law
Veale, M., Binns, R., & Edwards, L. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0083 · Cited by 278 · cited 3× · see citing works
“However, it turns out that this problem cannot be completely solved due to a lack of common frameworks and resources [131, 137, 157, 159].”
Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Chen, J., & Yang, D.-Y. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2310.20150 · Cited by 274
Graph representation learning: a survey
Chen, F., Wang, Y.-C., Wang, B., & Kuo, C.-C. J. (2020) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2020.13 · Cited by 270 · see citing works
Mantelero, A. (2013) · Computer Law & Security Review · doi:10.1016/j.clsr.2013.03.010 · Cited by 264 · see citing works
FedEraser: Enabling Efficient Client-Level Data Removal from Federated Learning Models
Liu, G., Ma, X., Yang, Y., Wang, C., & Liu, J. (2021) · 2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS) · doi:10.1109/iwqos52092.2021.9521274 · Cited by 248 · see citing works
A Survey on Graph Representation Learning Methods
Khoshraftar, S., & An, A. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3633518 · Cited by 247 · see citing works
Uncertainty in Neural Networks: Approximately Bayesian EnsemblingPreprint
Pearce, T., Leibfried, F., Brintrup, A., et al. (2018) · International Conference on Artificial Intelligence and Statistics · doi:10.48550/arxiv.1810.05546 · Cited by 239 · cited 1× · see citing works
“wrong output ( ZRF = 0) or random output ( ZRF = ( )”
Learning your identity and disease from research papers
Wang, R., Li, Y. F., Wang, X., Tang, H., & Zhou, X. (2009) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/1653662.1653726 · Cited by 206 · see citing works
Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models
Zhang, Y., Chen, X., Jia, J., et al. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2405.15234 · Cited by 206
Fast Yet Effective Machine Unlearning
Tarun, A. K., Chundawat, V. S., Mandal, M., & Kankanhalli, M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3266233 · Cited by 203 · cited 2× · see citing works
“) has 𝑟 = 𝐶 \ 𝐶 𝑓 .”
The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retraining
Liu, Y., Xu, L., Yuan, X., Wang, C., & Li, B. (2022) · IEEE INFOCOM 2022 - IEEE Conference on Computer Communications · doi:10.1109/infocom48880.2022.9796721 · Cited by 202 · see citing works
Distilling Causal Effect of Data in Class-Incremental Learning
Hu, X., Tang, K., Miao, C., Hua, X., & Zhang, H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00395 · Cited by 198 · cited 1× · see citing works
“However, this requires a contradiction analysis between old and new data [69].”
When Machine Unlearning Jeopardizes Privacy
Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., & Zhang, Y. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484756 · Cited by 195 · cited 9× · see citing works
“ShallowAttack [23] Python Pytorch Model-agnostic https://github.”
InfluentialFast Yet Effective Machine Unlearning
Tarun, A. K., Chundawat, V. S., Mandal, M., & Kankanhalli, M. (2021) · arXiv · doi:10.48550/arxiv.2111.08947 · Cited by 190 · see citing works
Knowledge Neurons in Pretrained Transformers
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.581 · Cited by 187 · cited 1× · see citing works
“com/AdityaGolatkar/SelectiveForgetting Neurons [29] Python Deep learning https://github.”
A Progressive Batching L-BFGS Method for Machine LearningPreprint
Bollapragada, R., Mudigere, D., Nocedal, J., et al. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1802.05374 · Cited by 182 · see citing works
Deep learning-based unlearning of dataset bias for mri harmonisation and confound removal
Dinsdale, N. K., Jenkinson, M., & Namburete, A. I. L. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2020.117689 · Cited by 178 · see citing works
Multiple Incremental Decremental Learning of Support Vector Machines
Karasuyama, M., & Takeuchi, I. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2048039 · Cited by 176 · see citing works
Making AI Forget You: Data Deletion in Machine Learning
Ginart, A., Guan, M. Y., Valiant, G., & Zou, J. (2019) · arXiv · doi:10.48550/arxiv.1907.05012 · Cited by 175 · cited 9× · see citing works
“Ideas similar to knowledge-adaptation priors were also investigated in [52, 171].”
InfluentialHumans forget, machines remember: Artificial intelligence and the Right to Be Forgotten
Li, T., Villaronga, E. F., & Kieseberg, P. (2017) · Computer Law and Security Review · doi:10.1016/j.clsr.2017.08.007 · Cited by 173 · cited 2× · see citing works
“However, it turns out that this problem cannot be completely solved due to a lack of common frameworks and resources [131, 137, 157, 159].”
Machine Unlearning of Features and Labels
Warnecke, A., Pirch, L., Wressnegger, C., & Rieck, K. (2023) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2023.23087 · Cited by 166 · see citing works
Chen, Y., Xiong, J., Xu, W., & Zuo, J. (2019) · Cluster Computing · doi:10.1007/s10586-018-1772-4 · Cited by 166 · cited 4× · see citing works
“Diabetes [12, 24, 166] is dataset of diabetic patients collected from two approaches: traditional paper documents and an automated recording system.”
InfluentialChundawat, V. S., Tarun, A. K., Mandal, M., & Kankanhalli, M. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3265506 · Cited by 162 · cited 8× · see citing works
“[26] proposed an error-minimizing noise to generated an approximate version of Dr so that the impaired model can be trained to mitigate the performance degradation.”
InfluentialCertified Data Removal from Machine Learning Models
Guo, C., Goldstein, T., Hannun, A., & van der Maaten, L. (2019) · arXiv · doi:10.48550/arxiv.1911.03030 · Cited by 160 · cited 5× · see citing works
“The formula for calculating an AIN value is 𝐴𝐼𝑁 = 𝑟 𝑡 ( 𝑀 𝑢 ,𝑀 𝑜𝑟𝑖𝑔 ,𝛼 ) where I( 𝑤 ; 𝐷 ) is the FIM that determines the quantities of information ( )”
InfluentialFeuerriegel, S., Dolata, M., & Schwabe, G. (2020) · Business & Information Systems Engineering · doi:10.1007/s12599-020-00650-3 · Cited by 160 · see citing works
Federated Unlearning via Class-Discriminative Pruning
Wang, J., Guo, S., Xie, X., & Qi, H. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3512222 · Cited by 157 · see citing works
A Survey on Federated Unlearning: Challenges, Methods, and Future Directions
Liu, Z., Jiang, Y., Shen, J., Peng, M., Lam, K., Yuan, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3679014 · Cited by 148 · see citing works
Unrolling SGD: Understanding Factors Influencing Machine Unlearning
Thudi, A., Deza, G., Chandrasekaran, V., & Papernot, N. (2022) · 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp53844.2022.00027 · Cited by 135 · see citing works
A Multi-Batch L-BFGS Method for Machine Learning
Berahas, A., Nocedal, J., & Takác, M. (2016) · Neural Information Processing Systems · doi:10.48550/arxiv.1605.06049 · Cited by 135
Zero-shot Knowledge Transfer via Adversarial Belief MatchingPreprint
Micaelli, P., & Storkey, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09768 · Cited by 134 · cited 1× · see citing works
“Hence, inspired by [107], Chundawat et al.”
Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., & Zhang, Y. (2022) · Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3548606.3559352 · Cited by 133 · see citing works
Mixed-Privacy Forgetting in Deep Networks
Golatkar, A., Achille, A., Ravichandran, A., Polito, M., & Soatto, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00085 · Cited by 125 · see citing works
Federated Unlearning for On-Device Recommendation
Yuan, W., Yin, H., Wu, F., Zhang, S., He, T., & Wang, H. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570463 · Cited by 125 · see citing works
Self-citationLifelong Anomaly Detection Through Unlearning
Du, M., Chen, Z., Liu, C., Oak, R., & Song, D. X. (2019) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3319535.3363226 · Cited by 125 · cited 6× · see citing works
“overtrained deep neural networks by actively unlearning useless, obsolete, or redundant data samples that could cause catastrophic forgetting [35, 53].”
InfluentialMachine unlearning: linear filtration for logit-based classifiers
Baumhauer, T., Schöttle, P., & Zeppelzauer, M. (2022) · Machine Learning · doi:10.1007/s10994-022-06178-9 · Cited by 123 · see citing works
Golatkar, A., Achille, A., & Soatto, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58526-6_23 · Cited by 122 · see citing works
Fair Attribute Classification through Latent Space De-biasing
Ramaswamy, V., Kim, S. S. Y., & Russakovsky, O. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00918 · Cited by 122 · see citing works
FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning
Sattler, F., Korjakow, T., Rischke, R., & Samek, W. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3129371 · Cited by 122 · see citing works
A geometric view of optimal transportation and generative model
Lei, N., Su, K., Cui, L., Yau, S., & Gu, X. D. (2018) · Computer Aided Geometric Design · doi:10.1016/j.cagd.2018.10.005 · Cited by 121 · cited 1× · see citing works
“This is also a special case of the optimal transportation cost between two probability distributions [86].”
KGA: A General Machine Unlearning Framework Based on Knowledge Gap AlignmentPreprint
Wang, L., Chen, T., Yuan, W., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.06535 · Cited by 121 · see citing works
Self-citationMachine Unlearning of Features and Labels
Warnecke, A., Pirch, L., Wressnegger, C., & Rieck, K. (2021) · arXiv · doi:10.48550/arxiv.2108.11577 · Cited by 119 · see citing works
FedRecovery: Differentially Private Machine Unlearning for Federated Learning Frameworks
Zhang, L., Zhu, T., Zhang, H., Xiong, P., & Zhou, W. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3297905 · Cited by 118 · see citing works
VeriFi: Towards Verifiable Federated Unlearning
Gao, X., Ma, X., Wang, J., Sun, Y., Li, B., Ji, S., Cheng, P., & Chen, J. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3382321 · Cited by 111 · cited 6× · see citing works
“[46] proposes a verification mechanism that uses a few communication rounds for”
InfluentialChen, C., Sun, F., Zhang, M., & Ding, B. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511997 · Cited by 110 · cited 3× · see citing works
“[18] proposed a partitionbased retraining process (smart retraining) to unlearn the model from the removed user behavior data.”
InfluentialFast Model Debias with Machine Unlearning
Chen, R.-Z., Yang, J.-F., Xiong, H., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2310.12560 · Cited by 110
Machine Unlearning: Linear Filtration for Logit-based Classifiers
Baumhauer, T., Schöttle, P., & Zeppelzauer, M. (2020) · arXiv · doi:10.48550/arxiv.2002.02730 · Cited by 110 · see citing works
PUMA: Performance Unchanged Model Augmentation for Training Data Removal
Wu, G., Hashemi, M., & Srinivasa, C. (2022) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2203.00846 · Cited by 108 · cited 4×
“However, influence functions depend on the current state of a learning algorithm [169].”
InfluentialA Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning
Wang, Z., Yang, E., Shen, L., & Huang, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3498346 · Cited by 101 · see citing works
Remember What You Want to Forget: Algorithms for Machine Unlearning
Sekhari, A., Acharya, J., Kamath, G., & Suresh, A. T. (2021) · arXiv · doi:10.48550/arxiv.2103.03279 · Cited by 99 · cited 4× · see citing works
“(Bounded/Certified Unlearning) Definition 1 ( 𝜖 -Approximate Unlearning). algorithm 𝐴 if ∀T ⊆ H ,𝐷 ∈ 𝑍 ∗ ,𝑧 ∈ 𝐷 :”
InfluentialDescent-to-Delete: Gradient-Based Methods for Machine Unlearning
Neel, S., Roth, A., & Sharifi-Malvajerdi, S. (2020) · arXiv · doi:10.48550/arxiv.2007.02923 · Cited by 96 · cited 8× · see citing works
“have some privacy leak, even though some provable guarantees are provided for special cases [57, 97, 106].”
InfluentialBackdoor Defense with Machine Unlearning
Liu, Y., Fan, M., Chen, C., Liu, X., Ma, Z., Wang, L., & Ma, J. (2022) · IEEE INFOCOM 2022 - IEEE Conference on Computer Communications · doi:10.1109/infocom48880.2022.9796974 · Cited by 95 · cited 2× · see citing works
“Note that, unlike using machine unlearning to mitigate adversarial attacks [89], adversarial machine unlearning is far stricter, as it concerns not only the model accuracy but also the privacy guarantees.”
Incremental and decremental training for linear classification
Tsai, C.-H., Lin, C.-Y., & Lin, C.-J. (2014) · Knowledge Discovery and Data Mining · doi:10.1145/2623330.2623661 · Cited by 91 · cited 1× · see citing works
“training load on the processor for support vectormachines (SVM) [16, 24, 37, 125, 154, 155] and linear classification [77, 78, 153].”
Total variation distance and the distribution of relative information
Verdú, S. (2014) · Information Theory and Applications Workshop · doi:10.1109/ita.2014.6804281 · Cited by 91 · cited 1×
“The formula for calculating an AIN value is 𝐴𝐼𝑁 = 𝑟 𝑡 ( 𝑀 𝑢 ,𝑀 𝑜𝑟𝑖𝑔 ,𝛼 ) where I( 𝑤 ; 𝐷 ) is the FIM that determines the quantities of information ( )”
Proof-of-Learning: Definitions and Practice
Jia, H., Yaghini, M., Choquette-Choo, C. A., Dullerud, N., Thudi, A., Chandrasekaran, V., & Papernot, N. (2021) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40001.2021.00106 · Cited by 89 · cited 1× · see citing works
“( ′ | 𝑤 ) is assumed to be a As a result, a candidate set of model parameters 𝑃𝑟 ( 𝑤 | 𝐷 ) is constructed by simply maximizing the posterior probability 𝑃𝑟 ( 𝑤 | 𝐷 ) , i.e.:”
HedgeCut: Maintaining Randomised Trees for Low-Latency Machine Unlearning
Schelter, S., Grafberger, S., & Dunning, T. (2021) · Proceedings of the 2021 International Conference on Management of Data · doi:10.1145/3448016.3457239 · Cited by 89 · cited 7× · see citing works
“[132] proposed an unlearning solution for extremely randomized trees by measuring the robustness of the split decisions.”
InfluentialApproximate Data Deletion from Machine Learning ModelsPreprint
Izzo, Z., Smart, M. A., Chaudhuri, K., & Zou, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10077 · Cited by 88 · cited 8× · see citing works
“They approximated a Hessian matrix computation with a project residual update [15, 73] that combines gradient methods with synthetic data.”
InfluentialAnalyzing Information Leakage of Updates to Natural Language Models
Zanella-Béguelin, S., Wutschitz, L., Tople, S., Rühle, V., Paverd, A., Ohrimenko, O., Köpf, B., & Brockschmidt, M. (2020) · Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3372297.3417880 · Cited by 86 · see citing works
Unlearnable Examples: Making Personal Data UnexploitablePreprint
Huang, H., Ma, X., Erfani, S., Bailey, J., & Wang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04898 · Cited by 85 · cited 6× · see citing works
“com/Hunter-DDM/knowledge-neurons Unlearnable [70] Python Deep learning https://github.”
InfluentialHeterogeneous Federated Knowledge Graph Embedding Learning and Unlearning
Zhu, X., Li, G., & Hu, W. P. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583305 · Cited by 75 · see citing works
Federated Unlearning: Guarantee the Right of Clients to Forget
Wu, L., Guo, S., Wang, J., Hong, Z., Zhang, J., & Ding, Y. (2022) · IEEE Network · doi:10.1109/mnet.001.2200198 · Cited by 73 · see citing works
Machine Unlearning for Random Forests
Brophy, J., & Lowd, D. (2020) · arXiv · doi:10.48550/arxiv.2009.05567 · Cited by 70 · cited 6× · see citing works
“com/schelterlabs/hedgecut DaRE-RF [12] Python Tree-based models https://github.”
InfluentialLearn to Forget: Machine Unlearning via Neuron Masking
Ma, Z., Liu, Y., Liu, X., Liu, J., Ma, J., & Ren, K. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3194884 · Cited by 68 · cited 8× · see citing works
“The 20 Newsgroups dataset [90] is a collection of around 20,000 newsgroup documents (almost) equally distributed among 20 newsgroups.”
InfluentialGupta, V., Jung, C., Neel, S., Roth, A., Sharifi-Malvajerdi, S., & Waites, C. (2021) · arXiv · doi:10.48550/arxiv.2106.04378 · Cited by 67 · cited 2× · see citing works
“( ′ | 𝑤 ) is assumed to be a As a result, a candidate set of model parameters 𝑃𝑟 ( 𝑤 | 𝐷 ) is constructed by simply maximizing the posterior probability 𝑃𝑟 ( 𝑤 | 𝐷 ) , i.e.:”
Influence Functions in Deep Learning Are FragilePreprint
Basu, S., Pope, P., & Feizi, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14651 · Cited by 65 · see citing works
Machine Unlearning for Image-to-Image Generative Models
Li, G., Hsu, H., Chen, C., et al. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2402.00351 · Cited by 65
Hard to Forget: Poisoning Attacks on Certified Machine Unlearning
Marchant, N. G., Rubinstein, B. I. P., & Alfeld, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20736 · Cited by 64 · cited 3× · see citing works
“(Bounded/Certified Unlearning) Definition 1 ( 𝜖 -Approximate Unlearning). algorithm 𝐴 if ∀T ⊆ H ,𝐷 ∈ 𝑍 ∗ ,𝑧 ∈ 𝐷 :”
UnlearnCanvas: A Stylized Image Dataset to Benchmark Machine Unlearning for Diffusion Models
Zhang, Y., Zhang, Y., Yao, Y., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2402.11846 · Cited by 64
Variational Bayesian Unlearning
Nguyen, Q. P., Low, B. K. H., & Jaillet, P. (2020) · arXiv · doi:10.48550/arxiv.2010.12883 · Cited by 62 · cited 1× · see citing works
“wrong output ( ZRF = 0) or random output ( ZRF = ( )”
Self-citationDeep Unlearning via Randomized Conditionally Independent Hessians
Mehta, R., Pal, S., Singh, V. P., & Ravi, S. N. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01017 · Cited by 62 · cited 2× · see citing works
“Softmax classifiers [6] ✗ ✓ ✓ ✗ – ✓ – – ✓ ✗ ✓ ✗ ✗ ✓ ✗ ✗ Linear models [73, 87] ✓ ✗ ✗ – ✗ – ✓ – ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ Tree-based models [132] ✗ – ✗ – ✗ ✗ – ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ Bayesian models [111] – ✓ ✗ ✗ ✗ – – ✓ – ✓ – ✓ ✗ ✗ ✗ ✗ DNN-based models [5, 54, 56, 57, 67, 105, 179] ✗ ✓ ✗ ✗ ✗ - - ✓ - - - ✓ ✗ ✗ ✗ ✗”
Chien, E., Pan, C., & Milenkovic, O. (2022) · arXiv.org · doi:10.48550/arxiv.2206.09140 · Cited by 61 · cited 2×
“5MB Classification [21, 24, 28] https://relational.”
Retaining Data from Streams of Social Platforms with Minimal Regret
Tam, N. T., Weidlich, M., Duong, T. C., et al. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/397 · Cited by 57
Self-citationAutomating Botnet Detection with Graph Neural NetworksPreprint
Zhou, J., Xu, Z., Rush, A. M., & Yu, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.06344 · Cited by 55 · see citing works
Federated Unlearning: How to Efficiently Erase a Client in FL?Preprint
Halimi, A., Kadhe, S., Rawat, A., & Baracaldo, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.05521 · Cited by 52 · see citing works
Athena: Probabilistic Verification of Machine Unlearning
Sommer, D. M., Song, L., Wagh, S., & Mittal, P. (2022) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2022-0072 · Cited by 51 · see citing works
Federated Unlearning with Knowledge Distillation
Wu, C., Zhu, S., & Mitra, P. (2022) · arXiv · doi:10.48550/arxiv.2201.09441 · Cited by 50 · cited 4× · see citing works
“Federated learning gives a unique setting for machine unlearning research [87, 90, 158, 162].”
InfluentialDetection of rumor conversations in Twitter using graph convolutional networks
Lotfi, S., Mirzarezaee, M., Hosseinzadeh, M., & Seydi, V. (2021) · Applied Intelligence · doi:10.1007/s10489-020-02036-0 · Cited by 50 · see citing works
RevFRF: Enabling Cross-Domain Random Forest Training With Revocable Federated Learning
Liu, Y., Ma, Z., Yang, Y., Liu, X., Ma, J., & Ren, K. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3104842 · Cited by 49 · see citing works
Hu, X., Li, D., Zheng, Z., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2308.08090 · Cited by 49
On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning
Thudi, A., Jia, H., Shumailov, I., & Papernot, N. (2021) · arXiv · doi:10.48550/arxiv.2110.11891 · Cited by 48 · cited 6× · see citing works
“Existing works define unlearning as obtaining a newmodel with an accuracy as good as if the model had retrainedwithout the data to be forgotten [55, 56, 58, 149].”
InfluentialTowards Effective and General Graph Unlearning via Mutual Evolution
Li, X.-K., Zhao, Y., Wu, Z.-Y., et al. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2401.11760 · Cited by 48
Learning with Selective Forgetting
Shibata, T., Irie, G., Ikami, D., & Mitsuzumi, Y. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/137 · Cited by 47 · cited 1× · see citing works
“) CIFAR 60K 163MB Classification 16+ papers ([58, 70, 136, 163], .”
Unlearning Protected User Attributes in Recommendations with Adversarial Training
Ganhör, C., Penz, D., Rekabsaz, N., Lesota, O., & Schedl, M. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531820 · Cited by 47 · see citing works
Unlearning Bias in Language Models by Partitioning Gradients
Yu, C., Jeoung, S., Kasi, A., Yu, P., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.375 · Cited by 46 · see citing works
How Does Data Augmentation Affect Privacy in Machine Learning?
Yu, D., Zhang, H., Chen, W., Yin, J., & Liu, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17284 · Cited by 45 · cited 3× · see citing works
“Data partition [2, 11] ✓ ✗ ✓ ✗ ✓ ✓ ✗ – ✗ ✓ ✓ ✓ ✗ – ✗ – Data augmentation [70, 135, 147, 173] ✗ ✓ ✗ ✗ ✗ – – – – ✗ – – ✗ ✓ ✗ ✗ Data influence [15, 119, 177] ✗ ✓ ✗ ✓ – – ✓ – ✓ ✓ – ✓ ✗ ✗ ✗ ✗”
MUSE: Machine Unlearning Six-Way Evaluation for Language ModelsPreprint
Shi, W., Lee, J., Huang, Y., Malladi, S., Zhao, J., Holtzman, A., Liu, D., Zettlemoyer, L., Smith, N. A., & Zhang, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.06460 · Cited by 44 · see citing works
Verifying in the Dark: Verifiable Machine Unlearning by Using Invisible Backdoor Triggers
Guo, Y., Zhao, Y., Hou, S., Wang, C., & Jia, X. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3328269 · Cited by 44 · see citing works
Fraboni, Y., Waerebeke, M. V., Scaman, K., Vidal, R., Kameni, L., & Lorenzi, M. (2022) · International Conference on Artificial Intelligence and Statistics · doi:10.48550/arxiv.2211.11656 · Cited by 42 · cited 1×
“The formula for calculating an AIN value is 𝐴𝐼𝑁 = 𝑟 𝑡 ( 𝑀 𝑢 ,𝑀 𝑜𝑟𝑖𝑔 ,𝛼 ) where I( 𝑤 ; 𝐷 ) is the FIM that determines the quantities of information ( )”
Measuring Forgetting of Memorized Training ExamplesPreprint
Jagielski, M., Thakkar, O. D., Tramèr, F., Ippolito, D., Lee, K., Carlini, N., Wallace, E., Song, S., Thakurta, A., Papernot, N., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.00099 · Cited by 41 · see citing works
ERM-KTP: Knowledge-Level Machine Unlearning via Knowledge Transfer
Lin, S., Zhang, X., Chen, C., Chen, X., & Susilo, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01929 · Cited by 41 · see citing works
Communication Efficient and Provable Federated Unlearning
Tao, Y., Wang, C.-L., Pan, M., et al. (2024) · Proceedings of the VLDB Endowment · doi:10.48550/arxiv.2401.11018 · Cited by 41
Qian, W., Zhao, C., Le, W., et al. (2023) · Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599526 · Cited by 40
Knowledge Removal in Sampling-based Bayesian Inference
Fu, S., He, F., & Tao, D.-C. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2203.12964 · Cited by 39 · cited 3×
“This certification can be achieved for some energy functions when formulating the evidence lower bound (ELBO) in Bayesian models [43, 44, 74].”
InfluentialTowards Probabilistic Verification of Machine Unlearning
Sommer, D. M., Song, L., Wagh, S., & Mittal, P. (2020) · arXiv · doi:10.48550/arxiv.2003.04247 · Cited by 38 · cited 4× · see citing works
“Athena [139, 140] Python Model-agnostic https://github.”
InfluentialA Survey on Machine Unlearning: Techniques and New Emerged Privacy Risks
Liu, H., Xiong, P., Zhu, T.-Q., et al. (2024) · Journal of Information Security and Applications · doi:10.48550/arxiv.2406.06186 · Cited by 37
Ren, Z. X., Baird, A., Han, J., Zhang, Z., & Schuller, B. W. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054087 · Cited by 34 · see citing works
Self-citationFast-FedUL: A Training-Free Federated Unlearning with Provable Skew Resilience
Huynh, T. T., Nguyen, T. B., Nguyen, P. L., et al. (2024) · ECML/PKDD · doi:10.48550/arxiv.2405.18040 · Cited by 34
Self-citationBonato, J., Cotogni, M., & Sabetta, L. (2024) · European Conference on Computer Vision · doi:10.48550/arxiv.2404.12922 · Cited by 34
Guo, T., Guo, S., Zhang, J., et al. (2022) · doi:10.48550/arxiv.2202.13295 · Cited by 34
Forget-SVGD: Particle-Based Bayesian Federated UnlearningPreprint
Gong, J., Simeone, O., Kassab, R., et al. (2021) · arXiv.org · doi:10.48550/arxiv.2111.12056 · Cited by 34 · see citing works
Federated UnlearningPreprint
Liu, G., Ma, X., Yang, Y., Wang, C., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13891 · Cited by 33 · cited 1× · see citing works
“• The degradation count: is defined as the ratio between the”
Efficient Repair of Polluted Machine Learning Systems via Causal Unlearning
Cao, Y., Yu, A. F., Aday, A., Stahl, E., Merwine, J., & Yang, J. (2018) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3196494.3196517 · Cited by 33 · cited 1× · see citing works
“For example, a data pollution attack might affect millions of data items, but only a few of them can be detected by human experts or SOTA detection methods [14].”
Exact-Fun: An Exact and Efficient Federated Unlearning Approach
Xiong, Z., Li, W., Li, Y., & Cai, Z. (2023) · Proceedings · doi:10.1109/icdm58522.2023.00188 · Cited by 33 · see citing works
Incremental and Decremental Learning for Linear Support Vector Machines
Romero, E., Barrio, I., & Belanche, L. (2007) · International Conference on Artificial Neural Networks · doi:10.1007/978-3-540-74690-4_22 · Cited by 33 · see citing works
Bayesian Neural Networks with Weight Sharing Using Dirichlet Processes
Roth, W., & Pernkopf, F. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2884905 · Cited by 33 · cited 1× · see citing works
“wrong output ( ZRF = 0) or random output ( ZRF = ( )”
Aldaghri, N., Mahdavifar, H., & Beirami, A. (2021) · IEEE Access · doi:10.1109/access.2021.3090019 · Cited by 32 · cited 1× · see citing works
“The formula for calculating an AIN value is 𝐴𝐼𝑁 = 𝑟 𝑡 ( 𝑀 𝑢 ,𝑀 𝑜𝑟𝑖𝑔 ,𝛼 ) where I( 𝑤 ; 𝐷 ) is the FIM that determines the quantities of information ( )”
Markov Chain Monte Carlo-Based Machine Unlearning
Nguyen, Q. P., Oikawa, R., Divakaran, D. M., Chan, M. C., & Low, B. K. H. (2022) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3488932.3517406 · Cited by 32 · see citing works
Self-citationEfficiently Forgetting What You Have Learned in Graph Representation Learning via Projection
Cong, W., & Mahdavi, M. (2023) · International Conference on Artificial Intelligence and Statistics · doi:10.48550/arxiv.2302.08990 · Cited by 32
Digital forgetting in large language models: a survey of unlearning methods
Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11078-6 · Cited by 31 · see citing works
BFU: Bayesian Federated Unlearning with Parameter Self-Sharing
Wang, W., Tian, Z., Zhang, C., Liu, A., & Yu, S. (2023) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3579856.3590327 · Cited by 31 · see citing works
FAST: Adopting Federated Unlearning to Eliminating Malicious Terminals at Server Side
Guo, X., Wang, P., Qiu, S., Song, W., Zhang, Q., Wei, X., & Zhou, D. (2023) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2023.3343117 · Cited by 31 · see citing works
Deep Autoencoding Topic Model With Scalable Hybrid Bayesian Inference
Zhang, H., Chen, B., Cong, Y., Guo, D., Liu, H., & Zhou, M. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3003660 · Cited by 31 · cited 1× · see citing works
“wrong output ( ZRF = 0) or random output ( ZRF = ( )”
Few-Shot Unlearning by Model Inversion
Yoon, Y., Nam, J., Yun, H., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2205.15567 · Cited by 31
Right to Be Forgotten in the Age of Machine Learning
Dang, Q.-V. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-71782-7_35 · Cited by 30 · cited 2× · see citing works
“Many privacy-preserving regulations have been enacted recently that involve the right to be forgotten” [11, 31], such as the European Union’s General Data Protection Regulation (GDPR) [97] and the California Consumer Privacy Act [115].”
Proof of Unlearning: Definitions and Instantiation
Weng, J., Yao, S., Du, Y., Huang, J., Weng, J., & Wang, C. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3358993 · Cited by 30 · see citing works
Learn to Unlearn: Insights Into Machine Unlearning
Qu, Y., Yuan, X., Ding, M., Ni, W., Rakotoarivelo, T., & Smith, D. B. (2024) · Computer · doi:10.1109/mc.2023.3333319 · Cited by 30 · see citing works
Kim, H., Lee, S., & Woo, S. S. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2312.16823 · Cited by 30
Prototype Learning for Interpretable Respiratory Sound Analysis
Ren, Z. X., Nguyên, T. T., & Nejdl, W. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747014 · Cited by 29 · see citing works
Self-citationPrIU: A Provenance-Based Approach for Incrementally Updating Regression Models
Wu, Y., Tannen, V., & Davidson, S. B. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3380571 · Cited by 28 · see citing works
FRAMU: Attention-Based Machine Unlearning Using Federated Reinforcement Learning
Shaik, T., Tao, X., Li, L., Xie, H., Cai, T., Zhu, X., & Li, Q. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3382726 · Cited by 28 · see citing works
Online Forgetting Process for Linear Regression Models
Li, Y., Wang, C., & Cheng, G. (2020) · International Conference on Artificial Intelligence and Statistics · doi:10.48550/arxiv.2012.01668 · Cited by 28 · cited 1×
“wrong output ( ZRF = 0) or random output ( ZRF = ( )”
Learning Parameter Distributions to Detect Concept Drift in Data Streams
Haug, J., & Kasneci, G. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412499 · Cited by 28 · see citing works
Anatomizing Bias in Facial Analysis
Singh, R., Majumdar, P., Mittal, S., & Mayank Vatsa, A. (2022) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v36i11.21500 · Cited by 27 · cited 1× · see citing works
“and unbiased feature representation [108, 120, 141, 165], where, machine unlearning, e.”
JUDO: Just-in-time rumour detection in streaming social platforms
Nguyen, T. T., Nguyên, T. T., Nguyen, T. T., Vo, B., Jo, J. H., & Nguyen, Q. V. H. (2021) · Information Sciences · doi:10.1016/j.ins.2021.04.018 · Cited by 27 · see citing works
Self-citationMachine Unlearning via Algorithmic Stability
Ullah, E., Mai, T., Rao, A., Rossi, R., & Arora, R. (2021) · arXiv · doi:10.48550/arxiv.2102.13179 · Cited by 26 · cited 4× · see citing works
“Differential privacy [62] ✗ ✓ – – – ✓ ✓ – ✓ ✓ – ✓ ✗ ✗ ✗ ✓ Certified removal [55, 59, 109, 156] – ✓ ✗ ✗ – – ✓ ✓ ✓ ✓ – ✓ ✗ ✗ ✗ ✓ Statistical query learning [13] – ✓ ✗ – ✗ ✓ ✓ – ✓ – – ✓ ✗ ✗ ✗ – Decremental learning [24, 52] ✗ – ✗ – – ✗ ✓ ✓ – – – ✓ ✗ ✗ ✗ ✗ Knowledge adaptation [26] ✗ ✓ – – – – – – ✗ ✗ – ✓ – – – – Paramete…”
InfluentialUnlearning Graph Classifiers with Limited Data Resources
Pan, C., Chien, E., & Milenković, O. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583547 · Cited by 26 · see citing works
Making Users Indistinguishable: Attribute-wise Unlearning in Recommender Systems
Li, Y., Chen, C., Zheng, X., Zhang, Y., Han, Z., Meng, D., & Wang, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612418 · Cited by 26 · see citing works
Example-based Explanations with Adversarial Attacks for Respiratory Sound Analysis
Chang, Y., Ren, Z., NGUYEN, T. V., et al. (2022) · Interspeech · doi:10.48550/arxiv.2203.16141 · Cited by 25
Self-citationOn Training Deep Neural Networks Using a Streaming Approach
Duda, P., Jaworski, M., Cader, A., et al. (2019) · Journal of Artificial Intelligence and Soft Computing Research · doi:10.2478/jaiscr-2020-0002 · Cited by 25 · see citing works
Deletion inference, reconstruction, and compliance in machine (un)learning
Gao, J., Garg, S., Mahmoody, M., & Vasudevan, P. N. (2022) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2022-0079 · Cited by 22 · see citing works
Multicategory Incremental Proximal Support Vector Classifiers
Tveit, A., & Hetland, M. L. (2003) · International Conference on Knowledge-Based Intelligent Information & Engineering Systems · doi:10.1007/978-3-540-45224-9_54 · Cited by 22
To be forgotten or to be fair: unveiling fairness implications of machine unlearning methods
Zhang, D., Pan, S., Hoang, T., Xing, Z., Staples, M., Xu, X., Yao, L., Lu, Q., & Zhu, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00398-y · Cited by 21 · see citing works
An Introduction to Machine Unlearning
Mercuri, S., Khraishi, R., Okhrati, R., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2209.00939 · Cited by 21 · cited 1×
“[103] ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗”
Thudi, A., Shumailov, I., Boenisch, F., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2202.12232 · Cited by 21
In-Context Unlearning: Language Models as Few Shot Unlearners
Pawelczyk, M., Neel, S., & Lakkaraju, H. (2023) · arXiv · doi:10.48550/arxiv.2310.07579 · Cited by 20 · see citing works
MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning
Cheng, J., & Amiri, H. (2024) · arXiv.org · doi:10.48550/arxiv.2406.14796 · Cited by 20
Machine Unlearning: Its Nature, Scope, and Importance for a “Delete Culture”
Floridi, L. (2023) · Philosophy & Technology · doi:10.1007/s13347-023-00644-5 · Cited by 20 · see citing works
Who's Harry Potter? Approximate Unlearning in LLMs
Eldan, R., & Russinovich, M. (2023) · arXiv · doi:10.48550/arxiv.2310.02238 · Cited by 19 · see citing works
Get Rid of Your Trail : Remotely Erasing Backdoors in Federated Learning
Alam, M., Lamri, H., & Maniatakos, M. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3465441 · Cited by 19 · see citing works
Towards efficient and effective unlearning of large language models for recommendation
Wang, H., Lin, J., Chen, B., Yang, Y., Tang, R., Zhang, W., & Yu, Y. H. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40044-2 · Cited by 19 · see citing works
Incremental and Decremental Proximal Support Vector Classification using Decay Coefficients
Tveit, A., Hetland, M. L., & Engum, H. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-45228-7_42 · Cited by 19 · see citing works
Ren, Z., Han, J., Cummins, N., et al. (2020) · Interspeech · doi:10.21437/interspeech.2020-1869 · Cited by 19
Self-citationMachine Unlearning for Image Retrieval: A Generative Scrubbing Approach
Zhang, P.-F., Bai, G., Huang, Z., & Xu, X.-S. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3548378 · Cited by 18 · cited 2× · see citing works
“Softmax classifiers [6] ✗ ✓ ✓ ✗ – ✓ – – ✓ ✗ ✓ ✗ ✗ ✓ ✗ ✗ Linear models [73, 87] ✓ ✗ ✗ – ✗ – ✓ – ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ Tree-based models [132] ✗ – ✗ – ✗ ✗ – ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ Bayesian models [111] – ✓ ✗ ✗ ✗ – – ✓ – ✓ – ✓ ✗ ✗ ✗ ✗ DNN-based models [5, 54, 56, 57, 67, 105, 179] ✗ ✓ ✗ ✗ ✗ - - ✓ - - - ✓ ✗ ✗ ✗ ✗”
Have You Forgotten? A Method to Assess if Machine Learning Models Have Forgotten Data
Liu, X., & Tsaftaris, S. A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-59710-8_10 · Cited by 18 · see citing works
DeltaBoost: Gradient Boosting Decision Trees with Efficient Machine Unlearning
Wu, Z., Zhu, J., Li, Q., & He, B. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3589313 · Cited by 18 · cited 1× · see citing works
“wrong output ( ZRF = 0) or random output ( ZRF = ( )”
Verifiable and Provably Secure Machine Unlearning
Eisenhofer, T., Riepel, D., Chandrasekaran, V., Ghosh, E., Ohrimenko, O., & Papernot, N. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00033 · Cited by 16 · cited 1× · see citing works
“[41] proposed a cryptography-informed approach with verifiable proofs, i.”
Decoupling the Class Label and the Target Concept in Machine Unlearning
Zhu, J.-N., Han, B., Yao, J., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2406.08288 · Cited by 16
SAFE: Machine Unlearning With Shard Graphs
Dukler, Y., Bowman, B., Achille, A., Golatkar, A., Swaminathan, A. D., & Soatto, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01569 · Cited by 15 · cited 1× · see citing works
“The formula for calculating an AIN value is 𝐴𝐼𝑁 = 𝑟 𝑡 ( 𝑀 𝑢 ,𝑀 𝑜𝑟𝑖𝑔 ,𝛼 ) where I( 𝑤 ; 𝐷 ) is the FIM that determines the quantities of information ( )”
FedCIO: Efficient Exact Federated Unlearning with Clustering, Isolation, and One-shot Aggregation
Qiu, H., Wang, Y., Xu, Y., Cui, L., & Shen, Z. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386788 · Cited by 15 · see citing works
FairSISA: Ensemble Post-Processing to Improve Fairness of Unlearning in LLMs
Kadhe, S., Halimi, A., Rawat, A., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.07420 · Cited by 15
Tiwary, P., Guha, A., Panda, S., et al. (2023) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2309.14054 · Cited by 15
SSSE: Efficiently Erasing Samples from Trained Machine Learning ModelsPreprint
Peşte, A., Alistarh, D., & Lampert, C. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03860 · Cited by 14 · cited 1× · see citing works
“The formula for calculating an AIN value is 𝐴𝐼𝑁 = 𝑟 𝑡 ( 𝑀 𝑢 ,𝑀 𝑜𝑟𝑖𝑔 ,𝛼 ) where I( 𝑤 ; 𝐷 ) is the FIM that determines the quantities of information ( )”
Knowledge-Adaptation PriorsPreprint
Khan, M. E., & Swaroop, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08769 · Cited by 14 · see citing works
VeriFi: Towards Verifiable Federated UnlearningPreprint
Gao, X., Ma, X., Wang, J., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12709 · Cited by 14 · see citing works
A survey of security and privacy issues of machine unlearning
Chen, A., Li, Y., Zhao, C., & Huai, M. (2025) · AI Magazine · doi:10.1002/aaai.12209 · Cited by 13 · see citing works
Flagged first, then by citations
18 of the 263 references couldn’t be matched to a known paper.
Research works printed but not linked: 19
- [15][15] Gert Cauwenberghs et al. 2000. Incremental and decremental support vector machine learning. NIPS 13 (2000). -
- [22][22] Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. 2021. Graph unlearning. arXiv preprint arXiv:2103.14991 (2021). - · arXiv
- [26][26] Jiali Cheng, George Dasoulas, Huan He, Chirag Agarwal, and Marinka Zitnik. 2023. GNNDelete: A General Strategy for Unlearning in Graph Neural Networks. In ICLR . -
- [36][36] Nicola K Dinsdale, Mark Jenkinson, et al. 2020. Unlearning scanner bias for mri harmonisation. In MICCAI . 369-378. -
- [46][46] Stefan Feuerriegel, Mateusz Dolata, and Gerhard Schwabe. 2020. Fair AI. Business & information systems engineering 62, 4 (2020), 379-384. -
- [48][48] Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. 2014. Privacy in pharmacogenetics: An { End-to-End } case study of personalized warfarin dosing. In USENIX Security . 17-32. -
- [55][55] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. 2019. Combining Neural Networks with Personalized PageRank for Classification on Graphs. In ICLR . -
- [65][65] Ruocheng Guo, Lu Cheng, Jundong Li, P Richard Hahn, and Huan Liu. 2020. A survey of learning causality with data: Problems and methods. CSUR 53, 4 (2020), 1-37. -
- [69][69] William L Hamilton. 2020. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning 14, 3 (2020), 1-159. -
- [85][85] Masayuki Karasuyama and Ichiro Takeuchi. 2010. Multiple incremental decremental learning of support vector machines. IEEE Transactions on Neural Networks 21, 7 (2010), 1048-1059. -
- [123][123] Salvatore Mercuri, Raad Khraishi, Ramin Okhrati, Devesh Batra, Conor Hamill, Taha Ghasempour, and Andrew Nowlan. 2022. An introduction to machine unlearning. arXiv preprint arXiv:2209.00939 (2022). - · arXiv
- [125][125] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. 2020. Learning from failure: De-biasing classifier from biased classifier. NIPS 33 (2020), 20673-20684. -
- [130][130] Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. 2022. A Survey of Machine Unlearning. arXiv preprint arXiv:2209.02299 (2022). , , -
- [134][134] Stuart L Pardau. 2018. The California consumer privacy act: Towards a Europeanstyle privacy regime in the United States. J. Tech. L. & Pol'y 23 (2018), 68. -
- [157][157] S Shan, E Wenger, J Zhang, H Li, H Zheng, and BY Zhao. 2020. Protecting Personal Privacy against Una uthorized Deep Learning Models. In USENIX Security . 1-16. -
- [163][163] Abhijeet Singh and Abhineet Anand. 2017. Data leakage detection using cloud computing. IJECS 6, 4 (2017). -
- [165][165] Yash Sinha, Murari Mandal, and Mohan Kankanhalli. 2024. Multi-Modal Recommendation Unlearning. arXiv preprint arXiv:2405.15328 (2024). - · arXiv
- [189][189] Cheng-Long Wang, Mengdi Huai, and Di Wang. 2023. Inductive graph unlearning. In USENIX . 3205-3222. -
- [204][204] Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S. Yu. 2023. Machine Unlearning: A Survey. CSUR 56, 1 (2023), 36. -
Cited by
Sorted by citations
Longo, L., Brčić, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 648 · see citing works
“While the interest in the machine unlearning paradigm [280] has been on the rise [281], it is unclear how to efficiently ensure that data owned by a particular user is unlearned by a given large-scale generative model, so that it can be ensured that no instance like that of the user claiming their…”
BackgroundXu, H., Zhu, T., Zhang, L., Zhou, W., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3603620 · Cited by 164 · see citing works
A Survey on Federated Unlearning: Challenges, Methods, and Future Directions
Liu, Z., Jiang, Y., Shen, J., Peng, M., Lam, K., Yuan, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3679014 · Cited by 148 · see citing works
“A detailed comparison of related FU surveys is summarized in Table 1.”
Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity
Novelli, C., Casolari, F., Hacker, P., Spedicato, G., & Floridi, L. (2024) · Computer Law & Security Review · doi:10.1016/j.clsr.2024.106066 · Cited by 124 · see citing works
FedRecovery: Differentially Private Machine Unlearning for Federated Learning Frameworks
Zhang, L., Zhu, T., Zhang, H., Xiong, P., & Zhou, W. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3297905 · Cited by 118 · see citing works
“It stems from the increasingly urgent requirement of individuals for data protection, and is one of the centrepieces of machine learning research [1], [2].”
BackgroundMachine Unlearning: Solutions and Challenges
Xu, J., Wu, Z., Wang, C., & Jia, X. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3379240 · Cited by 114 · see citing works
“a) Data Erasure Completeness:: This metric evaluates how thoroughly the unlearning algorithm makes the model remove the target data.”
BackgroundFast Machine Unlearning without Retraining through Selective Synaptic Dampening
Foster, J. D., Schoepf, S., & Brintrup, A. M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29092 · Cited by 91 · see citing works
A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning
Wang, Z., Yang, E., Shen, L., & Huang, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3498346 · Cited by 101 · see citing works
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metrics
Romandini, N., Mora, A., Mazzocca, C., Montanari, R., & Bellavista, P. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3478334 · Cited by 76 · see citing works
“Most published surveys primarily focus on MU [6], [11], [14], [15], [16], while FU is only covered from a general perspective with a limited amount of references.”
Training data influence analysis and estimation: a survey
Hammoudeh, Z., & Lowd, D. (2024) · Machine Learning · doi:10.1007/s10994-023-06495-7 · Cited by 80 · see citing works
Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Chen, J., & Yang, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.738 · Cited by 52 · see citing works
Reproducing Whisper-Style Training Using An Open-Source Toolkit And Publicly Available Data
Peng, Y., Tian, J., Yan, B., Berrebbi, D., Chang, X., Li, X., Shi, J., Arora, S., Chen, W., Sharma, R., et al. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389676 · Cited by 40 · see citing works
“Fourthly, OWSM provides a valuable testbed for investigating and exploring various machine learning problems such as data imbalance, continual learning [61], adversarial robustness [62], and machine unlearning [63].”
BackgroundDigital forgetting in large language models: a survey of unlearning methods
Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11078-6 · Cited by 31 · see citing works
Algorithms that forget: Machine unlearning and the right to erasure
Juliussen, B. A., Rui, J. P., & Johansen, D. (2023) · Computer law & security review · doi:10.1016/j.clsr.2023.105885 · Cited by 29 · see citing works
Proof of Unlearning: Definitions and Instantiation
Weng, J., Yao, S., Du, Y., Huang, J., Weng, J., & Wang, C. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3358993 · Cited by 30 · see citing works
“Note that the SISA framework can also be categorized as a data-driven unlearning approach according to a recent machine unlearning survey [53].”
MethodScore-VAE: Root Cause Analysis for Federated-Learning-Based IoT Anomaly Detection
Fan, J., Tang, G., Wu, K., Zhao, Z., Zhou, Y., & Huang, S. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3289814 · Cited by 26 · see citing works
“Third,theprocess ofacquiringnewknowledgeandself-renewaloftenrenders themodeltemporarilyinoperative[18],leadingtosignificant inconvenienceandpotentialsafetyrisksineverydaylife.”
Zhou, J., Li, H., Liao, X., Zhang, B., He, W., Li, Z., Zhou, L., & Gao, X. (2023) · Nature Communications · doi:10.1038/s41467-023-41703-x · Cited by 22 · see citing works
An overview of machine unlearning
Li, C., Jiang, H., Chen, J., Zhao, Y. L., Fu, S., Jing, F., & Guo, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100254 · Cited by 21 · see citing works
HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
Gutiérrez, B. J., Shu, Y., Gu, Y., Yasunaga, M., & Su, Y. (2024) · arXiv · doi:10.48550/arxiv.2405.14831 · Cited by 152 · see citing works
Reclaiming the Digital Commons: A Public Data Trust for Training Data
Chan, A., Bradley, H., & Rajkumar, N. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604658 · Cited by 23 · see citing works
Kim, H., Lee, S. T., & Woo, S. S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30118 · Cited by 21 · see citing works
Adversarial Item Promotion on Visually-Aware Recommender Systems by Guided Diffusion
Chen, L., Yuan, W., Chen, T., Ye, G., Nguyen, Q. V. H., & Yin, H. (2024) · ACM Transactions on Information Systems · doi:10.1145/3666088 · Cited by 25 · see citing works
Self-citationContinual Forgetting for Pre-Trained Vision Models
Zhao, H., Ni, B., Fan, J., Wang, Y., Chen, Y., Meng, G., & Zhang, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02705 · Cited by 23 · see citing works
“Machine unlearning involves retraining or modifying machine learning models to diminish or eradicate the influence of previously acquired patterns or biases, aiming to enhance the models’ fairness and safety [5, 6, 19, 48, 62, 72].”
BackgroundA survey on machine unlearning: Techniques and new emerged privacy risks
Liu, H., Xiong, P., Zhu, T., & Yu, P. S. (2025) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104010 · Cited by 21 · see citing works
FedCIO: Efficient Exact Federated Unlearning with Clustering, Isolation, and One-shot Aggregation
Qiu, H., Wang, Y., Xu, Y., Cui, L., & Shen, Z. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386788 · Cited by 15 · see citing works
Multiclass Unlearning for Image Classification via Weight Filtering
Poppi, S., Sarto, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · IEEE Intelligent Systems · doi:10.1109/mis.2024.3412742 · Cited by 14 · see citing works
“Among other techniques which can alter the internal knowledge representation of a model, like knowledge distillation, model pruning, or incremental learning, Machine Unlearning [2] emerges as the sole technique which can delete private training data.”
Longo, L., Brčić, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19775 · Cited by 14 · see citing works
“While the interest in the machine unlearning paradigm [280] has been on the rise [281], it is unclear how to efficiently ensure that data owned by a particular user is unlearned by a given large-scale generative model, so that it can be ensured that no instance like that of the user claiming their…”
BackgroundKumar, R., Prakash, R., Kumar, N., Chinara, S., & Mohammed, M. A. (2025) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-025-10436-z · Cited by 14 · see citing works
Privacy Adhering Machine Un-learning in NLP
Kumar, V. B., Gangadharaiah, R., & Roth, D. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.25 · Cited by 13 · see citing works
GraphGuard: Detecting and Counteracting Training Data Misuse in Graph Neural Networks
Wu, B. Y., Zhang, H., Yang, X., Wang, S., Xue, M., Pan, S., & Yuan, X. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24441 · Cited by 15 · see citing works
Strategic Data Revocation in Federated Unlearning
Ding, N., Wei, E., & Berry, R. (2024) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom52122.2024.10621201 · Cited by 12 · see citing works
“We first introduce the objectives and process of federated learning and unlearning, and then we specify users’ strategies and payoffs, respectively.”
MethodClass-wise federated unlearning: Harnessing active forgetting with teacher–student memory generation
Li, Y., Zhang, J., Liu, Y., & Chen, C. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113353 · Cited by 11 · see citing works
“This requirement has spurred the development of machine unlearning techniques, which aim to remove the influence of specific data samples from trained models (Nguyen et al., 2022).”
Feature Unlearning for Pre-trained GANs and VAEs
Moon, S., Cho, S., & Kim, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30138 · Cited by 13 · see citing works
“Unlearning scenarios can vary depending on the requirements (Nguyen et al. 2022).”
BackgroundTask-Aware Machine Unlearning and Its Application in Load Forecasting
Xu, W., & Teng, F. (2024) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2024.3376828 · Cited by 9 · see citing works
“1) Unlearning Completeness vs Model Performance: Although retraining is usually not a viable option, it is broadly agreed that the golden rule for unlearning is to minimize the distance between unlearnt and retrained models [30].”
BackgroundTo See or Not to See: Understanding the Tensions of Algorithmic Curation for Visual Arts
Srinivasan, R. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658917 · Cited by 9 · see citing works
Navigating the Ethical Landscape: Considerations in Implementing AI-ML Systems in Human Resources
Basnet, S. (2024) · International Journal of Research Publication and Reviews · doi:10.55248/gengpi.5.0324.0755 · Cited by 9 · see citing works
The Right to Be Forgotten in Artificial Intelligence: Issues, Approaches, Limitations and Challenges
Lobo, J. L., Gil-López, S., & Ser, J. D. (2023) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai54212.2023.00085 · Cited by 6 · see citing works
“We finish this manuscript by traversing several AI research areas motivated by different aspects of the RTBF : a) Machine Unlearning: Ideally, the unlearning paradigm [2] would allow users to remove their data from the model without needing to retrain it from scratch.”
BackgroundMethodFeaUn: Feature unlearning in vertical federated learning for IIoT against feature inference attacks
Lu, Z., Zhang, X., Li, T., Wang, Y., Li, G., & Liu, Z. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131110 · Cited by 6 · see citing works
The Price of Forgetting: Data Redemption Mechanism Design for Machine Unlearning
Cui, Y., & Cheung, M. H. (2024) · International Conference on Communications · doi:10.1109/icc51166.2024.10622287 · Cited by 6 · see citing works
“The accuracy degradation can be exponential [10] when more data is unlearned, which is referred to as catastrophic unlearning [13], and how to mitigate it naturally is still an open question [14].”
BackgroundA Decision-Making Process to Implement the ‘Right to Be Forgotten’ in Machine Learning
Hawkins, K., Alhuwaish, N., Belguith, S., Vranaki, A., & Charlesworth, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61089-9_2 · Cited by 5 · see citing works
A Study Regarding Machine Unlearning on Facial Attribute Data
Gündoğdu, E., Unal, A., & Ünal, G. (2024) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg59268.2024.10581972 · Cited by 4 · see citing works
“The current research on machine unlearning can be categorized into two main approaches: exact and approximate methods [22], each addressing the challenges posed by catastrophic unlearning and stochasticity, thereby contributing to more effective unlearning techniques.”
BackgroundToward Highly-Efficient and Accurate Services QoS Prediction via Machine Unlearning
Zeng, Y., Xu, J., Li, Y., Chen, C., Dai, Q., & Du, Z. (2023) · IEEE Access · doi:10.1109/access.2023.3291410 · Cited by 4 · see citing works
“To achieve this demand, the machine unlearning mechanism [3] has shown its promise.”
BackgroundLearning to Unlearn for Bayesian Personalized Ranking via Influence Function
Chen, J., Zhang, H., Li, H., & Li, Y. (2025) · Chinese Journal of Electronics · doi:10.23919/cje.2023.00.417 · Cited by 3 · see citing works
“Recommendation unlearning is a process designed to eliminate the impact of a specified set of training data upon request from a trained recommender model [15].”
Learning and Unlearning to Operate Profitable Secure Electric Vehicle Charging
Lee, S., & Choi, D. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3396524 · Cited by 3 · see citing works
“A concise review of various MuL methods was presented in [21].”
MethodFedNPC: A Federated Learning Framework for Large Language Models in Game NPCs
Hong, M., Zhang, K., Zhang, S., & He, Z. (2023) · IEEE Student Conference on Research and Development (SCOReD) · doi:10.1109/scored60679.2023.10563705 · Cited by 3 · see citing works
“While machine unlearning has proven to be an effective alternative compared to model retraining [24], Federated Learning can offer a more refined solution to this issue.”
BackgroundHeterogeneous Decentralized Machine Unlearning with Seed Model DistillationPreprint
Ye, G., Chen, T., Nguyen, Q. V. H., Huang, Z. H., Zhou, X., & Yin, H. (2023) · Research Square · doi:10.21203/rs.3.rs-2439540/v1 · Cited by 4 · see citing works
Self-citationNeuroscience-inspired continuous learning: a sustainable approach to AI energy challengePreprint
Xiao, D. (2023) · doi:10.31219/osf.io/twn9q · Cited by 3 · see citing works
Gao, H.-A., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu, S., Qiu, J., Qi, X., Wu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.21046 · Cited by 3 · see citing works
DiEvD-SF: Disruptive Event Detection Using Continual Machine Learning With Selective Forgetting
Seetha, A., Chouhan, S. S., Pilli, E. S., Raychoudhury, V., & Saha, S. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3364544 · Cited by 2 · see citing works
“Machine unlearning is a novel paradigm that enables the removal or forgetting of specific knowledge or patterns previously acquired by a ML model [11].”
BackgroundMethodEntUn: Mitigating the forget-retain dilemma in unlearning via entropy
Jung, D. (2025) · ICT Express · doi:10.1016/j.icte.2025.06.007 · Cited by 3 · see citing works
QoSEraser: A Data Erasable Framework for Web Service QoS Prediction
Zeng, Y., Li, Y., Xia, Z., Du, Z., Wang, J., Lian, R., & Xu, J. (2023) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse60056.2023.00022 · Cited by 2 · see citing works
“With the common forgetting frameworks, most of them [2]–[4], [26], [27] achieve suboptimal performance because of the reduced time expense obtained.”
BackgroundTran, H. N., Nguyen, N. T., Nguyen, N. V., Nguyen, H. X., & Nguyen, A. D. (2025) · Engineering Science and Technology an International Journal · doi:10.1016/j.jestch.2025.101996 · Cited by 2 · see citing works
Enhancing partition distinction: A contrastive policy to recommendation unlearning
Li, L., Zhuo, S., Lin, H., He, J., Qiu, W., Zhang, Q., Wang, C., & Huang, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107667 · Cited by 2 · see citing works
Federated Unlearning With Reinforcement Learning: Adaptive Privacy Preservation for Clients
Gao, K., Zhu, T., Ye, D., Gao, L., & Zhou, W. (2025) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104164 · Cited by 3 · see citing works
Using Shadow Models to Protect Private Data on Machine Unlearning
Lin, H.-C., & Yang, S. (2023) · IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan) · doi:10.1109/icce-taiwan58799.2023.10226994 · Cited by 1 · see citing works
When Contrastive Learning Meets Graph Unlearning: Graph Contrastive Unlearning for Link Prediction
Yang, T.-H., & Li, C. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386624 · Cited by 1 · see citing works
Ethical considerations for multimodal artificial intelligence in healthcare
Kostick, K. M., Wagner, J. K., Cabrera, L. Y., & Mandl, K. D. (2026) · AI and Ethics · doi:10.1007/s43681-026-01259-0 · Cited by 1 · see citing works
“By integrating these approaches, the scientific and policy communities can move beyond a “theater of anonymity” [61] toward substantive safeguards that better reflect the realities of multimodal AI.”
DeRDaVa: Deletion-Robust Data Valuation for Machine Learning
Xiao, T., Sim, R. H. L., Fan, J., & Low, B. K. H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29462 · Cited by 1 · see citing works
Generative AI for Trustworthy, Open, and Equitable Scholarship
Bourg, C., Kriegsman, S., Lindsay, N., Sardis, H., Stalberg, E., & Altman, M. (2024) · doi:10.21428/e4baedd9.567bfd15 · Cited by 1 · see citing works
Daten-Fairness als Daten-Gerechtigkeit by Design
Bieker, F., & Hansen, M. (2023) · Nomos Verlagsgesellschaft mbH & Co. KG eBooks · doi:10.5771/9783748938743-29
6th International Conference on Advanced Research Methods and Analytics - CARMA 2024
Conference, C., Domenech, J., Vicente, M. R., & de Pedraza, P. (2024) · doi:10.4995/carma2024.2024.19017
Class-Wise Federated Unlearning: Harnessing Active Forgetting with Teacher-Student Memory GenerationPreprint
Li, Y., Zhang, J., Liu, Y., & Chen, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5071728
Protection of Personal Data in Computerized Accounting Programs
Yüzbaşıoğlu, M. A. (2024) · European Journal of Formal Sciences and Engineering · doi:10.26417/sd8tfs54
ResNet34-Based Galaxy Morphology Classification with Machine UnlearningPreprint
Potdukhe, N., Sabde, P., Aaglave, P., Upadhye, M., Barmate, S., & Ratnaparkhi, S. (2026) · Research Square · doi:10.21203/rs.3.rs-9269285/v1
Approach for Class-Wise Unlearning of Data Along with Learning of New Data
Guru, D. S., & Shivaprasad, D. L. (2026) · Smart innovation, systems and technologies · doi:10.1007/978-3-032-18974-5_14
- Authors
- Nguyen, T. T., Huynh, T. T., Ren, Z., Nguyen, P. L., Liew, A. W.-C., Yin, H., & Nguyen, Q. V. H.
- Affiliations
- VinUniversity (VN)
- University of Bremen (DE)
- Trường Đại học Khoa học và Công nghệ Hà Nội (VN)
- Griffith University (AU)
- The University of Queensland (AU)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Volume
- 16
- Issue
- 5
- Pages
- 1-46
- ISSN
- 2157-6904 (ISSN-L)
- arXiv
- 2209.02299
- OpenAlex
- W4412565294
- Cited by
- 193
- References
- 263
- Citations by year
- 2026: 75 · 2025: 53 · 2024: 45 · 2023: 20
- FWCI
- 51.16
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Intelligent Systems and Technology (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Griffith Research Online (submittedVersion)
- arXiv (Cornell University)
- Funders
- Australian Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- machine unlearning, right to be forgotten, privacy, adversarial attacks, training data reconstruction, personal data protection, privacy-preserving machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
