Hard to Forget: Poisoning Attacks on Certified Machine Unlearning
Open accessAbstract
The right to erasure requires removal of a user's information from data held by organizations, with rigorous interpretations extending to downstream products such as learned models. Retraining from scratch with the particular user's data omitted fully removes its influence on the resulting model, but comes with a high computational cost. Machine "unlearning" mitigates the cost incurred by full retraining: instead, models are updated incrementally, possibly only requiring retraining when approximation errors accumulate. Rapid progress has been made towards privacy guarantees on the indistinguishability of unlearned and retrained models, but current formalisms do not place practical bounds on computation. In this paper we demonstrate how an attacker can exploit this oversight, highlighting a novel attack surface introduced by machine unlearning. We consider an attacker aiming to increase the computational cost of data removal. We derive and empirically investigate a poisoning attack on certified machine unlearning where strategically designed training data triggers complete retraining when removed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A membership inference attack or a poisoning attack would reveal a significant amount of detailed information about the unlearned samples.
“Thus, a membership inference attack or a poisoning attack would reveal a significant amount of detailed information about the unlearned samples [78, 108].”
From Machine Unlearning: A Survey (2024) · cites this paper as [108]
The Abstract (p. 1) and Related Work (p. 7) describe this paper’s poisoning attack as increasing the computational cost of unlearning, not revealing information about erased samples. Related Work mentions privacy vulnerabilities involving differencing attacks, but does not say that membership inference or poisoning reveals detailed information about unlearned samples. I searched for membership inference and privacy leakage from poisoning attacks. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 5× · see citing works
“Most of the results reported above are for Binary-MNIST.”
InfluentialBertsekas, D. P. (1997) · Journal of the Operational Research Society · doi:10.1038/sj.jors.2600425 · Cited by 11,969 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · cited 2× · see citing works
“We note that the signed gradient has been used as alternative to the gradient when generating adversarial examples (Goodfellow, Shlens, and Szegedy, 2015; Madry et al., 2018).”
On the limited memory BFGS method for large scale optimization
Liu, D. C., & Nocedal, J. (1989) · Mathematical Programming · doi:10.1007/bf01589116 · Cited by 8,684 · cited 1× · see citing works
“We assumed the organization and the attacker both used L-BFGS (D. C. Liu and Nocedal, 1989) to fit their models.”
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 2× · see citing works
“The `p-norm constraints that we impose on the attacker’s perturbations are commonly used to generate adversarial examples (Szegedy et al. 2014; Goodfellow, Shlens, and Szegedy 2015).”
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108 · cited 2× · see citing works
“For instance, Fashion-MNIST has a wider variation of intensities across all pixels compared to MNIST, which may make ℓ 1 -bounded perturbations less effective.”
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 2× · see citing works
“Such concerns are justified by a legitimate privacy risk: training data can be practically reconstructed from models (Fredrikson, Jha, and Ristenpart, 2015; Shokri et al., 2017; Veale, Binns, and Edwards, 2018).”
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961 · cited 1× · see citing works
“Such concerns are justified by a legitimate privacy risk: training data can be practically reconstructed from models (Fredrikson, Jha, and Ristenpart, 2015; Shokri et al., 2017; Veale, Binns, and Edwards, 2018).”
Minimization of functions having Lipschitz continuous first partial derivatives
Armijo, L. (1966) · Pacific Journal of Mathematics · doi:10.2140/pjm.1966.16.1 · Cited by 2,009 · cited 1× · see citing works
“Following Armijo (1966) we set 𝜏 = 𝑐 = .”
(2013) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · Cited by 1,959 · see citing works
Our Data, Ourselves: Privacy Via Distributed Noise Generation
Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., & Naor, M. (2006) · Lecture Notes in Computer Science · doi:10.1007/11761679_29 · Cited by 1,835 · cited 4× · see citing works
“They identified a connection to differential privacy (Dwork et al. 2006) and suggested adversarial robustness as a direction for future work.”
InfluentialUnderstanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · cited 3× · see citing works
“The perturbation is computed as where 𝐷 is the initial training data, 𝐷 (cid:48) ⊂ 𝐷 is the data to erase, 𝜽 are the initial model parameters and is the influence function (Cook and Weisberg, 1980; Koh and Liang, 2017).”
Poisoning Attacks against Support Vector Machines
Biggio, B., Nelson, B., & Laskov, P. (2012) · arXiv · doi:10.48550/arxiv.1206.6389 · Cited by 1,331 · cited 2× · see citing works
“Following standard threat models in the data poisoning literature (Biggio, Nelson, and Laskov, 2012; Muñoz-González et al., 2017; Shafahi et al., 2018; Shen, Xiaolei Zhu, and Ma, 2019; W. R. Huang et al., 2020) we consider attackers with white-box and grey-box access.”
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194 · cited 2× · see citing works
“In this paper we consider an active adversary as is more typical in the adversarial learning field (L. Huang et al., 2011).”
Self-citationEfficient projections onto the l 1 -ball for learning in high dimensions
Duchi, J. C., Shalev‐Shwartz, S., Singer, Y., & Chandra, T. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390191 · Cited by 1,187 · cited 1× · see citing works
“For instance, Duchi et al. (2008) provides a linear-time projection algorithm onto the ℓ 1 -ball.”
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
Chen, X., Liu, C., Li, B., Lu, K., & Song, D. (2017) · arXiv · doi:10.48550/arxiv.1712.05526 · Cited by 1,039 · cited 1× · see citing works
“The perturbation is computed as where 𝐷 is the initial training data, 𝐷 (cid:48) ⊂ 𝐷 is the data to erase, 𝜽 are the initial model parameters and is the influence function (Cook and Weisberg, 1980; Koh and Liang, 2017).”
Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2021) · 2021 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40001.2021.00019 · Cited by 810 · cited 4× · see citing works
“They also demonstrated an attack on the SISA algorithm (Bourtoule et al., 2021).”
InfluentialOn Detecting Adversarial PerturbationsPreprint
Metzen, J. H., Genewein, T., Fischer, V., & Bischoff, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04267 · Cited by 747 · cited 2× · see citing works
“Metzen et al. (2017) for instance, have explored detection of adversarial perturbations.”
Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural NetworksPreprint
Shafahi, A., Huang, W., Najibi, M., Suciu, O., Studer, C., Dumitraş, T., & Goldstein, T. (2018) · arXiv (Cornell University) · doi:10.3929/ethz-b-000460348 · Cited by 626 · see citing works
Towards Making Systems Forget with Machine Unlearning
Cao, Y., & Yang, J. (2015) · 2015 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2015.35 · Cited by 622 · cited 1× · see citing works
“Recent interest in efficient data erasure for learned models was spawned by Cao and Yang (2015), who were motivated by privacy and security applications.”
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization
Muñoz-González, L., Biggio, B., Demontis, A., Paudice, A., Wongrassamee, V., Lupu, E. C., & Roli, F. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140451 · Cited by 549 · cited 3× · see citing works
“Following standard threat models in the data poisoning literature (Biggio, Nelson, and Laskov, 2012; Muñoz-González et al., 2017; Shafahi et al., 2018; Shen, Xiaolei Zhu, and Ma, 2019; W. R. Huang et al., 2020) we consider attackers with white-box and grey-box access.”
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
Golatkar, A., Achille, A., & Soatto, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00932 · Cited by 394 · cited 2× · see citing works
“One might adopt an unlearning algorithm that never resorts to retraining from scratch (Golatkar, Achille, and Soatto, 2020a).”
Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners
Mei, S., & Zhu, X. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9569 · Cited by 346 · cited 1× · see citing works
“This is a bilevel optimization problem (Mei and Zhu 2015) if the learning algorithm A solves a deterministic optimization problem.”
Characterizations of an empirical influence function for detecting influential cases in regression
Cook, R. D., & Weisberg, S. (1980) · Technometrics · doi:10.1080/00401706.1980.10486199 · Cited by 341 · cited 1× · see citing works
“The perturbation is computed as where 𝐷 is the initial training data, 𝐷 (cid:48) ⊂ 𝐷 is the data to erase, 𝜽 are the initial model parameters and is the influence function (Cook and Weisberg, 1980; Koh and Liang, 2017).”
A Backdoor Attack Against LSTM-Based Text Classification Systems
Dai, J., Chen, C., & Li, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2941376 · Cited by 298 · see citing works
Algorithms that remember: model inversion attacks and data protection law
Veale, M., Binns, R., & Edwards, L. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0083 · Cited by 278 · cited 2× · see citing works
“Veale, Binns, and Edwards (2018) argue that models could be legally classified as personal data, which agrees with the draft guidance from the UK ICO (2020) that “erasure of the data... may not be possible without retraining the model... or deleting the model altogether”.”
Transferable Clean-Label Poisoning Attacks on Deep Neural Nets
Zhu, C., Huang, W., Shafahi, A., Li, H., Taylor, G., Studer, C., & Goldstein, T. (2019) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000459774 · Cited by 190 · see citing works
Making AI Forget You: Data Deletion in Machine Learning
Ginart, A., Guan, M. Y., Valiant, G., & Zou, J. (2019) · arXiv · doi:10.48550/arxiv.1907.05012 · Cited by 175 · cited 4× · see citing works
“Ginart et al. (2019) were first to introduce an approximate definition of unlearning—requiring that it be statistically indistinguishable from retraining.”
InfluentialTowards Large Yet Imperceptible Adversarial Image Perturbations With Perceptual Color Distance
Zhao, Z., Liu, Z., & Larson, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00112 · Cited by 166 · see citing works
Certified Data Removal from Machine Learning Models
Guo, C., Goldstein, T., Hannun, A., & van der Maaten, L. (2019) · arXiv · doi:10.48550/arxiv.1911.03030 · Cited by 160 · see citing works
Transferable Clean-Label Poisoning Attacks on Deep Neural NetsPreprint
Zhu, C., Huang, W., Shafahi, A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.05897 · Cited by 133 · cited 2× · see citing works
“Two commonly studied variants of data poisoning are availability attacks, which aim to degrade model test performance (e.g., Biggio, Nelson, and Laskov 2012; Muñoz González et al. 2017; Koh and Liang 2017; Shafahi et al. 2018; Zhu et al. 2019; Huang et al. 2020), and backdoor attacks, which aim to cause misclassificat…”
On the Suitability of Lp-Norms for Creating and Preventing Adversarial Examples
Sharif, M., Bauer, L., & Reiter, M. K. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00211 · Cited by 128 · cited 1× · see citing works
“While ℓ 𝑝 -norm constraints are easy to work with, they have been criticized for failing to capture perceptual distortion (Sharif, Bauer, and Reiter, 2018), which has lead to work on more effective perturbation constraints (Zhao, Z. Liu, and Larson, 2020; Ballet et al., 2019).”
Golatkar, A., Achille, A., & Soatto, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58526-6_23 · Cited by 122 · see citing works
MetaPoison: Practical General-purpose Clean-label Data PoisoningPreprint
Huang, W., Geiping, J., Fowl, L., Taylor, G., & Goldstein, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00225 · Cited by 102 · see citing works
Remember What You Want to Forget: Algorithms for Machine Unlearning
Sekhari, A., Acharya, J., Kamath, G., & Suresh, A. T. (2021) · arXiv · doi:10.48550/arxiv.2103.03279 · Cited by 99 · cited 4× · see citing works
“Subsequent work (Guo et al. 2020; Neel, Roth, and Sharifi-Malvajerdi 2021; Sekhari et al. 2021) has adopted similar approximate definitions to provide certified unlearning guarantees for strongly-convex learning problems.”
InfluentialDescent-to-Delete: Gradient-Based Methods for Machine Unlearning
Neel, S., Roth, A., & Sharifi-Malvajerdi, S. (2020) · arXiv · doi:10.48550/arxiv.2007.02923 · Cited by 96 · see citing works
HedgeCut: Maintaining Randomised Trees for Low-Latency Machine Unlearning
Schelter, S., Grafberger, S., & Dunning, T. (2021) · Proceedings of the 2021 International Conference on Management of Data · doi:10.1145/3448016.3457239 · Cited by 89 · cited 1× · see citing works
“Examples of such models include ensembles of randomized decision trees (Schelter, Grafberger, and Dunning, 2021), variants of 𝑘 -means clustering (Ginart et al., 2019), and a modeling framework called SISA that takes advantage of data sharding and caching (Bourtoule et al., 2021).”
Unlearnable Examples: Making Personal Data UnexploitablePreprint
Huang, H., Ma, X., Erfani, S., Bailey, J., & Wang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04898 · Cited by 85 · cited 2× · see citing works
“2021) and protecting data from being used to train models (Huang et al. 2021).”
Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning AttacksPreprint
Schwarzschild, A., Goldblum, M., Gupta, A. K., Dickerson, J. P., & Goldstein, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12557 · Cited by 78 · cited 1× · see citing works
“Our work builds on standard methods for data poisoning —a class of attacks that manipulate learned models by modifying or augmenting training data (see survey Schwarzschild et al., 2021).”
Imperceptible Adversarial Attacks on Tabular DataPreprint
Ballet, V., Renard, X., Aigrain, J., Laugel, T., Pascal Frossard, B., & Detyniecki, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03274 · Cited by 72 · cited 2× · see citing works
“While `p-norm constraints are easy to work with, they have been criticized for failing to capture perceptual distortion (Sharif, Bauer, and Reiter 2018), which has lead to work on more effective perturbation constraints (Zhao, Liu, and Larson 2020; Ballet et al. 2019).”
Characterizations of an Empirical Influence Function for Detecting Influential Cases in Regression
Cook, R. D., & Weisberg, S. (1980) · Technometrics · doi:10.2307/1268187 · Cited by 68 · see citing works
Gupta, V., Jung, C., Neel, S., Roth, A., Sharifi-Malvajerdi, S., & Waites, C. (2021) · arXiv · doi:10.48550/arxiv.2106.04378 · Cited by 67 · cited 2× · see citing works
“M. Chen et al. (2020) demonstrated that a series of published unlearned models are vulnerable to differencing attacks if not appropriately treated.”
Chambers, J. M. (1971) · Journal of the American Statistical Association · doi:10.1080/01621459.1971.10482338 · Cited by 63 · see citing works
Data Poisoning Attacks against Online LearningPreprint
Wang, Y., & Chaudhuri, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08994 · Cited by 52 · cited 2× · see citing works
“There has been some work on data poisoning for online learning (Wang and Chaudhuri 2018; Zhang, Zhu, and Lessard 2020), however existing work does not account for the ability of an attacker to request that their data be erased.”
Linear unlearning for cross-validation
Hansen, L. K., & Larsen, J. O. (1996) · Advances in Computational Mathematics · doi:10.1007/bf02124747 · Cited by 45 · cited 2× · see citing works
“Efficient unlearning algorithms are known for some canonical models under this definition, including linear regression (Chambers 1971; Hansen and Larsen 1996), naive Bayes and non-parametric models such as k-nearest neighbors (Schelter 2020).”
TensorClog: An Imperceptible Poisoning Attack on Deep Neural Network Applications
Shen, J., Zhu, X., & Ma, D. (2019) · IEEE Access · doi:10.1109/access.2019.2905915 · Cited by 43 · cited 1× · see citing works
“Following standard threat models in the data poisoning literature (Biggio, Nelson, and Laskov, 2012; Muñoz-González et al., 2017; Shafahi et al., 2018; Shen, Xiaolei Zhu, and Ma, 2019; W. R. Huang et al., 2020) we consider attackers with white-box and grey-box access.”
Zhang, X., Zhu, X., & Lessard, L. (2020) · Cited by 40 · see citing works
A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network InferencePreprint
Hong, S., Kaya, Y., Modoranu, I.-V., & Dumitraş, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02432 · Cited by 39 · cited 2× · see citing works
“While adversarial examples are predominantly used for evasion attacks, they have also been deployed for other purposes.”
InfluentialTowards Probabilistic Verification of Machine Unlearning
Sommer, D. M., Song, L., Wagh, S., & Mittal, P. (2020) · arXiv · doi:10.48550/arxiv.2003.04247 · Cited by 38 · cited 1× · see citing works
“Sommer et al. (2020) considered verification of unlearning—an important privacy feature for users who cannot trust organizations to erase their data.”
Robust Stopping Criteria for Dykstra's Algorithm
Birgin, E. G., & Raydan, M. (2005) · SIAM Journal on Scientific Computing · doi:10.1137/03060062x · Cited by 37 · cited 2× · see citing works
“𝑡 ← 𝑐 (cid:104) G , Δ X ⊲ Frobenius inner prod. return 𝜂 9: end procedure Algorithm 5 Projection operator onto the intersection of 𝐽 convex sets Ω = 𝐽𝑗 = 1 Ω based on Dykstra’s algorithm (Birgin and Raydan, 2005).”
When Machine Unlearning Jeopardizes Privacy
Chen, M., Zhang, Z., Wang, T., Michael Backes, A., Humbert, M., & Zhang, Y. (2021) · Figshare · doi:10.60882/cispa.24613773 · Cited by 33 · see citing works
When Machine Unlearning Jeopardizes Privacy
Chen, M., Zhang, Z., Wang, T., et al. (2021) · Figshare · doi:10.60882/cispa.24613773.v1 · Cited by 30 · see citing works
A backdoor attack against LSTM-based text classification systemsPreprint
Dai, J., & Chen, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12457 · Cited by 19 · see citing works
"Amnesia" - Machine Learning Models That Can Forget User Data Very Fast.
Schelter, S. (2020) · Conference on Innovative Data Systems Research · Cited by 14 · see citing works
Hidden Trigger Backdoor Attacks
Saha, A., Subramanya, A., & Pirsiavash, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6871 · Cited by 13 · cited 1× · see citing works
“…2017; Koh and Liang, 2017; Shafahi et al., 2018; C. Zhu et al., 2019; W. R. Huang et al., 2020), and backdoor attacks , which aim to cause misclassification of test-time samples that contain a trigger (e.g. X. Chen et al., 2017; Dai, C. Chen, and Li, 2019; Saha, Subramanya, and Pirsiavash, 2020).”
Towards Large yet Imperceptible Adversarial Image Perturbations with Perceptual Color DistancePreprint
Zhao, Z., Liu, Z., & Larson, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02466 · Cited by 10 · see citing works
The Geometry of Sign Gradient DescentPreprint
Balles, L., Pedregosa, F., & Roux, N. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08056 · Cited by 7 · cited 1× · see citing works
“Both are reasonable approaches—which is more effective likely depends on the geometry of the problem (Balles, Pedregosa, and Roux, 2020).”
Provable Defense Against Delusive PoisoningPreprint
Tao, L., Feng, L., Yi, J., Huang, S.-J., & Chen, S. (2021) · arXiv (Cornell University) · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Huang, W. R.; Geiping, J.; Fowl, L.; Taylor, G.; and Goldstein, T. 2020. MetaPoison: Practical General-purpose Cleanlabel Data Poisoning. In Advances in Neural Information Processing Systems , volume 33, 12080-12091.
- Marchant, N. G.; Rubinstein, B. I. P.; and Alfeld, S. 2021. Hard to Forget: Poisoning Attacks on Certified Machine Unlearning. arXiv:2109.08266. · arXiv
- Neel, S.; Roth, A.; and Sharifi-Malvajerdi, S. 2021. Descentto-Delete: Gradient-Based Methods for Machine Unlearning. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory , volume 132, 931-962.
Cited by
Sorted by citations
A Survey of Machine Unlearning
Nguyên, T. T., Huynh, T. T., Ren, Z. X., Nguyen, P. L., Liew, A. W., Yin, H., & Nguyen, Q. V. H. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3749987 · Cited by 303 · see citing works
“(Bounded/Certified Unlearning) Definition 1 ( 𝜖 -Approximate Unlearning). algorithm 𝐴 if ∀T ⊆ H ,𝐷 ∈ 𝑍 ∗ ,𝑧 ∈ 𝐷 :”
Fast Yet Effective Machine Unlearning
Tarun, A. K., Chundawat, V. S., Mandal, M., & Kankanhalli, M. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3266233 · Cited by 203 · see citing works
“Some researchers have studied the unintended privacy risks resulting from the existing unlearning methods [34], [35].”
BackgroundFast Yet Effective Machine Unlearning
Tarun, A. K., Chundawat, V. S., Mandal, M., & Kankanhalli, M. (2021) · arXiv · doi:10.48550/arxiv.2111.08947 · Cited by 190 · see citing works
Xu, H., Zhu, T., Zhang, L., Zhou, W., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3603620 · Cited by 164 · see citing works
A Review on Machine Unlearning
Zhang, H., Nakamura, T., Isohara, T., & Sakurai, K. (2023) · SN Computer Science · doi:10.1007/s42979-023-01767-4 · Cited by 81 · see citing works
“• Data Poisoning: The attacker can mix specific malicious data in the request process, which can affect the model training and subsequent model inference through the feedback of the service process to achieve the effect of interfering with the model [9, 10].”
A Survey of Machine Unlearning
Nguyen, T. T., Huynh, T. T., Ren, Z., Nguyen, P. L., Liew, A. W.-C., Yin, H., & Nguyen, Q. V. H. (2022) · arXiv · doi:10.48550/arxiv.2209.02299 · Cited by 193 · see citing works
“(Bounded/Certified Unlearning) Definition 1 ( 𝜖 -Approximate Unlearning). algorithm 𝐴 if ∀T ⊆ H ,𝐷 ∈ 𝑍 ∗ ,𝑧 ∈ 𝐷 :”
Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomy
Shaik, T., Tao, X., Xie, H., Li, L., Zhu, X., & Li, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3486109 · Cited by 78 · see citing works
FedME 2 : Memory Evaluation & Erase Promoting Federated Unlearning in DTMN
Xia, H., Xu, S., Pei, J., Zhang, R., Yu, Z. H., Zou, W., Wang, L., & Liu, C. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2023.3310049 · Cited by 60 · see citing works
“(2) Model memory erasure [15], [16], [17], [18].”
BackgroundMethodInfluentialGIF: A General Graph Unlearning Strategy via Influence Function
Wu, J., Yang, Y., Qian, Y., Sui, Y., Wang, X., & He, X. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583521 · Cited by 58 · see citing works
Threats, Attacks, and Defenses in Machine Unlearning: A Survey
Liu, Z., Ye, H., Chen, C., Zheng, Y., & Lam, K. (2025) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2025.3543483 · Cited by 51 · see citing works
“In the context that a model has been compromised after the detection of poisoning attacks, machine unlearning presents itself as an intuitive approach for model recovery, enabling the targeted removal of identified poisoned data.”
Learn What You Want to Unlearn: Unlearning Inversion Attacks against Machine Unlearning
Hu, H., Wang, S., Dong, T., & Xue, M. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00248 · Cited by 44 · see citing works
“However, in model inversion attacks, the target model is static (i.e, does not involve two models), and Slow-down unlearning attack [13] Efficiency vulnerability Camouflaged poisoning attack [26] Data poisoning vulnerability Over-unlearning attacks [14] Fidelity vulnerability Membership inference…”
BackgroundInfluentialGraph neural networks: a survey on the links between privacy and security
Guan, F., Zhu, T., Zhou, W., & Choo, K. R. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10656-4 · Cited by 42 · see citing works
Machine Unlearning: Taxonomy, Metrics, Applications, Challenges, and Prospects
Li, N., Zhou, C., Gao, Y., Chen, H., Zhang, Z., Kuang, B., & Fu, A. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3530988 · Cited by 53 · see citing works
“For example, Marchant et al. [133] introduced carefully crafted noise to trigger the unlearning process far more than typically required, significantly increasing computational and time costs, akin to traditional denial-of-service attacks.”
No Matter How You Slice It: Machine Unlearning with SISA Comes at the Expense of Minority Classes
Koch, K., & Soll, M. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00047 · Cited by 34 · see citing works
Backdoor Attacks via Machine Unlearning
Liu, Z., Wang, T., Huai, M., & Miao, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29321 · Cited by 32 · see citing works
Learn to Unlearn: Insights Into Machine Unlearning
Qu, Y., Yuan, X., Ding, M., Ni, W., Rakotoarivelo, T., & Smith, D. B. (2024) · Computer · doi:10.1109/mc.2023.3333319 · Cited by 30 · see citing works
A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services
Hu, H., Wang, S., Chang, J., Zhong, H., Sun, R., Hao, S., Zhu, H., & Xue, M. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24252 · Cited by 37 · see citing works
An overview of machine unlearning
Li, C., Jiang, H., Chen, J., Zhao, Y. L., Fu, S., Jing, F., & Guo, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100254 · Cited by 21 · see citing works
Machine Unlearning: Its Nature, Scope, and Importance for a “Delete Culture”
Floridi, L. (2023) · Philosophy & Technology · doi:10.1007/s13347-023-00644-5 · Cited by 20 · see citing works
“2022; Marchant, Rubinstein, and Alfeld 2022) because if a machine learning model is compromied by the injection of some pollution data into its dataset, it becomes necessary to remove such data from the dataset and retrain it.”
BackgroundSupporting Trustworthy AI Through Machine Unlearning
Hine, E., Novelli, C., Taddeo, M., & Floridi, L. (2024) · Science and Engineering Ethics · doi:10.1007/s11948-024-00500-5 · Cited by 19 · see citing works
“MU algo‑ rithms could be vulnerable to attacks that increase computation costs (Marchant et al., 2022) or use malicious unlearning requests to skew the model—for example, if a model is made to unlearn a label introduced for debiasing efforts, or a set of datapoints that would, if deleted, cause the…”
BackgroundGet Rid of Your Trail : Remotely Erasing Backdoors in Federated Learning
Alam, M., Lamri, H., & Maniatakos, M. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3465441 · Cited by 19 · see citing works
“Approximate unlearning is achieved by employing techniques such as influence functions to iteratively estimate the effect of the data points targeted for removal on the model parameters and subsequently adjusting the model parameters based on the estimated influence of the data point [39], [58], [59].”
MethodMUter: Machine Unlearning on Adversarially Trained Models
Liu, J., Xue, M., Lou, J., Zhang, X., Xiong, L., & Qin, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00451 · Cited by 16 · see citing works
A survey on machine unlearning: Techniques and new emerged privacy risks
Liu, H., Xiong, P., Zhu, T., & Yu, P. S. (2025) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104010 · Cited by 21 · see citing works
Sheng, X., Bao, W., & Ge, L. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679817 · Cited by 15 · see citing works
A survey of security and privacy issues of machine unlearning
Chen, A., Li, Y., Zhao, C., & Huai, M. (2025) · AI Magazine · doi:10.1002/aaai.12209 · Cited by 13 · see citing works
Anti-Backdoor Model: A Novel Algorithm to Remove Backdoors in a Non-Invasive Way
Chen, C., Hong, H., Xiang, T., & Xie, M. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3436508 · Cited by 9 · see citing works
“In contrast to conventional data poisoning [23], [24], [25], the backdoor poisoning attack does not compromise the accuracy of the original task, but introduces hidden triggers that activate only under specific circumstances within the attacked model.”
ResultIdentifying a Training-Set Attack's Target Using Renormalized Influence Estimation
Hammoudeh, Z., & Lowd, D. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3559335 · Cited by 10 · see citing works
“For learning environments where certified training data deletion is possible [22, 43], retraining (Alg.”
BackgroundThe Price of Unlearning: Identifying Unlearning Risk in Edge Computing
Zhang, L., Zhu, T., Xiong, P., & Zhou, W. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3662184 · Cited by 8 · see citing works
“Marchant et al. [30] focused on increasing the computation cost of machine unlearning.”
Exact and Efficient Unlearning for Large Language Model-Based Recommendation
Hu, Z., Zhang, Y., Xiao, M., Wang, W., Feng, F., & He, X. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3594687 · Cited by 17 · see citing works
“Toachieveefficientunlearningwhilepreserving high recommendation performance, we propose an Adapter Partition and Aggregation (APA) approach, which consists of 1) a balanced semantic-aware data partition method, and 2) a sample-adaptive method.”
Implications of Privacy Regulations on Video Surveillance Systems
Kansal, K., Wong, Y., & Kankanhalli, M. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3706108 · Cited by 7 · see citing works
TruVRF: Toward Triple-Granularity Verification on Machine Unlearning
Zhou, C., Gao, Y., Fu, A., Chen, K., Zhang, Z., Xue, M., Dai, Z., Ji, S., & Zhang, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3565991 · Cited by 7 · see citing works
“However, this method involves unacceptable computational overhead, particularly for large-scale datasets, as highlighted [29], [30], [33], which is di ffi cult to use.”
Huang, A., Cai, Z., & Xiong, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3570989 · Cited by 7 · see citing works
“Although the above methods show great potential to eliminate undesirable concepts, many works [107], [108], [109] reveal the vulnerability of machine unlearning to different attacks.”
Exploring Unlearning Methods to Ensure the Privacy, Security, and Usability of Recommender Systems
Leysen, J. (2023) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3604915.3608862 · Cited by 7 · see citing works
Machine Unlearning Through Fine-Grained Model Parameters Perturbation
Zuo, Z., Tang, Z., Li, K., & Datta, A. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3528551 · Cited by 7 · see citing works
“Byselectivelyperturbingasmall Byselectivelyperturbingasmall subsetofparameters,weaimtoachievethedesiredeffectof machineunlearningefficientlywhilealsoreducingtheimpact ontheoverallmodelperformance.”
Machine Unlearning Methodology Based on Stochastic Teacher Network
Zhang, X., Wang, J., Cheng, N., Sun, Y., Zhang, C., & Xiao, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46677-9_18 · Cited by 7 · see citing works
Fuzzy rough unlearning model for feature selection
Tang, Y., Zhao, S., Chen, H., Li, C., Zhai, J., & Zhou, Q. (2023) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2023.109102 · Cited by 6 · see citing works
Evaluating Machine Unlearning: Applications, Approaches, and Accuracy
Ali, Z., Muhammad, A., Adnan, R., Alkhalifah, T., & Aslam, S. (2024) · Engineering Reports · doi:10.1002/eng2.13081 · Cited by 5 · see citing works
Unlearning Attacks for Regression Learning
Chen, J., Shi, W., Lin, W., Wang, C., Liu, W., Sun, H., & Liu, G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3553821 · Cited by 4 · see citing works
“Furthermore, Marchant et al. [25] aimed to slow down the unlearning process by adding crafted poisoned data.”
Backdoor Online Tracing With Evolving Graphs
Jia, C., Chen, J., Ji, S., Cheng, Y., Zheng, H., & Xuan, Q. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3488517 · Cited by 3 · see citing works
“Simultaneously, identifying backdoor threats early on allows for the implementation of more efficient mitigation strategies [2] (enhance the defense effect and reduce the side-effect on the benign performance).”
Continuous Verification of Catastrophic Recalling in Machine Unlearning via Adversarial Testing
Qu, Y., Ji, Z., Cui, L., Zhang, C., Liu, L., & Tian, Z. (2024) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc63484.2024.00056 · Cited by 3 · see citing works
“Poisoning attacks can also be used as a certified mechanism [24], and the attacks include both white-box and gray-box attacks that specifically target a unique class of machine oblivious learning models, in particular certified removal and its variants.”
Ensuring User Privacy and Model Security via Machine Unlearning: A Review
Tang, Y., Cai, Z., Liu, Q., Zhou, T., & Ni, Q. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.032307 · Cited by 3 · see citing works
Gao, K., Zhu, T., Ye, D., Liu, B., & Zhou, W. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3630811 · Cited by 2 · see citing works
Tran, H. N., Nguyen, N. T., Nguyen, N. V., Nguyen, H. X., & Nguyen, A. D. (2025) · Engineering Science and Technology an International Journal · doi:10.1016/j.jestch.2025.101996 · Cited by 2 · see citing works
Aliyeva, L., Abdullayev, N., Zarbiyeva, S., Huseynov, I., & Suleymanov, U. (2024) · IEEE International Conference on Application of Information and Communication Technologies (AICT) · doi:10.1109/aict61888.2024.10740413 · Cited by 1 · see citing works
“Projected gradient descent by Marchant et al. [32], and attack on federated learning by Sun et al. [68] are proposed frameworks for data poisoning.”
BackgroundTang, L., & Joshi, J. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00040 · Cited by 1 · see citing works
Trojan Attack on Machine Unlearning: Security Risk of the Right to be Forgotten
Zhang, L., Zhu, T., Xiong, P., & Zhou, W. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3567848 · Cited by 1 · see citing works
“Marchant et al. [29] focused on increasing the computation cost of machine unlearning.”
Understanding the Robustness of Machine-Unlearning Models
Zhang, G., Xu, F., Bandara, H. M. N. D., Chen, S., & Sui, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9101-2_16 · Cited by 1 · see citing works
A2E: Black-Box Anti-Adversarial Example Based Watermarking to Verify Federated Unlearning
Gao, X., Wang, J., Zhang, Z., Chen, J., Cheng, P., & Chen, J. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3598987 · Cited by 1 · see citing works
FUBA: Backdoor Federated Learning via Federated Unlearning
Sheng, X., Bao, W., Guo, Y., & Fu, S. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3630110 · Cited by 1 · see citing works
“Some studies have investigated the use of unlearning techniques for poisoning [67], slow-down [68], or selective-forgetting attacks [69], [70], which involve attack models different from ours.”
Tackling Sequential Entanglement in Split Unlearning
Etheridge, A., Zhu, M., Liu, X., & Huang, C. (2025) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr67560.2025.00053 · Cited by 1 · see citing works
Differential Privacy with Data Removal for Online Happiness Assessment
Luo, Y., Lin, L., Wu, X., & Tao, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7241-4_11
Selective Noise-Aided Machine Unlearning with Deep Feature Visualization
Shaheryar, M., Lee, J. T., & Jung, S. K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-77389-1_8
DT-UPD: User Privacy Data Protection Through Distribution Transformation in Unlearning Cloud Service
Sun, S., Wu, F., Zhang, Y., Cui, L., Gao, L., & Yu, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1548-3_5
Inversion Triplet - A Contrastive Backdoor Mitigation Method for Self-Supervised Vision Encoders
Vo, H. K., Tian, Z., Zhang, C., Zheng, X., & Yu, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8295-9_20
FELEMN: Toward Efficient Feature-Level Machine Unlearning for Exact Privacy Protection
Wang, Z., Yu, Y., Li, M., Lou, J., Wang, N., Gu, Y., Su, S., Liu, Y., Jiang, H. L., & Tian, Z. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3613659
“Besides, some researchers state that the approximate scheme still has serious security flaws [11], [12], [13].”
PAMUS: An Entropy-Loss-Based Poisoning Attack for Undermining Machine Unlearning
Jiang, X., Ma, Y., Xu, W., & Wen, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-00828-2_7
Hard to Forget: Poisoning Attacks on Certified Machine Unlearning
Association for Artificial Intelligence 2022, Alfeld, S., Marchant, N. G., & Rubinstein, B. (2022) · Underline Science Inc. · doi:10.48448/ep63-m808
Self-citationJi, Z., Qu, Y., Gao, L., & Zhang, T. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3055-7_15
CAUA: A Realistic and Effective Attack on Machine Unlearning Under Limited Information
Zhang, J., Zhang, Y., Cui, J., Xie, X., Fan, C., & Li, B. (2025) · Annual Computer Security Applications Conference · doi:10.1109/acsac67867.2025.00080
“Marchant et al. [20] introduced an attack using projected gradient descent (PGD) to add perturbations to images, which are then uploaded to servers to interfere with the unlearning process in linear models.”
ZIUM: Zero-Shot Intent-Aware Adversarial Attack on Unlearned Models
Yook, H. J., Jhun, G. S., Cho, J. H., Jeon, M., Kim, D., Kim, T. H., & Lee, Y. K. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00374
“Several methods leveraging adversarial attacks on MU have been proposed [6, 12, 27, 28, 43].”
Meng, X. (2026) · Big data management · doi:10.1007/978-981-95-5787-5_9
How Much Can We Forget? On Boosting Removal Capacity in Machine Unlearning
Lu, X., Li, L., Zhang, D., Shi, K., & Tao, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1947-6_13
Ghasemi, M. (2026) · Artificial Intelligence in Geosciences · doi:10.1016/j.aiig.2026.100248
Machine unlearning through model forgetting techniques: application area and emerging methods
Büyüktanır, B., Kucur, E. N., Kuyumcu, Z., Ozdemir, N., Altınkaya, Ş., Baydoğmus, G. K., & Yıldız, K. (2026) · Cluster Computing · doi:10.1007/s10586-026-06300-9
“Marchant et al. [ 98 ] developed an attack targeting Certified Unlearning processes.”
A Review on Machine Unlearning Algorithms and Privacy Protection
Aliprantis, J., & Kalloniatis, C. (2026) · Electronics · doi:10.3390/electronics15184057
Vuseghesa, F. K., Messai, M.-L., & Bentayeb, F. (2026) · SN Computer Science · doi:10.1007/s42979-026-05397-4
“To mitigate poisoning attacks, existing studies primarily focus on traditional ML security measures, such as adversarial training [23, 24] but they don’t offer a complete solution due to their lack of generalizability to new attacks and accuracy preservation.”
- Authors
- Marchant, N. G., Rubinstein, B. I. P., & Alfeld, S.
- Affiliations
- The University of Melbourne (AU)
- Amherst College (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2022-06-28
- Volume
- 36
- Issue
- 7
- Pages
- 7691-7700
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W3199974954
- Cited by
- 64
- References
- 58
- Citations by year
- 2026: 6 · 2025: 25 · 2024: 19 · 2023: 12 · 2022: 2
- FWCI
- 8.00
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- Commonwealth Scientific and Industrial Research Organisation (DATA61)
- Department of Defence, Australian Government
- University of Melbourne (LE170100200)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Keywords
- machine unlearning, poisoning attacks, right to be forgotten, data deletion, privacy guarantees
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
