Deep Unlearning via Randomized Conditionally Independent Hessians
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Recent legislation has led to interest in machine unlearning, i. e., removing specific training samples from a predictive model as if they never existed in the training dataset. Unlearning may also be required due to corrupted/adversarial data or simply a user's updated privacy requirement. For models which require no training (k-NN), simply deleting the closest original sample can be effective. But this idea is inapplicable to models which learn richer representations. Recent ideas leveraging optimization-based updates scale poorly with the model dimension d, due to inverting the Hessian of the loss function. We use a variant of a new conditional independence coefficient, L-CODEC, to identify a subset of the model parameters with the most semantic overlap on an individual sample level. Our approach completely avoids the need to invert a (possibly) huge matrix. By utilizing a Markov blanket selection, we premise that L-CODEC is also suitable for deep unlearning, as well as other applications in vision. Compared to alternatives, L-CODEC makes approximate unlearning possible in settings that would otherwise be infeasible, including vision models used for face recognition, person reidentification and NLP models that may require unlearning samples identified for exclusion. Code is available at https://github.com/vsingh-group/LCODEC-deep-unlearning
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389
Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2021) · 2021 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40001.2021.00019 · Cited by 810 · cited 1×
“Other recent work has taken alternative views of unlearning, which do not require/operate under probabilistic frameworks, see [4, 31].”
BackgroundThe Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629
Towards Making Systems Forget with Machine Unlearning
Cao, Y., & Yang, J. (2015) · 2015 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2015.35 · Cited by 622 · cited 1×
“SVMs by [25,34], Na ¨ ıve Bayes Classifiers by [5], and k -means methods by [14] have all been studied.”
MethodEternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
Golatkar, A., Achille, A., & Soatto, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00932 · Cited by 394
A New Coefficient of Correlation
Chatterjee, S. (2021) · Journal of the American Statistical Association · doi:10.1080/01621459.2020.1758115 · Cited by 374 · cited 1×
“An altered version of this setup also gives us a form of explainability, where we can apply sensitivity analysis to each input feature or pixel and estimate its effect on the output via a similarly randomized version of the Chatterjee rank coefficient T (A,B), proposed by [8].”
MethodMaking AI Forget You: Data Deletion in Machine Learning
Ginart, A., Guan, M. Y., Valiant, G., & Zou, J. (2019) · arXiv · doi:10.48550/arxiv.1907.05012 · Cited by 175 · cited 2×
“SVMs by [23, 28], Naı̈ve Bayes Classifiers by [5], and k-means methods by [13] have all been studied.”
MethodCertified Data Removal from Machine Learning Models
Guo, C., Goldstein, T., Hannun, A., & van der Maaten, L. (2019) · arXiv · doi:10.48550/arxiv.1911.03030 · Cited by 160 · cited 2×
“[19, 22] provide updates to linear models and the last layers of networks, and [15, 16] provide updates based on linearizations that work over the full network, and followup work by [14] presents a scheme to unlearn under an assumption that some samples will not need to be removed.”
BackgroundMethodMixed-Privacy Forgetting in Deep Networks
Golatkar, A., Achille, A., Ravichandran, A., Polito, M., & Soatto, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00085 · Cited by 125 · cited 2×
“[19, 22] provide updates to linear models and the last layers of networks, and [15, 16] provide updates based on linearizations that work over the full network, and followup work by [14] presents a scheme to unlearn under an assumption that some samples will not need to be removed.”
BackgroundMethod
Cited by
A Survey of Machine Unlearning
Nguyen, T. T., Huynh, T. T., Ren, Z., Nguyen, P. L., Liew, A. W.-C., Yin, H., & Nguyen, Q. V. H. (2022) · arXiv · doi:10.48550/arxiv.2209.02299 · Cited by 193
“Softmax classifiers [6] ✗ ✓ ✓ ✗ – ✓ – – ✓ ✗ ✓ ✗ ✗ ✓ ✗ ✗ Linear models [73, 87] ✓ ✗ ✗ – ✗ – ✓ – ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ Tree-based models [132] ✗ – ✗ – ✗ ✗ – ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ Bayesian models [111] – ✓ ✗ ✗ ✗ – – ✓ – ✓ – ✓ ✗ ✗ ✗ ✗ DNN-based models [5, 54, 56, 57, 67, 105, 179] ✗ ✓ ✗ ✗ ✗ - - ✓ - - - ✓ ✗ ✗ ✗ ✗”
KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment
Wang, L., Chen, T., Yuan, W., Zeng, X., Wong, K.-F., & Yin, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.740 · Cited by 25
Patient Similarity Learning with Selective Forgetting
Qian, W., Zhao, C., Shao, H., Chen, M., Wang, F., & Huai, M. (2022) · 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9995016 · Cited by 11
TruVRF: Toward Triple-Granularity Verification on Machine Unlearning
Zhou, C., Gao, Y., Fu, A., Chen, K., Zhang, Z., Xue, M., Dai, Z., Ji, S., & Zhang, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3565991 · Cited by 7
Making Recommender Systems Forget: Learning and Unlearning for Erasable Recommendation
Li, Y., Zheng, X., Chen, C., & Liu, J. (2022) · arXiv · doi:10.48550/arxiv.2203.11491 · Cited by 43
Metadata
- Authors
- Mehta, R., Pal, S., Singh, V., & Ravi, S. N.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Volume
- 2022
- Pages
- 10412-10421
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312595359
- Cited by
- 60
- References
- 12
- FWCI
- 7.04
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.