CoChat
Sign Up
PreprintCited by 62Privacy-Preserving Technologies in DataGaussian Processes and Bayesian InferenceAdversarial Robustness in Machine Learning

Variational Bayesian Unlearning

Open access
Nguyen, Quoc Phong, Low, Bryan Kian Hsiang, Jaillet, Patrick·2020·arXiv
DOI 10.48550/arxiv.2010.12883arXiv 2010.12883
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper studies the problem of approximately unlearning a Bayesian model from a small subset of the training data to be erased. We frame this problem as one of minimizing the Kullback-Leibler divergence between the approximate posterior belief of model parameters after directly unlearning from erased data vs. the exact posterior belief from retraining with remaining data. Using the variational inference (VI) framework, we show that it is equivalent to minimizing an evidence upper bound which trades off between fully unlearning from erased data vs. not entirely forgetting the posterior belief given the full data (i.e., including the remaining data); the latter prevents catastrophic unlearning that can render the model useless. In model training with VI, only an approximate (instead of exact) posterior belief given the full data can be obtained, which makes unlearning even more challenging. We propose two novel tricks to tackle this challenge. We empirically demonstrate our unlearning methods on Bayesian models such as sparse Gaussian process and logistic regression using synthetic and real-world datasets.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.12883

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 33

  1. [1][1] C. M. Bishop. Pattern Recognition and Machine Learning . Springer, 2006. -
  2. [2][2] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. J. American Statistical Association , 112(518):859-877, 2017. -
  3. [3][3] L. Bourtoule, V. Chandrasekaran, C. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot. Machine unlearning. arXiv:1912.03817, 2019. - · arXiv
  4. [4][4] Y. Cao and J. Yang. Towards making systems forget with machine unlearning. In Proc. IEEE S&P , pages 463-480, 2015. -
  5. [5][5] J. Chen, N. Cao, B. K. H. Low, R. Ouyang, C. K.-Y. Tan, and P. Jaillet. Parallel Gaussian process regression with low-rank covariance matrix approximations. In Proc. UAI , pages 152-161, 2013. -
  6. [6][6] J. Chen, B. K. H. Low, P. Jaillet, and Y. Yao. Gaussian process decentralized data fusion and active sensing for spatiotemporal traffic modeling and prediction in mobility-on-demand systems. IEEE Trans. Autom. Sci. Eng. , 12:901-921, 2015. -
  7. [7][7] J. Chen, B. K. H. Low, and C. K.-Y. Tan. Gaussian process-based decentralized data fusion and active sensing for mobility-on-demand system. In Proc. RSS , 2013. -
  8. [8][8] J. Chen, B. K. H. Low, C. K.-Y. Tan, A. Oran, P. Jaillet, J. M. Dolan, and G. S. Sukhatme. Decentralized data fusion and active sensing with mobile sensors for modeling and predicting spatiotemporal traffic phenomena. In Proc. UAI , pages 163-173, 2012. -
  9. [9][9] M. Du, Z. Chen, C. Liu, R. Oak, and D. Song. Lifelong anomaly detection through unlearning. In Proc. CCS , pages 1283-1297, 2019. -
  10. [11][11] Y. Gal and M. van der Wilk. Variational inference in sparse Gaussian process regression and latent variable models-a gentle tutorial. arXiv preprint arXiv , 1402, 2014. -
  11. [12][12] A. Ginart, M. Guan, G. Valiant, and J. Y. Zou. Making AI forget you: Data deletion in machine learning. In Proc. NeurIPS , pages 3513-3526, 2019. -
  12. [13][13] A. Golatkar, A. Achille, and S. Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep neural networks. In Proc. CVPR , 2020. -
  13. [14][14] C. Guo, T. Goldstein, A. Hannun, and L. van der Maaten. Certified data removal from machine learning models. arXiv:1911.03030, 2019. - · arXiv
  14. [15][15] J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. In Proc. UAI , pages 282-290, 2013. -
  15. [16][16] Q. M. Hoang, T. N. Hoang, and B. K. H. Low. A generalized stochastic variational Bayesian hyperparameter learning framework for sparse spectrum Gaussian process regression. In Proc. AAAI , pages 2007-2014, 2017. -
  16. [17][17] Q. M. Hoang, T. N. Hoang, B. K. H. Low, and C. Kingsford. Collective model fusion for multiple black-box experts. In Proc. ICML , pages 2742-2750, 2019. -
  17. [18][18] T. N. Hoang, Q. M. Hoang, and B. K. H. Low. A unifying framework of anytime sparse Gaussian process regression models with stochastic variational inference for big data. In Proc. ICML , pages 569-578, 2015. -
  18. [19][19] T. N. Hoang, Q. M. Hoang, and B. K. H. Low. A distributed variational inference framework for unifying parallel sparse Gaussian process regression models. In Proc. ICML , pages 382-391, 2016. -
  19. [20][20] T. N. Hoang, Q. M. Hoang, B. K. H. Low, and J. P. How. Collective online learning of Gaussian processes in massive multi-agent systems. In Proc. AAAI , pages 7850-7857, 2019. -
  20. [21][21] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling. Improved variational inference with inverse autoregressive flow. In Proc. NeurIPS , pages 4743-4751, 2016. -
  21. [22][22] B. K. H. Low, N. Xu, J. Chen, K. K. Lim, and E. B. Özgül. Generalized online sparse Gaussian processes with application to persistent mobile robot localization. In Proc. ECML/PKDD Nectar Track , pages 499-503, 2014. -
  22. [23][23] B. K. H. Low, J. Yu, J. Chen, and P. Jaillet. Parallel Gaussian process regression for big data: Low-rank representation meets Markov approximation. In Proc. AAAI , pages 2821-2827, 2015. -
  23. [24][24] A. Mantelero. The EU proposal for a general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review , 29(3):229-235, 2013. -
  24. [25][25] R. Ouyang and B. K. H. Low. Gaussian process decentralized data fusion meets transfer learning in large-scale distributed cooperative perception. In Proc. AAAI , pages 3876-3883, 2018. -
  25. [26][26] G. Papamakarios, T. Pavlakou, and I. Murray. Masked autoregressive flow for density estimation. In Proc. NeurIPS , pages 2338-2347, 2017. -
  26. [27][27] J. Quiñonero-Candela and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process regression. JMLR , 6:1939-1959, 2005. -
  27. [28][28] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning . MIT Press, 2006. -
  28. [29][29] D. J. Rezende and S. Mohamed. Variational inference with normalizing flows. In Proc. ICML , pages 1530-1538, 2015. -
  29. [30][30] S. Schelter. 'Amnesia' - Towards machine learning models that can forget user data very fast. In Proc. International Workshop on Applied AI for Database Systems and Applications , 2019. -
  30. [31][31] T. Teng, J. Chen, Y. Zhang, and B. K. H. Low. Scalable variational Bayesian kernel selection for sparse Gaussian process regression. In Proc. AAAI , pages 5997-6004, 2020. -
  31. [32][32] N. Xu, B. K. H. Low, J. Chen, K. K. Lim, and E. B. Özgül. GP-Localize: Persistent mobile robot localization using online sparse Gaussian process observation model. In Proc. AAAI , pages 2585-2592, 2014. -
  32. [33][33] H. Yu, Y. Chen, Z. Dai, K. H. Low, and P. Jaillet. Implicit posterior variational inference for deep Gaussian processes. In Proc. NeurIPS , pages 14475-14486, 2019. -
  33. [34][34] H. Yu, T. N. Hoang, B. K. H. Low, and P. Jaillet. Stochastic variational inference for Bayesian sparse Gaussian process regression. In Proc. IJCNN , 2019.

Cited by

Sorted by citations

Authors
Nguyen, Q. P., Low, B. K. H., & Jaillet, P.
Affiliations
  • National University of Singapore (SG)
  • Massachusetts Institute of Technology (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
33
Pages
16025-16036
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.12883
arXiv
2010.12883
OpenAlex
W3105631581
Cited by
62
References
0
Citations by year
2025: 8 · 2024: 24 · 2023: 16 · 2022: 5 · 2021: 9
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
unlearning, data deletion, variational inference, Kullback-Leibler divergence, evidence lower bound, posterior beliefs, sparse Gaussian processes, logistic regression, privacy-preserving machine learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review