A Methodology for Formalizing Model-Inversion Attacks
The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Abstract
Confidentiality of training data induced by releasing machine-learning models, and has recently received increasing attention. Motivated by existing MI attacks and other previous attacks that turn out to be MI "in disguise," this paper initiates a formal study of MI attacks by presenting a game-based methodology. Our methodology uncovers a number of subtle issues, and devising a rigorous game-based definition, analogous to those in cryptography, is an interesting avenue for future work. We describe methodologies for two types of attacks. The first is for black-box attacks, which consider an adversary who infers sensitive values with only oracle access to a model. The second methodology targets the white-box scenario where an adversary has some additional knowledge about the structure of a model. For the restricted class of Boolean models and black-box attacks, we characterize model invertibility using the concept of influence from Boolean analysis in the noiseless case, and connect model invertibility with stable influence in the noisy case. Interestingly, we also discovered an intriguing phenomenon, which we call "invertibility interference," where a highly invertible model quickly becomes highly non-invertible by adding little noise. For the white-box case, we consider a common phenomenon in machine-learning models where the model is a sequential composition of several sub-models. We show, quantitatively, that even very restricted communication between layers could leak a significant amount of information. Perhaps more importantly, our study also unveils unexpected computational power of these restricted communication channels, which, to the best of our knowledge, were not previously known.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Calibrating Noise to Sensitivity in Private Data Analysis
Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006) · Lecture Notes in Computer Science · doi:10.1007/11681878_14 · Cited by 7,272
Cited by
A Survey of Machine and Deep Learning Methods for Internet of Things (IoT) Security
Al-Garadi, M. A., Mohamed, A., Al-Ali, A. K., Du, X., Ali, I., & Guizani, M. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2988293 · Cited by 1,392
“The last possible attack is inversion, which exploits the application program interfaces presented to the users by the current ML platform to collect roughly the necessary information about the pre-trained ML models [280], [285].”
MethodTrojaning Attack on Neural Networks
Liu, Y., Ma, S., Aafer, Y., Lee, W.-C., Zhai, J., Wang, W., & Zhang, X. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23291 · Cited by 1,270
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
Yeom, S., Giacomelli, I., Fredrikson, M., & Jha, S. (2018) · 2018 IEEE 31st Computer Security Foundations Symposium (CSF) · doi:10.1109/csf.2018.00027 · Cited by 1,001
Self-citationThe Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks
Zhang, Y., Jia, R., Pei, H., Wang, W., Li, B., & Song, D. (2019) · arXiv · doi:10.48550/arxiv.1911.07135 · Cited by 465
The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks
Zhang, Y., Jia, R., Pei, H., Wang, W., Li, B., & Song, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00033 · Cited by 449
Differential Privacy for Deep and Federated Learning: A Survey
Ouadrhiri, A. E., & Abdelhadi, A. (2022) · IEEE Access · doi:10.1109/access.2022.3151670 · Cited by 445
“Since, a malicious user (or the server) could reconstruct the users’ local dataset using only the local gradient sent to the server [10], [122], [123].”
BackgroundA Survey on Security Threats and Defensive Techniques of Machine Learning: A Data Driven View
Liu, Q., Li, P., Zhao, W., Cai, W., Yu, S., & Leung, V. C. M. (2018) · IEEE Access · doi:10.1109/access.2018.2805680 · Cited by 421
Advances in Adversarial Attacks and Defenses in Computer Vision: A Survey
Akhtar, N., Mian, A., Kardan, N., & Shah, M. (2021) · IEEE Access · doi:10.1109/access.2021.3127960 · Cited by 302
A Survey of Privacy Attacks in Machine Learning
Rigaki, M., & Garcia, S. (2024) · ACM Computing Surveys · doi:10.1145/3624010 · Cited by 285
“The papers are discussed in two sections: attacks on centralized learning and attacks on distributed learning.”
BackgroundPrivacy in Deep Learning: A Survey
Mireshghallah, F., Taram, M., Vepakomma, P., Singh, A., Raskar, R., & Esmaeilzadeh, H. (2020) · arXiv · doi:10.48550/arxiv.2004.12254 · Cited by 106
“Model inversion and attribute inference attacks are against attribute privacy, where an adversary tries to infer sensitive attributes of given data instances from a released model and the instance’s non-sensitive attributes [47].”
BackgroundKhosravy, M., Nakamura, K., Hirose, Y., Nitta, N., & Babaguchi, N. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3140687 · Cited by 94
Survey: Leakage and Privacy at Inference Time
Jegorova, M., Kaul, C., Mayor, C., O'Neil, A. Q., Weir, A., Murray-Smith, R., & Tsaftaris, S. A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229593 · Cited by 77
Understanding Unintended Memorization in Language Models Under Federated Learning
Thakkar, O. D., Ramaswamy, S., Mathews, R., & Beaufays, F. (2021) · Proceedings of the Third Workshop on Privacy in Natural Language Processing · doi:10.18653/v1/2021.privatenlp-1.1 · Cited by 39
Training Production Language Models without Memorizing User Data
Ramaswamy, S., Thakkar, O., Mathews, R., Andrew, G., McMahan, H. B., & Beaufays, F. (2020) · arXiv · doi:10.48550/arxiv.2009.10031 · Cited by 37
A comprehensive survey and taxonomy on privacy-preserving deep learning
Tran, A.-T., Luong, T.-D., & Huynh, V.-N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127345 · Cited by 21
Understanding Unintended Memorization in Federated Learning
Thakkar, O., Ramaswamy, S., Mathews, R., & Beaufays, F. (2020) · arXiv · doi:10.48550/arxiv.2006.07490
Metadata
- Authors
- Wu, X., Fredrikson, M., Jha, S., & Naughton, J. F.
- Venue
- 2016 IEEE 29th Computer Security Foundations Symposium (CSF)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 355-370
- ISSN
- 1063-6900 (ISSN-L)
- OpenAlex
- W2512472178
- Cited by
- 147
- References
- 1
- FWCI
- 4.17
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.