The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Open accessAbstract
Machine-learning (ML) algorithms are increasingly utilized in privacy-sensitive applications such as predicting lifestyle choices, making medical diagnoses, and facial recognition. In a model inversion attack, recently introduced in a case study of linear classifiers in personalized medicine by Fredrikson et al., adversarial access to an ML model is abused to learn sensitive genomic information about individuals. Whether model inversion attacks apply to settings outside theirs, however, is unknown. We develop a new class of model inversion attack that exploits confidence values revealed along with predictions. Our new attacks are applicable in a variety of settings, and we explore two in depth: decision trees for lifestyle surveys as used on machine-learning-as-a-service systems and neural networks for facial recognition. In both cases confidence values are revealed to those with the ability to make prediction queries to models. We experimentally show attacks that are able to estimate whether a respondent in a lifestyle survey admitted to cheating on their significant other and, in the other context, show how to recover recognizable images of people's faces given only their name and access to the ML model. We also initiate experimental exploration of natural countermeasures, investigating a privacy-aware decision tree training algorithm that is a simple variant of CART learning, as well as revealing only rounded confidence values. The lesson that emerges is that one can avoid these kinds of MI attacks with negligible degradation to utility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Privacy and security risks can be significantly reduced by restricting the training data to only the device side, and thus avoiding the privacy issues as in [195], incurred by uploading training data to the cloud.”
From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [195]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Such model inversion or reconstruction attacks can cause catastrophic data leakage: it has been shown that images can be reconstructed with impressive accuracy and detail 55 , allowing visualization of the original training data.”
From Secure, privacy-preserving and federated machine learning in medical imaging (2020) · cites this paper as [55]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this attack, attackers aim to reconstruct the training data using their ability to constantly query the model [84];”
From Generative AI in medical practice: in-depth exploration of privacy and security challenges (2024) · cites this paper as [84]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, in [195], personal information can be purposely obtained from training data by making use of the privacy leaking of a trained classifier.”
From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [195]
The cited source says what the claim attributes to it.
In the citing paper
“In addition, they use a model inversion attack to evaluate verifiability [6].”
From Machine Unlearning: A Survey (2024) · cites this paper as [6]
In the citing paper
“Due to the sensitivity of Batch Normalization (BN) layers58 when dealing with different clients in a non-IID setting, we found the best model performance to occur when keeping the pre-trained ResNet34 with spatial attention47 parameters fixed during FL training (i.e. using a learning rate of zero for those layers).”
From Federated learning for predicting clinical outcomes in patients with COVID-19 (2021) · cites this paper as [47]
In the citing paper
“For FL, there remains the potential risk of model ‘inversion’47 or even the reconstruction of training images from the model gradients themselves48.”
From Federated learning for predicting clinical outcomes in patients with COVID-19 (2021) · cites this paper as [47]
In the citing paper
“The advanced privacy attacks such as model inversion [19] and model extraction attack [20] have shown that the model parameters stored in the blockchain are not enough to protect the privacy of raw clinical data.”
From A Blockchain‐Based Federated Learning Method for Smart Healthcare (2021) · cites this paper as [19]
In the citing paper
“Under the CDP model, the aggregator releases the aggregated value with an expected additive error of at most Θ(1 /ϵ ) to ensure ϵ -DP ( e.g. , using the Laplace mechanism [96]).”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [96]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
Dwork, C. (2006) · Lecture notes in computer science · doi:10.1007/11787006_1 · Cited by 6,125
Bradski, G. R. (2000) · Medical Entomology and Zoology · Cited by 5,994
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412
Robust De-anonymization of Large Sparse Datasets
Narayanan, A., & Shmatikov, V. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.33 · Cited by 2,372
(2011) · IEEE Software · doi:10.1109/ms.2011.67 · Cited by 2,038
Estimation of the Warfarin Dose with Clinical and Pharmacogenetic Data
The International Warfarin Pharmacogenetics Consortium (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0809329 · Cited by 1,573
Homer, N., Szelinger, S., Redman, M., Duggan, D., Tembe, W., Muehling, J., Pearson, J. V., Stephan, D. A., Nelson, S. F., & Craig, D. W. (2008) · PLoS Genetics · doi:10.1371/journal.pgen.1000167 · Cited by 1,221
Revealing information while preserving privacy
Dinur, I., & Kobbi Nissim, B. (2003) · ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS) · doi:10.1145/773153.773173 · Cited by 993
Can machine learning be secure?
Barreno, M., Nelson, B. A., Sears, R., Joseph, A. D., & Tygar, J. D. (2006) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/1128817.1128824 · Cited by 852
Research works printed but not linked: 5
Cited by
Sorted by citations
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00482 · Cited by 7,781
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
“For classic (non-federated) models of computation, understanding a model’s susceptibility to attacks is an active and challenging research area [167, 357, 91, 293].”
BackgroundDeep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
Practical Secure Aggregation for Privacy-Preserving Machine Learning
Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., & Seth, K. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3133982 · Cited by 3,956
Metadata
- Authors
- Fredrikson, M., Jha, S., & Ristenpart, T.
- Affiliations
- Carnegie Mellon University (US)
- University of Wisconsin–Madison (US)
- Cornell Tech (US)
- Venue
- Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2015-10-12
- Pages
- 1322-1333
- ISSN
- 1543-7221 (ISSN-L)
- OpenAlex
- W2051267297
- Semantic Scholar
- d1b9a3b11e6c9571a1553556f82b605b2b4baec3
- Cited by
- 2,961
- References
- 27
- Citations by year
- 2026: 256 · 2025: 389 · 2024: 427 · 2023: 459 · 2022: 374 · 2021: 423 · 2020: 315 · 2019: 190 · 2018: 85 · 2017: 31 · 2016: 11
- FWCI
- 23.55
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGSAC Conference on Computer and Communications Security (CCS) (publishedVersion)
- Funders
- Defense Advanced Research Projects Agency (FA8650-10-C-7088)
- National Science Foundation (CNS-1546033 CNS-1330308 CNS-1065134)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Keywords
- model inversion attacks, confidence scores, privacy attacks on machine learning, decision tree, artificial neural networks, face recognition, lifestyle survey, machine learning as a service, CART, genetic privacy, countermeasures
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
