The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Open accessAbstract
Machine-learning (ML) algorithms are increasingly utilized in privacy-sensitive applications such as predicting lifestyle choices, making medical diagnoses, and facial recognition. In a model inversion attack, recently introduced in a case study of linear classifiers in personalized medicine by Fredrikson et al., adversarial access to an ML model is abused to learn sensitive genomic information about individuals. Whether model inversion attacks apply to settings outside theirs, however, is unknown. We develop a new class of model inversion attack that exploits confidence values revealed along with predictions. Our new attacks are applicable in a variety of settings, and we explore two in depth: decision trees for lifestyle surveys as used on machine-learning-as-a-service systems and neural networks for facial recognition. In both cases confidence values are revealed to those with the ability to make prediction queries to models. We experimentally show attacks that are able to estimate whether a respondent in a lifestyle survey admitted to cheating on their significant other and, in the other context, show how to recover recognizable images of people's faces given only their name and access to the ML model. We also initiate experimental exploration of natural countermeasures, investigating a privacy-aware decision tree training algorithm that is a simple variant of CART learning, as well as revealing only rounded confidence values. The lesson that emerges is that one can avoid these kinds of MI attacks with negligible degradation to utility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Privacy and security risks can be significantly reduced by restricting the training data to only the device side, and thus avoiding the privacy issues as in [195], incurred by uploading training data to the cloud.”
From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [195]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Such model inversion or reconstruction attacks can cause catastrophic data leakage: it has been shown that images can be reconstructed with impressive accuracy and detail 55 , allowing visualization of the original training data.”
From Secure, privacy-preserving and federated machine learning in medical imaging (2020) · cites this paper as [55]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this attack, attackers aim to reconstruct the training data using their ability to constantly query the model [84];”
From Generative AI in medical practice: in-depth exploration of privacy and security challenges (2024) · cites this paper as [84]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, in [195], personal information can be purposely obtained from training data by making use of the privacy leaking of a trained classifier.”
From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [195]
The cited source says what the claim attributes to it.
In the citing paper
“In addition, they use a model inversion attack to evaluate verifiability [6].”
From Machine Unlearning: A Survey (2024) · cites this paper as [6]
In the citing paper
“Due to the sensitivity of Batch Normalization (BN) layers58 when dealing with different clients in a non-IID setting, we found the best model performance to occur when keeping the pre-trained ResNet34 with spatial attention47 parameters fixed during FL training (i.e. using a learning rate of zero for those layers).”
From Federated learning for predicting clinical outcomes in patients with COVID-19 (2021) · cites this paper as [47]
In the citing paper
“For FL, there remains the potential risk of model ‘inversion’47 or even the reconstruction of training images from the model gradients themselves48.”
From Federated learning for predicting clinical outcomes in patients with COVID-19 (2021) · cites this paper as [47]
In the citing paper
“The advanced privacy attacks such as model inversion [19] and model extraction attack [20] have shown that the model parameters stored in the blockchain are not enough to protect the privacy of raw clinical data.”
From A Blockchain‐Based Federated Learning Method for Smart Healthcare (2021) · cites this paper as [19]
In the citing paper
“Under the CDP model, the aggregator releases the aggregated value with an expected additive error of at most Θ(1 /ϵ ) to ensure ϵ -DP ( e.g. , using the Laplace mechanism [96]).”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [96]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
Dwork, C. (2006) · Lecture notes in computer science · doi:10.1007/11787006_1 · Cited by 6,125
Bradski, G. R. (2000) · Medical Entomology and Zoology · Cited by 5,994
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412
Robust De-anonymization of Large Sparse Datasets
Narayanan, A., & Shmatikov, V. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.33 · Cited by 2,372
(2011) · IEEE Software · doi:10.1109/ms.2011.67 · Cited by 2,038
Estimation of the Warfarin Dose with Clinical and Pharmacogenetic Data
The International Warfarin Pharmacogenetics Consortium (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0809329 · Cited by 1,573
Homer, N., Szelinger, S., Redman, M., Duggan, D., Tembe, W., Muehling, J., Pearson, J. V., Stephan, D. A., Nelson, S. F., & Craig, D. W. (2008) · PLoS Genetics · doi:10.1371/journal.pgen.1000167 · Cited by 1,221
Revealing information while preserving privacy
Dinur, I., & Kobbi Nissim, B. (2003) · ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS) · doi:10.1145/773153.773173 · Cited by 993
Can machine learning be secure?
Barreno, M., Nelson, B. A., Sears, R., Joseph, A. D., & Tygar, J. D. (2006) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/1128817.1128824 · Cited by 852
Flagged first, then by citations
Research works printed but not linked: 5
Cited by
Sorted by citations
Items 101–126 of 126
Reliable customer analysis using federated learning and exploring deep-attention edge intelligence
Ahmed, U., Srivastava, G., & Lin, J. C.-W. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2021.08.028 · Cited by 52
Athena: Probabilistic Verification of Machine Unlearning
Sommer, D. M., Song, L., Wagh, S., & Mittal, P. (2022) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2022-0072 · Cited by 51
PUMA: Performance Unchanged Model Augmentation for Training Data Removal
Wu, G., Hashemi, M., & Srinivasa, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20846 · Cited by 61
Privacy-aware and Resource-saving Collaborative Learning for Healthcare in Cloud Computing
Hao, M., Li, H., Xu, G., Liu, Z., & Chen, Z. (2020) · ICC 2020 - 2020 IEEE International Conference on Communications (ICC) · doi:10.1109/icc40277.2020.9148979 · Cited by 52
Wu, B., Zhao, S., Sun, G., Zhang, X., Su, Z., Zeng, C., & Liu, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00220 · Cited by 44
Ho, T.-T., Tran, K.-D., & Huang, Y. (2022) · Sensors · doi:10.3390/s22103728 · Cited by 44
Data Poisoning in Sequential and Parallel Federated Learning
Nuding, F., & Mayer, R. (2022) · Proceedings of the 2022 ACM on International Workshop on Security and Privacy Analytics · doi:10.1145/3510548.3519372 · Cited by 42
Privacy-Preserving Tensor Factorization for Collaborative Health Data Analysis
Ma, J., Zhang, Q., Lou, J., Ho, J. C., Xiong, L., & Jiang, X. (2019) · Proceedings of the 28th ACM International Conference on Information and Knowledge Management · doi:10.1145/3357384.3357878 · Cited by 51
Understanding Unintended Memorization in Language Models Under Federated Learning
Thakkar, O. D., Ramaswamy, S., Mathews, R., & Beaufays, F. (2021) · Proceedings of the Third Workshop on Privacy in Natural Language Processing · doi:10.18653/v1/2021.privatenlp-1.1 · Cited by 39
Practical Attribute Reconstruction Attack Against Federated Learning
Chen, C., Lyu, L., Yu, H., & Chen, G. (2024) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2022.3159236 · Cited by 30
Federated Learning-Based Cell-Free Massive MIMO System for Privacy-Preserving
Zhang, J., Zhang, J., Ng, D. W. K., & Ai, B. (2023) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3225812 · Cited by 30
Preventing generation of verbatim memorization in language models gives a false sense of privacy
Ippolito, D., Tramer, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette Choo, C., & Carlini, N. (2023) · Proceedings of the 16th International Natural Language Generation Conference · doi:10.18653/v1/2023.inlg-main.3 · Cited by 28
“For example, Fredrikson et al. (2015) consider “model inversion”, where an image is successfully recovered from the model if it is identi-fiable to a human worker.”
BackgroundCommunication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications
Park, J., Samarakoon, S., Elgabli, A., Kim, J., Bennis, M., Kim, S.-L., & Debbah, M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3055679 · Cited by 25
Adaptive Differential Privacy Mechanism Based on Entropy Theory for Preserving Deep Neural Networks
Zhang, X., Yang, F., Guo, Y., Yu, H., Wang, Z., & Zhang, Q. (2023) · Mathematics · doi:10.3390/math11020330 · Cited by 24
A comprehensive survey and taxonomy on privacy-preserving deep learning
Tran, A.-T., Luong, T.-D., & Huynh, V.-N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127345 · Cited by 21
FREED: An Efficient Privacy-Preserving Solution for Person Re-IDentification
Zhao, B., Li, Y., Liu, X., Pang, H. H., & Deng, R. H. (2022) · 2022 IEEE Conference on Dependable and Secure Computing (DSC) · doi:10.1109/dsc54232.2022.9888863 · Cited by 23
Harnessing wireless channels for scalable and privacy-preserving federated learning
Elgabli, A., Park, J., Issaid, C. B., & Bennis, M. (2021) · IEEE Transactions on Communications · doi:10.1109/tcomm.2021.3078783 · Cited by 14
Differentially Private Knowledge Distillation for Mobile Analytics
Lyu, L., & Chen, C.-H. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401259 · Cited by 14
Making Machine Learning Forget
Shintre, S., Roundy, K. A., & Dhaliwal, J. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-21752-5_6 · Cited by 13
Through the Looking Glass: Learning to Attribute Synthetic Text Generated by Language Models
Munir, S., Batool, B., Shafiq, Z., Srinivasan, P., & Zaffar, F. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.155 · Cited by 11
A Lifecycle-Oriented Survey of Emerging Threats and Vulnerabilities in Large Language Models
de Maio, C., Di Gisi, M., Fenza, G., Gallo, M., & Loia, V. (2025) · IEEE Access · doi:10.1109/access.2025.3619764 · Cited by 10
Interpretable and Differentially Private Predictions
Harder, F., Bauer, M., & Park, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5827 · Cited by 6
Graves, L., Nagisetty, V., & Ganesh, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17371 · Cited by 5
Person identity shift for privacy-preserving person re-identification
Dou, S., Jiang, X., Zhao, Q., Wang, Y., Li, D., & Zhao, C. (2026) · Science China Information Sciences · doi:10.1007/s11432-023-4583-x · Cited by 2
Islam, N., & Zulkernine, M. (2025) · arXiv · doi:10.48550/arxiv.2503.01089 · Cited by 9
“Nonetheless, GCN-based models utilizing FL often neglect the underlying topological structure of the traffic networks, which can lead to potential privacy breach.”
Forgetting Private Textual Sequences in Language Models Via Leave-One-Out Ensemble
Liu, Z., & Kalinli, O. (2024) · ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp48485.2024.10446299 · Cited by 1
Metadata
- Authors
- Fredrikson, M., Jha, S., & Ristenpart, T.
- Affiliations
- Carnegie Mellon University (US)
- University of Wisconsin–Madison (US)
- Cornell Tech (US)
- Venue
- Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2015-10-12
- Pages
- 1322-1333
- ISSN
- 1543-7221 (ISSN-L)
- OpenAlex
- W2051267297
- Semantic Scholar
- d1b9a3b11e6c9571a1553556f82b605b2b4baec3
- Cited by
- 2,961
- References
- 27
- Citations by year
- 2026: 256 · 2025: 389 · 2024: 427 · 2023: 459 · 2022: 374 · 2021: 423 · 2020: 315 · 2019: 190 · 2018: 85 · 2017: 31 · 2016: 11
- FWCI
- 23.55
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGSAC Conference on Computer and Communications Security (CCS) (publishedVersion)
- Funders
- Defense Advanced Research Projects Agency (FA8650-10-C-7088)
- National Science Foundation (CNS-1546033 CNS-1330308 CNS-1065134)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Keywords
- model inversion attacks, confidence scores, privacy attacks on machine learning, decision tree, artificial neural networks, face recognition, lifestyle survey, machine learning as a service, CART, genetic privacy, countermeasures
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
