Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Open accessAbstract
Machine-learning (ML) algorithms are increasingly utilized in privacy-sensitive applications such as predicting lifestyle choices, making medical diagnoses, and facial recognition. In a model inversion attack, recently introduced in a case study of linear classifiers in personalized medicine by Fredrikson et al., adversarial access to an ML model is abused to learn sensitive genomic information about individuals. Whether model inversion attacks apply to settings outside theirs, however, is unknown. We develop a new class of model inversion attack that exploits confidence values revealed along with predictions. Our new attacks are applicable in a variety of settings, and we explore two in depth: decision trees for lifestyle surveys as used on machine-learning-as-a-service systems and neural networks for facial recognition. In both cases confidence values are revealed to those with the ability to make prediction queries to models. We experimentally show attacks that are able to estimate whether a respondent in a lifestyle survey admitted to cheating on their significant other and, in the other context, show how to recover recognizable images of people's faces given only their name and access to the ML model. We also initiate experimental exploration of natural countermeasures, investigating a privacy-aware decision tree training algorithm that is a simple variant of CART learning, as well as revealing only rounded confidence values. The lesson that emerges is that one can avoid these kinds of MI attacks with negligible degradation to utility.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Privacy and security risks can be significantly reduced by restricting training data to device side, avoiding privacy issues from uploading training data to cloud.
“Privacy and security risks can be significantly reduced by restricting the training data to only the device side, and thus avoiding the privacy issues as in [195], incurred by uploading training data to the cloud.”
From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [195]
The Introduction describes cloud ML services to which customers upload datasets and warns that providers could expose sensitive training data through insider attacks or system compromises. It does not evaluate keeping training data on devices or establish that doing so significantly reduces privacy and security risks; its ML APIs section also describes confidentiality risks when attackers cannot access the training data directly. Page numbers are unavailable in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measuredModel inversion or reconstruction attacks can cause catastrophic data leakage: images can be reconstructed with impressive accuracy and detail, allowing visualization of the original training data.
“Such model inversion or reconstruction attacks can cause catastrophic data leakage: it has been shown that images can be reconstructed with impressive accuracy and detail 55 , allowing visualization of the original training data.”
From Secure, privacy-preserving and federated machine learning in medical imaging (2020) · cites this paper as [55]
The Summary and Conclusion report extracting recognizable images of training subjects from facial-recognition models. Section 5.2.2 reports sharp, recognizable features for the best-performing model, but says other reconstructions were faint or blurry; the paper does not establish accurate, detailed recovery of the original training images themselves. (LLM, conf=0.86)
Experimental — accuracy not yet measuredIn model inversion attacks, attackers aim to reconstruct the training data using their ability to constantly query the model.
“In this attack, attackers aim to reconstruct the training data using their ability to constantly query the model [84];”
From Generative AI in medical practice: in-depth exploration of privacy and security challenges (2024) · cites this paper as [84]
The Summary and Section 5.2 describe model inversion attacks that use model access to infer sensitive features or reconstruct recognizable faces of people represented in the training set. The ML APIs section confirms that attackers can make prediction queries. But the paper does not establish that these attacks reconstruct the training data as a whole or recover exact training records; its face reconstructions are approximations. (LLM, conf=0.84)
Experimental — accuracy not yet measuredPersonal information can be purposely obtained from training data by exploiting privacy leaking of a trained classifier.
“For example, in [195], personal information can be purposely obtained from training data by making use of the privacy leaking of a trained classifier.”
From Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2020) · cites this paper as [195]
Baumhauer et al. [56] use a model inversion attack to evaluate verifiability.
“In addition, they use a model inversion attack to evaluate verifiability [6].”
From Machine Unlearning: A Survey (2024) · cites this paper as [6]
In their federated learning experiments, the best model performance occurred when the pre-trained ResNet34 with spatial attention parameters were kept fixed (learning rate zero) due to Batch Normalization sensitivity in non-IID settings.
“Due to the sensitivity of Batch Normalization (BN) layers58 when dealing with different clients in a non-IID setting, we found the best model performance to occur when keeping the pre-trained ResNet34 with spatial attention47 parameters fixed during FL training (i.e. using a learning rate of zero for those layers).”
From Federated learning for predicting clinical outcomes in patients with COVID-19 (2021) · cites this paper as [47]
Federated learning carries potential risks of model inversion or reconstruction of training images from model gradients.
“For FL, there remains the potential risk of model ‘inversion’47 or even the reconstruction of training images from the model gradients themselves48.”
From Federated learning for predicting clinical outcomes in patients with COVID-19 (2021) · cites this paper as [47]
Advanced privacy attacks such as model inversion and model extraction have shown that model parameters stored in the blockchain are not enough to protect the privacy of raw clinical data.
“The advanced privacy attacks such as model inversion [19] and model extraction attack [20] have shown that the model parameters stored in the blockchain are not enough to protect the privacy of raw clinical data.”
From A Blockchain‐Based Federated Learning Method for Smart Healthcare (2021) · cites this paper as [19]
Under the CDP model, the aggregator releases the aggregated value with an expected additive error of at most Θ(1/ε) to ensure ε-DP.
“Under the CDP model, the aggregator releases the aggregated value with an expected additive error of at most Θ(1 /ϵ ) to ensure ϵ -DP ( e.g. , using the Laplace mechanism [96]).”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [96]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · see citing works
Bradski, G. R. (2000) · Medical Entomology and Zoology · Cited by 5,994 · see citing works
Dwork, C. (2006) · Lecture notes in computer science · doi:10.1007/11787006_1 · Cited by 5,362 · see citing works
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412 · see citing works
Robust De-anonymization of Large Sparse Datasets
Narayanan, A., & Shmatikov, V. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.33 · Cited by 2,372 · see citing works
(2011) · IEEE Software · doi:10.1109/ms.2011.67 · Cited by 2,038 · see citing works
Estimation of the Warfarin Dose with Clinical and Pharmacogenetic Data
The International Warfarin Pharmacogenetics Consortium (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0809329 · Cited by 1,573 · see citing works
Homer, N., Szelinger, S., Redman, M., Duggan, D., Tembe, W., Muehling, J., Pearson, J. V., Stephan, D. A., Nelson, S. F., & Craig, D. W. (2008) · PLoS Genetics · doi:10.1371/journal.pgen.1000167 · Cited by 1,221 · see citing works
Revealing information while preserving privacy
Dinur, I., & Nissim, K. (2003) · ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (PODS) · doi:10.1145/773153.773173 · Cited by 1,003 · see citing works
Can machine learning be secure?
Barreno, M., Nelson, B. A., Sears, R., Joseph, A. D., & Tygar, J. D. (2006) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/1128817.1128824 · Cited by 852 · see citing works
Lowd, D., & Meek, C. (2005) · Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining · doi:10.1145/1081870.1081950 · Cited by 755 · see citing works
Smith, T. (2021) · The Encyclopedia of Research Methods in Criminology and Criminal Justice · doi:10.1002/9781119111931.ch11 · Cited by 687 · see citing works
Simple Demographics Often Identify People Uniquely
Sweeney, L. (2000) · Figshare · doi:10.1184/r1/6625769.v1 · Cited by 664 · see citing works
Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing.
Fredrikson, M., Lantz, E., Jha, S., Lin, S. M., Page, D., & Ristenpart, T. (2014) · PubMed · Cited by 658 · see citing works
Self-citationLearning hierarchical representations for face verification with convolutional deep belief networks
Huang, G. B., Lee, H., & Miller, E. G. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247968 · Cited by 408 · see citing works
Pylearn2: a machine learning research libraryPreprint
Goodfellow, I., Warde-Farley, D., Lamblin, P., Dumoulin, V., Mirza, M., Pascanu, R., Bergstra, J., Bastien, F., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1308.4214 · Cited by 245 · see citing works
The price of privacy and the limits of LP decoding
Dwork, C., McSherry, F., & Talwar, K. (2007) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/1250790.1250804 · Cited by 215 · see citing works
Learning your identity and disease from research papers
Wang, R., Li, Y. F., Wang, X., Tang, H., & Zhou, X. (2009) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/1653662.1653726 · Cited by 206 · see citing works
Genomic privacy and limits of individual detection in a pool
Sankararaman, S., Obozinski, G., Jordan, M. I., & Halperin, E. (2009) · Nature Genetics · doi:10.1038/ng.436 · Cited by 204 · see citing works
The disclosure of diagnosis codes can breach research participants' privacy
Loukides, G., Denny, J. C., & Malin, B. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.002725 · Cited by 141 · see citing works
Li, N., Qardaji, W. H., Su, D., Wu, Y., & Yang, W. (2013) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2508859.2516686 · Cited by 129 · see citing works
Kasiviswanathan, S. P., Rudelson, M., Smith, A., & Ullman, J. (2010) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/1806689.1806795 · Cited by 104 · see citing works
Personal privacy vs population privacy
Cormode, G. (2011) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2020408.2020598 · Cited by 94 · see citing works
Open source biometric recognition
Klontz, J. C., Klare, B. F., Klum, S., Jain, A. K., & Burge, M. J. (2013) · IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS) · doi:10.1109/btas.2013.6712754 · Cited by 94 · see citing works
Economic Analysis of the Digital Economy
Avi Goldfarb, B., Greenstein, S. M., & Tucker, C. E. (2015) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226206981.001.0001 · Cited by 84 · see citing works
Chi, C., Street, W. N., Robinson, J. G., & Crawford, M. A. (2012) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2012.07.011 · Cited by 58 · see citing works
Secure Authentication for Face Recognition
Dabbah, M. A., Woo, W. L., & Dlay, S. S. (2007) · doi:10.1109/ciisp.2007.369304 · Cited by 57 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00482 · Cited by 7,781 · see citing works
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772 · see citing works
“For classic (non-federated) models of computation, understanding a model’s susceptibility to attacks is an active and challenging research area [167, 357, 91, 293].”
BackgroundDeep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578 · see citing works
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · see citing works
Practical Secure Aggregation for Privacy-Preserving Machine Learning
Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., & Seth, K. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3133982 · Cited by 3,956 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Federated Learning in Mobile Edge Networks: A Comprehensive Survey
Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y.-C., Yang, Q., Niyato, D., & Miao, C. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2986024 · Cited by 2,565 · see citing works
Federated Learning With Differential Privacy: Algorithms and Performance Analysis
Wei, K., Li, J., Ding, M., Ma, C., Yang, H. H., Farokhi, F., Jin, S., Quek, T. Q. S., & Vincent Poor, H. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.2988575 · Cited by 2,400 · see citing works
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733 · see citing works
Exploiting Unintended Feature Leakage in Collaborative Learning
Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00029 · Cited by 1,734 · see citing works
Nasr, M., Shokri, R., & Houmansadr, A. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00065 · Cited by 1,760 · see citing works
Zhu, L., & Han, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-63076-8_2 · Cited by 1,906 · see citing works
Secure, privacy-preserving and federated machine learning in medical imaging
Kaissis, G. A., Makowski, M. R., Rückert, D., & Braren, R. F. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0186-1 · Cited by 1,501 · see citing works
Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
Wang, X., Han, Y., Leung, V. C. M., Niyato, D., Yan, X., & Chen, X. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2970550 · Cited by 1,498 · see citing works
“For example, in [195], personal information can be purposely obtained from training data by making use”
BackgroundA Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., & He, B. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3124599 · Cited by 1,407 · see citing works
A Survey of Machine and Deep Learning Methods for Internet of Things (IoT) Security
Al-Garadi, M. A., Mohamed, A., Al-Ali, A. K., Du, X., Ali, I., & Guizani, M. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2988293 · Cited by 1,392 · see citing works
Sattler, F., Muller, K.-R., & Samek, W. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3015958 · Cited by 1,355 · see citing works
Trojaning Attack on Neural Networks
Liu, Y., Ma, S., Aafer, Y., Lee, W.-C., Zhai, J., Wang, W., & Zhang, X. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23291 · Cited by 1,270 · see citing works
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
Yeom, S., Giacomelli, I., Fredrikson, M., & Jha, S. (2018) · 2018 IEEE 31st Computer Security Foundations Symposium (CSF) · doi:10.1109/csf.2018.00027 · Cited by 1,001 · see citing works
Self-citationBeyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning
Wang, Z., Song, M., Zhang, Z., Song, Y., Wang, Q., & Qi, H. (2019) · IEEE INFOCOM 2019 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2019.8737416 · Cited by 881 · see citing works
Federated learning for predicting clinical outcomes in patients with COVID-19
Dayan, I., Roth, H. R., Zhong, A., Harouni, A., Gentili, A., Abidin, A. Z., Liu, A., Costa, A. B., Wood, B. J., Tsai, C.-S., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01506-3 · Cited by 842 · see citing works
“For FL, there remains the potential risk of model ‘inversion’ 47 or even the reconstruction of training images from the model gradients themselves 48 .”
BackgroundManipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning
Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., & Li, B. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00057 · Cited by 861 · see citing works
A Survey on Federated Learning for Resource-Constrained IoT Devices
Imteaj, A., Thakker, U., Wang, S., Li, J., & Amini, M. H. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3095077 · Cited by 774 · see citing works
Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2021) · 2021 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40001.2021.00019 · Cited by 810 · see citing works
Challenges in deploying machine learning: A survey of case studies
Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2023) · ACM Computing Surveys · doi:10.1145/3533378 · Cited by 638 · see citing works
Efficient and Privacy-Enhanced Federated Learning for Industrial Artificial Intelligence
Hao, M., Li, H., Luo, X., Xu, G., Yang, H., & Liu, S. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2945367 · Cited by 569 · see citing works
Privacy and Robustness in Federated Learning: Attacks and Defenses
Lyu, L., Yu, H., Ma, X., Chen, C., Sun, L., Zhao, J., Yang, Q., & Yu, P. S. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3216981 · Cited by 548 · see citing works
End-to-end privacy preserving deep learning on multi-institutional medical imaging
Kaissis, G., Ziller, A., Passerat-Palmbach, J., Ryffel, T., Usynin, D., Trask, A., Lima, I., Mancuso, J., Jungmann, F., Steinborn, M.-M., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00337-8 · Cited by 513 · see citing works
Machine Learning Models that Remember Too Much
Song, C., Ristenpart, T., & Shmatikov, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134077 · Cited by 526 · see citing works
Self-citationGanju, K., Wang, Q., Yang, W., Gunter, C. A., & Borisov, N. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243834 · Cited by 475 · see citing works
Regulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466 · see citing works
“Third, we highlight key data protection risks under the GDPR, with a particular focus on model inversion [27-29].”
BackgroundPrivacy-Preserving Machine Learning: Threats and Solutions
Al-Rubaie, M., & Chang, J. M. (2019) · IEEE Security & Privacy · doi:10.1109/msec.2018.2888775 · Cited by 444 · see citing works
Lyu, L., Yu, H., Zhao, J., & Yang, Q. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-63076-8_1 · Cited by 444 · see citing works
Machine Learning with Membership Privacy using Adversarial Regularization
Nasr, M., Shokri, R., & Houmansadr, A. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243855 · Cited by 458 · see citing works
LOGAN: Membership Inference Attacks Against Generative Models
Hayes, J., Melis, L., Danezis, G., & De Cristofaro, E. (2019) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2019-0008 · Cited by 433 · see citing works
Privacy preservation in federated learning: An insightful survey from the GDPR perspective
Truong, N., Sun, K., Wang, S., Guitton, F., & Guo, Y. (2021) · Computers & Security · doi:10.1016/j.cose.2021.102402 · Cited by 399 · see citing works
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
Golatkar, A., Achille, A., & Soatto, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00932 · Cited by 394 · see citing works
Generative Adversarial Networks: A Survey Toward Private and Secure Applications
Cai, Z., Xiong, Z., Xu, H., Wang, P., Li, W., & Pan, Y. (2022) · ACM Computing Surveys · doi:10.1145/3459992 · Cited by 376 · see citing works
Deep Learning applications for COVID-19
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-020-00392-9 · Cited by 350 · see citing works
Biscotti: A blockchain system for private and secure federated learning
Shayan, M., Fung, C., Yoon, C. J. M., & Beschastnikh, I. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3044223 · Cited by 354 · see citing works
Generative AI in medical practice: in-depth exploration of privacy and security challenges
Chen, Y., & Esmaeilzadeh, P. (2024) · Journal of Medical Internet Research · doi:10.2196/53008 · Cited by 353 · see citing works
“In this attack, attackers aim to reconstruct the training data using their ability to constantly query the model [84]; (4) membership inference attacks—an attacker gains access to a set of real patient records and tries to determine whether any of the real patients are included in the training set…”
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., & Goldstein, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3162397 · Cited by 362 · see citing works
DeepInspect: A Black-box Trojan Detection and Mitigation Framework for Deep Neural Networks
Chen, H., Fu, C., Zhao, J., & Koushanfar, F. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/647 · Cited by 314 · see citing works
Rauniyar, A., Hagos, D. H., Jha, D., Håkegård, J. E., Bagci, U., Rawat, D. B., & Vlassov, V. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3329061 · Cited by 315 · see citing works
Model inversion attacks against collaborative inference
He, Z., Zhang, T., & Lee, R. B. (2019) · Proceedings of the 35th Annual Computer Security Applications Conference · doi:10.1145/3359789.3359824 · Cited by 313 · see citing works
GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
Chen, D., Yu, N., Zhang, Y., & Fritz, M. (2020) · Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3372297.3417238 · Cited by 311 · see citing works
Federated learning in a medical context: A systematic literature review
Pfitzner, B., Steckhan, N., & Arnrich, B. (2021) · ACM Transactions on Internet Technology · doi:10.1145/3412357 · Cited by 288 · see citing works
Algorithms that remember: model inversion attacks and data protection law
Veale, M., Binns, R., & Edwards, L. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0083 · Cited by 278 · see citing works
A Trustworthy Privacy Preserving Framework for Machine Learning in Industrial IoT Systems
Arachchige, P. C. M., Bertok, P., Khalil, I., Liu, D., Camtepe, S., & Atiquzzaman, M. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2974555 · Cited by 236 · see citing works
Privacy for Free: Wireless Federated Learning via Uncoded Transmission With Adaptive Power Control
Liu, D., & Simeone, O. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3036948 · Cited by 234 · see citing works
A Taxonomy of Attacks on Federated Learning
Jere, M. S., Farnan, T., & Koushanfar, F. (2021) · IEEE Security & Privacy · doi:10.1109/msec.2020.3039941 · Cited by 217 · see citing works
When Machine Learning Meets Privacy in 6G: A Survey
Sun, Y., Liu, J., Wang, J., Cao, Y., & Kato, N. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.3011561 · Cited by 211 · see citing works
Privacy Risks of General-Purpose Language Models
Pan, X., Zhang, M., Ji, S., & Yang, M. (2020) · 2020 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40000.2020.00095 · Cited by 188 · see citing works
More Than Privacy: Applying Differential Privacy in Key Areas of Artificial Intelligence
Zhu, T., Ye, D., Wang, W., Zhou, W., & Yu, P. S. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3014246 · Cited by 219 · see citing works
When Machine Unlearning Jeopardizes Privacy
Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., & Zhang, Y. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3484756 · Cited by 195 · see citing works
POSEIDON: Privacy-Preserving Federated Neural Network Learning
Sav, S., Pyrgelis, A., Troncoso-Pastoriza, J. R., Froelicher, D., Bossuat, J.-P., Sousa, J. S., & Hubaux, J.-P. (2021) · Proceedings 2021 Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.24119 · Cited by 162 · see citing works
Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning
Salem, A., Bhattacharya, A., Backes, M., Fritz, M., & Zhang, Y. (2020) · Universität des Saarlandes · doi:10.22028/d291-47370 · Cited by 157 · see citing works
Xu, H., Zhu, T., Zhang, L., Zhou, W., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3603620 · Cited by 164 · see citing works
“This is because membership inference attacks [5] and model inversion attacks [6] can reveal information about the speciic contents of a training dataset.”
BackgroundMethodLocal and Central Differential Privacy for Robustness and Privacy in Federated Learning
Naseri, M., Hayes, J., & De Cristofaro, E. (2022) · Proceedings 2022 Network and Distributed System Security Symposium · doi:10.14722/ndss.2022.23054 · Cited by 188 · see citing works
Rodríguez-Barroso, N., Stipcich, G., Jiménez-López, D., Ruiz-Millán, J. A., Martínez-Cámara, E., González-Seco, G., Luzón, M. V., Veganzones, M. A., & Herrera, F. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.07.009 · Cited by 148 · see citing works
Li, A., Sun, J., Zeng, X., Zhang, M., Li, H., & Chen, Y. (2021) · Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/3485730.3485929 · Cited by 139 · see citing works
Unrolling SGD: Understanding Factors Influencing Machine Unlearning
Thudi, A., Deza, G., Chandrasekaran, V., & Papernot, N. (2022) · 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp53844.2022.00027 · Cited by 135 · see citing works
Challenges and future directions of secure federated learning: a survey
Zhang, K., Song, X., Zhang, C., & Yu, S. (2022) · Frontiers of Computer Science · doi:10.1007/s11704-021-0598-z · Cited by 127 · see citing works
Kalapaaking, A. P., Khalil, I., & Yi, X. (2024) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2023.3268186 · Cited by 122 · see citing works
Smart Wearables for the Detection of Cardiovascular Diseases: A Systematic Literature Review
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2023) · Sensors · doi:10.3390/s23020828 · Cited by 120 · see citing works
A Blockchain‐Based Federated Learning Method for Smart Healthcare
Chang, Y., Fang, C., & Sun, W. (2021) · Computational Intelligence and Neuroscience · doi:10.1155/2021/4376418 · Cited by 119 · see citing works
GAN Enhanced Membership Inference: A Passive Local Attack in Federated Learning
Zhang, J., Zhang, J., Chen, J., & Yu, S. (2020) · ICC 2020 - 2020 IEEE International Conference on Communications (ICC) · doi:10.1109/icc40277.2020.9148790 · Cited by 133 · see citing works
Golatkar, A., Achille, A., & Soatto, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58526-6_23 · Cited by 122 · see citing works
A Survey on Differentially Private Machine Learning [Review Article]
Gong, M., Xie, Y., Pan, K., Feng, K., & Qin, A. K. (2020) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2020.2976185 · Cited by 115 · see citing works
Differential Privacy Preserving of Training Model in Wireless Big Data with Edge Computing
Du, M., Wang, K., Xia, Z., & Zhang, Y. (2020) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2018.2829886 · Cited by 111 · see citing works
Reviewing Federated Machine Learning and Its Use in Diseases Prediction
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2023) · Sensors · doi:10.3390/s23042112 · Cited by 114 · see citing works
Pixel-Based Image Encryption Without Key Management for Privacy-Preserving Deep Neural Networks
Sirichotedumrong, W., Kinoshita, Y., & Kiya, H. (2019) · IEEE Access · doi:10.1109/access.2019.2959017 · Cited by 105 · see citing works
Privacy in Deep Learning: A Survey
Mireshghallah, F., Taram, M., Vepakomma, P., Singh, A., Raskar, R., & Esmaeilzadeh, H. (2020) · arXiv · doi:10.48550/arxiv.2004.12254 · Cited by 106 · see citing works
Privacy and Security of Big Data in AI Systems: A Research and Standards Perspective
Dilmaghani, S., Brust, M. R., Danoy, G., Cassagnes, N., Pecero, J., & Bouvry, P. (2019) · 2019 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata47090.2019.9006283 · Cited by 111 · see citing works
History, development, and principles of large language models: an introductory survey
Wang, Z., Chu, Z., Doan, T. V., Ni, S., Yang, M., & Zhang, W. (2025) · AI and Ethics · doi:10.1007/s43681-024-00583-7 · Cited by 109 · see citing works
Federated learning in medical imaging: Part II: methods, challenges, and considerations
Darzidehkalani, E., Ghasemi-rad, M., & van Ooijen, P. M. A. (2022) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2022.03.016 · Cited by 103 · see citing works
Security, Privacy, and Usability in Continuous Authentication: A Survey
Baig, A. F., & Eskeland, S. (2021) · Sensors · doi:10.3390/s21175967 · Cited by 98 · see citing works
ARCANE: An Efficient Architecture for Exact Machine Unlearning
Yan, H., Li, X., Guo, Z., Li, H., Li, F., & Lin, X. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/556 · Cited by 97 · see citing works
Khosravy, M., Nakamura, K., Hirose, Y., Nitta, N., & Babaguchi, N. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3140687 · Cited by 94 · see citing works
Privacy-preserving and communication-efficient federated learning in Internet of Things
Fang, C., Guo, Y., Hu, Y., Ma, B., Feng, L., & Yin, A. (2021) · Computers & Security · doi:10.1016/j.cose.2021.102199 · Cited by 93 · see citing works
Analyzing Information Leakage of Updates to Natural Language Models
Zanella-Béguelin, S., Wutschitz, L., Tople, S., Rühle, V., Paverd, A., Ohrimenko, O., Köpf, B., & Brockschmidt, M. (2020) · Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3372297.3417880 · Cited by 86 · see citing works
Record and Reward Federated Learning Contributions with Blockchain
Martinez, I., Francis, S., & Hafid, A. S. (2019) · 2019 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC) · doi:10.1109/cyberc.2019.00018 · Cited by 92 · see citing works
Model Evasion Attack on Intrusion Detection Systems using Adversarial Machine Learning
Ayub, M. A., Johnson, W. A., Talbert, D. A., & Siraj, A. (2020) · 2020 54th Annual Conference on Information Sciences and Systems (CISS) · doi:10.1109/ciss48834.2020.1570617116 · Cited by 86 · see citing works
A Systematic Review of Adversarial Machine Learning Attacks, Defensive Controls and Technologies
Malik, J., Muthalagu, R., & Pawar, P. M. (2024) · IEEE Access · doi:10.1109/access.2024.3423323 · Cited by 76 · see citing works
AI system engineering key challenges and lessons learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobiezky, F., Zellinger, W., Brunner, D., Kumar, M., & Moser, B. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make3010004 · Cited by 71 · see citing works
Survey: Leakage and Privacy at Inference Time
Jegorova, M., Kaul, C., Mayor, C., O'Neil, A. Q., Weir, A., Murray-Smith, R., & Tsaftaris, S. A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229593 · Cited by 77 · see citing works
Data-free Knowledge Distillation for Object Detection
Chawla, A., Yin, H., Molchanov, P., & Alvarez, J. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00333 · Cited by 70 · see citing works
Hard to Forget: Poisoning Attacks on Certified Machine Unlearning
Marchant, N. G., Rubinstein, B. I. P., & Alfeld, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20736 · Cited by 64 · see citing works
“Such concerns are justified by a legitimate privacy risk: training data can be practically reconstructed from models (Fredrikson, Jha, and Ristenpart, 2015; Shokri et al., 2017; Veale, Binns, and Edwards, 2018).”
Property Inference from Poisoning
Mahloujifar, S., Ghosh, E., & Chase, M. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833623 · Cited by 67 · see citing works
Zhou, J., Chen, S., Wu, Y., Li, H., Zhang, B., Zhou, L., Hu, Y., Xiang, Z., Li, Z., Chen, N., et al. (2024) · Science Advances · doi:10.1126/sciadv.adh8601 · Cited by 63 · see citing works
A survey for federated learning evaluations: Goals and measures
Chai, D., Wang, L., Yang, L., Zhang, J., Chen, K., & Yang, Q. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3382002 · Cited by 64 · see citing works
EW-Tune: A Framework for Privately Fine-Tuning Large Language Models with Differential Privacy
Behnia, R., Ebrahimi, M. R., Pacheco, J., & Padmanabhan, B. (2022) · 2022 IEEE International Conference on Data Mining Workshops (ICDMW) · doi:10.1109/icdmw58026.2022.00078 · Cited by 60 · see citing works
Secure and robust machine learning for healthcare: A survey
Qayyum, A., Qadir, J., Bilal, M., & Al-Fuqaha, A. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3013489 · Cited by 59 · see citing works
Froelicher, D., Troncoso-Pastoriza, J. R., Sousa, J. S., & Hubaux, J.-P. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.2976612 · Cited by 59 · see citing works
Generative Ai in EU Law: Liability, Privacy, Intellectual Property, and Cybersecurity
Novelli, C., Casolari, F., Hacker, P., Spedicato, G., & Floridi, L. (2024) · doi:10.2139/ssrn.4821952 · Cited by 57 · see citing works
Privacy-Preserving Federated Learning Model for Healthcare Data
Islam, T. U., Ghasemi, R., & Mohammed, N. (2022) · 2022 IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc54503.2022.9720752 · Cited by 56 · see citing works
FENCE: Feasible Evasion Attacks on Neural Networks in Constrained Environments
Chernikova, A., & Oprea, A. (2022) · ACM Transactions on Privacy and Security · doi:10.1145/3544746 · Cited by 52 · see citing works
Reliable customer analysis using federated learning and exploring deep-attention edge intelligence
Ahmed, U., Srivastava, G., & Lin, J. C.-W. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2021.08.028 · Cited by 52 · see citing works
Athena: Probabilistic Verification of Machine Unlearning
Sommer, D. M., Song, L., Wagh, S., & Mittal, P. (2022) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2022-0072 · Cited by 51 · see citing works
PUMA: Performance Unchanged Model Augmentation for Training Data Removal
Wu, G., Hashemi, M., & Srinivasa, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20846 · Cited by 61 · see citing works
- Authors
- Fredrikson, M., Jha, S., & Ristenpart, T.
- Affiliations
- Carnegie Mellon University (US)
- University of Wisconsin–Madison (US)
- Cornell Tech (US)
- Venue
- Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2015-10-12
- Pages
- 1322-1333
- ISSN
- 1543-7221 (ISSN-L)
- OpenAlex
- W2051267297
- Semantic Scholar
- d1b9a3b11e6c9571a1553556f82b605b2b4baec3
- Cited by
- 2,961
- References
- 27
- Citations by year
- 2026: 256 · 2025: 389 · 2024: 427 · 2023: 459 · 2022: 374 · 2021: 423 · 2020: 315 · 2019: 190 · 2018: 85 · 2017: 31 · 2016: 11 · 2012: 1
- FWCI
- 23.55
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGSAC Conference on Computer and Communications Security (CCS) (publishedVersion)
- Funders
- Defense Advanced Research Projects Agency (FA8650-10-C-7088)
- National Science Foundation (CNS-1546033 CNS-1330308 CNS-1065134)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Keywords
- model inversion attacks, confidence scores, privacy attacks on machine learning, decision tree, artificial neural networks, face recognition, lifestyle survey, machine learning as a service, CART, genetic privacy, countermeasures
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
