The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples
Abstract
In a membership inference attack, an attacker aims to infer whether a data sample is in a target classifier's training dataset or not. Specifically, given a black-box access to the target classifier, the attacker trains a binary classifier, which takes a data sample's confidence score vector predicted by the target classifier as an input and predicts the data sample to be a member or non-member of the target classifier's training dataset. Membership inference attacks pose severe privacy and security threats to the training dataset. Most existing defenses leverage differential privacy when training the target classifier or regularize the training process of the target classifier. These defenses suffer from two key limitations: 1) they do not have formal utility-loss guarantees of the confidence score vectors, and 2) they achieve suboptimal privacy-utility tradeoffs. In this work, we propose MemGuard,the first defense with formal utility-loss guarantees against black-box membership inference attacks. Instead of tampering the training process of the target classifier, MemGuard adds noise to each confidence score vector predicted by the target classifier. Our key observation is that attacker uses a classifier to predict member or non-member and classifier is vulnerable to adversarial examples.Based on the observation, we propose to add a carefully crafted noise vector to a confidence score vector to turn it into an adversarial example that misleads the attacker's classifier. Specifically, MemGuard works in two phases. In Phase I, MemGuard finds a carefully crafted noise vector that can turn a confidence score vector into an adversarial example, which is likely to mislead the attacker's classifier to make a random guessing at member or non-member. We find such carefully crafted noise vector via a new method that we design to incorporate the unique utility-loss constraints on the noise vector. In Phase II, MemGuard adds the noise vector to the confidence score vector with a certain probability, which is selected to satisfy a given utility-loss budget on the confidence score vector. Our experimental results on three datasets show that MemGuard can effectively defend against membership inference attacks and achieve better privacy-utility tradeoffs than existing defenses. Our work is the first one to show that adversarial examples can be used as defensive mechanisms to defend against membership inference attacks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2014) · Cited by 33,419
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Calibrating Noise to Sensitivity in Private Data Analysis
Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006) · Lecture Notes in Computer Science · doi:10.1007/11681878_14 · Cited by 7,272
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 3×
“sarial example to evade the defense classifier. In particular, s is a normal example and s+r is an adversarial example. The adversarial machine learning community has developed many algorithms (e.g., [10, 19, 31, 35, 39, 40, 50, 63]) to find adversarial examples. However, these algorithms are insufficient to our probl…”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
Practical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P., Wu, X., Jha, S., & Swami, A. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.41 · Cited by 2,982
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961
Universal Adversarial Perturbations
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., & Frossard, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.17 · Cited by 2,680
Privacy-preserving deep learning
Shokri, R., & Shmatikov, V. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813687 · Cited by 2,356 · cited 1×
“arn a model [12, 24, 30], or the gradient in each iteration of gradient descent or stochastic gradient descent that is used to minimize the objective function [1, 6, 60, 66, 70]. Shokri and Shmatikov [57] designed a differential privacy method for collaborative learning of deep neural networks. Limitations. Existing d…”
MethodEnsemble Adversarial Training: Attacks and Defenses
Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., & McDaniel, P. (2017) · arXiv · doi:10.48550/arxiv.1705.07204 · Cited by 2,176
Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
Athalye, A., Carlini, N., & Wagner, D. (2018) · arXiv · doi:10.48550/arxiv.1802.00420 · Cited by 2,172 · cited 2×
“Nevertheless, in our experiments, we will consider the attacker uses adversarial training to train its attack classifier, as adversarial training was considered to be the most empirically robust method against adversarial examples so far [3].”
MethodFeature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
Xu, W., Evans, D., & Qi, Y. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23198 · Cited by 1,768
Nasr, M., Shokri, R., & Houmansadr, A. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00065 · Cited by 1,760
Exploiting Unintended Feature Leakage in Collaborative Learning
Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00029 · Cited by 1,734
Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
Papernot, N., McDaniel, P., & Goodfellow, I. (2016) · arXiv · doi:10.48550/arxiv.1605.07277 · Cited by 1,326
Delving into Transferable Adversarial Examples and Black-box Attacks
Liu, Y., Chen, X., Liu, C., & Song, D. (2016) · arXiv · doi:10.48550/arxiv.1611.02770 · Cited by 1,310
Homer, N., Szelinger, S., Redman, M., Duggan, D., Tembe, W., Muehling, J., Pearson, J. V., Stephan, D. A., Nelson, S. F., & Craig, D. W. (2008) · PLoS Genetics · doi:10.1371/journal.pgen.1000167 · Cited by 1,221
MagNet: A Two-Pronged Defense against Adversarial Examples
Meng, D., & Chen, H. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134057 · Cited by 1,006
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
Yeom, S., Giacomelli, I., Fredrikson, M., & Jha, S. (2018) · 2018 IEEE 31st Computer Security Foundations Symposium (CSF) · doi:10.1109/csf.2018.00027 · Cited by 1,001
(2011) · PubMed · Cited by 990
Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., & Backes, M. (2019) · Proceedings 2019 Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23119 · Cited by 852
Self-citationStealing Machine Learning Models via Prediction APIs
Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016) · arXiv · doi:10.48550/arxiv.1609.02943 · Cited by 796
Cross-VM side channels and their use to extract private keys
Zhang, Y., Juels, A., Reiter, M. K., & Ristenpart, T. (2012) · Proceedings of the 2012 ACM conference on Computer and communications security · doi:10.1145/2382196.2382230 · Cited by 787
Self-citationPrivacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing.
Fredrikson, M., Lantz, E., Jha, S., Lin, S. M., Page, D., & Ristenpart, T. (2014) · PubMed · Cited by 641
Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds
Bassily, R., Smith, A., & Thakurta, A. (2014) · 2014 IEEE 55th Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2014.56 · Cited by 621
SoK: Security and Privacy in Machine Learning
Papernot, N., McDaniel, P., Sinha, A., & Wellman, M. P. (2018) · 2018 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2018.00035 · Cited by 502
Stochastic gradient descent with differentially private updates
Song, S., Chaudhuri, K., & Sarwate, A. D. (2013) · 2013 IEEE Global Conference on Signal and Information Processing · doi:10.1109/globalsip.2013.6736861 · Cited by 501
Website fingerprinting in onion routing based anonymization networks
Panchenko, A., Niessen, L., Zinnen, A., & Engel, T. (2011) · Proceedings of the 10th annual ACM workshop on Privacy in the electronic society · doi:10.1145/2046556.2046570 · Cited by 501
Ganju, K., Wang, Q., Yang, W., Gunter, C. A., & Borisov, N. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243834 · Cited by 475 · cited 1×
“Several works [2, 16, 36] studied property inference attacks, which aim to infer a certain property (e.”
BackgroundStealing Hyperparameters in Machine Learning
Wang, B., & Gong, N. Z. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00038 · Cited by 470
Self-citationMachine Learning with Membership Privacy using Adversarial Regularization
Nasr, M., Shokri, R., & Houmansadr, A. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243855 · Cited by 458
SoK: Towards the Science of security and privacy in machine learning
Papernot, N., McDaniel, P. D., Sinha, A., & Wellman, M. P. (2018) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 427
Herrmann, D., Wendolsky, R., & Federrath, H. (2009) · Proceedings of the 2009 ACM workshop on Cloud computing security · doi:10.1145/1655008.1655013 · Cited by 424
Ateniese, G., Mancini, L. V., Spognardi, A., Villani, A., Vitali, D., & Felici, G. (2015) · International Journal of Security and Networks · doi:10.1504/ijsn.2015.071829 · Cited by 395
Touching from a distance: website fingerprinting attacks and defenses
Cai, X., Zhang, X. C., Joshi, B., & Johnson, R. (2012) · Proceedings of the 2012 ACM conference on Computer and communications security · doi:10.1145/2382196.2382260 · Cited by 394
Effective attacks and provable defenses for website fingerprinting
Wang, T., Cai, X., Nithyanand, R., Johnson, R., & Goldberg, I. (2014) · Cited by 340
A Critical Evaluation of Website Fingerprinting Attacks
Juarez, M., Afroz, S., Acar, G., Diaz, C., & Greenstadt, R. (2014) · Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2660267.2660368 · Cited by 327
Evaluating Differentially Private Machine Learning in Practice
Jayaraman, B., & Evans, D. (2019) · arXiv · doi:10.48550/arxiv.1902.08874 · Cited by 265
On the Feasibility of Internet-Scale Author Identification
Narayanan, A., Paskov, H., Gong, N. Z., Bethencourt, J., Stefanov, E., Shin, E. C. R., & Song, D. (2012) · 2012 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2012.46 · Cited by 261
Self-citationPrivacy Risks of Securing Machine Learning Models against Adversarial Examples
Song, L., Shokri, R., & Mittal, P. (2019) · Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3319535.3354211 · Cited by 229 · cited 1×
“Multiple studies have shown that such black-box ML classifier is vulnerable to membership inference attacks [43, 56, 58, 59].”
BackgroundAttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine LearningPreprint
Jia, J., & Gong, N. Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04810 · Cited by 222
Self-citationKnock Knock, Who's There? Membership Inference on Aggregate Location Data
Pyrgelis, A., Troncoso, C., & Cristofaro, E. D. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23183 · Cited by 210
Differentially Private Model Publishing for Deep Learning
Yu, L., Liu, L., Pu, C., Gursoy, M. E., & Truex, S. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00019 · Cited by 195
Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classification
Cao, X., & Gong, N. Z. (2017) · Proceedings of the 33rd Annual Computer Security Applications Conference · doi:10.1145/3134600.3134606 · Cited by 168
Self-citationUnderstanding Membership Inferences on Well-Generalized Learning Models
Long, Y., Bindschaedler, V., Wang, L., Bu, D., Wang, X., Tang, H., Gunter, C. A., & Chen, K. (2018) · arXiv · doi:10.48550/arxiv.1802.04889 · Cited by 166
Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning
Salem, A., Bhattacharya, A., Backes, M., Fritz, M., & Zhang, Y. (2020) · Universität des Saarlandes · doi:10.22028/d291-47370 · Cited by 157
Self-citationWhen Coding Style Survives Compilation: De-anonymizing Programmers from Executable Binaries
Caliskan, A., Yamaguchi, F., Dauber, E., Harang, R., Rieck, K., Greenstadt, R., & Narayanan, A. (2018) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23304 · Cited by 151
walk2friends: Inferring Social Links from Mobility Profiles
Michael Backes, A., Humbert, M., Pang, J., & Zhang, Y. (2017) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3133956.3133972 · Cited by 147
Self-citationTowards Practical Differentially Private Convex Optimization
Iyengar, R., Near, J. P., Song, D. X., Thakkar, O. D., Thakurta, A., & Wang, L. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00001 · Cited by 145
Membership Privacy in MicroRNA-based Studies
Michael Backes, A., Berrang, P., Humbert, M., & Manoharan, P. (2016) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2976749.2978355 · Cited by 136
You Are What You Like! Information Leakage Through Users' Interests
Chaabane, A., Ács, G., & Kâafar, M. A. (2012) · Cited by 131
Differentially Private Empirical Risk Minimization: Efficient Algorithms and Tight Error BoundsPreprint
Bassily, R., Smith, A., & Thakurta, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.7085 · Cited by 100
Towards Measuring Membership PrivacyPreprint
Long, Y., Bindschaedler, V., & Gunter, C. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.09136 · Cited by 95
Back to the Drawing Board: Revisiting the Design of Optimal Location Privacy-preserving Mechanisms
Oya, S., Troncoso, C., & Pérez‐González, F. (2017) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3133956.3134004 · Cited by 87
Gong, N. Z., & Liu, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05893 · Cited by 84
Self-citation(2017) · doi:10.1145/3038912.3052695 · Cited by 78
Misleading Authorship Attribution of Source Code using Adversarial LearningPreprint
Quiring, E., Maier, A., & Rieck, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12386 · Cited by 73
LOGAN: Evaluating Privacy Leakage of Generative Models Using Generative Adversarial Networks.Preprint
Hayes, J., Melis, L., Danezis, G., & Cristofaro, E. D. (2017) · arXiv (Cornell University) · Cited by 59
Statistical Privacy for Streaming Traffic
Zhang, X., Hamm, J., Reiter, M. K., & Zhang, Y. (2019) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23210 · Cited by 53
Self-citationMBeacon: Privacy-Preserving Beacons for DNA Methylation Data
Hagestedt, I., Zhang, Y., Humbert, M., Berrang, P., Tang, H., Wang, X., & Michael Backes, A. (2019) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23064 · Cited by 48
Self-citationKifer, D., Smith, A., & Thakurta, A. (2012) · Conference on Learning Theory · Cited by 45
Towards Reverse-Engineering Black-Box Neural NetworksPreprint
Oh, S. J., Augustin, M., Schiele, B., & Fritz, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01768 · Cited by 42
Zhang, Y., Humbert, M., Rahman, T., Li, C., Pang, J., & Michael Backes, A. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186095 · Cited by 35
Self-citationJia, J., & Gong, N. Z. (2020) · Springer eBooks · doi:10.1007/978-3-030-33432-1_2 · Cited by 25
Self-citationAdversarial Binaries for Authorship IdentificationPreprint
Meng, X., Miller, B. P., & Jha, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.08316 · Cited by 12
Under the Hood of Membership Inference Attacks on Aggregate Location Time-Series.Preprint
Pyrgelis, A., Troncoso, C., & Cristofaro, E. D. (2019) · arXiv (Cornell University) · Cited by 11
Cited by
Membership Inference Attacks on Machine Learning: A Survey
Hu, H., Salcic, Z., Sun, L., Dobbie, G., Yu, P. S., & Zhang, X. (2022) · ACM Computing Surveys · doi:10.1145/3523273 · Cited by 564
Membership Inference Attacks From First Principles
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., & Tramèr, F. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833649 · Cited by 502
Security and Privacy Challenges of Large Language Models: A Survey
Das, B. C., Amini, M. H., & Wu, Y. (2025) · ACM Computing Surveys · doi:10.1145/3712001 · Cited by 445
GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
Chen, D., Yu, N., Zhang, Y., & Fritz, M. (2020) · Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3372297.3417238 · Cited by 311
Self-citationA Survey of Privacy Attacks in Machine Learning
Rigaki, M., & Garcia, S. (2024) · ACM Computing Surveys · doi:10.1145/3624010 · Cited by 285
Metadata
- Authors
- Jia, J., Salem, A., Backes, M., Zhang, Y., & Gong, N. Z.
- Venue
- Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2019-11-06
- Pages
- 259-274
- OpenAlex
- W2983140679
- Cited by
- 324
- References
- 72
- FWCI
- 13.42
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.