CoChat
Sign Up
Cited by 852Adversarial Robustness in Machine LearningPrivacy-Preserving Technologies in DataSecurity and Verification in Computing

ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models

Open access
Salem, Ahmed, Zhang, Yang, Humbert, Mathias, Berrang, Pascal, Fritz, Mario, Backes, Michael·2019·Proceedings 2019 Network and Distributed System Security Symposium
DOI 10.14722/ndss.2019.23119
Open this paper in CoChat — free account

Abstract

Machine learning (ML) has become a core component of many real-world applications and training data is a key factor that drives current progress.This huge success has led Internet companies to deploy machine learning as a service (MLaaS).Recently, the first membership inference attack has shown that extraction of information on the training set is possible in such MLaaS settings, which has severe security and privacy implications.However, the early demonstrations of the feasibility of such attacks have many assumptions on the adversary, such as using multiple so-called shadow models, knowledge of the target model structure, and having a dataset from the same distribution as the target model's training data.We relax all these key assumptions, thereby showing that such attacks are very broadly applicable at low cost and thereby pose a more severe risk than previously thought.We present the most comprehensive study so far on this emerging and developing threat using eight diverse datasets which show the viability of the proposed attacks across domains.In addition, we propose the first effective defense mechanisms against such broader class of membership inference attacks that maintain a high level of utility of the ML model.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.14722/ndss.2019.23119

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Membership inference is the most well-established privacy attack and arguably constitutes a bigger privacy threat than class-level attacks, such as the attacks in [7, 59] and model inversion [18, 19].

    “We emphasize that membership inference is the most wellestablished privacy attack and arguably constitutes a bigger privacy threat [13,28,30,37,46,48,60,66,67,75] than classlevel attacks, such as the attacks in [7, 59] and model inversion [18, 19].”

    From When Machine Unlearning Jeopardizes Privacy (2021) · cites this paper as [60]

    The Introduction describes severe consequences of membership inference, and the Conclusion calls membership privacy a severe threat across eight datasets. The paper does not establish that membership inference is the most well-established privacy attack or compare its threat with class-level attacks or model inversion. No page numbers are available in our copy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Exploiting plain posterior information performs poorly for membership inference on well-generalized target models.

    “For the well-generalized target models, exploiting the plain posterior information has shown to perform poorly in terms of membership inference [60, 64].”

    From When Machine Unlearning Jeopardizes Privacy (2021) · cites this paper as [60]

  • Classical membership inference, which uses plain posterior information, performs well on overfitted target models.

    “This is consistent with the conclusion of previous studies [60, 64] that classical membership inference (which uses plain posterior information) performs well on overfitted target models.”

    From When Machine Unlearning Jeopardizes Privacy (2021) · cites this paper as [60]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. M. Backes, P. Berrang, M. Bieg, R. Eils, C. Herrmann, M. Humbert, and I. Lehmann, "Identifying Personal DNA Methylation Profiles by Genotype Inference," in Proceedings of the 2017 IEEE Symposium on Security and Privacy (S&P). IEEE, 2017, pp. 957-976. 13, 14
  2. M. Backes, P. Berrang, M. Humbert, and P. Manoharan, "Membership Privacy in MicroRNA-based Studies," in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2016, pp. 319-330. 1, 13
  3. M. Backes, M. Humbert, J. Pang, and Y. Zhang, "walk2friends: Inferring Social Links from Mobility Profiles," in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2017, pp. 1943-1957. 9
  4. P. Berrang, M. Humbert, Y. Zhang, I. Lehmann, R. Eils, and M. Backes, "Dissecting Privacy Risks in Biomedical Data," in Proceedings of the 2018 IEEE European Symposium on Security and Privacy (Euro S&P). IEEE, 2018. 8
  5. K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, "Practical Secure Ag- gregation for Privacy-Preserving Machine Learning," in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2017, pp. 1175-1191. 13, 14
  6. R. Bost, R. A. Popa, S. Tu, and S. Goldwasser, "Machine Learning Classification over Encrypted Data," in Proceedings of the 2015 Net- work and Distributed System Security Symposium (NDSS). Internet Society, 2015. 13, 14
  7. X. Cao and N. Z. Gong, "Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classification," in Proceedings of the 2017 Annual Computer Security Applications Conference (ACSAC). ACM, 2017, pp. 278-287. 14
  8. Q. Chen, C. Xiang, M. Xue, B. Li, N. Borisov, D. Kaarfar, and H. Zhu, "Differentially Private Data Generative Models," CoRR abs/1812.02274, 2018. 14 · arXiv
  9. N. Dowlin, R. Gilad-Bachrach, K. Laine, K. Lauter, M. Naehrig, and J. Wernsing, "CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy," in Proceedings of the 2016 International Conference on Machine Learning (ICML). JMLR, 2016, pp. 201-210. 13
  10. Z. Erkin, M. Franz, J. Guajardo, S. Katzenbeisser, I. Lagendijk, and T. Toft, "Privacy-Preserving Face Recognition," Symposium on Privacy Enhancing Technologies Symposium, 2009. 13
  11. M. Fredrikson, S. Jha, and T. Ristenpart, "Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures," in Proceedings of the 2015 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2015, pp. 1322-1333. 1, 13
  12. M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, and T. Ristenpart, "Privacy in Pharmacogenetics: An End-to-End Case Study of Person- alized Warfarin Dosing," in Proceedings of the 2014 USENIX Security Symposium (USENIX Security). USENIX, 2014, pp. 17-32. 9, 13
  13. L. Hanzlik, Y. Zhang, K. Grosse, A. Salem, M. Augustin, M. Backes, and M. Fritz, "MLCapsule: Guarded Offline Deployment of Machine Learning as a Service," CoRR abs/1808.00590, 2018. 14 · arXiv
  14. J. Hayes, L. Melis, G. Danezis, and E. D. Cristofaro, "LOGAN: Evaluating Privacy Leakage of Generative Models Using Generative Adversarial Networks," CoRR abs/1705.07663, 2017. 13 · arXiv
  15. Y. Huang, L. Malka, D. Evans, and J. Katz, "Efficient Privacy- Preserving Biometric Identification," in Proceedings of the 2011 Net- work and Distributed System Security Symposium (NDSS). Internet Society, 2011. 13
  16. T. Hunt, C. Song, R. Shokri, V. Shmatikov, and E. Witchel, "Ch- iron: Privacy-preserving Machine Learning as a Service," CoRR abs/1803.05961, 2018. 14 · arXiv
  17. J. Jia and N. Z. Gong, "AttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine Learning," in Pro- ceedings of the 2018 USENIX Security Symposium (USENIX Security). USENIX, 2018. 13
  18. B. Li and Y. Vorobeychik, "Scalable Optimization of Randomized Oper- ational Decisions in Adversarial Classification Settings," in Proceedings of the 2015 International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2015, pp. 599-607. 13
  19. J. Liu, M. Juuti, Y. Lu, and N. Asokan, "Oblivious Neural Network Predictions via MiniONN Transformations," in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2017, pp. 619-631. 13
  20. Y. Long, V. Bindschaedler, and C. A. Gunter, "Towards Measuring Membership Privacy," CoRR abs/1712.09136, 2017. 13 · arXiv
  21. Y. Long, V. Bindschaedler, L. Wang, D. Bu, X. Wang, H. Tang, C. A. Gunter, and K. Chen, "Understanding Membership Inferences on Well- Generalized Learning Models," CoRR abs/1802.04889, 2018. 13 · arXiv
  22. L. Melis, C. Song, E. D. Cristofaro, and V. Shmatikov, "Inference Attacks Against Collaborative Learning," CoRR abs/1805.04049, 2018. 14 · arXiv
  23. P. Mohassel and Y. Zhang, "SecureML: A System for Scalable Privacy- Preserving Machine Learning," in Proceedings of the 2017 IEEE Symposium on Security and Privacy (S&P). IEEE, 2017, pp. 19-38. 13
  24. M. Nasr, R. Shokri, and A. Houmansadr, "Machine Learning with Mem- bership Privacy using Adversarial Regularization," in Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2018. 14
  25. S. J. Oh, M. Augustin, B. Schiele, and M. Fritz, "Towards Reverse- Engineering Black-Box Neural Networks," in Proceedings of the 2018 International Conference on Learning Representations (ICLR), 2018. 1, 4
  26. S. J. Oh, M. Fritz, and B. Schiele, "Adversarial Image Perturbation for Privacy Protection -A Game Theory Perspective," in Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017, pp. 1482-1491. 13
  27. J. Pang and Y. Zhang, "DeepCity: A Feature Learning Framework for Mining Location Check-Ins," in Proceedings of the 2017 International Conference on Weblogs and Social Media (ICWSM). AAAI, 2017, pp. 652-655. 9
  28. N. Papernot, P. D. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, "Practical Black-Box Attacks Against Machine Learning," in Proceedings of the 2017 ACM Asia Conference on Computer and Communications Security (ASIACCS). ACM, 2017, pp. 506-519. 13
  29. A. Pyrgelis, C. Troncoso, and E. D. Cristofaro, "Knock Knock, Who's There? Membership Inference on Aggregate Location Data," in Proceedings of the 2018 Network and Distributed System Security Symposium (NDSS). Internet Society, 2018. 1, 9, 13
  30. B. D. Rouhani, H. Chen, and F. Koushanfar, "DeepSigns: A Generic Watermarking Framework for IP Protection of Deep Learning Models," CoRR abs/1804.00750, 2018. 14 · arXiv
  31. B. D. Rouhani, M. S. Riazi, and F. Koushanfar, "DeepSecure: Scalable Provably-Secure Deep Learning," CoRR abs/1705.08963, 2017. 14 · arXiv
  32. C. Song, T. Ristenpart, and V. Shmatikov, "Machine Learning Models that Remember Too Much," in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2017, pp. 587-601. 14
  33. O. Suciu, R. Mȃrginean, Y. Kaya, H. D. III, and T. Dumitras ¸, "When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks," CoRR abs/1803.06975, 2018. 14 · arXiv
  34. F. Tramer and D. Boneh, "Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware," CoRR abs/1806.03287, 2018. 14 · arXiv
  35. F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, "Ensemble Adversarial Training: Attacks and Defenses," in Proceedings of the 2017 International Conference on Learning Representations (ICLR), 2017. 13
  36. F. Tramér, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, "Stealing Machine Learning Models via Prediction APIs," in Proceedings of the 2016 USENIX Security Symposium (USENIX Security). USENIX, 2016, pp. 601-618. 1, 4, 13
  37. Y. Vorobeychik and B. Li, "Optimal Randomized Classification in Ad- versarial Settings," in Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems (AAMAS), 2014, pp. 485-492. 13
  38. B. Wang and N. Z. Gong, "Stealing Hyperparameters in Machine Learning," in Proceedings of the 2018 IEEE Symposium on Security and Privacy (S&P). IEEE, 2018. 1, 4, 14
  39. W. Xu, D. Evans, and Y. Qi, "Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks," in Proceedings of the 2018 Network and Distributed System Security Symposium (NDSS). Internet Society, 2018. 13
  40. D. Yang, D. Zhang, and B. Qu, "Participatory Cultural Mapping Based on Collective Behavior Data in Location-Based Social Networks," ACM Transactions on Intelligent Systems and Technology, 2016. 3
  41. Y. Zhang, M. Humbert, T. Rahman, C.-T. Li, J. Pang, and M. Backes, "Tagvisor: A Privacy Advisor for Sharing Hashtags," in Proceedings of the 2018 Web Conference (WWW). ACM, 2018, pp. 287-296. 9, 13
  42. Y. Zhang, M. Humbert, B. Surma, P. Manoharan, J. Vreeken, and M. Backes, "CTRL+Z: Recovering Anonymized Social Graphs," CoRR abs/1711.05441, 2017. · arXiv

Cited by

Sorted by citations

Authors
Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., & Backes, M.
Affiliations
  • Helmholtz Center for Information Security (DE)
  • ETH Zurich (CH)
ORCID
Venue
Proceedings 2019 Network and Distributed System Security Symposium
Publisher
Internet Society
Published
2019
DOI
10.14722/ndss.2019.23119
OpenAlex
W2807096445
Cited by
852
References
8
Citations by year
2026: 74 · 2025: 126 · 2024: 139 · 2023: 142 · 2022: 114 · 2021: 122 · 2020: 92 · 2019: 36 · 2018: 6 · 2017: 1
FWCI
34.81
Type
conference paper
Language
en
Open access
Locations
  • Network and Distributed System Security Symposium (publishedVersion)
  • SERVAL (Université de Lausanne) (publishedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion)
  • IRIS (submittedVersion)
  • Figshare (submittedVersion, cc-by)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
  • arXiv (Cornell University)
  • CISPA Helmholtz Center
Funders
European Commission (610150, 2007–2013, FP7 2007-2013, FP7/2007)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
Keywords
membership inference attacks, machine learning privacy, training data privacy, machine learning as a service, privacy-preserving machine learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review