CoChat
Sign Up
PreprintCited by 873Adversarial Robustness in Machine LearningPrivacy-Preserving Technologies in DataSecurity and Verification in Computing

ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models

Open access
Salem, Ahmed, Zhang, Yang, Humbert, Mathias, Berrang, Pascal, Fritz, Mario, Backes, Michael·2018·arXiv
DOI 10.48550/arxiv.1806.01246arXiv 1806.01246
Open this paper in CoChat — free account

Abstract

Machine learning (ML) has become a core component of many real-world applications and training data is a key factor that drives current progress. This huge success has led Internet companies to deploy machine learning as a service (MLaaS). Recently, the first membership inference attack has shown that extraction of information on the training set is possible in such MLaaS settings, which has severe security and privacy implications. However, the early demonstrations of the feasibility of such attacks have many assumptions on the adversary, such as using multiple so-called shadow models, knowledge of the target model structure, and having a dataset from the same distribution as the target model's training data. We relax all these key assumptions, thereby showing that such attacks are very broadly applicable at low cost and thereby pose a more severe risk than previously thought. We present the most comprehensive study so far on this emerging and developing threat using eight diverse datasets which show the viability of the proposed attacks across domains. In addition, we propose the first effective defense mechanisms against such broader class of membership inference attacks that maintain a high level of utility of the ML model.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1806.01246

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • General ML models have been shown to memorize and leak data from the training dataset, and this also holds for LLMs.

    “General ML models have been shown to memorize and leak data from the training dataset, and this also holds for LLMs.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Salem et al 2018

    The Introduction (page not provided) describes membership-inference attacks that reveal whether a data item was used to train an ML model, supporting the claim that models can leak information about training data. The paper does not establish that models reproduce memorized training data, and the passages on its experiments concern CNNs and other classifiers, not LLMs. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 63 · Cited by 873

Research works printed but not linked: 50

  1. M. Backes, P. Berrang, M. Bieg, R. Eils, C. Herrmann, M. Humbert, and I. Lehmann, "Identifying Personal DNA Methylation Profiles by Genotype Inference," in Proceedings of the 2017 IEEE Symposium on Security and Privacy (S&P). IEEE, 2017, pp. 957-976. 13, 14
  2. M. Backes, P. Berrang, M. Humbert, and P. Manoharan, "Membership Privacy in MicroRNA-based Studies," in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2016, pp. 319-330. 1, 13
  3. M. Backes, M. Humbert, J. Pang, and Y. Zhang, "walk2friends: Inferring Social Links from Mobility Profiles," in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2017, pp. 1943-1957. 9
  4. P. Berrang, M. Humbert, Y. Zhang, I. Lehmann, R. Eils, and M. Backes, "Dissecting Privacy Risks in Biomedical Data," in Proceedings of the 2018 IEEE European Symposium on Security and Privacy (Euro S&P). IEEE, 2018. 8
  5. K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, "Practical Secure Ag- gregation for Privacy-Preserving Machine Learning," in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2017, pp. 1175-1191. 13, 14
  6. R. Bost, R. A. Popa, S. Tu, and S. Goldwasser, "Machine Learning Classification over Encrypted Data," in Proceedings of the 2015 Net- work and Distributed System Security Symposium (NDSS). Internet Society, 2015. 13, 14
  7. X. Cao and N. Z. Gong, "Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classification," in Proceedings of the 2017 Annual Computer Security Applications Conference (ACSAC). ACM, 2017, pp. 278-287. 14
  8. N. Carlini and D. Wagner, "Towards Evaluating the Robustness of Neural Networks," in Proceedings of the 2017 IEEE Symposium on Security and Privacy (S&P). IEEE, 2017, pp. 39-57. 13
  9. Q. Chen, C. Xiang, M. Xue, B. Li, N. Borisov, D. Kaarfar, and H. Zhu, "Differentially Private Data Generative Models," CoRR abs/1812.02274, 2018. 14 · arXiv
  10. N. Dowlin, R. Gilad-Bachrach, K. Laine, K. Lauter, M. Naehrig, and J. Wernsing, "CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy," in Proceedings of the 2016 International Conference on Machine Learning (ICML). JMLR, 2016, pp. 201-210. 13
  11. Z. Erkin, M. Franz, J. Guajardo, S. Katzenbeisser, I. Lagendijk, and T. Toft, "Privacy-Preserving Face Recognition," Symposium on Privacy Enhancing Technologies Symposium, 2009. 13
  12. M. Fredrikson, S. Jha, and T. Ristenpart, "Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures," in Proceedings of the 2015 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2015, pp. 1322-1333. 1, 13
  13. M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, and T. Ristenpart, "Privacy in Pharmacogenetics: An End-to-End Case Study of Person- alized Warfarin Dosing," in Proceedings of the 2014 USENIX Security Symposium (USENIX Security). USENIX, 2014, pp. 17-32. 9, 13
  14. A. Gascón, P. Schoppmann, B. Balle, M. Raykova, J. Doerner, S. Zahur, and D. Evans, "Privacy-Preserving Distributed Linear Regression on High-Dimensional Data," Symposium on Privacy Enhancing Technolo- gies Symposium, 2017. 13
  15. I. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and Harness- ing Adversarial Examples," in Proceedings of the 2015 International Conference on Learning Representations (ICLR), 2015. 1
  16. L. Hanzlik, Y. Zhang, K. Grosse, A. Salem, M. Augustin, M. Backes, and M. Fritz, "MLCapsule: Guarded Offline Deployment of Machine Learning as a Service," CoRR abs/1808.00590, 2018. 14 · arXiv
  17. J. Hayes, L. Melis, G. Danezis, and E. D. Cristofaro, "LOGAN: Evaluating Privacy Leakage of Generative Models Using Generative Adversarial Networks," CoRR abs/1705.07663, 2017. 13 · arXiv
  18. N. Homer, S. Szelinger, M. Redman, D. Duggan, W. Tembe, J. Muehling, J. V. Pearson, D. A. Stephan, S. F. Nelson, and D. W. Craig, "Resolving Individuals Contributing Trace Amounts of DNA to Highly Complex Mixtures Using High-Density SNP Genotyping Microarrays," PLOS Genetics, 2008. 13
  19. Y. Huang, L. Malka, D. Evans, and J. Katz, "Efficient Privacy- Preserving Biometric Identification," in Proceedings of the 2011 Net- work and Distributed System Security Symposium (NDSS). Internet Society, 2011. 13
  20. T. Hunt, C. Song, R. Shokri, V. Shmatikov, and E. Witchel, "Ch- iron: Privacy-preserving Machine Learning as a Service," CoRR abs/1803.05961, 2018. 14 · arXiv
  21. M. Jagielski, A. Oprea, B. Biggio, C. Liu, C. Nita-Rotaru, and B. Li, "Manipulating Machine Learning: Poisoning Attacks and Countermea- sures for Regression Learning," in Proceedings of the 2018 IEEE Symposium on Security and Privacy (S&P). IEEE, 2018. 14
  22. J. Jia and N. Z. Gong, "AttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine Learning," in Pro- ceedings of the 2018 USENIX Security Symposium (USENIX Security). USENIX, 2018. 13
  23. B. Li and Y. Vorobeychik, "Scalable Optimization of Randomized Oper- ational Decisions in Adversarial Classification Settings," in Proceedings of the 2015 International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2015, pp. 599-607. 13
  24. J. Liu, M. Juuti, Y. Lu, and N. Asokan, "Oblivious Neural Network Predictions via MiniONN Transformations," in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2017, pp. 619-631. 13
  25. Y. Long, V. Bindschaedler, and C. A. Gunter, "Towards Measuring Membership Privacy," CoRR abs/1712.09136, 2017. 13 · arXiv
  26. Y. Long, V. Bindschaedler, L. Wang, D. Bu, X. Wang, H. Tang, C. A. Gunter, and K. Chen, "Understanding Membership Inferences on Well- Generalized Learning Models," CoRR abs/1802.04889, 2018. 13 · arXiv
  27. L. Melis, C. Song, E. D. Cristofaro, and V. Shmatikov, "Inference Attacks Against Collaborative Learning," CoRR abs/1805.04049, 2018. 14 · arXiv
  28. P. Mohassel and Y. Zhang, "SecureML: A System for Scalable Privacy- Preserving Machine Learning," in Proceedings of the 2017 IEEE Symposium on Security and Privacy (S&P). IEEE, 2017, pp. 19-38. 13
  29. M. Nasr, R. Shokri, and A. Houmansadr, "Machine Learning with Mem- bership Privacy using Adversarial Regularization," in Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2018. 14
  30. S. J. Oh, M. Augustin, B. Schiele, and M. Fritz, "Towards Reverse- Engineering Black-Box Neural Networks," in Proceedings of the 2018 International Conference on Learning Representations (ICLR), 2018. 1, 4
  31. S. J. Oh, M. Fritz, and B. Schiele, "Adversarial Image Perturbation for Privacy Protection -A Game Theory Perspective," in Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017, pp. 1482-1491. 13
  32. J. Pang and Y. Zhang, "DeepCity: A Feature Learning Framework for Mining Location Check-Ins," in Proceedings of the 2017 International Conference on Weblogs and Social Media (ICWSM). AAAI, 2017, pp. 652-655. 9
  33. N. Papernot, P. D. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, "Practical Black-Box Attacks Against Machine Learning," in Proceedings of the 2017 ACM Asia Conference on Computer and Communications Security (ASIACCS). ACM, 2017, pp. 506-519. 13
  34. N. Papernot, P. D. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, "The Limitations of Deep Learning in Adversarial Settings," in Proceedings of the 2016 IEEE European Symposium on Security and Privacy (Euro S&P). IEEE, 2016, pp. 372-387. 13
  35. A. Pyrgelis, C. Troncoso, and E. D. Cristofaro, "Knock Knock, Who's There? Membership Inference on Aggregate Location Data," in Proceedings of the 2018 Network and Distributed System Security Symposium (NDSS). Internet Society, 2018. 1, 9, 13
  36. B. D. Rouhani, H. Chen, and F. Koushanfar, "DeepSigns: A Generic Watermarking Framework for IP Protection of Deep Learning Models," CoRR abs/1804.00750, 2018. 14 · arXiv
  37. B. D. Rouhani, M. S. Riazi, and F. Koushanfar, "DeepSecure: Scalable Provably-Secure Deep Learning," CoRR abs/1705.08963, 2017. 14 · arXiv
  38. R. Shokri, M. Stronati, C. Song, and V. Shmatikov, "Membership Inference Attacks Against Machine Learning Models," in Proceedings of the 2017 IEEE Symposium on Security and Privacy (S&P). IEEE, 2017, pp. 3-18. 1, 2, 3, 4, 6, 7, 10, 11, 13
  39. C. Song, T. Ristenpart, and V. Shmatikov, "Machine Learning Models that Remember Too Much," in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2017, pp. 587-601. 14
  40. O. Suciu, R. Mȃrginean, Y. Kaya, H. D. III, and T. Dumitras ¸, "When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks," CoRR abs/1803.06975, 2018. 14 · arXiv
  41. F. Tramer and D. Boneh, "Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware," CoRR abs/1806.03287, 2018. 14 · arXiv
  42. F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, "Ensemble Adversarial Training: Attacks and Defenses," in Proceedings of the 2017 International Conference on Learning Representations (ICLR), 2017. 13
  43. F. Tramér, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, "Stealing Machine Learning Models via Prediction APIs," in Proceedings of the 2016 USENIX Security Symposium (USENIX Security). USENIX, 2016, pp. 601-618. 1, 4, 13
  44. Y. Vorobeychik and B. Li, "Optimal Randomized Classification in Ad- versarial Settings," in Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems (AAMAS), 2014, pp. 485-492. 13
  45. B. Wang and N. Z. Gong, "Stealing Hyperparameters in Machine Learning," in Proceedings of the 2018 IEEE Symposium on Security and Privacy (S&P). IEEE, 2018. 1, 4, 14
  46. W. Xu, D. Evans, and Y. Qi, "Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks," in Proceedings of the 2018 Network and Distributed System Security Symposium (NDSS). Internet Society, 2018. 13
  47. D. Yang, D. Zhang, and B. Qu, "Participatory Cultural Mapping Based on Collective Behavior Data in Location-Based Social Networks," ACM Transactions on Intelligent Systems and Technology, 2016. 3
  48. S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha, "Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting," in Pro- ceedings of the 2018 IEEE Computer Security Foundations Symposium (CSF). IEEE, 2018. 4, 8, 13
  49. Y. Zhang, M. Humbert, T. Rahman, C.-T. Li, J. Pang, and M. Backes, "Tagvisor: A Privacy Advisor for Sharing Hashtags," in Proceedings of the 2018 Web Conference (WWW). ACM, 2018, pp. 287-296. 9, 13
  50. Y. Zhang, M. Humbert, B. Surma, P. Manoharan, J. Vreeken, and M. Backes, "CTRL+Z: Recovering Anonymized Social Graphs," CoRR abs/1711.05441, 2017. · arXiv

Authors
Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., & Backes, M.
Affiliations
  • Helmholtz Center for Information Security (DE)
  • ETH Zurich (CH)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
DOI
10.48550/arxiv.1806.01246
arXiv
1806.01246
OpenAlex
W2807096445
Cited by
873
References
63
Citations by year
2026: 77 · 2025: 126 · 2024: 141 · 2023: 143 · 2022: 116 · 2021: 127 · 2020: 97 · 2019: 38 · 2018: 7 · 2017: 1
FWCI
35.38
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Network and Distributed System Security Symposium (publishedVersion)
  • SERVAL (Université de Lausanne) (publishedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion)
  • IRIS (submittedVersion)
  • Figshare (submittedVersion, cc-by)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
  • arXiv (Cornell University)
  • CISPA Helmholtz Center
Funders
European Commission (610150, 2007–2013, FP7 2007-2013, FP7/2007)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
Keywords
membership inference attacks, machine learning privacy, training data privacy, machine learning as a service, privacy-preserving machine learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review