The DOI matches its Crossref record; 50 of the 55 references it cites were found in a registry or an index.
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
Open accessAbstract
Machine learning algorithms, when applied to sensitive data, pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through the models' structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role. This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks and begins to shed light on what other factors may be in play. Finally, we explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous studies [45, 64, 75] have shown that overfitted models are more susceptible to classical membership inference attacks, while well-generalized models are almost immune to them.”
From When Machine Unlearning Jeopardizes Privacy (2021) · cites this paper as [75]
In the citing paper
“We emphasize that membership inference is the most wellestablished privacy attack and arguably constitutes a bigger privacy threat [13,28,30,37,46,48,60,66,67,75] than classlevel attacks, such as the attacks in [7, 59] and model inversion [18, 19].”
From When Machine Unlearning Jeopardizes Privacy (2021) · cites this paper as [75]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
(2012) · Cited by 9,062
Dwork, C. (2006) · Lecture notes in computer science · doi:10.1007/11787006_1 · Cited by 6,125
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961
Self-citationUnderstanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Estimation of the Warfarin Dose with Clinical and Pharmacogenetic Data
The International Warfarin Pharmacogenetics Consortium (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0809329 · Cited by 1,573
Identifying Personal Genomes by Surname Inference
Gymrek, M., McGuire, A. L., Golan, D., Halperin, E., & Erlich, Y. (2013) · Science · doi:10.1126/science.1229566 · Cited by 1,228
Homer, N., Szelinger, S., Redman, M., Duggan, D., Tembe, W., Muehling, J., Pearson, J. V., Stephan, D. A., Nelson, S. F., & Craig, D. W. (2008) · PLoS Genetics · doi:10.1371/journal.pgen.1000167 · Cited by 1,221
Revealing information while preserving privacy
Dinur, I., & Kobbi Nissim, B. (2003) · ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS) · doi:10.1145/773153.773173 · Cited by 993
(2011) · PubMed · Cited by 990
O’Donnell, R. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139814782 · Cited by 984
Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds
Bassily, R., Smith, A., & Thakurta, A. (2014) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2014.56 · Cited by 695
Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing.
Fredrikson, M., Lantz, E., Jha, S., Lin, S. M., Page, D., & Ristenpart, T. (2014) · PubMed · Cited by 658
Self-citationMachine Learning Models that Remember Too Much
Song, C., Ristenpart, T., & Shmatikov, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134077 · Cited by 526
Ateniese, G., Mancini, L. V., Spognardi, A., Villani, A., Vitali, D., & Felici, G. (2015) · International Journal of Security and Networks · doi:10.1504/ijsn.2015.071829 · Cited by 518
Bousquet, O., Elisseeff, A., & Ron, D. (2001) · MPG.PuRe (Max Planck Society) · Cited by 493
A Systematic Review of Re-Identification Attacks on Health Data
El Emam, K., Jonker, E., Arbuckle, L., & Malin, B. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028071 · Cited by 427
Zhang, J., Zhang, Z., Xiao, X., Yang, Y., & Winslett, M. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2350229.2350253 · Cited by 417
Brickell, J., & Shmatikov, V. (2008) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1401890.1401904 · Cited by 332
Learnability, Stability and Uniform Convergence
Shalev‐Shwartz, S., Shamir, O., Srebro, N., & Sridharan, K. (2010) · doi:10.5555/1756006.1953019 · Cited by 328
"You Might Also Like:" Privacy Risks of Collaborative Filtering
Calandrino, J. A., Kilzer, A., Narayanan, A., Felten, E. W., & Shmatikov, V. (2011) · 2011 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2011.40 · Cited by 311
Preserving Statistical Validity in Adaptive Data Analysis
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/2746539.2746580 · Cited by 291
Regulation of gene expression in the mammalian eye and its relevance to eye disease
Scheetz, T. E., Kim, K. A., Swiderski, R. E., Philp, A. R., Braun, T. A., Knudtson, K. L., Dorrance, A. M., DiBona, G. F., Huang, J., Casavant, T. L., et al. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0602562103 · Cited by 283
The price of privacy and the limits of LP decoding
Dwork, C., McSherry, F., & Talwar, K. (2007) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/1250790.1250804 · Cited by 223
Privacy risks from genomic data-sharing beacons
Shringarpure, S. S., & Bustamante, C. D. (2015) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2015.09.010 · Cited by 211
Data-dependent Initializations of Convolutional Neural NetworksPreprint
Krähenbühl, P., Doersch, C., Donahue, J., & Darrell, T. (2015) · International Conference on Learning Representations · doi:10.48550/arxiv.1511.06856 · Cited by 211
Learning your identity and disease from research papers
Wang, R., Li, Y. F., Wang, X., Tang, H., & Zhou, X. (2009) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/1653662.1653726 · Cited by 209
Security of Symmetric Encryption against Mass Surveillance
Bellare, M., Paterson, K. G., & Rogaway, P. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44371-2_1 · Cited by 201
Genomic privacy and limits of individual detection in a pool
Sankararaman, S., Obozinski, G., Jordan, M. I., & Halperin, E. (2009) · Nature Genetics · doi:10.1038/ng.436 · Cited by 197
Algorithmic stability for adaptive data analysis
Bassily, R., Kobbi Nissim, B., Smith, A., Steinke, T., Stemmer, U., & Ullman, J. (2016) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/2897518.2897566 · Cited by 179
A Methodology for Formalizing Model-Inversion Attacks
Wu, X., Fredrikson, M., Jha, S., & Naughton, J. F. (2016) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf.2016.32 · Cited by 159
Self-citationGeneralization in Adaptive Data Analysis and Holdout Reuse
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015) · arXiv · doi:10.48550/arxiv.1506.02629 · Cited by 152
Li, N., Qardaji, W. H., Su, D., Wu, Y., & Yang, W. (2013) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2508859.2516686 · Cited by 133
Differentially Private Feature Selection via Stability Arguments, and the Robustness of the Lasso
Thakurta, A., & Smith, A. (2013) · Conference on Learning Theory · Cited by 111
Subversion-Resilient Signature Schemes
Ateniese, G., Magri, B., & Venturi, D. (2015) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2810103.2813635 · Cited by 110
Kasiviswanathan, S. P., Rudelson, M., Smith, A., & Ullman, J. (2010) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/1806689.1806795 · Cited by 107
Personal privacy vs population privacy
Cormode, G. (2011) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2020408.2020598 · Cited by 98
Mass-surveillance without the State
Bellare, M., Jaeger, J., & Kane, D. M. (2015) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2810103.2813681 · Cited by 90
Differentially Private M-Estimators
Lei, L., & Jing, J. (2011) · Cited by 87
Economic Analysis of the Digital Economy
Avi Goldfarb, B., Greenstein, S. M., & Tucker, C. E. (2015) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226206981.001.0001 · Cited by 84
Wang, Y.-X., Lei, J., & Fienberg, S. E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.06309 · Cited by 79
On-Average KL-Privacy and Its Equivalence to Generalization for Max-Entropy Mechanisms
Wang, Y.-X., Lei, J., & Fienberg, S. E. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45381-1_10 · Cited by 48
The Power of Linear Reconstruction Attacks
Kasiviswanathan, S. P., Rudelson, M., & Smith, A. (2013) · ACM-SIAM Symposium on Discrete Algorithms (SODA) · doi:10.1137/1.9781611973105.102 · Cited by 26
Revisiting Differentially Private Regression: Lessons From Learning Theory and their ConsequencesPreprint
Wu, X., Fredrikson, M., Wu, W., Jha, S., & Naughton, J. F. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.06388 · Cited by 18
Self-citationSecurity of Linear Secret-Sharing Schemes Against Mass Surveillance
Giacomelli, I., Olimid, R. F., & Ranellucci, S. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-26823-1_4 · Cited by 7
Self-citationChapter 10 - Estimation of Treatment Effects from Combined Data
Komarova, T. V., Nekipelov, D., & Yakovlev, E. (2015) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226206981.003.0010 · Cited by 4
Research works printed but not linked: 6
- goodfellow, 2016, Deep Learning
- chollet, 2017, Keras Deep learning library for theano and tensorflow
- 2006, Netfix Prize
- yeom, 2017, Privacy risk in machine learning: Analyzing the connection to overfitting, CoRR
- lecun, 1998, The MNIST Database of Handwritten Digits
- neuberg, 2017, Personal photos model
Cited by
Sorted by citations
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
“This can be approached theoretically as in [425], although this often yields loose, vacuous bounds for realistic models.”
BackgroundExploiting Unintended Feature Leakage in Collaborative Learning
Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00029 · Cited by 1,734
Nasr, M., Shokri, R., & Houmansadr, A. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00065 · Cited by 1,760
Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., & Backes, M. (2019) · Proceedings 2019 Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23119 · Cited by 852
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
Metadata
- Authors
- Yeom, S., Giacomelli, I., Fredrikson, M., & Jha, S.
- Affiliations
- Carnegie Mellon University (US)
- University of Wisconsin–Madison (US)
- Venue
- 2018 IEEE 31st Computer Security Foundations Symposium (CSF)
- Publisher
- IEEE
- Published
- 2018-07
- Pages
- 268-282
- ISSN
- 1063-6900 (ISSN-L)
- OpenAlex
- W2795435272
- Cited by
- 1,001
- References
- 55
- Citations by year
- 2026: 114 · 2025: 176 · 2024: 141 · 2023: 155 · 2022: 144 · 2021: 137 · 2020: 81 · 2019: 43 · 2018: 8
- FWCI
- 27.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Security Foundations Workshop (CSFW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Keywords
- machine learning privacy, overfitting, membership inference attacks, attribute inference attacks, training data leakage, sensitive data, privacy risks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
