CoChat
Sign Up
Cited by 1,001Privacy-Preserving Technologies in DataAdversarial Robustness in Machine LearningPrivacy, Security, and Data Protection

Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Open access
Yeom, Samuel, Giacomelli, Irene, Fredrikson, Matt, Jha, Somesh·2018·2018 IEEE 31st Computer Security Foundations Symposium (CSF)
DOI 10.1109/csf.2018.00027
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning algorithms, when applied to sensitive data, pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through the models' structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role. This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks and begins to shed light on what other factors may be in play. Finally, we explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks.

Integrity signals

Registry

Crossref · 10.1109/csf.2018.00027

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Overfitted models are more susceptible to classical membership inference attacks, while well-generalized models are almost immune to them.

    “Previous studies [45, 64, 75] have shown that overfitted models are more susceptible to classical membership inference attacks, while well-generalized models are almost immune to them.”

    From When Machine Unlearning Jeopardizes Privacy (2021) · cites this paper as [75]

  • Membership inference is the most well-established privacy attack and arguably constitutes a bigger privacy threat than class-level attacks, such as the attacks in [7, 59] and model inversion [18, 19].

    “We emphasize that membership inference is the most wellestablished privacy attack and arguably constitutes a bigger privacy threat [13,28,30,37,46,48,60,66,67,75] than classlevel attacks, such as the attacks in [7, 59] and model inversion [18, 19].”

    From When Machine Unlearning Jeopardizes Privacy (2021) · cites this paper as [75]

Cites

Flagged first, then by citations

5 of the 55 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Yeom, S., Giacomelli, I., Fredrikson, M., & Jha, S.
Affiliations
  • Carnegie Mellon University (US)
  • University of Wisconsin–Madison (US)
ORCID
Venue
2018 IEEE 31st Computer Security Foundations Symposium (CSF)
Publisher
IEEE
Published
2018-07
Pages
268-282
ISSN
1063-6900 (ISSN-L)
DOI
10.1109/csf.2018.00027
OpenAlex
W2795435272
Cited by
1,001
References
55
Citations by year
2026: 114 · 2025: 176 · 2024: 141 · 2023: 155 · 2022: 144 · 2021: 137 · 2020: 81 · 2019: 43 · 2018: 8 · 2017: 2
FWCI
27.47
Type
conference paper
Language
en
Open access
Locations
  • IEEE Computer Security Foundations Workshop (CSFW) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
Keywords
machine learning privacy, overfitting, membership inference attacks, attribute inference attacks, training data leakage, sensitive data, privacy risks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review