Cited by 192Privacy-Preserving Technologies in DataAdversarial Robustness in Machine LearningPrivacy, Security, and Data Protection

Privacy Risks of General-Purpose Language Models

Open access
Pan, Xudong, Zhang, Mi, Ji, Shouling, Yang, Min·2020·2020 IEEE Symposium on Security and Privacy (SP)

Abstract

Recently, a new paradigm of building general-purpose language models (e.g., Google’s Bert and OpenAI’s GPT-2) in Natural Language Processing (NLP) for text feature extraction, a standard procedure in NLP systems that converts texts to vectors (i.e., embeddings) for downstream modeling, has arisen and starts to find its application in various downstream NLP tasks and real world systems (e.g., Google’s search engine [6]). To obtain general-purpose text embeddings, these language models have highly complicated architectures with millions of learnable parameters and are usually pretrained on billions of sentences before being utilized. As is widely recognized, such a practice indeed improves the state-of-the-art performance of many downstream NLP tasks. However, the improved utility is not for free. We find the text embeddings from general-purpose language models would capture much sensitive information from the plain text. Once being accessed by the adversary, the embeddings can be reverse-engineered to disclose sensitive information of the victims for further harassment. Although such a privacy risk can impose a real threat to the future leverage of these promising NLP tools, there are neither published attacks nor systematic evaluations by far for the mainstream industry-level language models. To bridge this gap, we present the first systematic study on the privacy risks of 8 state-of-the-art language models with 4 diverse case studies. By constructing 2 novel attack classes, our study demonstrates the aforementioned privacy risks do exist and can impose practical threats to the application of general-purpose language models on sensitive data covering identity, genome, healthcare and location. For example, we show the adversary with nearly no prior knowledge can achieve about 75% accuracy when inferring the precise disease site from Bert embeddings of patients’ medical descriptions. As possible countermeasures, we propose 4 different defenses (via rounding, differential privacy, adversarial training and subspace projection) to obfuscate the unprotected embeddings for mitigation purpose. With extensive evaluations, we also provide a preliminary analysis on the utility-privacy trade-off brought by each defense, which we hope may foster future mitigation researches.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/sp40000.2020.00095

Cites

77 of 77 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

9 of the 86 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016, http://www.deeplearningbook.org.
  2. N. Japkowicz and S. Stephen, "The class imbalance problem: A system- atic study," Intell. Data Anal., vol. 6, pp. 429-449, 2002.

Cited by

Sorted by citations

Authors
Pan, X., Zhang, M., Ji, S., & Yang, M.
Affiliations
  • Fudan University (CN)
  • Zhejiang University (CN)
Venue
2020 IEEE Symposium on Security and Privacy (SP)
Publisher
IEEE
Published
2020-05
Pages
1314-1331
ISSN
1081-6011 (ISSN-L)
OpenAlex
W3046764764
Cited by
188
References
86
Citations by year
2026: 20 · 2025: 50 · 2024: 43 · 2023: 36 · 2022: 21 · 2021: 12 · 2020: 6
FWCI
7.16
Type
conference paper
Language
en
Open access
Locations
IEEE Symposium on Security and Privacy (S&P) (publishedVersion, public-domain)
Funders
  • National Natural Science Foundation of China (61772466, U1936215, and U1836202, U1936215, U1836210, U1636204, 61972099, U1836213, U1836202, 61772466, U1736208, 2018YFB0804102)
  • Ant Financial Services Group
  • National Key Research and Development Program of China (2018YFB0804102)
  • Natural Science Foundation of Zhejiang Province (LR19F020003, U1936215)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
Keywords
privacy risks, text embeddings, sensitive information disclosure, BERT, GPT-2, differential privacy, adversarial training, subspace projection, privacy-utility trade-off

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review