CoChat
Sign Up
Cited by 1,734Privacy-Preserving Technologies in DataAdversarial Robustness in Machine LearningSecurity and Verification in Computing

Exploiting Unintended Feature Leakage in Collaborative Learning

Open access
Melis, Luca, Song, Congzheng, De Cristofaro, Emiliano, Shmatikov, Vitaly·2019·2019 IEEE Symposium on Security and Privacy (SP)
DOI 10.1109/sp.2019.00029
Open this paper in CoChat — free accountOpen PDF

Abstract

Collaborative machine learning and related techniques such as federated learning allow multiple participants, each with his own training dataset, to build a joint model by training locally and periodically exchanging model updates. We demonstrate that these updates leak unintended information about participants' training data and develop passive and active inference attacks to exploit this leakage. First, we show that an adversarial participant can infer the presence of exact data points -- for example, specific locations -- in others' training data (i.e., membership inference). Then, we show how this adversary can infer properties that hold only for a subset of the training data and are independent of the properties that the joint model aims to capture. For example, he can infer when a specific person first appears in the photos used to train a binary gender classifier. We evaluate our attacks on a variety of tasks, datasets, and learning configurations, analyze their limitations, and discuss possible defenses.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/sp.2019.00029

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Membership inference is the most well-established privacy attack and arguably constitutes a bigger privacy threat than class-level attacks, such as the attacks in [7, 59] and model inversion [18, 19].

    “We emphasize that membership inference is the most wellestablished privacy attack and arguably constitutes a bigger privacy threat [13,28,30,37,46,48,60,66,67,75] than classlevel attacks, such as the attacks in [7, 59] and model inversion [18, 19].”

    From When Machine Unlearning Jeopardizes Privacy (2021) · cites this paper as [46]

  • Federated learning alone does not provide a meaningful privacy guarantee.

    “While FL attempts to provide privacy by keeping private data on client devices, it does not provide a meaningful privacy guarantee.”

    From Federated Learning and Differential Privacy for Medical Image Analysis (2021) · cites this paper as [17]

  • Updated model parameters sent from clients to a centralized server can contain private information, such that even individual data points can be reconstructed.

    “Updated model parameters are still sent from the clients to a centralized server, and these can contain private information 16 , such that even individual data points can be reconstructed 17 .”

    From Federated Learning and Differential Privacy for Medical Image Analysis (2021) · cites this paper as [17]

  • The leakage of unintended features will expose collaborative learning to powerful inference attacks.

    “Melis et al. [136] founded that the leakage of unintended features will expose collaborative learning to powerful inference attacks.”

    From Federated Deep Learning for Cyber Security in the Internet of Things: Concepts, Applications, and Experimental Analysis (2021) · cites this paper as [136]

  • Property inference attacks assume that the adversary has auxiliary training data that are correctly labelled with the target property.

    “Property inference attacks assume that the adversary has auxiliary training data that are correctly labelled with the target property.”

    From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [28]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. N. Carlini, C. Liu, J. Kos, Ú. Erlingsson, and D. Song. The Secret Sharer: Measuring unintended neural network memorization & extracting secrets. arXiv:1802.08232, 2018. · arXiv
  2. T. M. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanara- man. Project Adam: Building an efficient and scalable deep learning training system. In OSDI, 2014.
  3. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. Le, et al. Large scale distributed deep networks. In NIPS, 2012.
  4. H. Edwards and A. Storkey. Censoring representations with an adversary. In ICLR, 2016.
  5. M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, and T. Ristenpart. Privacy in pharmacogenetics: An end- to-end case study of personalized Warfarin dosing. In USENIX Security, 2014.
  6. I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio. Deep Learning. MIT Press, 2016.
  7. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014.
  8. J. Hamm, Y. Cao, and M. Belkin. Learning privately from multiparty data. In ICML, 2016.
  9. Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally. Deep gradient compression: Reducing the communica- tion bandwidth for distributed training. In ICLR, 2018.
  10. H. B. McMahan, E. Moore, D. Ramage, S. Hampson, et al. Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017.
  11. H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang. Learning differentially private language models without losing accuracy. In ICLR, 2018.
  12. S. J. Oh, M. Augustin, M. Fritz, and B. Schiele. Towards reverse-engineering black-box neural networks. In ICLR, 2018.
  13. S. A. Osia, A. S. Shamsabadi, A. Taheri, K. Katevas, S. Sajadmanesh, H. R. Rabiee, N. D. Lane, and H. Had- dadi. A hybrid deep learning architecture for privacy- preserving mobile analytics. arXiv:1703.02952, 2017. · arXiv
  14. N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar. Semi-supervised knowledge transfer for deep learning from private training data. In ICLR, 2017.
  15. N. Papernot, S. Song, I. Mironov, A. Raghunathan, K. Talwar, and Ú. Erlingsson. Scalable private learning with PATE. In ICLR, 2018.
  16. M. Pathak, S. Rane, and B. Raj. Multiparty differential privacy via aggregation of locally trained classifiers. In NIPS, 2010.
  17. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cour- napeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit- learn: Machine learning in Python. JMLR, 12, 2011.
  18. A. Pyrgelis, C. Troncoso, and E. De Cristofaro. Knock knock, who's there? membership inference on aggregate location data. In NDSS, 2018.
  19. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  20. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 2014.
  21. F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ris- tenpart. Stealing machine learning models via prediction APIs. In USENIX Security, 2016.
  22. L. van der Maaten and G. Hinton. Visualizing data using t-SNE. JMLR, 2008.
  23. B. Verhoeven and W. Daelemans. CLiPS Stylometry Investigation (CSI) Corpus: A Dutch corpus for the detection of age, gender, personality, sentiment and de- ception in text. In LREC, 2014.
  24. D. Yang, D. Zhang, and B. Qu. Participatory cultural mapping based on collective behavior in location based social networks. ACM TIST, 2015.
  25. M. Zinkevich, M. Weimer, L. Li, and A. J. Smola. Parallelized stochastic gradient descent. In NIPS, 2010.

Cited by

Sorted by citations

Related

Authors
Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V.
Affiliations
  • University College London (GB)
  • Cornell University (US)
  • Cornell Tech (US)
ORCID
Venue
2019 IEEE Symposium on Security and Privacy (SP)
Publisher
IEEE
Published
2019-05
Pages
691-706
ISSN
1081-6011 (ISSN-L)
DOI
10.1109/sp.2019.00029
OpenAlex
W2898291644
Cited by
1,734
References
37
Citations by year
2026: 156 · 2025: 257 · 2024: 354 · 2023: 294 · 2022: 240 · 2021: 247 · 2020: 143 · 2019: 42 · 2018: 1
FWCI
64.37
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Symposium on Security and Privacy (S&P) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1611770, 1704296)
  • Alan Turing Institute (EP/N510129/1., N510129)
  • Engineering and Physical Sciences Research Council (N510129, EP/N510129/, EP/N510129/1, EPSRC grant EP/N510129/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
Keywords
feature leakage, collaborative machine learning, federated learning, membership inference, inference attacks, privacy attacks, data privacy, privacy-preserving machine learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review