Survey: Leakage and Privacy at Inference Time
Open accessAbstract
Leakage of data from publicly available Machine Learning (ML) models is an area of growing significance since commercial and government applications of ML can draw on multiple sources of data, potentially including users' and clients' sensitive data. We provide a comprehensive survey of contemporary advances on several fronts, covering involuntary data leakage which is natural to ML models, potential malicious leakage which is caused by privacy attacks, and currently available defence mechanisms. We focus on inference-time leakage, as the most likely scenario for publicly available models. We first discuss what leakage is in the context of different data, tasks, and model architectures. We then propose a taxonomy across involuntary and malicious leakage, followed by description of currently available defences, assessment metrics, and applications. We conclude with outstanding challenges and open questions, outlining some promising directions for future research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs can inadvertently learn and propagate biases present in training data, requiring benchmarks that can detect and mitigate such issues.
“LLMs can inadvertently learn and propagate biases present in training data, requiring benchmarks that can detect and mitigate such issues [32].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 249 · Cited by 77
Research works printed but not linked: 27
- [17][17] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, 'Understanding deep learning requires rethinking generalization,' in ICLR , 2017. -
- [28][28] V. Nagarajan, C. Raffel, and I. J. Goodfellow, 'Theoretical insights into memorization in GANs,' NeurIPS Workshop , 2018. -
- [51][51] M. Jagielski, N. Carlini, D. Berthelot, A. Kurakin, and N. Papernot, 'High accuracy and high fidelity extraction of neural networks,' in USENIX , 2020. -
- [55][55] V. Chandrasekaran, K. Chaudhuri, I. Giacomelli, S. Jha, and S. Yan, 'Exploring connections between active learning and model extraction,' in USENIX , 2020, pp. 1309-1326. -
- [69][69] L. Zhu, Z. Liu, and S. Han, in NeurIPS . -
- [75][75] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, in NeurIPS . -
- [81][81] Y. Zhang, R. Jia, H. Pei, W. Wang, B. Li, and D. Song, 'The secret revealer: Generative modelinversion attacks against deep neural networks,' IEEE/CVF CVPR , pp. 250-258, 2020. -
- [82][82] L. Zhu, Z. Liu, and S. Han, in NeurIPS , pp. 14 77414 784. -
- [85][85] Z. Wang, M. Song, Z. Zhang, Y. Song, Q. Wang, and H. Qi, 'Beyond inferring class representatives: Userlevel privacy leakage from federated learning,' IEEE INFOCOM , pp. 2512-2520, 2019. -
- [86][86] B. Jayaraman and D. Evans, 'Evaluating differentially private machine learning in practice,' in USENIX , 2019. -
- [97][97] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen, and X. Chen, in NeurIPS . -
- [109][109] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. P´ oczos, R. R. Salakhutdinov, and A. J. Smola, 'Deep sets,' in NIPS , 2017. -
- [119][119] J. Yoon, D. Jarrett, and M. van der Schaar, 'Time-series generative adversarial networks,' in NeurIPS , vol. 32. Curran Associates, Inc., 2019, pp. 5508-5518. -
- [122][122] J. Jia, A. Salem, M. Backes, Y. Zhang, and N. Z. Gong, 'MemGuard: defending against black-box member- ship inference attacks via adversarial examples,' in SIGSAC , 2019, pp. 259-274. -
- [123][123] Z. Yang, B. Shao, B. Xuan, E.-C. Chang, and F. Zhang, 'Defending model inversion and membership inference attacks via prediction purification,' ArXiv , 2020. -
- [129][129] B. Mirzasoleiman, A. Karbasi, and A. Krause, 'Deletion-robust submodular maximization: Data summarization with 'the right to be forgotten',' in ICML , vol. 70. PMLR, 2017, pp. 2449-2458. -
- [132][132] L. Bourtoule, V. Chandrasekaran, C. A. ChoquetteChoo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, 'Machine unlearning,' CoRR , 2019. -
- [136][136] C. Guo, T. Goldstein, A. Y. Hannun, and L. van der Maaten, 'Certified data removal from machine learning models,' CoRR , 2019. -
- [139][139] H. Harutyunyan, A. Achille, G. Paolini, O. Majumder, A. Ravichandran, R. Bhotika, and S. Soatto, 'Estimating informativeness of samples with smooth unique information,' in ICLR , 2021. -
- [146][146] S. Tan, R. Caruana, G. Hooker, and Y. Lou, 'Distilland-compare: Auditing black-box models using transparent model distillation,' in AAAI/ACM Conference on AI, Ethics, and Society , ser. AIES'18. ACM, 2018. -
- [158][158] I. Mironov, 'R´ enyi differential privacy,' IEEE CSF , pp. 263-275, 2017. -
- [169][169] D. Wang, M. Ye, and J. Xu, 'Differentially private empirical risk minimization revisited: Faster and more general,' in NeurIPS , 2017, pp. 2722-2731. -
- [173][173] K. S. S. Kumar and M. P. Deisenroth, 'Differentially private empirical risk minimization with sparsityinducing norms,' CoRR , 2019. -
- [175][175] J. Hamm, Y. Cao, and M. Belkin, 'Learning privately from multiparty data,' in ICML , ser. JMLR Workshop and Conference Proceedings, vol. 48. JMLR, 2016. -
- [200][200] N. Roux, M. Schmidt, and F. Bach, in NeurIPS . -
- [206][206] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, and M. Bennis, 'Advances and open problems in federated learning,' CoRR , 2019. -
- [210][210] N. Papernot, M. Abadi, ´ U. Erlingsson, I. J. Goodfellow, and K. Talwar, 'Semi-supervised knowledge transfer for deep learning from private training data,' in ICLR , 2017. -
- Authors
- Jegorova, M., Kaul, C., Mayor, C., O'Neil, A. Q., Weir, A., Murray-Smith, R., & Tsaftaris, S. A.
- Affiliations
- Meta (United Kingdom) (GB)
- University of Glasgow (GB)
- National Health Service Scotland (GB)
- Canon Medical Research Europe, Edinburgh, U.K.
- Canon Medical Research Europe (United Kingdom) (GB)
- University of Glasgow, U.K.
- The Alan Turing Institute (GB)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022
- Volume
- 45
- Issue
- 7
- Pages
- 1-20
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3179862281
- Cited by
- 77
- References
- 249
- Citations by year
- 2026: 8 · 2025: 25 · 2024: 23 · 2023: 14 · 2022: 7
- FWCI
- 5.38
- Type
- journal article
- Language
- en
- Open access
- green
- eprints.gla.ac.uk
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion, cc-by-sa)
- arXiv (Cornell University) (submittedVersion, other-oa)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- UK Research and Innovation (104690)
- Royal Academy of Engineering (RCSRF1819\8\25)
- Innovate UK (104690)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Keywords
- privacy attacks, machine learning privacy, data leakage, privacy-preserving machine learning, sensitive data protection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
