Property Inference from Poisoning
Abstract
Property inference attacks consider an adversary who has access to a trained ML model and tries to extract some global statistics of the training data. In this work, we study property inference in scenarios where the adversary can maliciously control a part of the training data (poisoning data) with the goal of increasing the leakage. Previous works on poisoning attacks focused on trying to decrease the accuracy of models. Here, for the first time, we study poisoning attacks where the goal of the adversary is to increase the information leakage of the model. We show that poisoning attacks can boost the information leakage significantly and should be considered as a stronger threat model in sensitive applications where some of the data sources may be malicious.We theoretically prove that our attack can always succeed as long as the learning algorithm used has good generalization properties. Then we experimentally evaluate our on different datasets (Census dataset, Enron email dataset, MNIST and CelebA), properties (that are present in the training data as features, that are not present as features, and properties that are uncorrelated with the rest of the training data or classification task) and model architectures (including Resnet-18 and Resnet-50). We were able to achieve high attack accuracy with relatively low poisoning rate, namely, 2–3% poisoning in most of our experiments. We also evaluated our attacks on models trained with DP and we show that even with very small values for $\epsilon$, the attack is still quite successful1.1Code is available at https://github.com/smahloujifar/PropertyInferenceFromPoisoning.git
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Model poisoning backdoor attacks during the aggregation phase can modify the aggregation process by scaling the parameter of the uploaded malicious local model.
“Model poisoning backdoor attacks during the aggregation phase can modify the aggregation process by scaling the parameter of the uploaded malicious local model [181], [182].”
From Data and model poisoning backdoor attacks on wireless federated learning, and the defense mechanisms: A comprehensive survey (2024) · cites this paper as [182]
Cites
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 24,190 · see citing works
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Calibrating Noise to Sensitivity in Private Data Analysis
Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006) · Lecture Notes in Computer Science · doi:10.1007/11681878_14 · Cited by 7,272 · see citing works
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602 · see citing works
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · see citing works
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961 · see citing works
Nasr, M., Shokri, R., & Houmansadr, A. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00065 · Cited by 1,760 · see citing works
Exploiting Unintended Feature Leakage in Collaborative Learning
Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00029 · Cited by 1,734 · see citing works
How to backdoor federated learning
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018) · arXiv · doi:10.48550/arxiv.1807.00459 · Cited by 1,495 · see citing works
Poisoning Attacks against Support Vector Machines
Biggio, B., Nelson, B., & Laskov, P. (2012) · arXiv · doi:10.48550/arxiv.1206.6389 · Cited by 1,331 · see citing works
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
Chen, X., Liu, C., Li, B., Lu, K., & Song, D. (2017) · arXiv · doi:10.48550/arxiv.1712.05526 · Cited by 1,039 · see citing works
The Enron Corpus: A New Dataset for Email Classification Research
Klimt, B., & Yang, Y. (2004) · European Conference on Machine Learning · doi:10.1007/978-3-540-30115-8_22 · Cited by 832 · see citing works
Certified Robustness to Adversarial Examples with Differential Privacy
Lecuyer, M., Atlidakis, V., Geambasu, R., Hsu, D., & Jana, S. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00044 · Cited by 750 · see citing works
Analyzing Federated Learning through an Adversarial Lens
Bhagoji, A. N., Chakraborty, S., Mittal, P., & Calo, S. (2018) · arXiv · doi:10.48550/arxiv.1811.12470 · Cited by 703 · see citing works
Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds
Bassily, R., Smith, A., & Thakurta, A. (2014) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2014.56 · Cited by 645 · see citing works
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629 · see citing works
Stealing Hyperparameters in Machine Learning
Wang, B., & Gong, N. Z. (2018) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2018.00038 · Cited by 491 · see citing works
Ateniese, G., Mancini, L. V., Spognardi, A., Villani, A., Vitali, D., & Felici, G. (2015) · International Journal of Security and Networks · doi:10.1504/ijsn.2015.071829 · Cited by 482 · see citing works
Ganju, K., Wang, Q., Yang, W., Gunter, C. A., & Borisov, N. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243834 · Cited by 475 · cited 10× · see citing works
“In this attack, as in previous work [3, 17] we assume that the adversary can sample from D−, D+.”
BackgroundMethodResultInfluentialMachine Learning with Membership Privacy using Adversarial Regularization
Nasr, M., Shokri, R., & Houmansadr, A. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243855 · Cited by 458 · see citing works
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., & Goldstein, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3162397 · Cited by 362 · see citing works
Label-Only Membership Inference AttacksPreprint
Choquette-Choo, C. A., Tramèr, F., Carlini, N., & Papernot, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14321 · Cited by 215 · see citing works
Agnostic Estimation of Mean and Covariance
Lai, K. A., Rao, A. B., & Vempala, S. (2016) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2016.76 · Cited by 208 · see citing works
Auditing Data Provenance in Text-Generation Models
Song, C., & Shmatikov, V. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330885 · Cited by 192 · see citing works
Stronger Data Poisoning Attacks Break Data Sanitization Defenses
Koh, P. W., Steinhardt, J., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1811.00741 · Cited by 189 · see citing works
Analyzing Federated Learning through an Adversarial Lens
Bhagoji, A. N., Chakraborty, S., Mittal, P., & Calo, S. B. (2019) · International Conference on Machine Learning · Cited by 188 · see citing works
Systematic Evaluation of Privacy Risks of Machine Learning ModelsPreprint
Song, L., & Mittal, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10595 · Cited by 185 · see citing works
White-box vs Black-box: Bayes Optimal Strategies for Membership InferencePreprint
Sablayrolles, A., Douze, M., Ollivier, Y., Schmid, C., & Jeǵou, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11229 · Cited by 176 · see citing works
Sever: A Robust Meta-Algorithm for Stochastic OptimizationPreprint
Diakonikolas, I., Kamath, G., Kane, D. M., Li, J. Z., Steinhardt, J., & Stewart, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02815 · Cited by 162 · see citing works
Robust Estimation via Robust Gradient Estimation
Prasad, A., Suggala, A. S., Balakrishnan, S., & Ravikumar, P. (2020) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12364 · Cited by 128 · see citing works
Mahloujifar, S., Diochnos, D. I., & Mahmoody, M. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33014536 · Cited by 96 · see citing works
Self-citationAuditing Differentially Private Machine Learning: How Private is Private SGD?Preprint
Jagielski, M., Ullman, J., & Oprea, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07709 · Cited by 93 · see citing works
Subpopulation Data Poisoning Attacks
Jagielski, M., Severi, G., Pousette Harger, N., & Oprea, A. (2021) · Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3460120.3485368 · Cited by 86 · see citing works
Blockwise p-Tampering Attacks on Cryptographic Primitives, Extractors, and Learners
Mahloujifar, S., & Mahmoody, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-70503-3_8 · Cited by 19 · see citing works
Self-citationCan Adversarially Robust Learning Leverage Computational Hardness?Preprint
Mahloujifar, S., & Mahmoody, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.01407 · Cited by 12 · see citing works
Self-citationData Poisoning Attacks in Multi-Party Learning.
Mahloujifar, S., Mahmoody, M., & Mohammed, A. (2019) · International Conference on Machine Learning · Cited by 9 · see citing works
Self-citationLower Bounds for Adversarially Robust PAC LearningPreprint
Diochnos, D. I., Mahloujifar, S., & Mahmoody, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05815 · Cited by 8 · see citing works
Self-citationTowards Privacy and Security of Deep Learning Systems: A Survey.Preprint
He, Y., Meng, G., Chen, K., et al. (2019) · arXiv (Cornell University) · Cited by 8 · see citing works
Model-Targeted Poisoning Attacks: Provable Convergence and Certified Bounds.
Suya, F., Mahloujifar, S., Evans, D., & Tian, Y. (2020) · Cited by 7 · see citing works
Self-citationLearning under p-tampering poisoning attacks
Mahloujifar, S., Diochnos, D. I., & Mahmoody, M. (2019) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-019-09675-1 · Cited by 4 · see citing works
Self-citationObliviousness Makes Poisoning Adversaries WeakerPreprint
Garg, S., Jha, S., Mahloujifar, S., Mahmoody, M., & Thakurta, A. (2020) · arXiv (Cornell University) · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Rodríguez-Barroso, N., Jiménez-López, D., Luzón, M. V., Herrera, F., & Martínez-Cámara, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.011 · Cited by 296 · see citing works
Wan, Y., Qu, Y., Ni, W., Xiang, Y., Gao, L., & Hossain, E. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3361451 · Cited by 152 · see citing works
A Systematic Review of Adversarial Machine Learning Attacks, Defensive Controls and Technologies
Malik, J., Muthalagu, R., & Pawar, P. M. (2024) · IEEE Access · doi:10.1109/access.2024.3423323 · Cited by 76 · see citing works
Survey: Leakage and Privacy at Inference Time
Jegorova, M., Kaul, C., Mayor, C., O'Neil, A. Q., Weir, A., Murray-Smith, R., & Tsaftaris, S. A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229593 · Cited by 77 · see citing works
- Authors
- Mahloujifar, S., Ghosh, E., & Chase, M.
- Affiliations
- Princeton University (US)
- Microsoft (United States) (US)
- Venue
- 2022 IEEE Symposium on Security and Privacy (SP)
- Publisher
- IEEE
- Published
- 2022-05
- Pages
- 1120-1137
- ISSN
- 1081-6011 (ISSN-L)
- OpenAlex
- W3122816307
- Cited by
- 67
- References
- 42
- Citations by year
- 2026: 2 · 2025: 12 · 2024: 28 · 2023: 14 · 2022: 8 · 2021: 3
- FWCI
- 8.57
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Symposium on Security and Privacy (S&P) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Keywords
- property inference attacks, data poisoning, information leakage, machine learning privacy, adversarial training data, differential privacy, training data extraction, ResNet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
