CoChat
Sign Up
PreprintCited by 479Adversarial Robustness in Machine LearningPrivacy-Preserving Technologies in DataSecurity and Verification in Computing

Certified Defenses for Data Poisoning Attacks

Open access
Steinhardt, Jacob, Koh, Pang Wei, Liang, Percy·2017·arXiv
DOI 10.48550/arxiv.1706.03691arXiv 1706.03691
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine learning systems trained on user-provided data are susceptible to data poisoning attacks, whereby malicious users inject false training data with the aim of corrupting the learned model. While recent work has proposed a number of attacks and defenses, little is understood about the worst-case loss of a defense in the face of a determined attacker. We address this by constructing approximate upper bounds on the loss across a broad family of attacks, for defenders that first perform outlier removal followed by empirical risk minimization. Our approximation relies on two assumptions: (1) that the dataset is large enough for statistical concentration between train and test error to hold, and (2) that outliers within the clean (non-poisoned) data do not have a strong effect on the model. Our bound comes paired with a candidate attack that often nearly matches the upper bound, giving us a powerful tool for quickly assessing defenses on a given dataset. Empirically, we find that even under a simple defense, the MNIST-1-7 and Dogfish datasets are resilient to attack, while in contrast the IMDB sentiment dataset can be driven from 12% to 23% test error by adding only 3% poisoned data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1706.03691

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Introducing just 3 percent poisoned data can escalate the test error from 12 percent to 23 percent in the IMDB sentiment dataset.

    “Another concern is poisoned data, which can drastically degrade model performance; for instance, experiments with the IMDB sentiment dataset have demonstrated that introducing just 3 percent poisoned data can escalate the test error from 12 percent to 23 percent [174, 175].”

    From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [174]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. P. Awasthi, M. F. Balcan, and P. M. Long. The power of localization for efficiently learning linear separators with noise. In Symposium on Theory of Computing (STOC) , pages 449-458, 2014.
  2. J. F. Bard. Practical Bilevel Optimization: Algorithms and Applications . Springer, 1999.
  3. M. Barreno, B. Nelson, A. D. Joseph, and J. D. Tygar. The security of machine learning. Machine Learning , 81(2):121-148, 2010.
  4. V. Behzadan and A. Munir. Vulnerability of deep reinforcement learning to policy induction attacks. arXiv , 2017.
  5. K. Bhatia, P. Jain, and P. Kar. Robust regression via hard thresholding. In Advances in Neural Information Processing Systems (NIPS) , pages 721-729, 2015.
  6. B. Biggio, B. Nelson, and P. Laskov. Poisoning attacks against support vector machines. In International Conference on Machine Learning (ICML) , pages 1467-1474, 2012.
  7. B. Biggio, I. Pillai, S. R. Bulò, D. Ariu, M. Pelillo, and F. Roli. Is data clustering in adversarial settings secure? In Workshop on Artificial Intelligence and Security (AISec) , 2013.
  8. B. Biggio, G. Fumera, and F. Roli. Security evaluation of pattern classifiers under attack. IEEE Transactions on Knowledge and Data Engineering , 26(4):984-996, 2014a.
  9. B. Biggio, K. Rieck, D. Ariu, C. Wressnegger, I. Corona, G. Giacinto, and F. Roli. Poisoning behavioral malware clustering. In Workshop on Artificial Intelligence and Security (AISec) , 2014b.
  10. B. Biggio, B. S. Rota, P. Ignazio, M. Michele, M. E. Zemene, P. Marcello, and R. Fabio. Poisoning complete-linkage hierarchical clustering. In Workshop on Structural, Syntactic, and Statistical Pattern Recognition , 2014c.
  11. M. A. Bishop. The art and science of computer security . Addison-Wesley Longman Publishing Co., Inc., 2002.
  12. M. Brückner and T. Scheffer. Stackelberg games for adversarial prediction problems. In SIGKDD , pages 547-555, 2011.
  13. M. Brückner, C. Kanzow, and T. Scheffer. Static prediction games for adversarial learning problems. Journal of Machine Learning Research (JMLR) , 13:2617-2654, 2012.
  14. C. Burkard and B. Lagesse. Analysis of causative attacks against SVMs learning from data streams. In International Workshop on Security And Privacy Analytics , 2017.
  15. N. Carlini, P. Mishra, T. Vaidya, Y. Zhang, M. Sherr, C. Shields, D. Wagner, and W. Zhou. Hidden voice commands. In USENIX Security , 2016.
  16. M. Charikar, J. Steinhardt, and G. Valiant. Learning from untrusted data. In Symposium on Theory of Computing (STOC) , 2017.
  17. Y. Chen, C. Caramanis, and S. Mannor. Robust high dimensional sparse regression and matching pursuit. arXiv , 2013.
  18. S. Diamond and S. Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research (JMLR) , 17(83):1-5, 2016.
  19. J. Gardiner and S. Nagaraja. On the security of machine learning in malware c&c detection: A survey. ACM Computing Surveys (CSUR) , 49(3), 2016.
  20. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS) , 2014.
  21. I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR) , 2015.
  22. S. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel. Adversarial attacks on neural network policies. arXiv , 2017.
  23. S. M. Kakade, K. Sridharan, and A. Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. In Advances in Neural Information Processing Systems (NIPS) , 2009.
  24. A. Kerckhoffs. La cryptographie militaire. Journal des sciences militaires , 9, 1883.
  25. A. R. Klivans, P. M. Long, and R. A. Servedio. Learning halfspaces with malicious noise. Journal of Machine Learning Research (JMLR) , 10:2715-2740, 2009.
  26. P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning (ICML) , 2017.
  27. A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv , 2016.
  28. R. Laishram and V. V. Phoha. Curie: A method for protecting SVM classifier from poisoning attack. arXiv , 2016.
  29. A. Lakhina, M. Crovella, and C. Diot. Diagnosing network-wide traffic anomalies. In ACM SIGCOMM Computer Communication Review , volume 34, pages 219-230, 2004.
  30. P. Laskov and N. Šrndi` c. Practical evasion of a learning-based classifier: A case study. In Symposium on Security and Privacy , 2014.
  31. B. Li, Y. Wang, A. Singh, and Y. Vorobeychik. Data poisoning attacks on factorization-based collaborative filtering. In Advances in Neural Information Processing Systems (NIPS) , 2016.
  32. Y. Lin, Z. Hong, Y. Liao, M. Shih, M. Liu, and M. Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv , 2017.
  33. J. Liu and X. Zhu. The teaching dimension of linear learners. Journal of Machine Learning Research (JMLR) , 17(162), 2016.
  34. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Association for Computational Linguistics (ACL) , 2011.
  35. S. Mei and X. Zhu. The security of latent Dirichlet allocation. In Artificial Intelligence and Statistics (AISTATS) , 2015a.
  36. V. Metsis, I. Androutsopoulos, and G. Paliouras. Spam filtering with naive Bayes - which naive Bayes? In CEAS , volume 17, pages 28-69, 2006.
  37. M. Mozaffari-Kermani, S. Sur-Kolay, A. Raghunathan, and N. K. Jha. Systematic poisoning attacks on and defenses for machine learning in healthcare. IEEE Journal of Biomedical and Health Informatics , 19(6):1893-1905, 2015.
  38. N. M. Nasrabadi, T. D. Tran, and N. Nguyen. Robust lasso with missing and grossly corrupted observations. In Advances in Neural Information Processing Systems (NIPS) , 2011.
  39. A. Newell, R. Potharaju, L. Xiang, and C. Nita-Rotaru. On the practicality of integrity attacks on document-level sentiment analysis. In Workshop on Artificial Intelligence and Security (AISec) , pages 83-93, 2014.
  40. J. Newsome, B. Karp, and D. Song. Paragraph: Thwarting signature learning by training maliciously. In International Workshop on Recent Advances in Intrusion Detection , 2006.
  41. N. H. Nguyen and T. D. Tran. Exact recoverability from dense corrupted observations via ℓ 1 -minimization. IEEE Transactions on Information Theory , 59(4):2017-2035, 2013.
  42. N. Papernot, P. McDaniel, and I. Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv , 2016a.
  43. N. Papernot, P. McDaniel, A. Sinha, and M. Wellman. Towards the science of security and privacy in machine learning. arXiv , 2016b.
  44. S. Park, J. Weimer, and I. Lee. Resilient linear classification: an approach to deal with attacks on training data. In International Conference on Cyber-Physical Systems , pages 155-164, 2017.
  45. B. Rubinstein, B. Nelson, L. Huang, A. D. Joseph, S. Lau, S. Rao, N. Taft, and J. Tygar. Antidote: Understanding and defending against poisoning of anomaly detectors. In ACM SIGCOMM Conference on Internet measurement conference , 2009.
  46. S. Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in Machine Learning , 4(2):107-194, 2011.
  47. J. Steinhardt, S. Wager, and P. Liang. The statistics of streaming sparse regression. arXiv preprint arXiv:1412.4182 , 2014. · arXiv
  48. J. Steinhardt, G. Valiant, and M. Charikar. Avoiding imposters and delinquents: Adversarial crowdsourcing and peer prediction. In Advances in Neural Information Processing Systems (NIPS) , 2016.
  49. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR) , 2014.
  50. F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart. Stealing machine learning models via prediction APIs. In USENIX Security , 2016.
  51. J. Vuurens, A. P. de Vries, and C. Eickhoff. How much spam can you take? An analysis of crowdsourcing results to increase accuracy. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval , 2011.
  52. G. Wang. Combating Attacks and Abuse in Large Online Communities . PhD thesis, University of California Santa Barbara, 2016.
  53. H. Xiao, H. Xiao, and C. Eckert. Adversarial label flips attack on support vector machines. In European Conference on Artificial Intelligence , 2012.
  54. H. Xiao, B. Biggio, G. Brown, G. Fumera, C. Eckert, and F. Roli. Is feature selection secure against training data poisoning? In International Conference on Machine Learning (ICML) , 2015a.
  55. L. Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research (JMLR) , 11:2543-2596, 2010.
  56. C. Yang, Q. Wu, H. Li, and Y. Chen. Generative poisoning attack method against neural networks. arXiv , 2017.
  57. Y. Zhou and M. Kantarcioglu. Modeling adversarial learning as nested Stackelberg games. In Pacific-Asia Conference on Knowledge Discovery and Data Mining , 2016.

Cited by

Sorted by citations

Authors
Steinhardt, J., Koh, P. W., & Liang, P.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1706.03691
arXiv
1706.03691
OpenAlex
W2626801932
Cited by
479
References
2
Citations by year
2026: 1 · 2025: 37 · 2024: 51 · 2023: 80 · 2022: 64 · 2021: 92 · 2020: 91 · 2019: 42 · 2018: 19 · 2017: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation
  • Open Philanthropy Project
  • Future of Life Institute
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
Keywords
data poisoning, defense evaluation, outlier removal, empirical risk minimization, adversarial robustness, MNIST

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review