CoChat
Sign Up
PreprintCited by 971Adversarial Robustness in Machine LearningMachine Learning and Data ClassificationPrivacy-Preserving Technologies in Data

Adversarial Examples Are Not Bugs, They Are Features

Open access
Ilyas, Andrew, Santurkar, Shibani, Tsipras, Dimitris, Engstrom, Logan, Tran, Brandon, Madry, Aleksander·2019·arXiv
DOI 10.48550/arxiv.1905.02175arXiv 1905.02175
Open this paper in CoChat — free accountOpen PDF

Abstract

Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and pervasiveness remain unclear. We demonstrate that adversarial examples can be directly attributed to the presence of non-robust features: features derived from patterns in the data distribution that are highly predictive, yet brittle and incomprehensible to humans. After capturing these features within a theoretical framework, we establish their widespread existence in standard datasets. Finally, we present a simple setting where we can rigorously tie the phenomena we observe in practice to a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1905.02175

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Ilyas et al. proved that robust models held a more plausible explanation than non-robust models.

    “Ilyas et al. [151] classified features based on their robustness i.e., the ability to counter adversarial examples, and proved that robust models held a more plausible explanation than non-robust models.”

    From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [151]

    Section 2 (p. 2) defines robust and non-robust features, and Section 4 (p. 9) shows that, in a specific theoretical model, a robust classifier's gradient can align better with the direction separating classes. The Conclusion (p. 10) links non-robust features to difficulty producing human-meaningful, faithful explanations. These passages support a connection between robustness and interpretability, but do not prove that robust models generally provide more plausible explanations than non-robust models. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. [Din+19] Gavin Weiguang Ding et al. 'On the Sensitivity of Adversarial Robustness to Input Data Distributions'. In: International Conference on Learning Representations . 2019.
  2. [Eng+19a] Logan Engstrom et al. 'A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features''. In: Distill (2019). https://distill.pub/2019/advex-bugs-discussion. DOI: 10 . 23915 / distill.00019 .
  3. [Eng+19b] Logan Engstrom et al. 'A Rotation and a Translation Suffice: Fooling CNNs with Simple Transformations'. In: International Conference on Machine Learning (ICML) . 2019.
  4. [FFF18] Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. 'Adversarial vulnerability for any classifier'. In: Advances in Neural Information Processing Systems (NeuRIPS) . 2018.
  5. [FMF16] Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. 'Robustness of classifiers: from adversarial to random noise'. In: Advances in Neural Information Processing Systems . 2016.
  6. [For+19] Nic Ford et al. 'Adversarial Examples Are a Natural Consequence of Test Error in Noise'. In: arXiv preprint arXiv:1901.10513 . 2019. · arXiv
  7. [Fur+18] Tommaso Furlanello et al. 'Born-Again Neural Networks'. In: International Conference on Machine Learning (ICML) . 2018.
  8. [Gil+18] Justin Gilmer et al. 'Adversarial spheres'. In: Workshop of International Conference on Learning Representations (ICLR) . 2018.
  9. [Goh19a] Gabriel Goh. 'A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Robust Feature Leakage'. In: Distill (2019). https://distill.pub/2019/advex-bugs-discussion/response2. DOI: 10.23915/distill.00019.2 . · DOI
  10. [Goh19b] Gabriel Goh. 'A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Two Examples of Useful, Non-Robust Features'. In: Distill (2019). https://distill.pub/2019/advexbugs-discussion/response-3. DOI: 10.23915/distill.00019.3 . · DOI
  11. [GSS15] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 'Explaining and Harnessing Adversarial Examples'. In: International Conference on Learning Representations (ICLR) . 2015.
  12. [HD19] Dan Hendrycks and Thomas G. Dietterich. 'Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations'. In: International Conference on Learning Representations (ICLR) . 2019.
  13. [He+17] Warren He et al. 'Adversarial example defense: Ensembles of weak defenses are not strong'. In: USENIX Workshop on Offensive Technologies (WOOT) . 2017.
  14. [HVD14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 'Distilling the Knowledge in a Neural Network'. In: Neural Information Processing Systems (NeurIPS) Deep Learning Workshop . 2014.
  15. [JLT18] Saumya Jetley, Nicholas Lord, and Philip Torr. 'With friends like these, who needs adversaries?' In: Advances in Neural Information Processing Systems (NeurIPS) . 2018.
  16. [KSJ19] Beomsu Kim, Junghoon Seo, and Taegyun Jeon. 'Bridging Adversarial Robustness and Gradient Interpretability'. In: International Conference on Learning Representations Workshop on Safe Machine Learning (ICLR SafeML) . 2019.
  17. [Lec+19] Mathias Lecuyer et al. 'Certified robustness to adversarial examples with differential privacy'. In: Symposium on Security and Privacy (SP) . 2019.
  18. [Liu+17] Yanpei Liu et al. 'Delving into Transferable Adversarial Examples and Black-box Attacks'. In: International Conference on Learning Representations (ICLR) . 2017.
  19. [LM00] Beatrice Laurent and Pascal Massart. 'Adaptive estimation of a quadratic functional by model selection'. In: Annals of Statistics . 2000.
  20. [Mad+18] Aleksander Madry et al. 'Towards deep learning models resistant to adversarial attacks'. In: International Conference on Learning Representations (ICLR) . 2018.
  21. [MDM18] Saeed Mahloujifar, Dimitrios I Diochnos, and Mohammad Mahmoody. 'The curse of concentration in robust learning: Evasion and poisoning attacks from concentration of measure'. In: AAAI Conference on Artificial Intelligence (AAAI) . 2018.
  22. [MV15] Aravindh Mahendran and Andrea Vedaldi. 'Understanding deep image representations by inverting them'. In: computer vision and pattern recognition (CVPR) . 2015.
  23. [Nak19a] Preetum Nakkiran. 'A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Examples are Just Bugs, Too'. In: Distill (2019). https://distill.pub/2019/advexbugs-discussion/response-5. DOI: 10.23915/distill.00019.5 . · DOI
  24. [Nak19b] Preetum Nakkiran. 'Adversarial robustness may be at odds with simplicity'. In: arXiv preprint arXiv:1901.00532 . 2019. · arXiv
  25. [Pap+17] Nicolas Papernot et al. 'Practical black-box attacks against machine learning'. In: Asia Conference on Computer and Communications Security . 2017.
  26. [PMG16] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. 'Transferability in Machine Learning: from Phenomena to Black-box Attacks using Adversarial Samples'. In: ArXiv preprint arXiv:1605.07277 . 2016. · arXiv
  27. [Rec+19] Benjamin Recht et al. 'Do CIFAR-10 Classifiers Generalize to CIFAR-10?' In: International Conference on Machine Learning (ICML) . 2019.
  28. [RSL18] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 'Certified defenses against adversarial examples'. In: International Conference on Learning Representations (ICLR) . 2018.
  29. [Sch+18] Ludwig Schmidt et al. 'Adversarially Robust Generalization Requires More Data'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2018.
  30. [Sha+19a] Ali Shafahi et al. 'Are adversarial examples inevitable?' In: International Conference on Learning Representations (ICLR) . 2019.
  31. [Sha+19b] Adi Shamir et al. 'A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance'. In: arXiv preprint arXiv:1901.10861 . 2019. · arXiv
  32. [SHS19] David Stutz, Matthias Hein, and Bernt Schiele. 'Disentangling Adversarial Robustness and Generalization'. In: Computer Vision and Pattern Recognition (CVPR) . 2019.
  33. [Smi+17] D. Smilkov et al. 'SmoothGrad: removing noise by adding noise'. In: ICML workshop on visualization for deep learning . 2017.
  34. [Sug+19] Arun Sai Suggala et al. 'Revisiting Adversarial Risk'. In: Conference on Artificial Intelligence and Statistics (AISTATS) . 2019.
  35. [Sze+14] Christian Szegedy et al. 'Intriguing properties of neural networks'. In: International Conference on Learning Representations (ICLR) . 2014.
  36. [TG16] Thomas Tanay and Lewis Griffin. 'A Boundary Tilting Perspective on the Phenomenon of Adversarial Examples'. In: ArXiv preprint arXiv:1608.07690 . 2016. · arXiv
  37. [Tra+17] Florian Tramer et al. 'The Space of Transferable Adversarial Examples'. In: ArXiv preprint arXiv:1704.03453 . 2017. · arXiv
  38. [Tsi+19] Dimitris Tsipras et al. 'Robustness May Be at Odds with Accuracy'. In: International Conference on Learning Representations (ICLR) . 2019.
  39. [Ues+18] Jonathan Uesato et al. 'Adversarial Risk and the Dangers of Evaluating Against Weak Attacks'. In: International Conference on Machine Learning (ICML) . 2018.
  40. [Wan+18] Tongzhou Wang et al. 'Dataset Distillation'. In: ArXiv preprint arXiv:1811.10959 . 2018. · arXiv
  41. [WK18] Eric Wong and J Zico Kolter. 'Provable defenses against adversarial examples via the convex outer adversarial polytope'. In: International Conference on Machine Learning (ICML) . 2018.
  42. [Xia+19] Kai Y. Xiao et al. 'Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability'. In: International Conference on Learning Representations (ICLR) . 2019.
  43. [Zou+18] Haosheng Zou et al. 'Geometric Universality of Adversarial Examples in Deep Learning'. In: Geometry in Machine Learning ICML Workshop (GIML) . 2018.

Cited by

Sorted by citations

Authors
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., & Madry, A.
Affiliations
Massachusetts Institute of Technology (US)
ORCID
Ilyas, Andrew
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
32
Pages
125-136
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1905.02175
arXiv
1905.02175
OpenAlex
W2942630857
Cited by
971
References
5
Citations by year
2026: 6 · 2025: 79 · 2024: 163 · 2023: 177 · 2022: 147 · 2021: 203 · 2020: 143 · 2019: 52 · 2018: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Funders
  • National Science Foundation (1607189, 1553428, 1815221, 1413920, 1563880, 1447786)
  • Intel Corporation
  • Microsoft
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
Keywords
adversarial examples, robustness, interpretability, machine learning robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review