Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images
Open accessAbstract
Deep neural networks (DNNs) have recently been achieving state-of-the-art performance on a variety of pattern-recognition tasks, most notably visual classification problems. Given that DNNs are now able to classify objects in images with near-human-level performance, questions naturally arise as to what differences remain between computer and human vision. A recent study revealed that changing an image (e.g. of a lion) in a way imperceptible to humans can cause a DNN to label the image as something else entirely (e.g. mislabeling a lion a library). Here we show a related result: it is easy to produce images that are completely unrecognizable to humans, but that state-of-the-art DNNs believe to be recognizable objects with 99.99% confidence (e.g. labeling with certainty that white noise static is a lion). Specifically, we take convolutional neural networks trained to perform well on either the ImageNet or MNIST datasets and then find images with evolutionary algorithms or gradient ascent that DNNs label with high confidence as belonging to each dataset class. It is possible to produce images totally unrecognizable to human eyes that DNNs believe with near certainty are familiar objects, which we call "fooling images" (more generally, fooling examples). Our results shed light on interesting differences between human vision and current DNNs, and raise questions about the generality of DNN computer vision.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The development of methods for enabling feature explainability from deep neural networks has been proposed as an essential component of a robust model validation procedure.
“The development of methods for enabling feature explainability from deep neural networks has become an active research area over the past few years, and has been proposed as an essential component of a robust model validation procedure, to ensure that the classification performance is being driven by relevant features as opposed to noise or artifacts in the data [16, 93-99].”
From EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces (2018) · cites this paper as [96]
The Discussion (p. 8) describes the paper’s method as a way to visualize learned features and says such tools help researchers understand what DNNs have learned. The paper also demonstrates confident classifications of unrecognizable images (Abstract, p. 1), but it does not propose feature explainability as an essential component of model validation or explicitly prescribe checking whether performance is driven by relevant features rather than noise or artifacts. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [1][1] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 , 2014. 2 - · arXiv
- [3][3] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. 1 -
- [4][4] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. arXiv preprint arXiv:1409.0575 , 2014. 2 - · arXiv
- [5][5] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842 , 2014. 2 Figure S4. 50 independent runs of evolution produced images that an MNIST DNN believes with 99.99% to be handwritten digits. Columns are digits. In each row are the final (best) images evolved for each class during that run. <!-- image --> Figure S5. Images found by directly maximizing an objective function consisting of the posterior probability (softmax output) added to a regularization term, here L2-regularization. Optimization begins at the ImageNet mean plus small Gaussian noise to break symmetry. When regularization is added, confidences are generally lower than 99.99% because the objective contains terms other than confidence. Here, the average is 98.591%. For clarity, images are shown with the mean subtracted. <!-- image --> Figure S6. As in Fig. S5, but with blurring and pseudo-L1-regularization, which is accomplished by setting the pixels with lowest norm to zero throughout the optimization. <!-- image --> Figure S7. As in Fig. S5, but with slightly more aggressive blurring than in Fig. S6. <!-- image --> Figure S8. Multiple images produced for each class in the manner of Fig. S5. Each column shows the result of a different local optimum, which was reached by starting at the ImageNet mean and adding different draws of small Gaussian noise. <!-- image --> Figure S9. Multiple images produced for each class in the manner of Fig. S7. Each column shows the result of a different local optimum, which was reached by starting at the ImageNet mean and adding different draws of small Gaussian noise. <!-- image --> · arXiv
Cited by
Sorted by citations
EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces
Lawhern, V. J., Solon, A. J., Waytowich, N. R., Gordon, S. M., Hung, C. P., & Lance, B. J. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aace8c · Cited by 5,115 · see citing works
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310 · see citing works
Hallucinating symmetric protein assemblies
Wicky, B. I. M., Milles, L. F., Courbet, A., Ragotte, R. J., Dauparas, J., Kinfu, E., Tipps, S., Kibler, R. D., Baek, M., DiMaio, F., et al. (2022) · Science · doi:10.1126/science.add1964 · Cited by 240 · see citing works
Waytowich, N., Lawhern, V. J., Garcia, J. O., Cummings, J., Faller, J., Sajda, P., & Vettel, J. M. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aae5d8 · Cited by 262 · see citing works
Structure learning and the posterior parietal cortex
Summerfield, C., Luyckx, F., & Sheahan, H. (2020) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2019.101717 · Cited by 118 · see citing works
- Authors
- Nguyen, A., Yosinski, J., & Clune, J.
- Affiliations
- University of Wyoming (US)
- Cornell University (US)
- ORCID
- Clune, Jeff
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2014
- Pages
- 427-436
- ISSN
- 1063-6919 (ISSN-L)
- arXiv
- 1412.1897
- OpenAlex
- W1932198206
- Cited by
- 3,258
- References
- 1
- Citations by year
- 2026: 73 · 2025: 210 · 2024: 313 · 2023: 323 · 2022: 318 · 2021: 460 · 2020: 401 · 2019: 407 · 2018: 348 · 2017: 211 · 2016: 124 · 2015: 60 · 2014: 8
- FWCI
- 139.54
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Aeronautics and Space Administration
- University of Wyoming
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- deep neural network robustness, adversarial examples, evolutionary algorithms, gradient ascent, ImageNet, MNIST, visual categorization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
