The DOI matches its Crossref record; 43 of the 45 references it cites were found in a registry or an index.
Towards Evaluating the Robustness of Neural Networks
Open accessAbstract
Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input x and any target classification t, it is possible to find a new input x' that is similar to x but classified as t. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from 95% to 0.5%. In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with 100% probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Carlini and Farid (2020) add imperceptible adversarial perturbations to the fake/real faces and show that even the state-of-the-art DeepFake detectors are vulnerable to both white-box and black-box attacks (Carlini and Wagner 2017; Brown et al. 2017) with significant accuracy reduction on the public datasets (Wang et al. 2020e; Frank et al. 2020).”
From Countering Malicious DeepFakes: Survey, Battleground, and Horizon (2022) · cites this paper as Carlini and Wagner 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing detection methods face challenges with generalization across datasets and vulnerability to adversarial attacks.”
From Comprehensive Evaluation of Deepfake Detection Models: Accuracy, Generalization, and Resilience to Adversarial Attacks (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Adversarial attacks have been widely studied against ML models [47,50] under both white-box [9, 21, 31, 55] and black-box assumptions [33, 43].”
From Adversarial Detection Avoidance Attacks: Evaluating the robustness of perceptual hashing-based client-side scanning (2021) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P. D., Wu, X., Jha, S., & Swami, A. (2016) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2016.41 · Cited by 3,251
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198
Deep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · Cited by 1,839
Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
Papernot, N., McDaniel, P., & Goodfellow, I. (2016) · arXiv · doi:10.48550/arxiv.1605.07277 · Cited by 1,326
A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots
Giusti, A., Guzzi, J., Ciresan, D. C., He, F.-L., Rodriguez, J. P., Fontana, F., Faessler, M., Forster, C., Schmidhuber, J., Caro, G. D., et al. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2015.2509024 · Cited by 665
Towards Deep Neural Network Architectures Robust to Adversarial ExamplesPreprint
Gu, S., & Rigazio, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.5068 · Cited by 658
All you need is a good initPreprint
Mishkin, D., & Matas, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06422 · Cited by 573
Globally Normalized Transition-Based Neural Networks
Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., & Collins, M. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1231 · Cited by 538
Malware classification with recurrent networks
Pascanu, R., Stokes, J. W., Sanossian, H. Y. Y., Marinescu, M., & Thomas, A. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178304 · Cited by 462
Large-scale malware classification using random projections and neural networks
Dahl, G. E., Stokes, J. W., Deng, L., & Yu, D. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638293 · Cited by 455
Carlini, N., Mishra, P., Vaidya, T., Zhang, Y., Sherr, M., Shields, C., Wagner, D., & Zhou, W. (2016) · USENIX Security Symposium · Cited by 455
Self-citationShaham, U., Yamada, Y., & Negahban, S. N. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.04.027 · Cited by 432
Adversarial Perturbations Against Deep Neural Networks for Malware ClassificationPreprint
Grosse, K., Papernot, N., Manoharan, P., Michael Backes, A., & McDaniel, P. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04435 · Cited by 413
Graham, B. (2014) · arXiv · doi:10.48550/arxiv.1412.6071 · Cited by 337
Learning with a Strong AdversaryPreprint
Huang, R., Xu, B. H., Schuurmans, D., & Szepesvári, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.03034 · Cited by 335
Measuring Neural Net Robustness with ConstraintsPreprint
Bastani, O., Ioannou, Y., Lampropoulos, L., Vytiniotis, D., Nori, A., & Criminisi, A. (2016) · arXiv (Cornell University) · doi:10.17863/cam.26484 · Cited by 266
Fast, Lean, and Accurate: Modeling Password Guessability Using Neural Networks
Melicher, W., Ur, B., Segreti, S. M., Komanduri, S., Bauer, L., Christin, N., & Cranor, L. F. (2016) · USENIX Security Symposium · Cited by 252
Neural Networks in Mobile Robot Motion
Janglová, D. (2004) · International Journal of Advanced Robotic Systems · doi:10.5772/5615 · Cited by 189
Yuan, Z., Lü, Y., Wang, Z., & Xue, Y. (2014) · ACM SIGCOMM Computer Communication Review · doi:10.1145/2740070.2631434 · Cited by 167
The Limitations of Deep Learning in Adversarial SettingsPreprint
Papernot, N., McDaniel, P. D., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.07528 · Cited by 160
DeepFool: a simple and accurate method to fool deep neural networksPreprint
Moosavi-Dezfooli, S.-M., Fawzi, A., & Pascal Frossard, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04599 · Cited by 154
On the Effectiveness of Defensive DistillationPreprint
Papernot, N., & McDaniel, P. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.05113 · Cited by 69
Neural Networks in Mobile Robot Motion
Janglová, D. (2005) · Pro Literatur Verlag, Germany eBooks · doi:10.5772/10672 · Cited by 11
Research works printed but not linked: 9
- papernot, 0, cleverhans v1 0 0 an adversarial machine learning library
- goodfellow, 2014, Explaining and Harnessing Adversarial Examples
- 10.1109/MSP.2012.2205597 · DOI · Paper page
- lecun, 1998, The MNIST Database of Handwritten Digits
- bringing, 2016, big neural networks to self-driving cars smartphones and drones
- maas, 2013, Rectifier nonlinearities improve neural network acoustic models, Proc ICML, 30
- szegedy, 2015, Rethinking the inception architecture for computer vision
- warde-farley, 2016, Advanced Structured Prediction
- huang, 2016, Safety verification of deep neural networks
Cited by
Sorted by citations
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Boosting Adversarial Attacks with Momentum
Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., & Li, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00957 · Cited by 3,334
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
One Pixel Attack for Fooling Deep Neural Networks
Su, J., Vargas, D. V., & Sakurai, K. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2890858 · Cited by 2,406
Metadata
- Authors
- Carlini, N., & Wagner, D.
- Affiliations
- University of California, Berkeley (US)
- Venue
- 2017 IEEE Symposium on Security and Privacy (SP)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 39-57
- ISSN
- 1081-6011 (ISSN-L)
- OpenAlex
- W2516574342
- Cited by
- 8,859
- References
- 45
- Citations by year
- 2026: 566 · 2025: 983 · 2024: 1222 · 2023: 1200 · 2022: 1111 · 2021: 1308 · 2020: 1181 · 2019: 801 · 2018: 393 · 2017: 86
- FWCI
- 306.99
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Symposium on Security and Privacy (S&P) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Cisco Systems
- International Science and Technology Center
- Multidisciplinary University Research Initiative (FA9550-12–1-0040)
- Air Force Office of Scientific Research (FA9550-, FA9550-12, FA9550‐12‐1, FA9550-12-1-0040)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- adversarial examples, defensive distillation, adversarial attacks, neural network robustness, attack transferability, distance measures, robustness analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
