Adversarial examples in the physical world
Open accessAbstract
Most existing machine learning classifiers are highly vulnerable to adversarial examples. An adversarial example is a sample of input data which has been modified very slightly in a way that is intended to cause a machine learning classifier to misclassify it. In many cases, these modifications can be so subtle that a human observer does not even notice the modification at all, yet the classifier still makes a mistake. Adversarial examples pose security concerns because they could be used to perform an attack on machine learning systems, even if the adversary has no access to the underlying model. Up to now, all previous work has assumed a threat model in which the adversary can feed data directly into the machine learning classifier. This is not always the case for systems operating in the physical world, for example those which are using signals from cameras and other sensors as input. This paper shows that even in such physical world scenarios, machine learning systems are vulnerable to adversarial examples. We demonstrate this by feeding adversarial images obtained from a cell-phone camera to an ImageNet Inception classifier and measuring the classification accuracy of the system. We find that a large fraction of adversarial examples are classified incorrectly even when perceived through the camera.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Adversarial attacks have been widely studied against ML models under both white-box and black-box assumptions.
“Adversarial attacks have been widely studied against ML models [47,50] under both white-box [9, 21, 31, 55] and black-box assumptions [33, 43].”
From Adversarial Detection Avoidance Attacks: Evaluating the robustness of perceptual hashing-based client-side scanning (2021) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 11
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR'2015, arXiv:1409.0473, 2015. · arXiv
- Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 387- 402. Springer, 2013.
- Nicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang, Micah Sherr, Clay Shields, David Wagner, and Wenchao Zhou. Hidden voice commands. In 25th USENIX Security Symposium (USENIX Security 16), Austin, TX, August 2016. USENIX Association. URL https://www.usenix.org/conference/usenixsecurity16/ technical-sessions/presentation/carlini.
- Nilesh Dalvi, Pedro Domingos, Sumit Sanghai, Deepak Verma, et al. Adversarial classification. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 99-108. ACM, 2004.
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS'2012). 2012.
- Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D Joseph, Benjamin IP Rubinstein, Udam Saini, Charles A Sutton, J Doug Tygar, and Kai Xia. Exploiting machine learning to subvert your spam filter. N. Papernot, P. McDaniel, and I. Goodfellow. Transferability in Machine Learning: from Phe- nomena to Black-Box Attacks using Adversarial Samples. ArXiv e-prints, May 2016b. URL http://arxiv.org/abs/1605.07277. · arXiv
- Nicolas Papernot, Patrick Drew McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against deep learning systems using adversarial examples. CoRR, abs/1602.02697, 2016a. URL http://arxiv.org/abs/1602.02697. · arXiv
- Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 23rd ACM SIGSAC Conference on Computer and Communications Security, October 2016. To appear.
- Daniel F Smith, Arnold Wiliem, and Brian C Lovell. Face recognition on consumer devices: Re- flections on replay attacks. IEEE Transactions on Information Forensics and Security, 10(4): 736-745, 2015.
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. ICLR, abs/1312.6199, 2014. URL http://arxiv.org/abs/1312.6199. · arXiv
- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. CoRR, abs/1512.00567, 2015. URL http://arxiv.org/abs/1512.00567. · arXiv
Cited by
Sorted by citations
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859 · see citing works
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · see citing works
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733 · see citing works
How to backdoor federated learning
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018) · arXiv · doi:10.48550/arxiv.1807.00459 · Cited by 1,495 · see citing works
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231 · see citing works
Feature denoising for improving adversarial robustness
Xie, C., Wu, Y., Maaten, L. V. D., Yuille, A. L., & He, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00059 · Cited by 940 · see citing works
Generative adversarial network: An overview of theory and applications
Aggarwal, A., Mittal, M., & Battineni, G. (2021) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2020.100004 · Cited by 783 · see citing works
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765 · see citing works
Certified Robustness to Adversarial Examples with Differential Privacy
Lecuyer, M., Atlidakis, V., Geambasu, R., Hsu, D., & Jana, S. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00044 · Cited by 750 · see citing works
HopSkipJumpAttack: A Query-Efficient Decision-Based Attack
Chen, J., Jordan, M. I., & Wainwright, M. J. (2020) · 2020 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40000.2020.00045 · Cited by 718 · see citing works
DÏoT: A Federated Self-learning Anomaly Detection System for IoT
Nguyen, T. D., Marchal, S., Miettinen, M., Fereidooni, H., Asokan, N., & Sadeghi, A.-R. (2019) · 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2019.00080 · Cited by 663 · see citing works
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582 · see citing works
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · see citing works
“The augmentation can be carried out in a brute-force manner by feeding both the original data and adversarial samples during training [201], or by using a regularization term to implicitly represent adversarial samples [134].”
BackgroundUnderstanding adversarial attacks on deep learning based medical image analysis systems
Ma, X., Niu, Y., Gu, L., Wang, Y., Zhao, Y., Bailey, J., & Lu, F. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2020.107332 · Cited by 553 · see citing works
Recent Advances in Adversarial Training for Adversarial Robustness
Bai, T., Luo, J., Zhao, J., Wen, B., & Wang, Q. (2021) · arXiv · doi:10.48550/arxiv.2102.01356 · Cited by 490 · see citing works
“, 2015], iterative FGSM [Kurakin et al., 2016], and Projected Gradient Descent (PGD) attack [Madry et al.”
BackgroundMethodArtificial Intelligence in Society
OECD (2019) · doi:10.1787/eedfee77-en · Cited by 487 · see citing works
Development and arealization of the cerebral cortex
Cadwell, C. R., Bhaduri, A., Mostajo-Radji, M. A., Keefe, M. G., & Nowakowski, T. J. (2019) · Neuron · doi:10.1016/j.neuron.2019.07.009 · Cited by 408 · see citing works
PRADA: Protecting Against DNN Model Stealing Attacks
Juuti, M., Szyller, S., Marchal, S., & Asokan, N. (2019) · 2019 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2019.00044 · Cited by 440 · see citing works
“VII, similarly evaluated on duplication round 7 using I-FGSM with step size (cid:15) = 64 / 255 .”
MethodInfluentialWhy do deep convolutional networks generalize so poorly to small image transformations?
Azulay, A., & Weiss, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.12177 · Cited by 428 · see citing works
“Despite the excellent performance of CNNs on object recognition, the vulnerability to adversarial attacks suggests that superficial changes can result in highly non-human shifts in prediction [43, 35, 26, 1, 15, 41, 3, 40].”
BackgroundMaking machine learning robust against adversarial inputs
Goodfellow, I., McDaniel, P., & Papernot, N. (2018) · Communications of the ACM · doi:10.1145/3134599 · Cited by 358 · see citing works
Self-citationBai, X., Wang, X., Liu, X., Liu, Q., Song, J., Sebe, N., & Kim, B. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108102 · Cited by 313 · see citing works
Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain
Rosenberg, I., Shabtai, A., Elovici, Y., & Rokach, L. (2022) · ACM Computing Surveys · doi:10.1145/3453158 · Cited by 295 · see citing works
Inductive biases for deep learning of higher-level cognition
Goyal, A., & Bengio, Y. (2022) · Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rspa.2021.0068 · Cited by 300 · see citing works
RobustBench: a standardized adversarial robustness benchmark
Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., & Hein, M. (2020) · arXiv · doi:10.48550/arxiv.2010.09670 · Cited by 284 · see citing works
SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems
Chou, E., Tramer, F., & Pellegrino, G. (2020) · 2020 IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw50608.2020.00025 · Cited by 281 · see citing works
Disentangling Adversarial Robustness and Generalization
Stutz, D., Hein, M., & Schiele, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00714 · Cited by 223 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
“For example, Reference [60] shows that a nearly-invisible image perturbation can lead a NN to classify an object for another; Reference [81] constructs abstract synthetic images that are classified arbitrarily; Reference [72] finds deconstructed versions of an image that are indistinguishable from the viewpoint of the…”
BackgroundAdversarial Deepfakes: Evaluating Vulnerability of Deepfake Detectors to Adversarial Examples
Hussain, S., Neekhara, P., Jere, M., Koushanfar, F., & McAuley, J. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00339 · Cited by 204 · see citing works
DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models
Sha, Z., Li, Z., Yu, N., & Zhang, Y. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3616588 · Cited by 175 · see citing works
Interpretation and visualization techniques for deep learning models in medical imaging
Huff, D. T., Weisman, A. J., & Jeraj, R. (2021) · Physics in Medicine & Biology · doi:10.1088/1361-6560/abcd17 · Cited by 172 · see citing works
Explainable artificial intelligence for cybersecurity: a literature survey
Charmet, F., Tanuwidjaja, H. C., Ayoubi, S., Gimenez, P.-F., Han, Y., Jmila, H., Blanc, G., Takahashi, T., & Zhang, Z. (2022) · Annals of Telecommunications · doi:10.1007/s12243-022-00926-7 · Cited by 167 · see citing works
Wasserstein Adversarial Examples via Projected Sinkhorn Iterations
Wong, E., Schmidt, F. R., & Kolter, J. Z. (2019) · arXiv · doi:10.48550/arxiv.1902.07906 · Cited by 145 · see citing works
“If we consider the (cid:96) ∞ ball B ∞ ( x, (cid:15) ) = { x + ∆ : (cid:107) ∆ (cid:107) ∞ ≤ (cid:15) } and use steepest descent with respect to the (cid:96) ∞ norm, then we recover the Basic Iterative Method originally presented by Kurakin et al. (2017).”
MethodInterpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks
Wagner, J., Köhler, J. M., Gindele, T., Hetzel, L., Wiedemer, J. T., & Behnke, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00931 · Cited by 144 · see citing works
Adversarially Robust Policy Learning: Active construction of physically-plausible perturbations
Mandlekar, A., Zhu, Y., Garg, A., Fei-Fei, L., & Savarese, S. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206245 · Cited by 116 · see citing works
Towards Proving the Adversarial Robustness of Deep Neural Networks
Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.257.3 · Cited by 115 · see citing works
Adversarial Threats to DeepFake Detection: A Practical Perspective
Neekhara, P., Dolhansky, B., Bitton, J., & Ferrer, C. C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00103 · Cited by 117 · see citing works
TSViz: Demystification of Deep Learning Models for Time-Series Analysis
Siddiqui, S. A., Mercier, D., Munir, M., Dengel, A., & Ahmed, S. (2019) · IEEE Access · doi:10.1109/access.2019.2912823 · Cited by 95 · see citing works
Universal Adversarial Perturbation via Prior Driven Uncertainty Approximation
Liu, H., Ji, R., Li, J., Zhang, B., Gao, Y., Wu, Y., & Huang, F. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00303 · Cited by 95 · see citing works
Abdel Hakeem, S. A., & Kim, H. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3558849 · Cited by 90 · see citing works
A Survey of Deep Learning Applications to Autonomous Vehicle Control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2962338 · Cited by 96 · see citing works
AI system engineering key challenges and lessons learned
Fischer, L., Ehrlinger, L., Geist, V., Ramler, R., Sobiezky, F., Zellinger, W., Brunner, D., Kumar, M., & Moser, B. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make3010004 · Cited by 71 · see citing works
Secure and robust machine learning for healthcare: A survey
Qayyum, A., Qadir, J., Bilal, M., & Al-Fuqaha, A. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3013489 · Cited by 59 · see citing works
AI-GAN: Attack-Inspired Generation of Adversarial Examples
Bai, T., Zhao, J., Zhu, J., Han, S., Chen, J., Li, B., & Kot, A. (2021) · 2021 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip42928.2021.9506278 · Cited by 58 · see citing works
Machine learning in NextG networks via generative adversarial networks
Ayanoglu, E., Davaslioglu, K., & Sagduyu, Y. E. (2022) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2022.3153004 · Cited by 55 · see citing works
Grey-box Adversarial Attack And Defence For Sentiment Classification
Xu, Y., Zhong, X., Jimeno Yepes, A., & Lau, J. H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.321 · Cited by 54 · see citing works
FENCE: Feasible Evasion Attacks on Neural Networks in Constrained Environments
Chernikova, A., & Oprea, A. (2022) · ACM Transactions on Privacy and Security · doi:10.1145/3544746 · Cited by 52 · see citing works
Joshi, S., Villalba, J., Zelasko, P., Moro-Velazquez, L., & Dehak, N. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3116438 · Cited by 53 · see citing works
Exploring Adversarial Attacks in Federated Learning for Medical Imaging
Darzi, E., Dubost, F., Sijtsema, N. M., & van Ooijen, P. M. A. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3423457 · Cited by 45 · see citing works
CUDA: Convolution-Based Unlearnable Datasets
Sadasivan, V. S., Soltanolkotabi, M., & Feizi, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00376 · Cited by 24 · see citing works
Robust Vision-Based Cheat Detection in Competitive Gaming
Jonnalagadda, A., Frosio, I., Schneider, S., McGuire, M., & Kim, J. (2021) · Proceedings of the ACM on Computer Graphics and Interactive Techniques · doi:10.1145/3451259 · Cited by 23 · see citing works
DEEPFAKER: A Unified Evaluation Platform for Facial Deepfake and Detection Models
Wang, L., Meng, X., Li, D., Zhang, X., Ji, S., & Guo, S. (2024) · ACM Transactions on Privacy and Security · doi:10.1145/3634914 · Cited by 11 · see citing works
Jain, S., Cretu, A.-M., & de Montjoye, Y.-A. (2021) · arXiv · doi:10.48550/arxiv.2106.09820 · Cited by 21 · see citing works
“Adversarial attacks have been widely studied against ML models [38, 41] under both white-box [6, 15, 23, 46] and black-box assumptions [25, 35].”
BackgroundBeyond Statistical Learning: Exact Learning Is Essential for General Intelligence
György, A., Lattimore, T., Lazić, N., & Szepesvári, C. (2025) · arXiv · doi:10.48550/arxiv.2506.23908
- Authors
- Kurakin, A., Goodfellow, I. J., & Bengio, S.
- Venue
- Artificial Intelligence Safety and Security
- Publisher
- Chapman and Hall/CRC
- Published
- 2018-07-27
- Pages
- 99-112
- OpenAlex
- W2460937040
- Cited by
- 5,375
- References
- 2
- Citations by year
- 2026: 240 · 2025: 604 · 2024: 688 · 2023: 729 · 2022: 643 · 2021: 821 · 2020: 766 · 2019: 520 · 2018: 284 · 2017: 72 · 2016: 6 · 2012: 2
- FWCI
- 894.75
- Type
- book chapter
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Chapman & Hall/CRC eBooks (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Keywords
- adversarial examples, physical-world attacks, adversarial robustness, image classification, InceptionV3, ImageNet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
