CoChat
Sign Up
Cited by 200Adversarial Robustness in Machine LearningDigital Media Forensic DetectionSecurity and Verification in Computing

A general framework for adversarial examples with objectives

Open access
Sharif, Mahmood, Bhagavatula, Sruti, Bauer, Lujo, Reiter, Michael K.·2019·ACM Transactions on Privacy and Security
DOI 10.1145/3317611
Open this paper in CoChat — free accountOpen PDF

Abstract

Images perturbed subtly to be misclassified by neural networks, called adversarial examples, have emerged as a technically deep challenge and an important concern for several application domains. Most research on adversarial examples takes as its only constraint that the perturbed images are similar to the originals. However, real-world application of these ideas often requires the examples to satisfy additional objectives, which are typically enforced through custom modifications of the perturbation process. In this article, we propose adversarial generative nets(AGNs), a general methodology to train a generator neural network to emit adversarial examples satisfying desired objectives. We demonstrate the ability of AGNs to accommodate a wide range of objectives, including imprecise ones difficult to model, in two application domains. In particular, we demonstrate physical adversarial examples—eyeglass frames designed to fool face recognition—with better robustness, inconspicuousness, and scalability than previous approaches, as well as a new attack to fool a handwritten-digit classifier.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3317611

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Face recognition systems can be easily spoofed by impersonation attackers wearing a small printed adversarial patch.

    “Research has shown that face recognition systems can be easily spoofed by impersonation attackers wearing a small printed adversarial patch [90, 30, 43, 73].”

    From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [73]

    The Introduction and Experiment Setup (pages unavailable in our copy) describe impersonation attacks against face-recognition DNNs using printed adversarial patterns affixed to eyeglasses worn by attackers. The source does not establish that face-recognition systems generally are easily spoofed by a small patch: it tests particular DNNs and eyeglass-based attacks under controlled conditions, and notes that some physical impersonation attempts failed. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Adversarial attacks against face recognition have recently been proved as a new type of threat.

    “Besides presentation attacks, adversarial attacks against face recognition have recently been proved as a new type of threat.”

    From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [73]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 68

  1. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein GAN. In Proceedings of the International Conference on Machine Learning (ICML'17).
  2. David Arthur and Sergei Vassilvitskii. 2007. k-means++: The advantages of careful seeding. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms (SODA'07).
  3. Anish Athalye and Nicholas Carlini. 2018. On the robustness of the CVPR 2018 white-box adversarial example de- fenses. arXiv:1804.03286 (2018). · arXiv
  4. Anish Athalye, Nicholas Carlini, and David Wagner. 2018. Obfuscated gradients give a false sense of security: Cir- cumventing defenses to adversarial examples. In Proceedings of the International Conference on Machine Learning (ICML'18).
  5. Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: An open source facial behavior analysis toolkit. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV'16). DOI: https://doi.org/10.1109/WACV.2016.7477553 · DOI
  6. Shumeet Baluja and Ian Fischer. 2018. Learning to attack: Adversarial transformation networks. In Proceedings of the AAAI Conference on Artificial Intelligence.
  7. Battista Biggio and Fabio Roli. 2018. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recogn. 84 (2018), 317-331. DOI:https://doi.org/10.1016/j.patcog.2018.07.023 · DOI
  8. Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. 2013. Evasion attacks against machine learning at test time. In Proceedings of the European Confer- ence on Machine Learning and Knowledge Discovery in Databases (ECML/PKDD'13). DOI:https://doi.org/10.1007/ 978-3-642-40994-3_25 · DOI · Paper page
  9. Nicholas Carlini and David Wagner. 2017. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the ACM Workshop on Artificial Intelligence and Security (AISec'17). DOI:https://doi.org/10. 1145/3128572.3140444 · DOI
  10. Nicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang, Micah Sherr, Clay Shields, David Wagner, and Wenchao Zhou. 2016. Hidden voice commands. In Proceedings of the USENIX Security Symposium.
  11. Yizheng Chen, Yacin Nadji, Athanasios Kountouras, Fabian Monrose, Roberto Perdisci, Manos Antonakakis, and Nikolaos Vasiloglou. 2017. Practical attacks against graph-based clustering. In Proceedings of the ACM SIGSAC Con- ference on Computer and Communications Security (CCS'17). DOI:https://doi.org/10.1145/3133956.3134083 · DOI
  12. Moustapha Cisse, Yossi Adi, Natalia Neverova, and Joseph Keshet. 2017. Houdini: Fooling deep structured prediction models. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS'17).
  13. Simon Eberz, Nicola Paoletti, Marc Roeschlin, Marta Kwiatkowska, I. Martinovic, and A. Patané. 2017. Broken hearted: How to attack ECG biometrics. In Proceedings of the ISOC Annual Network and Distributed System Security Symposium (NDSS'17).
  14. Alexei A. Efros and Thomas K. Leung. 1999. Texture synthesis by non-parametric sampling. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'99). DOI:https://doi.org/10.1109/ICCV.1999.790383 · DOI
  15. Logan Engstrom, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. 2017. A rotation and a translation suffice: Fooling CNNs with simple transformations. In Proceedings of the NeurIPS Machine Learning and Computer Security Workshop.
  16. Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song. 2018. Robust physical-world attacks on machine learning models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'18). DOI:https://doi.org/10.1109/CVPR.2018.00175 · DOI · Paper page
  17. Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. 2016. Robustness of classifiers: From adver- sarial to random noise. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS'16).
  18. Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner. 2017. Detecting adversarial samples from artifacts. arXiv:1703.00410 (2017). · arXiv
  19. Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. 2015. Model inversion attacks that exploit confidence infor- mation and basic countermeasures. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS'15). DOI:https://doi.org/10.1145/2810103.2813677 · DOI · Paper page
  20. Justin Gilmer, Ryan P. Adams, Ian Goodfellow, David Andersen, and George E. Dahl. 2018. Motivating the rules of the game for adversarial example research. arXiv:1807.06732 (2018). · arXiv
  21. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS'14).
  22. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press.
  23. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR'15).
  24. Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. 2017. On the (statis- tical) detection of adversarial examples. arXiv:1702.06280 (2017). · arXiv
  25. Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick McDaniel. 2017. Adversarial perturbations against deep neural networks for malware classification. In Proceedings of the European Symposium on Research in Computer Security (ESORICS'17).
  26. Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. 2018. Countering adversarial images using input transformations. In Proceedings of the International Conference on Learning Representations (ICLR'18).
  27. Adam Harvey. 2010. CV Dazzle: Camouflage from face detection. Master's thesis. New York University. Retreieved from http://cvdazzle.com.
  28. Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. 2015. Learning with a strong adversary. arXiv:1511.03034 (2015). · arXiv
  29. Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. 2018. Geometric robustness of deep networks: Analysis and improvement. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'18). DOI:https://doi.org/10.1109/CVPR.2018.00467 · DOI
  30. Harini Kannan, Alexey Kurakin, and Ian Goodfellow. 2018. Adversarial logit pairing. arXiv:1803.06373 (2018). · arXiv
  31. Alex Kantchelian, J. D. Tygar, and Anthony D. Joseph. 2016. Evasion and hardening of tree ensemble classifiers. In Proceedings of the International Conference on Machine Learning (ICML'16).
  32. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR'15).
  33. J. Zico Kolter and Eric Wong. 2018. Provable defenses against adversarial examples via the convex outer adversarial polytope. In Proceedings of the International Conference on Machine Learning (ICML'18).
  34. Neeraj Kumar, Alexander C. Berg, Peter N. Belhumeur, and Shree K. Nayar. 2009. Attribute and simile classifiers for face verification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'09). DOI:https: //doi.org/10.1109/ICCV.2009.5459250 · DOI
  35. Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2017. Adversarial examples in the physical world. In Proceedings of the International Conference on Learning Representations Workshop (ICLRW'17).
  36. Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Jun Zhu, and Xiaolin Hu. 2018. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'18). DOI:https://doi.org/10.1109/CVPR.2018.00191 · DOI
  37. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In Proceedings of the International Conference on Learning Represen- tations (ICLR'18).
  38. Tom Malzbender, Dan Gelb, and Hans Wolters. 2001. Polynomial texture maps. In Proceedings of the ACM Annual Con- ference on Computer Graphics and Interactive Techniques (SIGGRAPH'01). DOI:https://doi.org/10.1145/383259.383320 · DOI
  39. Dongyu Meng and Hao Chen. 2017. MagNet: A two-pronged defense against adversarial examples. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS'17). DOI:https://doi.org/10.1145/ 3133956.3134057 · DOI
  40. Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. 2017. On detecting adversarial perturba- tions. In Proceedings of the International Conference on Learning Representations (ICLR'17).
  41. Matthew Mirman, Timon Gehr, and Martin Vechev. 2018. Differentiable abstract interpretation for provably robust neural networks. In Proceedings of the International Conference on Machine Learning (ICML'18).
  42. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. 2016. Distributional smoothing with virtual adversarial training. In Proceedings of the International Conference on Learning Representations (ICLR'16).
  43. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. DeepFool: A simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16). DOI:https://doi.org/10.1109/CVPR.2016.282 · DOI · Paper page
  44. Nina Narodytska and Shiva Kasiviswanathan. 2017. Simple black-box adversarial attacks on deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW'17). DOI:https: //doi.org/10.1109/CVPRW.2017.172 · DOI
  45. George L. Nemhauser, Laurence A. Wolsey, and Marshall L. Fisher. 1978. An analysis of approximations for maxi- mizing submodular set functions. Math. Program. 14, 1 (1978), 265-294. DOI:https://doi.org/10.1007/BF01588971 · DOI · Paper page
  46. Jorge Nocedal. 1980. Updating quasi-Newton matrices with limited storage. Math. Comp. 35, 151 (1980), 773-782. DOI:https://doi.org/10.1090/S0025-5718-1980-0572855-7 · DOI
  47. Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. 2017. Practical black-box attacks against machine learning. In Proceedings of the ACM Symposium on Information, Computer and Communications Security (AsiaCCS'17). DOI:https://doi.org/10.1145/3052973.3053009 · DOI · Paper page
  48. Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. 2015. Deep face recognition. In Proceedings of the British Machine Vision Conference (BMVC'15). DOI:https://doi.org/10.5244/C.29.41 · DOI · Paper page
  49. Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. 2018. Generative adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'18). DOI:https://doi.org/10. 1109/CVPR.2018.00465 · DOI
  50. Alec Radford, Luke Metz, and Soumith Chintala. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR'16).
  51. Andras Rozsa, Ethan M. Rudd, and Terrance E. Boult. 2016. Adversarial diversity and hard positive generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW'16). DOI:https: //doi.org/10.1109/CVPRW.2016.58 · DOI
  52. Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J. Fleet. 2016. Adversarial manipulation of deep representations. In Proceedings of the International Conference on Learning Representations (ICLR'16).
  53. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS'16).
  54. Pouya Samangouei, Maya Kabkab, and Rama Chellappa. 2018. Defense-GAN: Protecting classifiers against adversarial attacks using generative models. In Proceedings of the International Conference on Learning Representations (ICLR'18).
  55. Adi Shamir, Itay Safran, Eyal Ronen, and Orr Dunkelman. 2019. A simple explanation for the existence of adversarial examples with small hamming distance. arXiv:1901.10861 (2019). · arXiv
  56. Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. 2016. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the ACM SIGSAC Conference on Computer and Commu- nications Security (CCS'16). DOI:https://doi.org/10.1145/2976749.2978392 · DOI · Paper page
  57. Vignesh Srinivasan, Arturo Marban, Klaus-Robert Müller, Wojciech Samek, and Shinichi Nakajima. 2018. Counter- strike: Defending deep learning architectures against adversarial samples by Langevin dynamics with supervised denoising autoencoder. arXiv:1805.12017 (2018). · arXiv
  58. Nedim Srndic and Pavel Laskov. 2014. Practical evasion of a learning-based classifier: A case study. In Proceedings of the IEEE Symposium on Security and Privacy (S&P'14). DOI:https://doi.org/10.1109/SP.2014.20 · DOI
  59. K. O. Stanley. 2007. Compositional pattern producing networks: A novel abstraction of development. Gen. Program. Evolv. Mach. 8, 2 (2007), 131-162. DOI:https://doi.org/10.1007/s10710-007-9028-8 · DOI
  60. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representa- tions (ICLR'14).
  61. Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. 2014. DeepFace: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'14). DOI:https://doi.org/10.1109/CVPR.2014.220 · DOI · Paper page
  62. A. Vedaldi and K. Lenc. 2015. MatConvNet-Convolutional neural networks for MATLAB. In Proceedings of the Annual ACM Conference on Multimedia (MM'15). DOI:https://doi.org/10.1145/2733373.2807412 · DOI · Paper page
  63. David Warde-Farley and Ian Goodfellow. 2016. Adversarial perturbations of deep neural networks. Perturbations, Optimization, and Statistics, Tamir Hazan, George Papandreou, and Daniel Tarlow (Eds.). MIT Press, 1-32.
  64. Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. 2018. Generating adversarial examples with adversarial networks. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'18). DOI:https://doi.org/10.24963/ijcai.2018/543 · DOI
  65. Weilin Xu, Yanjun Qi, and David Evans. 2016. Automatically evading classifiers. In Proceedings of the ISOC Annual Network and Distributed System Security Symposium (NDSS'16).
  66. Takayuki Yamada, Seiichi Gohshi, and Isao Echizen. 2013. Privacy Visor: Method based on light absorbing and re- flecting properties for preventing face image detection. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC'13). DOI:https://doi.org/10.1109/SMC.2013.271 · DOI
  67. Guoming Zhang, Chen Yan, Xiaoyu Ji, Taimin Zhang, Tianchen Zhang, and Wenyuan Xu. 2017. DolphinAttack: Inaudible voice commands. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS'17). DOI:https://doi.org/10.1145/3133956.3134052 · DOI
  68. Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR'18).

Cited by

Sorted by citations

Authors
Sharif, M., Bhagavatula, S., Bauer, L., & Reiter, M. K.
Affiliations
  • Carnegie Mellon University (US)
  • University of North Carolina at Chapel Hill (US)
ORCID
Venue
ACM Transactions on Privacy and Security
Publisher
Association for Computing Machinery (ACM)
Published
2019-06-10
Volume
22
Issue
3
Pages
1-30
ISSN
  • 2471-2566 (ISSN-L)
  • 2471-2574
DOI
10.1145/3317611
OpenAlex
W2932026309
Cited by
200
References
6
Citations by year
2026: 18 · 2025: 31 · 2024: 33 · 2023: 30 · 2022: 23 · 2021: 30 · 2020: 26 · 2019: 7 · 2018: 1 · 2016: 1
FWCI
9.92
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Privacy and Security (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • NSF (1801391 and 1801494)
  • Google and Nvidia, and from Lockheed Martin and NATO through Carnegie Mellon CyLab
  • CyLab Presidential Fellowship and a Symantec Research Labs Fellowship
  • National Security Agency (H9823018D0008)
  • Multidisciplinary University Research Initiative (MURI) Cyber Deception
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
Keywords
adversarial examples, adversarial perturbations, neural network robustness, eyeglass frames, handwritten digit recognition, generative adversarial networks, imperceptible perturbations, misclassification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review