CoChat
Sign Up
Cited by 2,406Adversarial Robustness in Machine LearningDigital Media Forensic DetectionSoftware Testing and Debugging Techniques

One Pixel Attack for Fooling Deep Neural Networks

Open access
Su, Jiawei, Vargas, Danilo Vasconcellos, Sakurai, Kouichi·2019·IEEE Transactions on Evolutionary Computation
DOI 10.1109/tevc.2019.2890858
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent research has revealed that the output of deep neural networks (DNNs) can be easily altered by adding relatively small perturbations to the input vector. In this paper, we analyze an attack in an extremely limited scenario where only one pixel can be modified. For that we propose a novel method for generating one-pixel adversarial perturbations based on differential evolution (DE). It requires less adversarial information (a black-box attack) and can fool more types of networks due to the inherent features of DE. The results show that 67.97% of the natural images in Kaggle CIFAR-10 test dataset and 16.04% of the ImageNet (ILSVRC 2012) test images can be perturbed to at least one target class by modifying just one pixel with 74.03% and 22.91% confidence on average. We also show the same vulnerability on the original CIFAR-10 dataset. Thus, the proposed attack explores a different take on adversarial machine learning in an extreme limited scenario, showing that current DNNs are also vulnerable to such low dimension attacks. Besides, we also illustrate an important application of DE (or broadly speaking, evolutionary computation) in the domain of adversarial machine learning: creating tools that can effectively generate low-cost adversarial attacks against neural networks for evaluating robustness.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tevc.2019.2890858

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Deep neural networks are vulnerable to adversarial examples, which are carefully chosen inputs that cause the network to change output without a visible change to a human.

    “On the other hand, it is worth noting that researchers have recently that noticed deep neural networks are vulnerable to adversarial examples, which are carefully chosen inputs that cause the network to change output without a visible change to a human (Fig. 15) [60–63].”

    From Convolutional neural networks: an overview and application in radiology (2018) · cites this paper as [62]

  • These studies indicate that the way artificial networks see and predict is different from the way we do.

    “Although the impact of adversarial examples in the medical domain is unknown, these studies indicate that the way artificial networks see and predict is different from the way we do.”

    From Convolutional neural networks: an overview and application in radiology (2018) · cites this paper as [62]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 56

  1. [1][1] M. Barreno, B. Nelson, A. D. Joseph, and J. Tygar. The security of machine learning. Machine Learning , 81(2): pp.121-148, 2010. -
  2. [2][2] M. Barreno, B. Nelson, R. Sears, A. D. Joseph, and J. D. Tygar. Can machine learning be secure? In Proceedings of the 2006 ACM Symposium on Information, computer and communications security , pp.16-25. ACM, 2006. -
  3. [3][3] J. Brest, S. Greiner, B. Boskovic, M. Mernik, and V. Zumer. Selfadapting control parameters in differential evolution: A comparative study on numerical benchmark problems. IEEE transactions on evolutionary computation , 10(6): pp.646-657, 2006. -
  4. [4][4] P. Civicioglu and E. Besdok. A conceptual comparison of the cuckoosearch, particle swarm optimization, differential evolution and artificial bee colony algorithms. Artificial intelligence review , pp.1-32, 2013. -
  5. [7][7] S. M. Moosavi Dezfooli, F. Alhussein and F. Pascal. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp.2574-2582, 2016. -
  6. [9][9] A. Fawzi, S. M. Moosavi Dezfooli, and P. Frossard. The robustness of deep networks: A geometrical perspective. IEEE Signal Processing Magazine, 34(6): pp.50-62. -
  7. [10][10] A. Fawzi, S.-M. Moosavi-Dezfooli, P. Frossard, and S. Soatto. Classification regions of deep neural networks. arXiv preprint arXiv:1705.09552 , 2017. - · arXiv
  8. [12][12] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, (1)4: pp. 7, University of Toronto.2009. -
  9. [13][13] M. Lin, Q. Chen, and S. Yan. Network in network. arXiv preprint arXiv:1312.4400 , 2013. - · arXiv
  10. [14][14] S. M. Moosavi Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial perturbations. In Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , number EPFLCONF-226156, 2017. -
  11. [15][15] N. Narodytska and S. Kasiviswanathan. Simple black-box adversarial attacks on deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pp.1310-1318. IEEE, 2017. -
  12. [16][16] A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp.427-436, 2015. -
  13. [17][17] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security , pp.506-519. ACM, 2017. -
  14. [19][19] A. Rozsa, E. M. Rudd, and T. E. Boult. Adversarial diversity and hard positive generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pp.25-32, 2016. -
  15. [21][21] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. - · arXiv
  16. [22][22] J. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (workshop track) . -
  17. [23][23] R. Storn and K. Price. Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. Journal of global optimization , 11(4): pp.341-359, 1997. -
  18. [24][24] S. Christian, Z. Wojciech, S. Ilya, b. Joan, E. Dumitru, G. Ian, F. Rob. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013. - · arXiv
  19. [25][25] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp.1701-1708, 2014. -
  20. [26][26] D. V. Vargas and J. Murata. Spectrum-diverse neuroevolution with unified neural models. IEEE transactions on neural networks and learning systems , 28(8):pp.1759-1773, 2017. -
  21. [27][27] D. V. Vargas, J. Murata, H. Takano, and A. C. B. Delbem. General subpopulation framework and taming the conflict inside populations. Evolutionary computation , 23(1):pp.1-36, 2015. -
  22. [28][28] D. Wei, B. Zhou, A. Torrabla, and W. Freeman. Understanding intraclass knowledge inside cnn. arXiv preprint arXiv:1507.02379 , 2015. - · arXiv
  23. [29][29] J. Yosinski, J. Clune, T. Fuchs, and H. Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 , 2015. - · arXiv
  24. [30][30] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision , pp.818-833. Springer, 2014. -
  25. [31][31] J. Su, D. Vargas, and K. Sakurai. One pixel attack for fooling deep neural networks. arXiv preprint arXiv:1710.08864 , 2017. - · arXiv
  26. [32][32] N. Hansen. The CMA evolution strategy: a comparing review In Towards a new evolutionary computation , pp.75-102. Springer, 2006. -
  27. [33][33] A. Moustafa, B. Bharathan, S. Mani. Did you hear that? Adversarial Examples Against Automatic Speech Recognition. arXiv preprint arXiv:1801.00554 , 2018. - · arXiv
  28. [34][34] P. Nicolas, M. Patrick, S. Ananthram, H. Richard. Crafting Adversarial Input Sequences for Recurrent Neural Networks. arXiv preprint arXiv:1604.08275 , 2016. - · arXiv
  29. [35][35] G. Kathrin, P. Nicolas, M. Praveen, B. Michael, M. Patrick. Adversarial Perturbations Against Deep Neural Networks for Malware Classification. arXiv preprint arXiv:1606.04435 , 2016. - · arXiv
  30. [36][36] E. Logan, T. Brandon, T. Dimitris, S. Ludwig, M. Aleksander. A Rotation and a Translation Suffice: Fooling CNNs with Simple Transformations. arXiv preprint arXiv:1712.02779 , 2017. - · arXiv
  31. [37][37] G. Kathrin, P. Nicolas, M. Praveen, B. Michael, M. Patrick. Adversarial Perturbations Against Deep Neural Networks for Malware Classification. arXiv preprint arXiv:1606.04435 , 2016. - · arXiv
  32. [38][38] M. Sharif, L. Bauer, MK. Reiter On the Suitability of Lp-norms for Creating and Preventing Adversarial Examples. arXiv preprint arXiv:1802.09653 , 2018. - · arXiv
  33. [39][39] X. Yuan, P. He, Q. Zhu, R. R. Bhat Adversarial Examples: Attacks and Defenses for Deep Learning. arXiv preprint arXiv:1712.07107 , 2017. - · arXiv
  34. [40][40] N. Papernot, P. McDaniel, X. Wu, S. Jha, A. Swami Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of IEEE Symposium on Security and Privacy (SP) , pp.17011708. -
  35. [41][41] R. Huang, B. Xu, D. Schuurmans, C. Szepesvri Learning with a strong adversary. arXiv preprint arXiv:1511.03034 , 2015. - · arXiv
  36. [42][42] W. Xu, D. Evans, Y. Qi Feature squeezing: Detecting adversarial examples in deep neural networks arXiv preprint arXiv:1704.01155 , 2017. - · arXiv
  37. [43][43] B. Liang et al. Detecting Adversarial Examples in Deep Networks with Adaptive Noise Reduction. arXiv preprint arXiv:1705.08378 , 2017. - · arXiv
  38. [44][44] K. O. Stanley, R. Miikkulainen. Evolving neural networks through augmenting topologies. In Evolutionary Computation , pp.99-127. -
  39. [45][45] N. Carlini, D. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security , pp.3-14. -
  40. [46][46] N. Carlini, D. Wagner. Defensive distillation is not robust to adversarial examples. arXiv preprint arXiv:1607.04311 , 2016. - · arXiv
  41. [48][48] H.G. Beyer and H.P. Schwefel. Evolution strategiesA comprehensive introduction. In Natural computing , pp.3-52. -
  42. [49][49] A.K. Qin and P.N. Suganthan. Self-adaptive differential evolution algorithm for numerical optimization. In The 2005 IEEE Congress on Evolutionary Computation , pp.1785-1791. -
  43. [50][50] N. Hansen, S. D. Mller, and P. Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). In Evol. Comput , pp.1-18. -
  44. [51][51] N. Hansen and A. Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of the 1996 IEEE Conference on Evolutionary Computation , pp.312-317. -
  45. [53][53] D. Whitley, S. Dominic, R. Das and C.W. Anderson. Genetic reinforcement learning for neurocontrol problems. In Machine Learning , pp.259-284. -
  46. [54][54] D.V. Vargas, H. Takano and J. Murata. Self organizing classifiers: first steps in structured evolutionary machine learning. In Evolutionary Intelligence , pp.57-72. -
  47. [55][55] D.V. Vargas, H. Takano and J. Murata. Self organizing classifiers and niched fitness. In Proceedings of the 15th annual conference on Genetic and evolutionary computation , pp.1109-1116. -
  48. [56][56] D.V. Vargas, H. Takano and J. Murata. Novelty-organizing team of classifiers-a team-individual multi-objective approach to reinforcement learning. In Proceedings of the SICE Annual Conference (SICE) , pp.1785-1792. -
  49. [57][57] D.V. Vargas, H. Takano and J. Murata. Novelty-organizing team of classifiers in noisy and dynamic environments. In 2015 IEEE Congress on Evolutionary Computation (CEC) , pp.2937-2944. -
  50. [58][58] R.J. Urbanowicz and J.H. Moore. ExSTraCS 2.0: description and evaluation of a scalable learning classifier system. In Evolutionary intelligence , pp.89-116. -
  51. [59][59] I.M. Alvarez, W.N. Browne and M. Zhang. Compaction for code fragment based learning classifier systems. In Australasian Conference on Artificial Life and Computational Intelligence , pp.41-53. -
  52. [60][60] N. Carlini and D. Wagner. Magnet and efficient defenses against adversarial attacks are not robust to adversarial examples. arXiv preprint arXiv:1711.08478 , 2017. - · arXiv
  53. [61][61] A. Abdolmaleki, B. Price, N. Lau, L.P. Reis and G. Neumann. Deriving and improving CMA-ES with information geometric trust regions. In Proceedings of the Genetic and Evolutionary Computation Conference , pp.657-664. -
  54. [62][62] K. Nishida and Y. Akimoto. PSA-CMA-ES: CMA-ES with population size adaptation. In Proceedings of the Genetic and Evolutionary Computation Conference , pp.865-872. -
  55. [63][63] M. Groves and J. Branke. Sequential sampling for noisy optimisation with CMA-ES. In Proceedings of the Genetic and Evolutionary Computation Conference , pp.1023-1030. -
  56. [65][65] Jiawei Su, Danilo Vasconcellos Vargas, Kouichi Sakurai One Pixel Attack for Fooling Deep Neural Networks. In IEEE Transactions on Evolutionary Computation , Vol.23 , Issue.5 , pp. 828-841. Publisher: IEEE. DOI: 10.1109/TEVC.2019.2890858.

Cited by

Sorted by citations

Authors
Su, J., Vargas, D. V., & Sakurai, K.
Affiliations
  • Kyushu University (JP)
  • Advanced Telecommunications Research Institute International (JP)
ORCID
Su, Jiawei
Venue
IEEE Transactions on Evolutionary Computation
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2019-10
Volume
23
Issue
5
Pages
828-841
ISSN
  • 1089-778X (ISSN-L)
  • 1941-0026
DOI
10.1109/tevc.2019.2890858
OpenAlex
W2765424254
Cited by
2,406
References
7
Citations by year
2026: 109 · 2025: 211 · 2024: 289 · 2023: 358 · 2022: 326 · 2021: 433 · 2020: 363 · 2019: 226 · 2018: 83 · 2017: 7 · 2012: 1
FWCI
153.74
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Evolutionary Computation (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Collaboration Hubs for International Program of SICORP
  • Japan Science and Technology Agency
  • Kyushu University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
one-pixel attack, adversarial perturbations, adversarial learning, deep neural network robustness, black-box attacks, differential evolution, adversarial attack generation, image classification, CIFAR-10, ImageNet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review