CoChat
Sign Up
Cited by 1,238Adversarial Robustness in Machine LearningSecurity and Verification in ComputingDigital Media Forensic Detection

BadNets: Evaluating Backdooring Attacks on Deep Neural Networks

Open access
Gu, Tianyu, Liu, Kang, Dolan-Gavitt, Brendan, Garg, Siddharth·2019·IEEE Access
DOI 10.1109/access.2019.2909068
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep learning-based techniques have achieved state-of-the-art performance on a wide variety of recognition and classification tasks. However, these networks are typically computationally expensive to train, requiring weeks of computation on many GPUs; as a result, many users outsource the training procedure to the cloud or rely on pre-trained models that are then fine-tuned for a specific task. In this paper, we show that the outsourced training introduces new security risks: an adversary can create a maliciously trained network (a backdoored neural network, or a BadNet) that has the state-of-the-art performance on the user's training and validation samples but behaves badly on specific attacker-chosen inputs. We first explore the properties of BadNets in a toy example, by creating a backdoored handwritten digit classifier. Next, we demonstrate backdoors in a more realistic scenario by creating a U.S. street sign classifier that identifies stop signs as speed limits when a special sticker is added to the stop sign; we then show in addition that the backdoor in our U.S. street sign detector can persist even if the network is later retrained for another task and cause a drop in an accuracy of 25% on average when the backdoor trigger is present. These results demonstrate that backdoors in neural networks are both powerful and-because the behavior of neural networks is difficult to explicate-stealthy. This paper provides motivation for further research into techniques for verifying and inspecting neural networks, just as we have developed tools for verifying and debugging software.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2019.2909068

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 48 · Cited by 1,238

Research works printed but not linked: 40

  1. T. Gu, B. Dolan-Gavitt, and S. Garg, ''Badnets: Identifying vulnerabilities in the machine learning model supply chain,'' in Proc. Neural Inf. Process. Symp. Workshop Mach. Learn. Secur. (MLSec), 2017, pp. 1-5. [Online]. Available: https://machine-learning-and-security.github.io/
  2. A. Graves, A.-R. Mohamed, and G. Hinton, ''Speech recognition with deep recurrent neural networks,'' in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2013, pp. 6645-6649.
  3. K. M. Hermann and P. Blunsom, ''Multilingual distributed representations without word alignment,'' in Proc. ICLR, Apr. 2014, pp. 1-9. [Online]. Available: http://arxiv.org/abs/1312.6173 · arXiv
  4. D. Bahdanau, K. Cho, and Y. Bengio. (2014). ''Neural machine trans- lation by jointly learning to align and translate.'' [Online]. Available: https://arxiv.org/abs/arXiv:1409.0473
  5. D. Silver et al., ''Mastering the game of Go with deep neural networks and tree search,'' Nature, vol. 529, no. 7587, pp. 484-489, 2016. doi: 10.1038/ nature16961. · DOI · Paper page
  6. A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, ''CNN features off-the-shelf: An astounding baseline for recognition,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW). Washington, DC, USA: IEEE Comput. Soc., Jun. 2014, pp. 512-519. doi: 10.1109/ CVPRW.2014.131. · DOI · Paper page
  7. K. Simonyan and A. Zisserman. (2014). ''Very deep convolutional networks for large-scale image recognition.'' [Online]. Available: https://arxiv.org/abs/1409.1556 · arXiv
  8. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, ''Rethinking the inception architecture for computer vision,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 2818-2826.
  9. C. Szegedy et al. (2013). ''Intriguing properties of neural networks.'' [Online]. Available: https://arxiv.org/abs/1312.6199 · arXiv
  10. M. Barreno, B. Nelson, R. Sears, A. D. Joseph, and J. D. Tygar, ''Can machine learning be secure?'' in Proc. ACM Symp. Inf., Comput. Commun. Secur. New York, NY, USA: ACM, 2006, pp. 16-25.
  11. N. Dalvi, P. Domingos, S. Sanghai, and D. Verma, ''Adversarial classifica- tion,'' in Proc. 10th ACM SIGKDD Int. Conf. Knowl. Discovery Data Min- ing (KDD). New York, NY, USA: ACM, 2004, pp. 99-108. doi: 10.1145/ 1014052.1014066. · DOI
  12. D. Lowd and C. Meek, ''Adversarial learning,'' in Proc. 11th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining (KDD). New York, NY, USA: ACM, 2005, pp. 641-647. doi: 10.1145/1081870.1081950. · DOI · Paper page
  13. D. Lowd and C. Meek, ''Good word attacks on statistical spam filters,'' in Proc. Conf. Email Anti-Spam (CEAS), 2005, pp. 1-8.
  14. G. L. Wittel and S. F. Wu, ''On attacking statistical spam filters,'' in Proc. Conf. Email Anti-Spam (CEAS), Mountain View, CA, USA, 2004, pp. 1-7.
  15. J. Newsome, B. Karp, and D. Song, ''Paragraph: Thwarting signature learn- ing by training maliciously,'' in Proc. 9th Int. Conf. Recent Adv. Intrusion Detection (RAID). Berlin, Germany: Springer-Verlag, 2006, pp. 81-105.
  16. S. P. Chung and A. K. Mok, ''Allergy attack against automatic signature generation,'' in Proc. 9th Int. Conf. Recent Adv. Intrusion Detection, 2006, pp. 61-80.
  17. S. P. Chung and A. K. Mok, ''Advanced allergy attacks: Does a corpus really help?'' in Proc. 10th Int. Conf. Recent Adv. Intrusion Detection, 2007, pp. 236-255.
  18. B. Biggio, B. Nelson, and P. Laskov, ''Poisoning attacks against sup- port vector machines,'' in Proc. 29th Int. Conf. Int. Conf. Mach. Learn. Madison, WI, USA: Omnipress, 2012, pp. 1467-1474.
  19. L. Huang, A. D. Joseph, B. Nelson, B. I. Rubinstein, and J. D. Tygar, ''Adversarial machine learning,'' in Proc. 4th ACM Workshop Secur. Artif. Intell. (AISec). New York, NY, USA: ACM, 2011, pp. 43-58. doi: 10.1145/ 2046684.2046692. · DOI · Paper page
  20. N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, ''Practical black-box attacks against machine learning,'' in Proc. ACM Asia Conf. Comput. Commun. Secur., 2016, pp. 506-519.
  21. S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, ''Univer- sal adversarial perturbations,'' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jul. 2016, pp. 1765-1773.
  22. I. Evtimov et al. (2017). ''Robust physical-world attacks on deep learning models,'' [Online]. Available: https://arxiv.org/abs/1707.08945 · arXiv
  23. X. Chen, C. Liu, B. Li, K. Lu, and D. Song. (2018). ''Targeted backdoor attacks on deep learning systems using data poisoning.'' [Online]. Avail- able: https://arxiv.org/abs/arXiv:1712.05526
  24. K. Liu, B. Dolan-Gavitt, and S. Garg, ''Fine-pruning: Defending against backdooring attacks on deep neural networks,'' in Proc. Int. Symp. Res. Attacks, Intrusions, Defenses. Cham, Switzerland: Springer, 2018, pp. 273-294.
  25. B. Wang et al., Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks. Piscataway, NJ, USA: IEEE, 2019.
  26. B. Tran, J. Li, and A. Madry, ''Spectral signatures in backdoor attacks,'' in Proc. Adv. Neural Inf. Process. Syst., 2018, pp. 8011-8021.
  27. S. Shen, S. Tople, and P. Saxena, ''AUROR: Defending against poisoning attacks in collaborative deep learning systems,'' in Proc. 32nd Annu. Conf. Comput. Secur. Appl. (ACSAC). New York, NY, USA: ACM, 2016, pp. 508-519. doi: 10.1145/2991079.2991125. · DOI · Paper page
  28. L. Muñoz-González et al., ''Towards poisoning of deep learning algo- rithms with back-gradient optimization,'' in Proc. 10th ACM Workshop Artif. Intell. Secur. New York, NY, USA: ACM, 2017, pp. 27-38. [Online]. Available: https://arxiv.org/abs/1708.08689 · arXiv
  29. Y. Ji, X. Zhang, S. Ji, X. Luo, and T. Wang, ''Model-reuse attacks on deep learning systems,'' in Proc. ACM SIGSAC Conf. Comput. Commun. Secur. (CCS), 2018, pp. 349-363.
  30. J. Schmidhuber, ''Deep learning in neural networks: An overview,'' Neural Netw., vol. 61, pp. 85-117, Jan. 2015.
  31. A. Blum and R. L. Rivest, ''Training a 3-node neural network is NP- complete,'' in Proc. Adv. Neural Inf. Process. Syst., 1989, pp. 494-501.
  32. X. Glorot, A. Bordes, and Y. Bengio, ''Domain adaptation for large-scale sentiment classification: A deep learning approach,'' in Proc. 28th Int. Conf. Mach. Learn. (ICML), 2011, pp. 513-520.
  33. F. Larsson, M. Felsberg, and P. E. Forssen, ''Correlating fourier descriptors of local patches for road sign recognition,'' IET Comput. Vis., vol. 5, no. 4, pp. 244-254, Jul. 2011.
  34. Y. LeCun et al., ''Learning algorithms for classification: A comparison on handwritten digit recognition,'' Neural Netw., Stat. Mech. Perspective, vol. 261, p. 276, Jan. 1995.
  35. Y. Zhang, P. Liang, and M. J. Wainwright. (2016). ''Convexified convolutional neural networks.'' [Online]. Available: https://arxiv.org/ abs/arXiv:1609.01000
  36. C. Chen, A. Seff, A. Kornhauser, and J. Xiao, ''Deepdriving: Learning affordance for direct perception in autonomous driving,'' in Proc. IEEE Int. Conf. Comput. Vis. (ICCV). Washington, DC, USA: IEEE Comput. Soc., Dec. 2015, pp. 2722-2730. doi: 10.1109/ICCV.2015.312. · DOI · Paper page
  37. S. Ren, K. He, R. Girshick, and J. Sun, ''Faster R-CNN: Towards real-time object detection with region proposal networks,'' in Proc. Adv. Neural Inf. Process. Syst., 2015, pp. 91-99.
  38. A. Møgelmose, D. Liu, and M. M. Trivedi, ''Traffic sign detection for U.S. roads: Remaining challenges and a case for tracking,'' in Proc. 17th Int. IEEE Conf. Intell. Transp. Syst. (ITSC), Oct. 2014, pp. 1394-1399.
  39. J. Zhang, M. Huang, X. Li, and X. Jin, ''A real-time chinese traffic sign detection algorithm based on modified YOLOv2,'' Algorithms, vol. 10, no. 4, p. 127, Nov. 2017.
  40. J. Samuel, N. Mathewson, J. Cappos, and R. Dingledine, ''Survivable key compromise in software update systems,'' in Proc. CCS, 2010, pp. 61-72 TIANYU GU received the bachelor's degree in electrical engineering from Fudan University, in 2016, and the M.S. degree in electrical and computer engineering from New York University, in 2018. He was a Research Assistant with the High Power Electronics Group, Fudan University, from 2013 to 2016, and a Visiting Scholar with Harvard University, in 2015.

Authors
Gu, T., Liu, K., Dolan-Gavitt, B., & Garg, S.
Affiliations
New York University (US)
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2019
Volume
7
Pages
47230-47244
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2019.2909068
OpenAlex
W2942091739
Cited by
1,238
References
48
Citations by year
2026: 205 · 2025: 283 · 2024: 245 · 2023: 188 · 2022: 136 · 2021: 108 · 2020: 59 · 2019: 14
FWCI
36.15
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Access (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Science Foundation (1801495)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
backdoor attacks, adversarial learning, handwritten digit recognition, neural network verification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review