The DOI matches its Crossref record; 47 of the 49 references it cites were found in a registry or an index.
Trojaning Attack on Neural Networks
Open accessAbstract
With the fast spread of machine learning techniques, sharing and adopting public machine learning models become very popular.This gives attackers many new opportunities.In this paper, we propose a trojaning attack on neural networks.As the models are not intuitive for human to understand, the attack features stealthiness.Deploying trojaned models can cause various severe consequences including endangering human lives (in applications like autonomous driving).We first inverse the neural network to generate a general trojan trigger, and then retrain the model with reversed engineered training data to inject malicious behaviors to the model.The malicious behaviors are only activated by inputs stamped with the trojan trigger.In our attack, we do not need to tamper with the original training process, which usually takes weeks to months.Instead, it takes minutes to hours to apply our attack.Also, we do not require the datasets that are used to train the model.In practice, the datasets are usually not shared due to privacy or copyright concerns.We use five different applications to demonstrate the power of our attack, and perform a deep analysis on the possible factors that affect the attack.The results show that our attack is highly effective and efficient.The trojaned behaviors can be successfully triggered (with nearly 100% possibility) without affecting its test accuracy for normal input and even with better accuracy on public dataset.Also, it only takes a small amount of time to attack a complex neuron network model.In the end, we also discuss possible defense against such attacks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“An alternative to this assumption would be incorporating an additional party or a public bulletin board (see, e.g., [398]) into the model that is known to the clients and trusted to not collude with the server.”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [398]
Experimental — accuracy not yet measuredIn the citing paper
“By sharing the model parameters instead of data, FL introduces new attack surfaces at training time by enhancing the capabilities of the adversary [16].”
From Vulnerabilities in federated learning (2021) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationLabaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,119
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
Huang, G. B., Mattar, M., Berg, T. L., & Miller, E. G. (2008) · HAL · Cited by 4,445
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
Practical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P. D., Wu, X., Jha, S., & Swami, A. (2016) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2016.41 · Cited by 3,234
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961
Pang, B., & Lee, L. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219855 · Cited by 2,213
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
Xu, W., Evans, D., & Qi, Y. (2018) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23198 · Cited by 2,204
Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition
Sharif, M., Bhagavatula, S., Bauer, L., & Reiter, M. K. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978392 · Cited by 1,530
Li, X., & Roth, D. (2002) · International Conference on Computational Linguistics · doi:10.3115/1072228.1072378 · Cited by 1,308
Age and gender classification using convolutional neural networks
Levi, G., & Hassncer, T. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301352 · Cited by 1,194
Evaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164
Convolutional Neural Networks for Sentence ClassificationPreprint
Kim, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5882 · Cited by 1,112
The MegaFace Benchmark: 1 Million Faces for Recognition at Scale
Kemelmacher-Shlizerman, I., Seitz, S. M., Miller, D., & Brossard, E. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.527 · Cited by 912
Learn++: an incremental learning algorithm for supervised neural networks
Polikar, R., Upda, L., Upda, S. S., & Honavar, V. (2001) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/5326.983933 · Cited by 888
Can machine learning be secure?
Barreno, M., Nelson, B. A., Sears, R., Joseph, A. D., & Tygar, J. D. (2006) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/1128817.1128824 · Cited by 852
Age and Gender Estimation of Unfiltered Faces
Eidinger, E., Enbar, R., & Hassner, T. (2014) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2014.2359646 · Cited by 830
Stealing Machine Learning Models via Prediction APIs
Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016) · arXiv · doi:10.48550/arxiv.1609.02943 · Cited by 796
Pang, B., & Lee, L. (2005) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0506075 · Cited by 744
Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing.
Fredrikson, M., Lantz, E., Jha, S., Lin, S. M., Page, D., & Ristenpart, T. (2014) · PubMed · Cited by 658
Towards Making Systems Forget with Machine Unlearning
Cao, Y., & Yang, J. (2015) · 2015 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2015.35 · Cited by 622
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization
Muñoz-González, L., Biggio, B., Demontis, A., Paudice, A., Wongrassamee, V., Lupu, E. C., & Roli, F. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140451 · Cited by 549
Visualizing Deep Convolutional Neural Networks Using Natural Pre-images
Mahendran, A., & Vedaldi, A. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0911-8 · Cited by 500
Carlini, N., Mishra, P., Vaidya, T., Zhang, Y., Sherr, M., Shields, C., Wagner, D., & Zhou, W. (2016) · USENIX Security Symposium · Cited by 455
Nguyen, A. M., Yosinski, J., & Clune, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03616 · Cited by 324
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection MethodsPreprint
Carlini, N., & Wagner, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07263 · Cited by 323
Is feature selection secure against training data poisoning?Preprint
Huang, X. X., Biggio, B., Gavin Brown, B., Fumera, G., Eckert, C. M., & Roli, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.07933 · Cited by 315
MagNet: a Two-Pronged Defense against Adversarial ExamplesPreprint
Meng, D., & Chen, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09064 · Cited by 299
Adversarial Example Defense: Ensembles of Weak Defenses are not Strong
He, W., Wei, J., Chen, X., Carlini, N., & Song, D. X. (2017) · Cited by 221
Generative Poisoning Attack Method Against Neural NetworksPreprint
Yang, C., Wu, Q., Li, H. H., & Chen, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01340 · Cited by 215
Latent Factor Guided Convolutional Neural Networks for Age-Invariant Face Recognition
Wen, Y., Li, Z., & Qiao, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.529 · Cited by 162
A Methodology for Formalizing Model-Inversion Attacks
Wu, X., Fredrikson, M., Jha, S., & Naughton, J. F. (2016) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf.2016.32 · Cited by 159
Poisoning behavioral malware clustering
Biggio, B., Rieck, K., Ariu, D., Wressnegger, C., Corona, I., Giacinto, G., & Roli, F. (2014) · doi:10.1145/2666652.2666666 · Cited by 139
Evading Classifiers by Morphing in the Dark
Dang, H., Huang, Y., & Chang, E.-C. (2017) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3133956.3133978 · Cited by 111
An incremental-learning neural network for the classification of remote-sensing images
Bruzzone, L., & Prieto, D. F. (1999) · Pattern Recognition Letters · doi:10.1016/s0167-8655(99)00091-4 · Cited by 62
Geigel, A. (2013) · Journal of Computer Security · doi:10.3233/jcs-2012-0460 · Cited by 38
Comparing Incremental Learning Strategies for Convolutional Neural Networks
Lomonaco, V., & Maltoni, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46182-3_15 · Cited by 25
Mishra, A. (2019) · doi:10.1002/9781119556749.ch15 · Cited by 23
Geigel, A. (2014) · NSUWorks (Nova Southeastern University) · doi:10.13140/rg.2.1.4138.7680 · Cited by 5
Research works printed but not linked: 2
- [26]26. D. Erhan, A. Courville, and Y. Bengio, "Understanding representations learned in deep architectures," Department dInformatique et Recherche Operationnelle, University of Montreal, QC, Canada, Tech. Rep, vol. 1355, 2010.
- [62]62. G. Zhang, C. Yan, X. Ji, T. Zhang, T. Zhang, and W. Xu, "Dolphinattack: Inaudible voice commands," in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS '17. New York, NY, USA: ACM, 2017, pp. 103-117.
Cited by
Sorted by citations
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
Self-citationExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
How to backdoor federated learning
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018) · arXiv · doi:10.48550/arxiv.1807.00459 · Cited by 1,495
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H.-T., & Zhao, B. Y. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00031 · Cited by 1,577
A survey on security and privacy of federated learning
Mothukuri, V., Parizi, R. M., Pouriyeh, S., Huang, Y., Dehghantanha, A., & Srivastava, G. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2020.10.007 · Cited by 1,480
Metadata
- Authors
- Liu, Y., Ma, S., Aafer, Y., Lee, W.-C., Zhai, J., Wang, W., & Zhang, X.
- Affiliations
- Purdue University West Lafayette (US)
- Nanjing University (CN)
- Venue
- Proceedings 2018 Network and Distributed System Security Symposium
- Publisher
- Internet Society
- Published
- 2018
- OpenAlex
- W2753783305
- Cited by
- 1,270
- References
- 49
- Citations by year
- 2026: 114 · 2025: 193 · 2024: 206 · 2023: 184 · 2022: 149 · 2021: 185 · 2020: 153 · 2019: 59 · 2018: 26
- FWCI
- 43.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- public-domain
- Locations
- Network and Distributed System Security Symposium (publishedVersion, public-domain)
- Rutgers University Community Repository (Rutgers University) (submittedVersion)
- Purdue e-Pubs (Purdue University System) (submittedVersion)
- Funders
- National Science Foundation (1748764, 1320444, 1409668)
- Defense Advanced Research Projects Agency (FA-8650-15-C-7562, FA8650-15C)
- Office of Naval Research (N000141712947, N000141410468)
- Sandia National Laboratories (1701331)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Hardware and Architecture › Physical Unclonable Functions (PUFs) and Hardware Security
- Keywords
- neural network security, stealthy attacks, model poisoning, autonomous driving safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
