A Synergetic Attack against Neural Network Classifiers combining Backdoor and Adversarial Examples
Open accessAbstract
The pervasiveness of neural networks (NNs) in critical computer vision and image processing applications makes them very attractive for adversarial manipulation. A large body of existing research thoroughly investigates two broad categories of attacks targeting the integrity of NN models. The first category of attacks, commonly called Adversarial Examples, perturbs the model’s inference by carefully adding noise into input examples. In the second category of attacks, adversaries try to manipulate the model during the training process by implanting Trojan backdoors. Researchers show that such attacks pose severe threats to the growing applications of NNs and propose several defenses against each attack type individually. However, such one-sided defense approaches leave potentially unknown risks in real-world scenarios when an adversary can unify different attacks to create new and more lethal ones bypassing existing defenses.In this work, we show how to jointly exploit adversarial perturbation and model poisoning vulnerabilities to practically launch a new stealthy attack, dubbed AdvTrojan. AdvTrojan is stealthy because it can be activated only when: 1) a carefully crafted adversarial perturbation is injected into the input examples during inference, and 2) a Trojan backdoor is implanted during the training process of the model. We leverage adversarial noise in the input space to move Trojan-infected examples across the model decision boundary, making it difficult to detect. The stealthiness behavior of AdvTrojan fools the users into accidentally trusting the infected model as a robust classifier against adversarial examples. AdvTrojan can be implemented by only poisoning the training data similar to conventional Trojan backdoor attacks. Our thorough analysis and extensive experiments on several benchmark datasets show that AdvTrojan can bypass existing defenses with a success rate close to 100% in most of our experimental scenarios and can be extended to attack federated learning as well as high-resolution images.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The adaptive learning rate defense requires individual update information from participants on the server side, is not compatible with SecAgg, and cannot guarantee participants' privacy security.
“Similar to the aggregation modification method, the proposed defense mechanism requires the individual to update information from the participant on the server side. Thus, it is not compatible with the SecAgg. The participants' privacy security cannot be guaranteed.”
From Data and model poisoning backdoor attacks on wireless federated learning, and the defense mechanisms: A comprehensive survey (2024) · cites this paper as [213]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863 · see citing works
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · see citing works
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602 · see citing works
On Assessing ML Model Robustness: A Methodological Framework (Academic Track)
Awadid, A., & Robert, B. (2025) · OASIcs : OpenAccess Series in Informatics · doi:10.4230/oasics.saia.2024.1 · Cited by 4,723 · see citing works
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P. D., Wu, X., Jha, S., & Swami, A. (2016) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2016.41 · Cited by 3,123 · see citing works
Adversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325 · see citing works
Ensemble Adversarial Training: Attacks and Defenses
Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., & McDaniel, P. (2017) · arXiv · doi:10.48550/arxiv.1705.07204 · Cited by 2,176 · see citing works
Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
Athalye, A., Carlini, N., & Wagner, D. (2018) · arXiv · doi:10.48550/arxiv.1802.00420 · Cited by 2,172 · see citing works
How to backdoor federated learning
Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2018) · arXiv · doi:10.48550/arxiv.1807.00459 · Cited by 1,495 · see citing works
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H.-T., & Zhao, B. Y. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00031 · Cited by 1,486 · see citing works
(2017) · neural information processing systems · Cited by 1,413 · see citing works
Trojaning Attack on Neural Networks
Liu, Y., Ma, S., Aafer, Y., Lee, W.-C., Zhai, J., Wang, W., & Zhang, X. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23291 · Cited by 1,270 · see citing works
BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
Gu, T., Dolan-Gavitt, B., & Garg, S. (2017) · arXiv · doi:10.48550/arxiv.1708.06733 · Cited by 1,061 · see citing works
Adversarial Examples Are Not Bugs, They Are Features
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., & Madry, A. (2019) · arXiv · doi:10.48550/arxiv.1905.02175 · Cited by 971 · see citing works
Certified Robustness to Adversarial Examples with Differential Privacy
Lecuyer, M., Atlidakis, V., Geambasu, R., Hsu, D., & Jana, S. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00044 · Cited by 750 · see citing works
Gao, Y., Xu, C., Wang, D., Chen, S., Ranasinghe, D. C., & Nepal, . (2019) · Annual Computer Security Applications Conference · doi:10.1145/3359789.3359790 · Cited by 719 · see citing works
DBA: Distributed Backdoor Attacks against Federated Learning
Xie, C., Huang, K., Chen, P., & Li, B. (2020) · International Conference on Learning Representations · Cited by 468 · see citing works
ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation
Liu, Y., Lee, W., Tao, G., Ma, S., Aafer, Y., & Zhang, X. (2019) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3319535.3363216 · Cited by 419 · see citing works
Practical Black-Box Attacks against Deep Learning Systems using Adversarial Examples.Preprint
Papernot, N., McDaniel, P. D., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2016) · arXiv (Cornell University) · Cited by 286 · see citing works
Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems
Doan, B. G., Abbasnejad, E., & Ranasinghe, D. C. (2020) · Annual Computer Security Applications Conference · doi:10.1145/3427228.3427264 · Cited by 263 · see citing works
Certified Adversarial Robustness with Additive NoisePreprint
Li, B., Chen, C., Wang, W., & Carin, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03113 · Cited by 225 · see citing works
Model-Reuse Attacks on Deep Learning Systems
Ji, Y., Zhang, X., Ji, S., Luo, X., & Wang, T. (2018) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3243734.3243757 · Cited by 174 · see citing works
STRIP: A Defence Against Trojan Attacks on Deep Neural NetworksPreprint
Gao, Y., Xu, C., Wang, D., Chen, S., Ranasinghe, D. C., & Nepal, . (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06531 · Cited by 127 · see citing works
Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation
Liao, C., Zhong, H., Squicciarini, A., Zhu, S., & Miller, D. (2018) · arXiv · doi:10.48550/arxiv.1808.10307 · Cited by 110 · see citing works
Improving the Generalization of Adversarial Training with Domain AdaptationPreprint
Song, C., He, K., Wang, L., & Hopcroft, J. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.00740 · Cited by 99 · see citing works
A Tale of Evil Twins: Adversarial Inputs versus Poisoned Models
Pang, R., Shen, H., Zhang, X., Ji, S., Vorobeychik, Y., Luo, X., Liu, A., & Wang, T. (2020) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3372297.3417253 · Cited by 85 · see citing works
Backdooring and Poisoning Neural Networks with Image-Scaling Attacks
Quiring, E., & Rieck, K. (2020) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw50608.2020.00024 · Cited by 74 · see citing works
Scalable Differential Privacy with Certified Robustness in Adversarial LearningPreprint
Phan, N., Thai, M. T., Hu, H., Jin, R., Sun, T., & Dou, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09822 · Cited by 36 · see citing works
Self-citationOn the Trade-off between Adversarial and Backdoor Robustness
Weng, C., Lee, Y.-T., & Wu, S.-H. (2020) · neural information processing systems · Cited by 27 · see citing works
ZK-GanDef: A GAN Based Zero Knowledge Adversarial Training Defense for Neural Networks
Liu, G., Khalil, I. M., & Khreishah, A. A. (2019) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2019.00021 · Cited by 21 · see citing works
Self-citation
Flagged first, then by citations
3 of the 37 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Ferrag, M. A., Friha, O., Kantarcı, B., Tihanyi, N., Cordeiro, L. C., Debbah, M. A., Hamouda, D., Al-Hawawreh, M. S., & Choo, K. R. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3317242 · Cited by 170 · see citing works
Wan, Y., Qu, Y., Ni, W., Xiang, Y., Gao, L., & Hossain, E. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3361451 · Cited by 152 · see citing works
“3 The evaluation metrics used in this case are BAR and MAR, where BAR denotes the probability that the global model is misled to classify backdoor inputs into the target label, and MAR evaluates the prediction accuracy of the backdoor-injected global model on clean inputs.”
MethodRed Alarm for Pre-trained Models: Universal Vulnerability to Neuron-level Backdoor Attacks
Zhang, Z., Xiao, G., Li, Y., Lv, T. S., Qi, F., Liu, Z., Wang, Y., Jiang, X., & Sun, M. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1377-5 · Cited by 70 · see citing works
Backdoor Attacks and Defenses Targeting Multi-Domain AI Models: A Comprehensive Review
Zhang, S., Pan, Y., Liu, Q., Yan, Z., Choo, K. R., & Wang, G. (2024) · ACM Computing Surveys · doi:10.1145/3704725 · Cited by 47 · see citing works
A concealed poisoning attack to reduce deep neural networks’ robustness against adversarial samples
Zheng, J., Chan, P. P. K., Chi, H., & He, Z. (2022) · Information Sciences · doi:10.1016/j.ins.2022.09.060 · Cited by 24 · see citing works
Poison-Tolerant Collaborative Filtering Against Poisoning Attacks on Recommender Systems
Baker, T., Li, T., Jia, J., Zhang, B., Tan, C., & Zomaya, A. Y. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3354462 · Cited by 11 · see citing works
FinBack: Infiltrating Backdoors into Gradient Compressors on Federated Learning
Tang, X., Wen, Y., Peng, L., Shen, M., Zhang, T., Weng, Y., Kang, J., & Niyato, D. T. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3633157 · Cited by 9 · see citing works
Verifying Binary Neural Networks on Continuous Input Space using Star Reachability
Ivashchenko, M., Choi, S.-W., Nguyen, L. V., & Tran, H.-D. (2023) · FME Workshop on Formal Methods in Software Engineering · doi:10.1109/formalise58978.2023.00009 · Cited by 4 · see citing works
An Adaptive Black-box Defense against Trojan Attacks on Text Data
Sharadgah, F., Khreishah, A. A., Al‐Ayyoub, M., Jararweh, Y., Liu, G., Khalil, I. M., Almutiry, M., & Saeed, N. (2021) · International Conference on Social Networks Analysis, Management and Security (SNAMS) · doi:10.1109/snams53716.2021.9732112 · Cited by 3 · see citing works
Self-citationMATFL: Defending Against Synergetic Attacks in Federated Learning
Yang, W. D., Peng, L., Tang, X., & Weng, Y. (2023) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics60724.2023.00072
The Sinister Ghost in the Wooden Horse: From Ancient Myths to Industrial Fraud by Trojan Attack
Eakin, M., Khosravy, M., Adams, A., & Witkowski, O. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-2329-7_45
- Authors
- Liu, G., Khalil, I., Khreishah, A., & Phan, N.
- Affiliations
- New Jersey Institute of Technology (US)
- Venue
- 2021 IEEE International Conference on Big Data (Big Data)
- Publisher
- IEEE
- Published
- 2021-12-15
- Pages
- 834-846
- OpenAlex
- W3197655726
- Cited by
- 11
- References
- 37
- Citations by year
- 2025: 2 · 2024: 3 · 2023: 4 · 2022: 1 · 2021: 1
- FWCI
- 1.04
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE International Conference on Big Data (BigData) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Keywords
- adversarial examples, model poisoning, neural network security, stealthy attacks, adversarial perturbations, high-resolution imagery, defense evasion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
