Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization
Open accessAbstract
A number of online services nowadays rely upon machine learning to extract valuable information from data collected in the wild. This exposes learning algorithms to the threat of data poisoning, i.e., a coordinate attack in which a fraction of the training data is controlled by the attacker and manipulated to subvert the learning process. To date, these attacks have been devised only against a limited class of binary learning algorithms, due to the inherent complexity of the gradient-based procedure used to optimize the poisoning points (a.k.a. adversarial training examples). In this work, we first extend the definition of poisoning attacks to multiclass problems. We then propose a novel poisoning algorithm based on the idea of back-gradient optimization, i.e., to compute the gradient of interest through automatic differentiation, while also reversing the learning procedure to drastically reduce the attack complexity. Compared to current poisoning strategies, our approach is able to target a wider class of learning algorithms, trained with gradient-based procedures, including neural networks and deep learning architectures. We empirically evaluate its effectiveness on several application examples, including spam filtering, malware detection, and handwritten digit recognition. We finally show that, similarly to adversarial test examples, adversarial training examples can also be transferred across different learning algorithms.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Under the LDP model, at least Ω(√n/ε) additive error in expectation must be incurred by any ε-DP mechanism for the same task.
“In contrast, under the LDP model, at least Ω( √ n/ϵ ) additive error in expectation must be incurred by any ϵ -DP mechanism for the same task [117], [111].”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [111]
When the aggregator is untrusted, Local Differential Privacy or Distributed Differential Privacy are needed to protect privacy of individuals.
“When the aggregator is untrusted which is often the case in distributed scenarios, Local Differential Privacy (LDP) [111] or Distributed Differential Privacy (DDP) are needed [112], [102] to protect privacy of individuals.”
From Privacy and Robustness in Federated Learning: Attacks and Defenses (2024) · cites this paper as [111]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993 · see citing works
Practical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495 · see citing works
Universal Adversarial Perturbations
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., & Frossard, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.17 · Cited by 2,680 · see citing works
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194 · see citing works
Evasion Attacks against Machine Learning at Test Time
Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., & Roli, F. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40994-3_25 · Cited by 890 · see citing works
Self-citationCan machine learning be secure?
Barreno, M., Nelson, B. A., Sears, R., Joseph, A. D., & Tygar, J. D. (2006) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/1128817.1128824 · Cited by 852 · see citing works
The security of machine learning
Barreno, M., Nelson, B. A., Joseph, A. D., & Tygar, J. D. (2010) · Machine Learning · doi:10.1007/s10994-010-5188-5 · Cited by 839 · see citing works
Fast Exact Multiplication by the Hessian
Pearlmutter, B. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.1.147 · Cited by 553 · see citing works
Certified Defenses for Data Poisoning Attacks
Steinhardt, J., Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1706.03691 · Cited by 479 · see citing works
Gradient-Based Optimization of Hyperparameters
Bengio, Y. (2000) · Neural Computation · doi:10.1162/089976600300015187 · Cited by 467 · see citing works
Security Evaluation of Pattern Classifiers under Attack
Biggio, B., Fumera, G., & Roli, F. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2013.57 · Cited by 412 · see citing works
Self-citationLearning in the Presence of Malicious Errors
Kearns, M. J., & Li, M. (1993) · SIAM Journal on Computing · doi:10.1137/0222052 · Cited by 340 · see citing works
Practical Evasion of a Learning-Based Classifier: A Case Study
Šrndić, N., & Laskov, P. (2014) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2014.20 · Cited by 340 · see citing works
ANTIDOTE: understanding and defending against poisoning of anomaly detectors
Rubinstein, B. I. P., Nelson, B., Huang, L., Joseph, A. D., Lau, S.-H., Rao, S., Taft, N., & Tygar, J. D. (2009) · Proceedings of the 9th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1644893.1644895 · Cited by 331 · see citing works
Exploiting machine learning to subvert your spam filter
Nelson, B. A., Barreno, M., Chi, F. J., Joseph, A. D., Rubinstein, B. I. P., Saini, U., Sutton, C. A., Tygar, J. D., & Xia, K. (2008) · Cited by 322 · see citing works
Malicious PDF detection using metadata and structural features
Smutz, C., & Stavrou, A. (2012) · Proceedings of the 28th Annual Computer Security Applications Conference · doi:10.1145/2420950.2420987 · Cited by 271 · see citing works
Automated Dynamic Analysis of Ransomware: Benefits, Limitations and use for DetectionPreprint
Sgandurra, D., Muñoz-González, L., Mohsen, R., & Lupu, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.03020 · Cited by 241 · see citing works
Self-citationPoisoning behavioral malware clustering
Biggio, B., Rieck, K., Ariu, D., Wressnegger, C., Corona, I., Giacinto, G., & Roli, F. (2014) · doi:10.1145/2666652.2666666 · Cited by 138 · see citing works
Self-citationBagging Classifiers for Fighting Poisoning Attacks in Adversarial Classification Tasks
Biggio, B., Corona, I., Fumera, G., Giacinto, G., & Roli, F. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21557-5_37 · Cited by 111 · see citing works
Self-citationIs data clustering in adversarial settings secure?
Biggio, B., Pillai, I., Bulò, S. R., Ariu, D., Pelillo, M., & Roli, F. (2013) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/2517312.2517321 · Cited by 105 · see citing works
Self-citationSecurity Analysis of Online Centroid Anomaly DetectionPreprint
Kloft, M., & Laskov, P. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1003.0078 · Cited by 83 · see citing works
Poisoning Complete-Linkage Hierarchical Clustering
Biggio, B., Bulò, S. R., Pillai, I., Mura, M. L., Zemene, E., Pelillo, M., & Roli, F. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44415-3_5 · Cited by 48 · see citing works
Self-citationWatanabe, O., & Yokomori, T. (1999) · Lecture notes in computer science · doi:10.1007/3-540-46769-6 · Cited by 15 · see citing works
Bshouty, N. H., Eiron, N., & Kushilevitz, E. (1999) · Lecture notes in computer science · doi:10.1007/3-540-46769-6_17 · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 16
- [8][8] Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012. Poisoning attacks against support vector machines, In 29th Int'l Conf. on Machine Learning, John Langford and Joelle Pineau (Eds.). Int'l Conf. on Machine Learning (ICML) , 1807-1814. -
- [13][13] C. Do, C.S. Foo, and A.Y. Ng. 2008. Efficient multiple hyperparameter learning for log-linear models. In Advances in Neural Information Processing Systems . 377-384. -
- [14][14] Justin Domke. 2012. Generic Methods for Optimization-Based Modeling. In 15th Int'l Conf. Artificial Intelligence and Statistics (Proceedings of Machine Learning Research) , Neil D. Lawrence and Mark Girolami (Eds.), Vol. 22. PMLR, La Palma, Canary Islands, 318-326. -
- [15][15] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations . -
- [17][17] Anthony D. Joseph, Pavel Laskov, Fabio Roli, J. Doug Tygar, and Blaine Nelson. 2013. Machine Learning Methods for Computer Security (Dagstuhl Perspectives Workshop 12371). Dagstuhl Manifestos 3, 1 (2013), 1-30. -
- [20][20] P. W. Koh and P. Liang. 2017. Understanding Black-box Predictions via Influence Functions. In International Conference on Machine Learning (ICML) . -
- [21][21] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. GradientBased Learning Applied to Document Recognition. In Proceedings of the IEEE , Vol. 86. 2278-2324. -
- [22][22] Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. 2015. Gradient-based Hyperparameter Optimization Through Reversible Learning. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37 (ICML'15) . JMLR.org, 2113-2122. -
- [23][23] Shike Mei and Xiaojin Zhu. 2015. Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners. In 29th AAAI Conf. Artificial Intelligence (AAAI '15) . -
- [26][26] Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D. Joseph, Benjamin I. P. Rubinstein, Udam Saini, Charles Sutton, J. D. Tygar, and Kai Xia. 2008. Exploiting machine learning to subvert your spam filter. In LEET'08: Proceedings of the 1st Usenix Workshop on Large-Scale Exploits and Emergent Threats . USENIX Association, Berkeley, CA, USA, 1-9. -
- [29][29] K.R. Patil, X. Zhu, L. Kopeć, and B.C. Love. 2014. Optimal teaching for limitedcapacity human learners. In Advances in Neural Information Processing Systems . 2465-2473. -
- [31][31] F. Pedregosa. 2016. Hyperparameter optimization with approximate gradient. In 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research) , Maria Florina Balcan and Kilian Q. Weinberger (Eds.), Vol. 48. PMLR, New York, New York, USA, 737-746. -
- [36][36] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations . http://arxiv. org/abs/1312.6199 -
- [38][38] GangWang,Tianyi Wang, Haitao Zheng, and Ben Y. Zhao. 2014. Man vs. Machine: Practical Adversarial Detection of Malicious Crowdsourcing Workers. In 23rd USENIX Security Symposium (USENIX Security 14) . USENIX Association, San Diego, CA. -
- [39][39] Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. 2015. Is Feature Selection Secure against Training Data Poisoning?. In JMLR W&CP - Proc. 32nd Int'l Conf. Mach. Learning (ICML) , Francis Bach and David Blei (Eds.), Vol. 37. 1689-1698. -
- [40][40] X. Zhu. 2013. Machine Teaching for Bayesian Learners in the Exponential Family. In Advances in Neural Information Processing Systems . 1905-1913.
Cited by
Sorted by citations
A survey on security and privacy of federated learning
Mothukuri, V., Parizi, R. M., Pouriyeh, S., Huang, Y., Dehghantanha, A., & Srivastava, G. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2020.10.007 · Cited by 1,480 · see citing works
Trojaning Attack on Neural Networks
Liu, Y., Ma, S., Aafer, Y., Lee, W.-C., Zhai, J., Wang, W., & Zhang, X. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23291 · Cited by 1,270 · see citing works
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks
Liu, K., Dolan-Gavitt, B., & Garg, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00470-5_13 · Cited by 1,055 · see citing works
Data Poisoning Attacks Against Federated Learning Systems
Tolpegin, V., Truex, S., Gursoy, M. E., & Liu, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58951-6_24 · Cited by 927 · see citing works
FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping
Cao, X., Fang, M., Liu, J., & Gong, N. Z. (2021) · Proceedings 2021 Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.24434 · Cited by 999 · see citing works
Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning
Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., & Li, B. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00057 · Cited by 861 · see citing works
Self-citationManipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning
Shejwalkar, V., & Houmansadr, A. (2021) · Proceedings 2021 Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.24498 · Cited by 615 · see citing works
Privacy and Robustness in Federated Learning: Attacks and Defenses
Lyu, L., Yu, H., Ma, X., Chen, C., Sun, L., Zhao, J., Yang, Q., & Yu, P. S. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3216981 · Cited by 548 · see citing works
PRADA: Protecting Against DNN Model Stealing Attacks
Juuti, M., Szyller, S., Marchal, S., & Asokan, N. (2019) · 2019 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2019.00044 · Cited by 440 · see citing works
Biscotti: A blockchain system for private and secure federated learning
Shayan, M., Fung, C., Yoon, C. J. M., & Beschastnikh, I. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2020.3044223 · Cited by 354 · see citing works
Weight Poisoning Attacks on Pretrained Models
Kurita, K., Michel, P., & Neubig, G. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.249 · Cited by 354 · see citing works
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., & Goldstein, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3162397 · Cited by 362 · see citing works
Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain
Rosenberg, I., Shabtai, A., Elovici, Y., & Rokach, L. (2022) · ACM Computing Surveys · doi:10.1145/3453158 · Cited by 295 · see citing works
PoisonGAN: Generative Poisoning Attacks Against Federated Learning in Edge Computing Systems
Zhang, J., Chen, B., Cheng, X., Binh, H. T. T., & Yu, S. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3023126 · Cited by 291 · see citing works
Shejwalkar, V., Houmansadr, A., Kairouz, P., & Ramage, D. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833647 · Cited by 292 · see citing works
Understanding Distributed Poisoning Attack in Federated Learning
Cao, D., Chang, S., Lin, Z., Liu, G., & Sun, D. (2019) · 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads47876.2019.00042 · Cited by 285 · see citing works
Vulnerabilities in federated learning
Bouacida, N., & Mohapatra, P. (2021) · IEEE Access · doi:10.1109/access.2021.3075203 · Cited by 248 · see citing works
LIRA: Learnable, Imperceptible and Robust Backdoor Attacks
Doan, K., Lao, Y., Zhao, W., & Li, P. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01175 · Cited by 211 · see citing works
Wang, T., Zhu, J.-Y., Torralba, A., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1811.10959 · Cited by 194 · see citing works
Deep Model Poisoning Attack on Federated Learning
Zhou, X., Xu, M., Wu, Y., & Zheng, N. (2021) · Future Internet · doi:10.3390/fi13030073 · Cited by 170 · see citing works
CONTRA: Defending Against Poisoning Attacks in Federated Learning
Awan, S., Luo, B., & Li, F. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-88418-5_22 · Cited by 182 · see citing works
Poisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00179 · Cited by 138 · see citing works
Poisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2023) · arXiv · doi:10.48550/arxiv.2302.10149 · Cited by 138 · see citing works
Blockchain analytics and artificial intelligence
Dillenberger, D. N., Novotny, P., Zhang, Q., Jayachandran, P., Gupta, H., Hans, S., Verma, D., Chakraborty, S., Thomas, J. J., Walli, M. M., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2900638 · Cited by 82 · see citing works
An Overview of Backdoor Attacks Against Deep Neural Networks and Possible Defences
Guo, W., Tondi, B., & Barni, M. (2022) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2022.3190213 · Cited by 91 · see citing works
Adversarial Label-Flipping Attack and Defense for Graph Neural Networks
Zhang, M., Hu, L., Shi, C., & Wang, X. (2020) · 2020 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm50108.2020.00088 · Cited by 74 · see citing works
A Systematic Review of Adversarial Machine Learning Attacks, Defensive Controls and Technologies
Malik, J., Muthalagu, R., & Pawar, P. M. (2024) · IEEE Access · doi:10.1109/access.2024.3423323 · Cited by 76 · see citing works
Hard to Forget: Poisoning Attacks on Certified Machine Unlearning
Marchant, N. G., Rubinstein, B. I. P., & Alfeld, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20736 · Cited by 64 · see citing works
“Following standard threat models in the data poisoning literature (Biggio, Nelson, and Laskov, 2012; Muñoz-González et al., 2017; Shafahi et al., 2018; Shen, Xiaolei Zhu, and Ma, 2019; W. R. Huang et al., 2020) we consider attackers with white-box and grey-box access.”
Hidden Trigger Backdoor Attacks
Saha, A., Subramanya, A., & Pirsiavash, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6871 · Cited by 13 · see citing works
Neural networks memorise personal information from one sample
Hartley, J., Sanchez, P. P., Haider, F., & Tsaftaris, S. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-48034-3 · Cited by 10 · see citing works
- Authors
- Muñoz-González, L., Biggio, B., Demontis, A., Paudice, A., Wongrassamee, V., Lupu, E. C., & Roli, F.
- Affiliations
- Imperial College London (GB)
- University of Cagliari (IT)
- Venue
- Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security
- Publisher
- ACM
- Published
- 2017-11-03
- Pages
- 27-38
- OpenAlex
- W2750650104
- Cited by
- 549
- References
- 25
- Citations by year
- 2026: 16 · 2025: 54 · 2024: 75 · 2023: 82 · 2022: 79 · 2021: 92 · 2020: 75 · 2019: 46 · 2018: 29 · 2017: 1
- FWCI
- 18.83
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Workshop on Artificial Intelligence and Security (AISec) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- UNICA IRIS Institutional Research Information System (University of Cagliari) (submittedVersion)
- CINECA IRIS Institutial Research Information System (University of Genoa) (submittedVersion)
- arXiv (Cornell University)
- Funders
- UK EPSRC (EP/L022729/1, EP/L016796/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Keywords
- data poisoning, deep learning security, model transferability, spam filtering, malware detection, handwritten digit recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
