The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
A taxonomy and survey of attacks against machine learning
Open accessAbstract
The majority of machine learning methodologies operate with the assumption that their environment is benign. However, this assumption does not always hold, as it is often advantageous to adversaries to maliciously modify the training (poisoning attacks) or test data (evasion attacks). Such attacks can be catastrophic given the growth and the penetration of machine learning applications in society. Therefore, there is a need to secure machine learning enabling the safe adoption of it in adversarial cases, such as spam filtering, malware detection, and biometric recognition. This paper presents a taxonomy and survey of attacks against systems that use machine learning. It organizes the body of knowledge in adversarial machine learning so as to identify the aspects where researchers from different fields can contribute to. The taxonomy identifies attacks which share key characteristics and as such can potentially be addressed by the same defence approaches. Thus, the proposed taxonomy makes it easier to understand the existing attack landscape towards developing defence mechanisms, which are not investigated in this survey. The taxonomy is also leveraged to identify open problems that can lead to new research areas within the field of adversarial machine learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“We observe that visual recognition is the second most investigated domain, which largely is due to the existence and the ease of use of well-known datasets such as the MNIST database of handwritten digits [36], ImageNet database [37].”
MethodLearning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
Practical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258
Boosting Adversarial Attacks with Momentum
Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., & Li, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00957 · Cited by 2,956
Advances in Neural Information Processing Systems 20
Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008) · neural information processing systems · Cited by 2,846
Artificial intelligence and statistics
Yu, B., & Kumbier, K. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700813 · Cited by 2,329
Adversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325
A database for handwritten text recognition research
Hull, J. J. (1994) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.291440 · Cited by 2,244
Robust physical-world attacks on deep learning visual classification
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00175 · Cited by 2,173
Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
Athalye, A., Carlini, N., & Wagner, D. (2018) · arXiv · doi:10.48550/arxiv.1802.00420 · Cited by 2,172 · cited 1×
“[72] investigated the prevalence of obfuscated gradients on neural networks in MNIST [56] and CIFAR10 [75] datasets.”
BackgroundThreat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
Akhtar, N., & Mian, A. (2018) · IEEE Access · doi:10.1109/access.2018.2807385 · Cited by 2,021
Evasion attacks against machine learning at test time
Biggio, B., Corona, I., Maiorca, D., Nelson, B. A., Srndic, N., Laskov, P., Giacinto, G., & Roli, F. (2017) · CINECA IRIS Institutial Research Information System (University of Genoa) · Cited by 1,870
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733
Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition
Sharif, M., Bhagavatula, S., Bauer, L., & Reiter, M. K. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978392 · Cited by 1,530
Wild patterns: Ten years after the rise of adversarial machine learning
Biggio, B., & Roli, F. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.023 · Cited by 1,409
Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
Papernot, N., McDaniel, P., & Goodfellow, I. (2016) · arXiv · doi:10.48550/arxiv.1605.07277 · Cited by 1,326
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194
Dalvi, N., Domingos, P., Mausam, Sanghai, S., & Verma, D. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014066 · Cited by 887
Can machine learning be secure?
Barreno, M., Nelson, B., Sears, R., Joseph, A. D., & Tygar, J. D. (2006) · Proceedings of the 2006 ACM Symposium on Information, computer and communications security · doi:10.1145/1128817.1128824 · Cited by 836
The security of machine learning
Barreno, M., Nelson, B., Joseph, A. D., & Tygar, J. D. (2010) · Machine Learning · doi:10.1007/s10994-010-5188-5 · Cited by 817
Lowd, D., & Meek, C. (2005) · Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining · doi:10.1145/1081870.1081950 · Cited by 755
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization
Muñoz-González, L., Biggio, B., Demontis, A., Paudice, A., Wongrassamee, V., Lupu, E. C., & Roli, F. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140451 · Cited by 549
Exploiting machine learning to subvert your spam filter
Nelson, B. A., Barreno, M., Chi, F. J., Joseph, A. D., Rubinstein, B. I. P., Saini, U., Sutton, C. A., Tygar, J. D., & Xia, K. (2008) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 481
Neural Networks (IJCNN), The 2011 International Joint Conference on
Pokharel, R., Seth, S., & Principe, J. C. (2011) · CiteSeer X (The Pennsylvania State University) · Cited by 429
Crafting adversarial input sequences for recurrent neural networks
Papernot, N., McDaniel, P., Swami, A., & Harang, R. (2016) · MILCOM 2016 - 2016 IEEE Military Communications Conference · doi:10.1109/milcom.2016.7795300 · Cited by 419
Security Evaluation of Pattern Classifiers under Attack
Biggio, B., Fumera, G., & Roli, F. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2013.57 · Cited by 411
Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on
Ferrari, V., Marín-Jiménez, M. J., & Zisserman, A. (2008) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 397
Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners
Mei, S., & Zhu, X. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9569 · Cited by 346 · cited 1×
“[87] presented an efficient solution for a broad class of poisoning attacks against spam email detection systems and wine quality modelling systems.”
BackgroundANTIDOTE: understanding and defending against poisoning of anomaly detectors
Rubinstein, B. I. P., Nelson, B., Huang, L., Joseph, A. D., Lau, S.-H., Rao, S., Taft, N., & Tygar, J. D. (2009) · Proceedings of the 9th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1644893.1644895 · Cited by 331
Adversarial Feature Selection Against Evasion Attacks
Zhang, F., Chan, P. P. K., Biggio, B., Yeung, D. S., & Roli, F. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2415032 · Cited by 243
Static prediction games for adversarial learning problems
Brückner, M., Kanzow, C., & Scheffer, T. (2012) · publish.UP (University of Potsdam) · doi:10.5555/2503308.2503326 · Cited by 240
Stackelberg games for adversarial prediction problems
Brückner, M., & Scheffer, T. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020495 · Cited by 229 · cited 1×
“…al. [41] Wang et al. [42] Demetrio et al. [43] Table 3: Comparative analysis of Intrusion Detection Systems’ Attack Evaluation Bruckner and Scheffer [47] in the context of spam email filtering study Stack-elberg games in which the learner acts as the leader committing to a predictive model, while…”
BackgroundEnhancing robustness of machine learning systems via data transformations
Bhagoji, A. N., Cullina, D., Sitawarin, C., & Mittal, P. (2018) · 2018 52nd Annual Conference on Information Sciences and Systems (CISS) · doi:10.1109/ciss.2018.8362326 · Cited by 217
Support vector machines under adversarial label contamination
Xiao, H., Biggio, B., Nelson, B., Xiao, H., Eckert, C., & Roli, F. (2015) · Neurocomputing · doi:10.1016/j.neucom.2014.08.081 · Cited by 216
Data Poisoning Attacks against Autoregressive Models
Alfeld, S., Zhu, X., & Barford, P. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10237 · Cited by 205
Learning Universal Adversarial Perturbations with Generative Models
Hayes, J., & Danezis, G. (2018) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00015 · Cited by 187
Adversary Resistant Deep Neural Networks with an Application to Malware Detection
Wang, Q., Guo, W., Zhang, K., Ororbia, A. G., Xing, X., Liu, X., & Giles, C. L. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098158 · Cited by 176
A taxonomy and survey of cyber-physical intrusion detection approaches for vehicles
Loukas, G., Karapistoli, E., Panaousis, E., Sarigiannidis, P. G., Bezemskij, A., & Vuong, T. P. (2018) · Ad Hoc Networks · doi:10.1016/j.adhoc.2018.10.002 · Cited by 156
Self-citationAdversarial support vector machine learning
Zhou, Y., Kantarcıoğlu, M., Thuraisingham, B. M., & Xi, B. (2012) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2339530.2339697 · Cited by 143
2008 IEEE International Conference on Data Mining Workshops
(2008) · doi:10.1109/icdmw14999.2008 · Cited by 142
Biggio, B., Fumera, G., Russu, P., Didaci, L., & Roli, F. (2015) · IEEE Signal Processing Magazine · doi:10.1109/msp.2015.2426728 · Cited by 119
Feature Cross-Substitution in Adversarial Classification
Li, B., & Vorobeychik, Y. (2014) · neural information processing systems · Cited by 111
Is data clustering in adversarial settings secure?
Biggio, B., Pillai, I., Bulò, S. R., Ariu, D., Pelillo, M., & Roli, F. (2013) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/2517312.2517321 · Cited by 109
A Survey of Adversarial Machine Learning in Cyber Warfare
Duddu, V. (2018) · Defence Science Journal · doi:10.14429/dsj.68.12371 · Cited by 94
2015 Systems and Information Engineering Design Symposium
(2015) · doi:10.1109/sieds34769.2015 · Cited by 90
Adversarial learning in credit card fraud detection
Zeager, M. F., Sridhar, A., Fogal, N., Adams, S., Brown, D. E., & Beling, P. A. (2017) · Systems and Information Engineering Design Symposium · doi:10.1109/sieds.2017.7937699 · Cited by 86
Detecting Poisoning Attacks on Machine Learning in IoT Environments
Baracaldo, N., Chen, B., Ludwig, H., Safavi, A., & Zhang, R. (2018) · IEEE International Congress on Internet of Things (ICIOT) · doi:10.1109/iciot.2018.00015 · Cited by 85
Randomized Prediction Games for Adversarial Machine Learning
Bulò, S. R., Biggio, B., Pillai, I., Pelillo, M., & Roli, F. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2593488 · Cited by 83
Proceedings of the 4th ACM workshop on Security and artificial intelligence
Chen, Y., Cárdenas, Á. A., Greenstadt, R., & Rubinstein, B. (2011) · doi:10.1145/2046684 · Cited by 80
A survey of game theoretic approach for adversarial machine learning
Zhou, Y., Kantarcıoğlu, M., & Xi, B. (2018) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1259 · Cited by 79
A Game Theoretical Model for Adversarial Learning
Liu, W., & Chawla, S. (2009) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2009.9 · Cited by 73
2018 IEEE Security and Privacy Workshops (SPW)
(2018) · doi:10.1109/spw45269.2018 · Cited by 73
A survey of practical adversarial example attacks
Sun, L., Tan, M., & Zhou, Z. G. (2018) · Cybersecurity · doi:10.1186/s42400-018-0012-9 · Cited by 68
Nash Equilibria of Static Prediction Games
Brückner, M., & Scheffer, T. (2009) · Cited by 59
Proceedings of the 2006 ACM Symposium on Information, computer and communications security
Lin, F., Lee, D.-T., Lin, B. P., Shieh, S., & Jajodia, S. (2006) · doi:10.1145/1128817 · Cited by 52
Safe Machine Learning and Defeating Adversarial Attacks
Rouhani, B. D., Samragh, M., Javidi, T., & Koushanfar, F. (2019) · IEEE Security & Privacy · doi:10.1109/msec.2018.2888779 · Cited by 48
Biggio, B., Corona, I., He, Z.-M., Chan, P. P. K., Giacinto, G., Yeung, D. S., & Roli, F. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-20248-8_15 · Cited by 47
People‐centric sensing in assistive healthcare: Privacy challenges and directions
Giannetsos, T., Dimitriou, T. D., & Prasad, N. R. (2011) · Security and Communication Networks · doi:10.1002/sec.313 · Cited by 41
Self-citationSEROSA: SERvice oriented security architecture for Vehicular Communications
Gisdakis, S., Laganà, M., Giannetsos, T., & Papadimitratos, P. (2013) · IEEE Vehicular Networking Conference (VNC) · doi:10.1109/vnc.2013.6737597 · Cited by 37
Self-citationProceedings of the 2013 ACM workshop on Artificial intelligence and security
Sadeghi, A.-R., Germany, C. 2. A. S. C. O. C. A. C. S. 2. B., Association for Computing Machinery-Digital Library, Audit and Control ACM Special Interest Group on Security, & SIGs, A. (2013) · doi:10.1145/2517312 · Cited by 32
Neural TrojansPreprint
Liu, Y., Xie, Y., & Srivastava, A. (2017) · arXiv (Cornell University) · Cited by 29
Large-scale strategic games and adversarial machine learning
Alpcan, T., Rubinstein, B. I. P., & Leckie, C. A. (2016) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2016.7798940 · Cited by 24
Jeong, J., Kwon, S., Hong, M., Kwak, J., & Shon, T. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-7262-8 · Cited by 22
Adversarial learning games with deep learning models
Chivukula, A. S., & Liu, W. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966196 · Cited by 18
Fraud Detection with Multi-Modal Attention and Correspondence Learning
Park, J., Kim, M.-H., Choi, S. B., Kweon, I. S., & Choi, D. (2019) · International Conference on Electronics, Information, and Communication (ICEIC) · doi:10.23919/elinfocom.2019.8706354 · Cited by 8
Unsupervised Learning for Trustworthy IoT
Banerjee, N., Giannetsos, T., Panaousis, E., & Took, C. C. (2018) · 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz-ieee.2018.8491672 · Cited by 3
Self-citation
Cited by
A Survey on Machine Learning Techniques for Cyber Security in the Last Decade
Shaukat, K., Luo, S., Varadharajan, V., Hameed, I. A., & Xu, M. (2020) · IEEE Access · doi:10.1109/access.2020.3041951 · Cited by 530
Digital Twin: A Comprehensive Survey of Security Threats
Alcaraz, C., & Lopez, J. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3171465 · Cited by 340
Design, Modeling and Implementation of Digital Twins
Segovia, M., & Garcia-Alfaro, J. (2022) · Sensors · doi:10.3390/s22145396 · Cited by 332
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., & Goldstein, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3162397 · Cited by 362
“A broad discussion concerning security for machine learning can be found in [10] and [11].”
BackgroundMachine learning algorithms for social media analysis: A survey
T.K., B., Annavarapu, C. S. R., & Bablani, A. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100395 · Cited by 276
Metadata
- Authors
- Pitropakis, N., Panaousis, E., Giannetsos, T., Anastasiadis, E., & Loukas, G.
- Venue
- Computer Science Review
- Publisher
- Elsevier BV
- Published
- 2019-11
- Volume
- 34
- Pages
- 100199
- ISSN
- 1574-0137 (ISSN-L)
- OpenAlex
- W2981446616
- Cited by
- 263
- References
- 76
- FWCI
- 12.55
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.