The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Intriguing Properties of Adversarial ML Attacks in the Problem Space
Open accessAbstract
Recent research efforts on adversarial ML have investigated problem-space attacks, focusing on the generation of real evasive objects in domains where, unlike images, there is no clear inverse mapping to the feature space (e.g., software). However, the design, comparison, and real-world implications of problem-space attacks remain underexplored.This paper makes two major contributions. First, we propose a novel formalization for adversarial ML evasion attacks in the problem-space, which includes the definition of a comprehensive set of constraints on available transformations, preserved semantics, robustness to preprocessing, and plausibility. We shed light on the relationship between feature space and problem space, and we introduce the concept of side-effect features as the byproduct of the inverse feature-mapping problem. This enables us to define and prove necessary and sufficient conditions for the existence of problem-space attacks. We further demonstrate the expressive power of our formalization by using it to describe several attacks from related literature across different domains.Second, building on our formalization, we propose a novel problem-space attack on Android malware that overcomes past limitations. Experiments on a dataset with 170K Android apps from 2017 and 2018 show the practical feasibility of evading a state-of-the-art malware classifier along with its hardened version. Our results demonstrate that "adversarial-malware as a service" is a realistic threat, as we automatically generate thousands of realistic and inconspicuous adversarial applications at scale, where on average it takes only a few minutes to generate an adversarial app. Yet, out of the 1600+ papers on adversarial ML published in the past six years, roughly 40 focus on malware [15]-and many remain only in the feature space.Our formalization of problem-space attacks paves the way to more principled research in this domain. We responsibly release the code and dataset of our novel attack to other researchers, to encourage future work on defenses in the problem space.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
(2016) · Cited by 11,679
(2017) · Cited by 11,625
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
Weiser, M. (1984) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1984.5010248 · Cited by 3,204
Drebin: Effective and Explainable Detection of Android Malware in Your Pocket
Arp, D., Spreitzenbarth, M., Hübner, M., Gascon, H., & Rieck, K. (2014) · Proceedings 2014 Network and Distributed System Security Symposium · doi:10.14722/ndss.2014.23247 · Cited by 2,331
Types and Programming Languages
Pierce, B. C. (2002) · Cited by 2,226
Evasion attacks against machine learning at test time
Biggio, B., Corona, I., Maiorca, D., Nelson, B. A., Srndic, N., Laskov, P., Giacinto, G., & Roli, F. (2017) · CINECA IRIS Institutial Research Information System (University of Genoa) · Cited by 1,870
Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition
Sharif, M., Bhagavatula, S., Bauer, L., & Reiter, M. K. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978392 · Cited by 1,530
Wild patterns: Ten years after the rise of adversarial machine learning
Biggio, B., & Roli, F. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.023 · Cited by 1,409
Arzt, S., Rasthofer, S., Fritz, C., Bodden, E., Bartel, A., Klein, J., Le Traon, Y., Octeau, D., & McDaniel, P. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594299 · Cited by 1,396
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194 · cited 4×
“We focus on evasion attacks [12, 16, 32], where the adversary modifies objects at test time to induce targeted misclassifications.”
BackgroundInfluentialAudio Adversarial Examples: Targeted Attacks on Speech-to-Text
Carlini, N., & Wagner, D. (2018) · 2018 IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00009 · Cited by 1,035
Linear-time algorithms for testing the satisfiability of propositional horn formulae
Dowling, W. F., & Gallier, J. H. (1984) · The Journal of Logic Programming · doi:10.1016/0743-1066(84)90014-1 · Cited by 1,013
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
Carlini, N., & Wagner, D. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140444 · Cited by 966
Soot: a Java bytecode optimization framework
Vallée-Rai, R., Co, P., Gagnon, E., Hendren, L., Lam, P., & Sundaresan, V. (2010) · CASCON First Decade High Impact Papers on - CASCON '10 · doi:10.1145/1925805.1925818 · Cited by 933
Evasion Attacks against Machine Learning at Test Time
Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., & Roli, F. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40994-3_25 · Cited by 890 · cited 7×
“Many formalizations have been proposed in the literature to describe feature-space attacks, either as optimization problems [12, 16] (see also §II-A for details) or game theoretic frameworks [21].”
BackgroundInfluentialDalvi, N., Domingos, P., Mausam, Sanghai, S., & Verma, D. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014066 · Cited by 887
AndroZoo: collecting millions of Android apps for the research community
Allix, K., Bissyandé, T. F., Klein, J., & Le Traon, Y. (2016) · Proceedings of the 13th International Conference on Mining Software Repositories · doi:10.1145/2901739.2903508 · Cited by 825
Limits of Static Analysis for Malware Detection
Moser, A., Kruegel, C., & Kirda, E. (2007) · Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007) · doi:10.1109/acsac.2007.21 · Cited by 813
Hard and easy distributions of SAT problems
Mitchell, D. G. M., Selman, B., & Levesque, H. J. (1992) · Cited by 799
Generating Natural Language Adversarial Examples
Alzantot, M., Sharma, Y., Elgohary, A., Ho, B.-J., Srivastava, M., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1316 · Cited by 787
Test pattern generation using Boolean satisfiability
Larrabee, T. (1992) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/43.108614 · Cited by 703
Weiser, M. (1981) · doi:10.5555/800078.802557 · Cited by 695
Adversarial Examples for Malware Detection
Grosse, K., Papernot, N., Manoharan, P., Backes, M., & McDaniel, P. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66399-9_4 · Cited by 542
On Evaluating Adversarial Robustness
Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I., Mądry, A., & Kurakin, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06705 · Cited by 524
Security Evaluation of Pattern Classifiers under Attack
Biggio, B., Fumera, G., & Roli, F. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2013.57 · Cited by 411
Compilers: Principles, Techniques, and Tools (2nd Edition)
Aho, A. V., Lam, M. S., Sethi, R., & Ullman, J. D. (2006) · Addison-Wesley Longman Publishing Co., Inc. eBooks · Cited by 387
TextBugger: Generating Adversarial Text Against Real-world Applications
Li, J., Ji, S., Du, T., Li, B., & Wang, T. (2019) · Proceedings 2019 Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23138 · Cited by 343
Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables
Kolosnjaji, B., Demontis, A., Biggio, B., Maiorca, D., Giacinto, G., Eckert, C., & Roli, F. (2018) · 2018 26th European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553214 · Cited by 335
LIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328
Good Word Attacks on Statistical Spam Filters.
Lowd, D., & Meek, C. (2005) · Conference on Email and Anti-Spam · Cited by 316
Generating hard satisfiability problems
Selman, B., Mitchell, D. G., & Levesque, H. J. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(95)00045-3 · Cited by 296
Yes, Machine Learning Can Be More Secure! A Case Study on Android Malware Detection
Demontis, A., Melis, M., Biggio, B., Maiorca, D., Arp, D., Rieck, K., Corona, I., Giacinto, G., & Roli, F. (2019) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2017.2700270 · Cited by 288
Malicious PDF detection using metadata and structural features
Smutz, C., & Stavrou, A. (2012) · Proceedings of the 28th Annual Computer Security Applications Conference · doi:10.1145/2420950.2420987 · Cited by 271
Adversarial attacks against intrusion detection systems: Taxonomy, solutions and open issues
Corona, I., Giacinto, G., & Roli, F. (2013) · Information Sciences · doi:10.1016/j.ins.2013.03.022 · Cited by 225
Generic Black-Box End-to-End Attack Against State of the Art API Call Based Malware Classifiers
Rosenberg, I., Shabtai, A., Rokach, L., & Elovici, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00470-5_23 · Cited by 194
Evading network anomaly detection systems: formal reasoning and practical techniques
Fogla, P., & Lee, W. (2006) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/1180405.1180414 · Cited by 192
Arzt, S., Rasthofer, S., Fritz, C., Bodden, E., Bartel, A., Klein, J., Traon, Y. L., Octeau, D., & McDaniel, P. (2014) · Open Repository and Bibliography (University of Luxembourg) · Cited by 185
Static detection of malicious JavaScript-bearing PDF documents
Laskov, P., & Šrndić, N. (2011) · Proceedings of the 27th Annual Computer Security Applications Conference · doi:10.1145/2076732.2076785 · Cited by 161
Automated software transplantation
Barr, E. T., Harman, M., Jia, Y., Marginean, A., & Petke, J. (2015) · International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771796 · Cited by 160
Detection of Malicious PDF Files Based on Hierarchical Document Structure.
Šrndić, N., & Laskov, P. (2013) · Network and Distributed System Security Symposium · Cited by 151
Evading Classifiers by Morphing in the Dark
Dang, H., Huang, Y., & Chang, E.-C. (2017) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3133956.3133978 · Cited by 133
Yang, W. M., Kong, D., Xie, T., & Gunter, C. A. (2017) · Annual Computer Security Applications Conference · doi:10.1145/3134600.3134642 · Cited by 132
Maiorca, D., Corona, I., & Giacinto, G. (2013) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/2484313.2484327 · Cited by 111
A Pattern Recognition System for Malicious PDF Files Detection
Maiorca, D., Giacinto, G., & Corona, I. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31537-4_40 · Cited by 101
HideNoSeek: Camouflaging Malicious JavaScript in Benign ASTs
Fass, A., Michael Backes, A., & Stock, B. (2019) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3319535.3345656 · Cited by 83
Explaining Black-box Android Malware Detection
Melis, M., Maiorca, D., Biggio, B., Giacinto, G., & Roli, F. (2018) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553598 · Cited by 61
(2019) · USENIX Security Symposium · Cited by 60
Evading Botnet Detectors Based on Flows and Random Forest with Adversarial Samples
Apruzzese, G., & Colajanni, M. (2018) · IEEE International Symposium on Network Computing and Applications (NCA) · doi:10.1109/nca.2018.8548327 · Cited by 55
Evaluating the effectiveness of Adversarial Attacks against Botnet Detectors
Apruzzese, G., Colajanni, M., & Marchetti, M. (2019) · IEEE International Symposium on Network Computing and Applications (NCA) · doi:10.1109/nca.2019.8935039 · Cited by 51
Adversarially Robust Malware Detection Using Monotonic Classification
Incer, I., Theodorides, M., Afroz, S., & Wagner, D. (2018) · ACM International Workshop on Security and Privacy Analytics (IWSPA) · doi:10.1145/3180445.3180449 · Cited by 50
Adversarial Machine Learning Beyond the Image Domain
Zizzo, G., Hankin, C., Maffeis, S., & Jones, K. (2019) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3316781.3323470 · Cited by 38
Evading classifiers in discrete domains with provable optimality guaranteesPreprint
Kulynych, B., Hayes, J., Samarin, N., & Troncoso, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.10939 · Cited by 29
Cited by
Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey
He, K., Kim, D. D., & Asghar, M. R. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3233793 · Cited by 450
Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain
Rosenberg, I., Shabtai, A., Elovici, Y., & Rokach, L. (2022) · ACM Computing Surveys · doi:10.1145/3453158 · Cited by 295
Explainable artificial intelligence for cybersecurity: a literature survey
Charmet, F., Tanuwidjaja, H. C., Ayoubi, S., Gimenez, P.-F., Han, Y., Jmila, H., Blanc, G., Takahashi, T., & Zhang, Z. (2022) · Annals of Telecommunications · doi:10.1007/s12243-022-00926-7 · Cited by 167
Deep Reinforcement Adversarial Learning Against Botnet Evasion Attacks
Apruzzese, G., Andreolini, M., Marchetti, M., Venturi, A., & Colajanni, M. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.3031843 · Cited by 117
The Threat of Offensive AI to Organizations
Mirsky, Y., Demontis, A., Kotak, J., Shankar, R., Gelei, D., Yang, L., Zhang, X., Pintor, M., Lee, W., Elovici, Y., et al. (2023) · Computers & Security · doi:10.1016/j.cose.2022.103006 · Cited by 97
Metadata
- Authors
- Pierazzi, F., Pendlebury, F., Cortellazzi, J., & Cavallaro, L.
- Venue
- 2020 IEEE Symposium on Security and Privacy (SP)
- Publisher
- IEEE
- Published
- 2020-05
- Pages
- 1332-1349
- ISSN
- 1081-6011 (ISSN-L)
- OpenAlex
- W2989093880
- Cited by
- 287
- References
- 59
- FWCI
- 14.31
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.