The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
LEMNA: Explaining Deep Learning based Security Applications
Open accessAbstract
While deep learning has shown a great potential in various domains, the lack of transparency has limited its application in security or safety-critical areas. Existing research has attempted to develop explanation techniques to provide interpretable explanations for each classification decision. Unfortunately, current methods are optimized for non-security tasks ( e.g., image analysis). Their key assumptions are often violated in security applications, leading to a poor explanation fidelity. In this paper, we propose LEMNA, a high-fidelity explanation method dedicated for security applications. Given an input data sample, LEMNA generates a small set of interpretable features to explain how the input sample is classified. The core idea is to approximate a local area of the complex deep learning decision boundary using a simple interpretable model. The local interpretable model is specially designed to (1) handle feature dependency to better work with security applications ( e.g., binary code analysis); and (2) handle nonlinear local boundaries to boost explanation fidelity. We evaluate our system using two popular deep learning applications in security (a malware classifier, and a function start detector for binary reverse-engineering). Extensive evaluations show that LEMNA's explanation has a much higher fidelity level compared to existing methods. In addition, we demonstrate practical use cases of LEMNA to help machine learning developers to validate model behavior, troubleshoot classification errors, and automatically patch the errors of the target models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859
Jolliffe, I. T. (1986) · Springer Series in Statistics · doi:10.1007/978-1-4757-1904-8 · Cited by 8,273
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694 · cited 1×
“As a side note, we want to clarify that machine learning explanation is completely different from feature selection methods such as Principal Component Analysis (PCA) [26], Sparse Coding [39] or Chi-square Statistics [49].”
MethodDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P., Wu, X., Jha, S., & Swami, A. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.41 · Cited by 2,982
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761
Sparsity and Smoothness Via the Fused Lasso
Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., & Knight, K. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00490.x · Cited by 2,724
Hierarchical Mixtures of Experts and the EM Algorithm
Jordan, M. I., & Jacobs, R. A. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.181 · Cited by 2,673
Drebin: Effective and Explainable Detection of Android Malware in Your Pocket
Arp, D., Spreitzenbarth, M., Hübner, M., Gascon, H., & Rieck, K. (2014) · Proceedings 2014 Network and Distributed System Security Symposium · doi:10.14722/ndss.2014.23247 · Cited by 2,331
Tutorial on maximum likelihood estimation
Myung, I. J. (2003) · Journal of Mathematical Psychology · doi:10.1016/s0022-2496(02)00028-7 · Cited by 1,802
Static Analysis of Shape in TensorFlow Programs
The Theano Development Team, Al‐Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2020.15 · Cited by 1,589
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
Finite Mixture Modeling with Mixture Outcomes Using the EM Algorithm
Muthén, B., & Shedden, K. (1999) · Biometrics · doi:10.1111/j.0006-341x.1999.00463.x · Cited by 1,486
Small sample size effects in statistical pattern recognition: Recommendations for practitioners
Raudys, S. J., & Jain, A. K. (1991) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.75512 · Cited by 1,384
Incremental and Decremental Support Vector Machine Learning
Cauwenberghs, G., & Poggio, T. (2000) · neural information processing systems · Cited by 1,377
Delving into Transferable Adversarial Examples and Black-box Attacks
Liu, Y., Chen, X., Liu, C., & Song, D. (2016) · arXiv · doi:10.48550/arxiv.1611.02770 · Cited by 1,310
A Deep Learning Approach for Network Intrusion Detection System
Javaid, A., Niyaz, Q., Sun, W., & Alam, M. (2016) · Proceedings of the 9th EAI International Conference on Bio-inspired Information and Communications Technologies (formerly BIONETICS) · doi:10.4108/eai.3-12-2015.2262516 · Cited by 1,274
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.494 · Cited by 1,137
Principal Component Analysis and Factor Analysis
Jolliffe, I. T. (1986) · Springer Series in Statistics · doi:10.1007/978-1-4757-1904-8_7 · Cited by 1,081
MagNet: A Two-Pronged Defense against Adversarial Examples
Meng, D., & Chen, H. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134057 · Cited by 1,006
Deep learning approach for Network Intrusion Detection in Software Defined Networking
Tang, T. A., Mhamdi, L., McLernon, D., Zaidi, S. A. R., & Ghogho, M. (2016) · 2016 International Conference on Wireless Networks and Mobile Communications (WINCOM) · doi:10.1109/wincom.2016.7777224 · Cited by 806
AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation
Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., & Vechev, M. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00058 · Cited by 799
BitBlaze: A New Approach to Computer Security via Binary Analysis
Song, D., Brumley, D., Yin, H., Caballero, J., Jager, I., Kang, M. G., Liang, Z., Newsome, J., Poosankam, P., & Saxena, P. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-89862-7_1 · Cited by 718
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687
Towards Making Systems Forget with Machine Unlearning
Cao, Y., & Yang, J. (2015) · 2015 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2015.35 · Cited by 622 · cited 2×
“A representative method is “machine unlearning” [10], which is to remove the influence of certain training data by transforming the standard training algorithms into a summation form.”
BackgroundMethod39 Dimensionality and sample size considerations in pattern recognition practice
Jain, A. K., & Chandrasekaran, B. (1982) · Handbook of Statistics · doi:10.1016/s0169-7161(82)02042-2 · Cited by 560
Angora: Efficient Fuzzing by Principled Search
Chen, P., & Chen, H. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00046 · Cited by 552
Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M. (2017) · arXiv · doi:10.48550/arxiv.1702.04595 · Cited by 487
Large-scale malware classification using random projections and neural networks
Dahl, G. E., Stokes, J. W., Deng, L., & Yu, D. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638293 · Cited by 439
Control flow integrity for COTS binaries
Zhang, M., & Sekar, R. (2013) · Cited by 409
Bayesian Modelling and Inference on Mixtures of Distributions
Marin, J.-M., Mengersen, K., & Robert, C. P. (2005) · Handbook of Statistics · doi:10.1016/s0169-7161(05)25016-2 · Cited by 367
Practical Evasion of a Learning-Based Classifier: A Case Study
Srndic, N., & Laskov, P. (2014) · 2014 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2014.20 · Cited by 346
Recognizing functions in binaries with neural networks
Shin, E. C. R., Song, D. X., & Moazzezi, R. (2015) · USENIX Security Symposium · Cited by 322
DevNet: A Deep Event Network for multimedia event detection and evidence recounting
Gan, C., Wang, N., Yang, Y., Yeung, D.-Y., & Hauptmann, A. G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298872 · Cited by 312
Malicious PDF detection using metadata and structural features
Smutz, C., & Stavrou, A. (2012) · Proceedings of the 28th Annual Computer Security Applications Conference · doi:10.1145/2420950.2420987 · Cited by 271
Detecting P2P botnets through network behavior analysis and machine learning
Saad, S., Traore, I., Ghorbani, A., Sayed, B., Zhao, D., Lu, W., Felix, J., & Hakimian, P. (2011) · 2011 Ninth Annual International Conference on Privacy, Security and Trust · doi:10.1109/pst.2011.5971980 · Cited by 267
Interpreting Blackbox Models via Model ExtractionPreprint
Bastani, O., Kim, C., & Bastani, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08504 · Cited by 258
Variable Selection in Finite Mixture of Regression Models
Khalili, A., & Chen, J. (2007) · Journal of the American Statistical Association · doi:10.1198/016214507000000590 · Cited by 237
Automatic Reverse Engineering of Malware Emulators
Sharif, M., Lanzi, A., Giffin, J., & Lee, W. (2009) · 2009 30th IEEE Symposium on Security and Privacy · doi:10.1109/sp.2009.27 · Cited by 204
Mitigating Adversarial Effects Through Randomization
Xie, C., Wang, J., Zhang, Z., Ren, Z., & Yuille, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01991 · Cited by 181
Adversary Resistant Deep Neural Networks with an Application to Malware Detection
Wang, Q., Guo, W., Zhang, K., Ororbia, A. G., Xing, X., Liu, X., & Giles, C. L. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098158 · Cited by 176
Self-citationMitigating Evasion Attacks to Deep Neural Networks via Region-based Classification
Cao, X., & Gong, N. Z. (2017) · Proceedings of the 33rd Annual Computer Security Applications Conference · doi:10.1145/3134600.3134606 · Cited by 168
BYTEWEIGHT: Learning to Recognize Functions in Binary Code
Bao, T., Burket, J., Woo, M., Turner, R., & Brumley, D. (2018) · Figshare · doi:10.1184/r1/6468464 · Cited by 160
Neural Nets Can Learn Function Type Signatures From Binaries
Chua, Z. L., Shen, S., Saxena, P., & Liang, Z. (2017) · USENIX Security Symposium · Cited by 118
Malicious Behavior Detection using Windows Audit Logs
Berlin, K., Slater, D., & Saxe, J. (2015) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/2808769.2808773 · Cited by 100
Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program FeaturesPreprint
Saxe, J., & Berlin, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.03096 · Cited by 81
Incremental and decremental training for linear classification
Tsai, C.-H., Lin, C.-Y., & Lin, C. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623661 · Cited by 75
MASHA-EL: A Multi-Agent System for Supporting Adaptive E-Learning
Garruzzo, S., Rosaci, D., & Sarnè, G. M. L. (2007) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2007.83 · Cited by 34
Incremental and Decremental Learning for Linear Support Vector Machines
Romero, E., Barrio, I., & Belanche, L. (2007) · International Conference on Artificial Neural Networks · doi:10.1007/978-3-540-74690-4_22 · Cited by 33
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv
Cited by
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., & Zhao, B. Y. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00031 · Cited by 1,368
Machine Learning and Deep Learning Methods for Intrusion Detection Systems: A Survey
Liu, H., & Lang, B. (2019) · Applied Sciences · doi:10.3390/app9204396 · Cited by 1,133
“These models report only the detection results and have no interpretable basis [93].”
BackgroundDeep Learning Based Vulnerability Detection: Are We There Yet?
Chakraborty, S., Krishna, R., Ding, Y., & Ray, B. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3087402 · Cited by 790
STRIP: a defence against trojan attacks on deep neural networks
Gao, Y., Xu, C., Wang, D., Chen, S., Ranasinghe, D. C., & Nepal, S. (2019) · Proceedings of the 35th Annual Computer Security Applications Conference · doi:10.1145/3359789.3359790 · Cited by 707
Deep Learning Based Attack Detection for Cyber-Physical System Cybersecurity: A Survey
Zhang, J., Pan, L., Han, Q.-L., Chen, C., Wen, S., & Xiang, Y. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2021.1004261 · Cited by 432
Metadata
- Authors
- Guo, W., Mu, D., Xu, J., Su, P., Wang, G., & Xing, X.
- Venue
- Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2018-10-15
- Pages
- 364-379
- OpenAlex
- W2890991187
- Cited by
- 287
- References
- 59
- FWCI
- 11.13
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.