The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Open accessAbstract
Deep learning (DL) systems are increasingly deployed in safety- and security-critical domains including self-driving cars and malware detection, where the correctness and predictability of a system's behavior for corner case inputs are of great importance. Existing DL testing depends heavily on manually labeled data and therefore often fails to expose erroneous behaviors for rare inputs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2014) · Cited by 33,419
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“aw inputs without any human guidance besides labeled training data. In recent years, DNNs have surpassed human performance in many application domains due to increasing availability of large datasets [20, 38, 47], specialized hardware [34, 50], and efficient training algorithms [31, 39, 66, 71]. A DNN consists of mult…”
Background(2010) · International Conference on Machine Learning · Cited by 13,995
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P., Wu, X., Jha, S., & Swami, A. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.41 · Cited by 2,982
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961
A Neural Algorithm of Artistic Style
Gatys, L., Ecker, A., & Bethge, M. (2016) · Journal of Vision · doi:10.1167/16.12.326 · Cited by 2,612
Drebin: Effective and Explainable Detection of Android Malware in Your Pocket
Arp, D., Spreitzenbarth, M., Hübner, M., Gascon, H., & Rieck, K. (2014) · Proceedings 2014 Network and Distributed System Security Symposium · doi:10.14722/ndss.2014.23247 · Cited by 2,331
Proceedings of the 13th International Conference on Neural Information Processing Systems
Leen, T. K., Dietterich, T. G., & Tresp, V. (2000) · Cited by 2,190
Proceedings for the 5th International Conference on Learning Representations
Srivastava, A., & Sutton, C. A. (2017) · Cited by 1,837
Understanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
Xu, W., Evans, D., & Qi, Y. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23198 · Cited by 1,768
Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks
Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_5 · Cited by 1,706
Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition
Sharif, M., Bhagavatula, S., Bauer, L., & Reiter, M. K. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978392 · Cited by 1,530
Understanding Neural Networks Through Deep Visualization
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,358
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · ACM SIGARCH Computer Architecture News · doi:10.1145/3140659.3080246 · Cited by 1,240
Safety Verification of Deep Neural Networks
Huang, X., Kwiatkowska, M., Wang, S., & Wu, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_1 · Cited by 818
Stealing Machine Learning Models via Prediction APIs
Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016) · arXiv · doi:10.48550/arxiv.1609.02943 · Cited by 796
Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
Li, C., & Wand, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.272 · Cited by 741
Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing.
Fredrikson, M., Lantz, E., Jha, S., Lin, S. M., Page, D., & Ristenpart, T. (2014) · PubMed · Cited by 641
Improving the Robustness of Deep Neural Networks via Stability Training
Zheng, S., Song, Y., Leung, T., & Goodfellow, I. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.485 · Cited by 624
Towards Making Systems Forget with Machine Unlearning
Cao, Y., & Yang, J. (2015) · 2015 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2015.35 · Cited by 622 · cited 1×
“Recently, the security and privacy aspects of machine learning have drawn significant attention from the researchers in both machine learning [26, 49, 72] and security [12, 21, 22, 55, 63, 74, 82] communities.”
BackgroundSelf-citationTowards Deep Neural Network Architectures Robust to Adversarial Examples
Gu, S., & Rigazio, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.5068 · Cited by 622
Finding and understanding bugs in C compilers
Yang, X., Chen, Y., Eide, E., & Regehr, J. (2011) · ACM SIGPLAN Notices · doi:10.1145/1993316.1993532 · Cited by 586
Measuring Neural Net Robustness with Constraints
Bastani, O., Ioannou, Y., Lampropoulos, L., Vytiniotis, D., Nori, A., & Criminisi, A. (2016) · neural information processing systems · doi:10.17863/cam.26484 · Cited by 472
(1998) · Cited by 469
Achieving Human Parity in Conversational Speech Recognition
Xiong, W., Droppo, J., Huang, X., Seide, F. T. B., Seltzer, M., Stolcke, A., Yu, D., & Zweig, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.05256 · Cited by 451
Droid-Sec: deep learning in android malware detection
Yuan, Z., Lu, Y., Wang, Z., & Xue, Y. (2014) · Proceedings of the 2014 ACM conference on SIGCOMM · doi:10.1145/2619239.2631434 · Cited by 371
Shaham, U., Yamada, Y., & Negahban, S. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.04.027 · Cited by 355
Practical Evasion of a Learning-Based Classifier: A Case Study
Srndic, N., & Laskov, P. (2014) · 2014 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2014.20 · Cited by 346
Learning to Generate Reviews and Discovering Sentiment
Radford, A., Jozefowicz, R., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1704.01444 · Cited by 330 · cited 1×
“It has recently been shown that each neuron in a DNN tends to independently extract a specific feature of the input instead of collaborating with other neurons for feature extraction [58, 87].”
BackgroundThe microsoft 2016 conversational speech recognition system
Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M., Stolcke, A., Yu, D., & Zweig, G. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953159 · Cited by 326
Adversarial Perturbations Against Deep Neural Networks for Malware Classification
Grosse, K., Papernot, N., Manoharan, P., Michael Backes, A., & McDaniel, P. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04435 · Cited by 299
Malicious PDF detection using metadata and structural features
Smutz, C., & Stavrou, A. (2012) · Proceedings of the 28th Annual Computer Security Applications Conference · doi:10.1145/2420950.2420987 · Cited by 271
An Abstraction-Refinement Approach to Verification of Artificial Neural Networks
Pulina, L., & Tacchella, A. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-14295-6_24 · Cited by 261
Mobile-sandbox: having a deeper look into android applications
Spreitzenbarth, M., Freiling, F., Echtler, F., Schreck, T., & Hoffmann, J. (2013) · Proceedings of the 28th Annual ACM Symposium on Applied Computing · doi:10.1145/2480362.2480701 · Cited by 250
Policy compression for aircraft collision avoidance systems
Julian, K. D., Lopez, J., Brush, J. S., Owen, M. P., & Kochenderfer, M. J. (2016) · 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc.2016.7778091 · Cited by 247
Toward Human Parity in Conversational Speech Recognition
Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M. L., Stolcke, A., Yu, D., & Zweig, G. (2017) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2017.2756440 · Cited by 245
Brubaker, C., Jana, S., Ray, B., Khurshid, S., & Shmatikov, V. (2014) · 2014 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2014.15 · Cited by 218
Self-citationAutomatically Evading Classifiers: A Case Study on PDF Malware Classifiers
Xu, W., Qi, Y., & Evans, D. (2016) · Proceedings 2016 Network and Distributed System Security Symposium · doi:10.14722/ndss.2016.23115 · Cited by 218
Artistic Style Transfer for Videos
Ruder, M., Dosovitskiy, A., & Brox, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-45886-1_3 · Cited by 193
Misleading worm signature generators using deliberate noise injection
Perdisci, R., Dagon, D., Lee, W., Fogla, P., & Sharif, M. (2006) · 2006 IEEE Symposium on Security and Privacy (S&P'06) · doi:10.1109/sp.2006.26 · Cited by 189
Coverage-directed differential testing of JVM implementations
Chen, Y., Su, T., Sun, C., Su, Z., & Zhao, J. (2016) · Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2908080.2908095 · Cited by 169
A Methodology for Formalizing Model-Inversion Attacks
Wu, X., Fredrikson, M., Jha, S., & Naughton, J. F. (2016) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf.2016.32 · Cited by 159
Parseval Networks: Improving Robustness to Adversarial Examples
Cissé, M., Bojanowski, P., Grave, É., Dauphin, Y., & Usunier, N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.08847 · Cited by 158
Randomized Differential Testing as a Prelude to Formal Verification
Groce, A., Holzmann, G. J., & Joshi, R. (2007) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2007.68 · Cited by 124
Extending Defensive DistillationPreprint
Papernot, N., & McDaniel, P. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.05264 · Cited by 122
NEZHA: Efficient Domain-Independent Differential Testing
Petsios, T., Tang, A., Stolfo, S. J., Keromytis, A. D., & Jana, S. (2017) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2017.27 · Cited by 115
Self-citationGuided differential testing of certificate validation in SSL/TLS implementations
Chen, Y., & Su, Z. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2786835 · Cited by 84
Privacy oracle: a system for finding application leaks with black box differential testing
Jung, J., Sheth, A. N., Greenstein, B., Wetherall, D. J., Maganis, G., & Kohno, T. (2008) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/1455770.1455806 · Cited by 83
HVLearn: Automated Black-Box Analysis of Hostname Verification in SSL/TLS Implementations
Sivakorn, S., Argyros, G., Pei, K., Keromytis, A. D., & Jana, S. (2017) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2017.46 · Cited by 75
Self-citationArgyros, G., Stais, I., Jana, S., Keromytis, A. D., & Kiayias, A. (2016) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2976749.2978383 · Cited by 51
Self-citationToward Guaranteed Illumination Models for Non-convex Objects
Zhang, Y., Mu, C., Kuo, H.-W., et al. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.120 · Cited by 8
Cited by
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733
Wild patterns: Ten years after the rise of adversarial machine learning
Biggio, B., & Roli, F. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.023 · Cited by 1,409
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
Self-citationSpeech emotion recognition using deep 1D & 2D CNN LSTM networks
Zhao, J., Mao, X., & Chen, L. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2018.08.035 · Cited by 1,044
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Metadata
- Authors
- Pei, K., Cao, Y., Yang, J., & Jana, S.
- Venue
- Proceedings of the 26th Symposium on Operating Systems Principles
- Publisher
- ACM
- Published
- 2017-10-14
- Pages
- 1-18
- OpenAlex
- W2616028256
- Cited by
- 1,217
- References
- 75
- FWCI
- 89.20
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.