The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Guiding Deep Learning System Testing Using Surprise Adequacy
Open accessAbstract
Deep Learning (DL) systems are rapidly being adopted in safety and security critical domains, urgently calling for ways to test their correctness and robustness. Testing of DL systems has traditionally relied on manual collection and labelling of data. Recently, a number of coverage criteria based on neuron activation values have been proposed. These criteria essentially count the number of neurons whose activation during the execution of a DL system satisfied certain properties, such as being above predefined thresholds. However, existing coverage criteria are not sufficiently fine grained to capture subtle behaviours exhibited by DL systems. Moreover, evaluations have focused on showing correlation between adversarial examples and proposed criteria rather than evaluating and guiding their use for actual testing of DL systems. We propose a novel test adequacy criterion for testing of DL systems, called Surprise Adequacy for Deep Learning Systems (SADL), which is based on the behaviour of DL systems with respect to their training data. We measure the surprise of an input as the difference in DL system's behaviour between the input and the training data (i.e., what was learnt during training), and subsequently develop this as an adequacy criterion: a good test input should be sufficiently but not overtly surprising compared to training data. Empirical evaluation using a range of DL systems from simple image classifiers to autonomous driving car platforms shows that systematic sampling of inputs based on their surprise can improve classification accuracy of DL systems against adversarial examples by up to 77.5% via retraining.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198
Learning Hierarchical Features for Scene Labeling
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.231 · Cited by 2,620
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876
Regression testing minimization, selection and prioritization: a survey
Yoo, S., & Harman, M. (2012) · Software Testing, Verification and Reliability · doi:10.1002/stv.430 · Cited by 1,322
Self-citationSoftware unit test coverage and adequacy
Zhu, H., Hall, P. A. V., & May, J. H. R. (1997) · ACM Computing Surveys · doi:10.1145/267580.267590 · Cited by 1,247
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
Multivariate Density Estimation: Theory, Practice, and Visualization
Scott, D. W. (2015) · Wiley Series in Probability and Statistics · doi:10.1002/9781118575574 · Cited by 1,165
Introduction to Software Testing
(2021) · Essentials of Software Testing · doi:10.1017/9781108974073.004 · Cited by 1,056
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
Carlini, N., & Wagner, D. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140444 · Cited by 966
On Using Very Large Target Vocabulary for Neural Machine Translation
Jean, S., Cho, K., Memisevic, R., & Bengio, Y. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1001 · Cited by 862
Wand, M. P., & Jones, M. C. (1994) · Chapman & Hall/CRC eBooks · doi:10.1201/b14876 · Cited by 763
Introduction to Software Testing
Ammann, P., & Offutt, J. (2016) · Cambridge University Press eBooks · doi:10.1017/9781316771273 · Cited by 607
Detection of Malicious Code Variants Based on Deep Learning
Cui, Z., Xue, F., Cai, X., Cao, Y., Wang, G.-G., & Chen, J. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2822680 · Cited by 597
Multivariate Density Estimation: Theory, Practice, and Visualization.
Weiss, R., & Scott, D. W. (1994) · Journal of the American Statistical Association · doi:10.2307/2291242 · Cited by 495
Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
Ma, X., Li, B., Wang, Y., Erfani, S., Wijewickrema, S. N. R., Schoenebeck, G. R., Song, D. X., Houle, M. E., & Bailey, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02613 · Cited by 440
DeepMutation: Mutation Testing of Deep Learning Systems
Ma, L., Zhang, F., Sun, J., Xue, M., Li, B., Juefei-Xu, F., Xie, C., Li, L., Liu, Y., Zhao, J., et al. (2018) · 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2018.00021 · Cited by 366
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
Carlini, N., & Wagner, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07263 · Cited by 307
Better Mixing via Deep Representations
Bengio, Y., Mesnil, G., Dauphin, Y., & Rifai, S. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4404 · Cited by 237
Test Set Diameter: Quantifying the Diversity of Sets of Test Cases
FELDT, R. H., Poulding, S., Clark, D., & Yoo, S. (2016) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2016.33 · Cited by 148
Self-citationRegression Testing Minimisation, Selection and Prioritisation - A Survey
Yoo, S., & Harman, M. (2009) · UCL Discovery (University College London) · Cited by 134
Self-citationAutomatic system testing of programs without test oracles
Murphy, C., Shen, K., & Kaiser, G. E. (2009) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/1572272.1572295 · Cited by 85
BIOSIGN™ : multi-parameter monitoring for early warning of patient deterioration
Tarassenko, L. (2005) · IEE International Seminar on Medical Applications of Signal Processing · doi:10.1049/ic:20050334 · Cited by 77
Metamorphic Testing and Beyond
Chen, T. Y., Kuo, F., Tse, T. H., & Zhou, Z. Q. (2005) · International Workshop on Software Technology and Engineering Practice (STEP) · doi:10.1109/step.2003.18 · Cited by 71
Combinatorial Testing for Deep Learning SystemsPreprint
Ma, L., Zhang, F., Xue, M., Li, B., Liu, Y., Zhao, J., & Wang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07723 · Cited by 50
Ma, L., Juefei-Xu, F., Sun, J., Chen, C., Su, T., Zhang, F., Xue, M., Li, B., Li, L., Liu, Y., et al. (2018) · arXiv (Cornell University) · Cited by 46
Detecting rare events using extreme value statistics applied to epileptic convulsions in children
Luca, S., Karsmakers, P., Cuppens, K., Croonenborghs, T., de Vel, A. V., Ceulemans, B., Lagae, L., Huffel, S. V., & Vanrumste, B. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2013.11.007 · Cited by 40
Metamorphic Testing of Stochastic Optimisation
Yoo, S. (2010) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2010.26 · Cited by 29
Self-citation
Cited by
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Wu, M., & Yi, X. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100270 · Cited by 532
DeepHunter: a coverage-guided fuzz testing framework for deep neural networks
Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., Zhao, J., Li, B., Yin, J., & See, S. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330579 · Cited by 426
Testing machine learning based systems: a systematic mapping
Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., & Tonella, P. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09881-0 · Cited by 251
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Metadata
- Authors
- Kim, J., Feldt, R., & Yoo, S.
- Venue
- 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 1039-1049
- OpenAlex
- W2888510757
- Cited by
- 430
- References
- 37
- FWCI
- 38.16
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.