The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Concolic testing for deep neural networks
Open accessAbstract
Concolic testing combines program execution and symbolic analysis to explore the execution paths of a software program. In this paper, we develop the first concolic testing approach for Deep Neural Networks (DNNs). More specifically, we utilise quantified linear arithmetic over rationals to express test requirements that have been studied in the literature, and then develop a coherent method to perform concolic testing with the aim of better coverage. Our experimental results show the effectiveness of the concolic testing approach in both achieving high coverage and finding adversarial examples.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2010) · International Conference on Machine Learning · Cited by 13,995
(2017) · International Conference on Machine Learning · Cited by 6,382
Multiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253
KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs
Cadar, C., Dunbar, D., & Engler, D. (2008) · doi:10.5555/1855741.1855756 · Cited by 2,620
Arjovsky, M., Chintala, S., & Bottou, L. (2017) · arXiv · doi:10.48550/arxiv.1701.07875 · Cited by 2,225
DART: directed automated random testing
Godefroid, P., Klarlund, N., & Sen, K. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065036 · Cited by 2,067
Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks
Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_5 · Cited by 1,706
CUTE: a concolic unit testing engine for C
Sen, K., Marinov, D., & Agha, G. (2005) · Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1081706.1081750 · Cited by 1,651
DART: directed automated random testing
Godefroid, P., Klarlund, N., & Sen, K. (2005) · Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation · doi:10.1145/1065010.1065036 · Cited by 1,570
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
Automated Whitebox Fuzz Testing.
Godefroid, P., Levin, M. Y., & Molnar, D. A. (2008) · Cited by 1,095
Safety Verification of Deep Neural Networks
Huang, X., Kwiatkowska, M., Wang, S., & Wu, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_1 · Cited by 818
Self-citationAI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation
Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., & Vechev, M. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00058 · Cited by 799
Derivative-Free and Blackbox Optimization
Audet, C., & Hare, W. (2017) · Springer series in operations research/Springer series in operations research and financial engineering · doi:10.1007/978-3-032-00906-7 · Cited by 672
Test input generation with java PathFinder
Visser, W., Pǎsǎreanu, C. S., & Khurshid, S. (2004) · Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis · doi:10.1145/1007512.1007526 · Cited by 501
Heuristics for Scalable Dynamic Test Generation
Burnim, J., & Sen, K. (2008) · 2008 23rd IEEE/ACM International Conference on Automated Software Engineering · doi:10.1109/ase.2008.69 · Cited by 412
A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction
Wiatowski, T., & Bolcskei, H. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2776228 · Cited by 399
Differentiable Abstract Interpretation for Provably Robust Neural Networks
Mirman, M., Gehr, T., & Vechev, M. (2018) · International Conference on Machine Learning · Cited by 318
CUTE: a concolic unit testing engine for C
Sen, K., Marinov, D., & Agha, G. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1095430.1081750 · Cited by 315
Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution
Xie, T., Marinov, D., Schulte, W., & Notkin, D. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-31980-1_24 · Cited by 287
Reachability Analysis of Deep Neural Networks with Provable Guarantees
Ruan, W., Huang, X., & Kwiatkowska, M. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/368 · Cited by 246
Self-citationFeature-Guided Black-Box Safety Testing of Deep Neural Networks
Wicker, M., Huang, X., & Kwiatkowska, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-89960-2_22 · Cited by 212
Self-citationTest input generation with java PathFinder
Visser, W., Păsăreanu, C. S., & Khurshid, S. (2004) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1013886.1007526 · Cited by 116
A game-based approximate verification of deep neural networks with provable guarantees
Wu, M., Wicker, M., Ruan, W., Huang, X., & Kwiatkowska, M. (2019) · Theoretical Computer Science · doi:10.1016/j.tcs.2019.05.046 · Cited by 111
Self-citationMa, L., Juefei-Xu, F., Sun, J., Chen, C., Su, T., Zhang, F., Xue, M., Li, B., Li, L., Liu, Y., et al. (2018) · arXiv (Cornell University) · Cited by 46
Global Robustness Evaluation of Deep Neural Networks with Provable Guarantees for the $L_0$ NormPreprint
Ruan, W., Wu, M., Sun, Y., Huang, X., Kroening, D., & Kwiatkowska, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.05805 · Cited by 28
Self-citation
Cited by
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Wu, M., & Yi, X. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100270 · Cited by 532
Self-citationDeepHunter: a coverage-guided fuzz testing framework for deep neural networks
Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., Zhao, J., Li, B., Yin, J., & See, S. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330579 · Cited by 426
Software Engineering for AI-Based Systems: A Survey
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 335
Qayyum, A., Usama, M., Qadir, J., & Al-Fuqaha, A. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2975048 · Cited by 257
“E. Testing of ML Models and Autonomous Vehicles 1) Behavior Testing of Models: In a recent study, Sun et al. proposed four novel testing criteria for verifying structural features of DNN using MC/DC 7 coverage criteria [218].”
Method
Metadata
- Authors
- Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., & Kroening, D.
- Venue
- Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2018-09-03
- Pages
- 109-119
- OpenAlex
- W2963913218
- Cited by
- 332
- References
- 27
- FWCI
- 32.26
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.