The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Deep learning library testing via effective model generation
Abstract
Deep learning (DL) techniques are rapidly developed and have been widely adopted in practice. However, similar to traditional software systems, DL systems also contain bugs, which could cause serious impacts especially in safety-critical domains. Recently, many research approaches have focused on testing DL models, while little attention has been paid for testing DL libraries, which is the basis of building DL models and directly affects the behavior of DL systems. In this work, we propose a novel approach, LEMON, to testing DL libraries. In particular, we (1) design a series of mutation rules for DL models, with the purpose of exploring different invoking sequences of library code and hard-to-trigger behaviors; and (2) propose a heuristic strategy to guide the model generation process towards the direction of amplifying the inconsistent degrees of the inconsistencies between different DL libraries caused by bugs, so as to mitigate the impact of potential noise introduced by uncertain factors in DL libraries. We conducted an empirical study to evaluate the effectiveness of LEMON with 20 release versions of 4 widely-used DL libraries, i.e., TensorFlow, Theano, CNTK, MXNet. The results demonstrate that LEMON detected 24 new bugs in the latest release versions of these libraries, where 7 bugs have been confirmed and one bug has been fixed by developers. Besides, the results confirm that the heuristic strategy for model generation indeed effectively guides LEMON in amplifying the inconsistent degrees for bugs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, model-level fuzzers can only cover a limited set of APIs with limited patterns [72], missing many diverse and interesting API sequences that can lead to bugs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [71]
Citations
Cites
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundAdversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,199
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundWhat every computer scientist should know about floating-point arithmetic
Goldberg, D. (1991) · ACM Computing Surveys · doi:10.1145/103162.103163 · Cited by 1,750 · cited 1×
“However, due to many uncertain factors in DL libraries, such as randomness and floating-point computing deviation [28], it is challenging to determine whether a found problem is a real bug in the library, or a problem caused by some uncertain factors.”
BackgroundAn Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110
Roulette-wheel selection via stochastic acceptance
Lipowski, A., & Lipowska, D. (2012) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2011.12.004 · Cited by 761
Robust log-based anomaly detection on unstable log data
Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundSelf-citationMarkov Chain Monte Carlo in Practice: A Roundtable Discussion
Kass, R. E., Carlin, B. P., Gelman, A., & Neal, R. M. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480547 · Cited by 704
Metamorphic Testing: A Review of Challenges and Opportunities
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2019) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 665
Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1
Zoubin Ghahramani, A., Welling, M., Cortes, C., Lawrence, N. D., & Weinberger, K. Q. (2014) · Cited by 656
Zhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238187 · Cited by 611 · cited 1×
“Furthermore, there are a great deal of researches focusing on testing DL models in the literature [10, 20, 31, 45ś47, 55, 62, 65, 66, 69].”
BackgroundDeepGauge: multi-granularity testing criteria for deep learning systems
Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238202 · Cited by 588 · cited 3×
“For example, Ma et al. [45] proposed a set of multi-granularity testing criteria, including neuron-level and layer-level coverage criteria, for DL models.”
Background(1998) · Cited by 469
Guiding Deep Learning System Testing Using Surprise Adequacy
Kim, J., Feldt, R., & Yoo, S. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00108 · Cited by 430
DeepHunter: a coverage-guided fuzz testing framework for deep neural networks
Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., Zhao, J., Li, B., Yin, J., & See, S. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330579 · Cited by 426 · cited 3×
“Besides, many of them proposed to find/generate adversarial inputs [52, 60, 66].”
BackgroundMethodMutation Testing Advances: An Analysis and Survey
(2019) · Advances in Computers · doi:10.1016/bs.adcom.2018.03.015 · Cited by 425
DeepMutation: Mutation Testing of Deep Learning Systems
Ma, L., Zhang, F., Sun, J., Xue, M., Li, B., Juefei-Xu, F., Xie, C., Li, L., Liu, Y., Zhao, J., et al. (2018) · 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2018.00021 · Cited by 366
Distributed deep learning networks among institutions for medical imaging
Chang, K., Balachandar, N., Lam, C., Yi, D., Brown, J., Beers, A., Rosen, B., Rubin, D. L., & Kalpathy-Cramer, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy017 · Cited by 340
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336
Concolic testing for deep neural networks
Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., & Kroening, D. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238172 · Cited by 332
Automatic Software Repair: A Bibliography
Monperrus, M. (2019) · ACM Computing Surveys · doi:10.1145/3105906 · Cited by 321
An empirical study on TensorFlow program bugs
Zhang, Y., Chen, Y., Cheung, S.-C., Xiong, Y., & Zhang, L. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213866 · Cited by 304 · cited 1×
“However, there is little attention on testing DL libraries, which may also contain bugs [32, 71] that directly affect the performance of DL models.”
BackgroundJia, Y., & Harman, M. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.04.016 · Cited by 287
Policy compression for aircraft collision avoidance systems
Julian, K. D., Lopez, J., Brush, J. S., Owen, M. P., & Kochenderfer, M. J. (2016) · 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc.2016.7778091 · Cited by 247 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundNIC: Detecting Adversarial Samples with Neural Network Invariant Checking
Ma, S., Liu, Y., Tao, G., Lee, W.-C., & Zhang, X. (2019) · Proceedings 2019 Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23415 · Cited by 235 · cited 1×
“, testing DL models by generating adversarial inputs [48, 52, 66] or measuring the testing adequacy of DL models [45, 46, 55].”
MethodChen, J., Patra, J., Pradel, M., Xiong, Y., Zhang, H., Hao, D., & Zhang, L. (2021) · ACM Computing Surveys · doi:10.1145/3363562 · Cited by 215
Self-citationMODE: automated neural network model debugging via state differential analysis and input selection
Ma, S., Liu, Y., Lee, W.-C., Zhang, X., & Grama, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236082 · Cited by 211
CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries
Pham, H. V., Lutellier, T., Qi, W., & Tan, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00107 · Cited by 190
Dwarakanath, A., Ahuja, M., Sikand, S., Rao, R. M., Bose, R. P. J. C., Dubash, N., & Podder, S. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213858 · Cited by 183
Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing
Wang, J., Dong, G., Sun, J., Wang, X., & Zhang, P. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00126 · Cited by 177
An Empirical Investigation of Incident Triage for Online Service Systems
Chen, J., He, X., Lin, Q., Xu, Y. D., Zhang, H., Hao, D., Gao, F., Xu, Z., Dang, Y., & Zhang, D. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00020 · Cited by 143
Self-citationGuo, Q., Chen, S., Xie, X., Ma, L., Hu, Q., Liu, H., Liu, Y., Zhao, J., & Li, X. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00080 · Cited by 141 · cited 1×
“Compared with traditional software systems, DL systems usually involve more complex components, e.g., platform/hardware infrastructure, deep learning libraries, models, source programs for training, and training and testing corpus [30].”
MethodAn empirical comparison of compiler testing techniques
Chen, J., Hu, W., Hao, D., Xiong, Y., Zhang, H., Zhang, L., & Xie, B. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884878 · Cited by 138 · cited 1×
“In traditional software systems, the test-oracle problem has been well studied and there are some ways of relieving the test-oracle problem [13, 17, 19, 22, 37, 49].”
MethodSelf-citationThreats to the validity of mutation-based test assessment
Papadakis, M., Henard, C., Harman, M., Jia, Y., & Traon, Y. L. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931040 · Cited by 135 · cited 1×
“Mutation testing has been extensively studied in traditional software systems [9, 33ś35, 50, 53, 54, 68].”
MethodTest generation via Dynamic Symbolic Execution for mutation testing
Zhang, L., Xie, T., Zhang, L., Tillmann, N., de Halleux, J., & Mei, H. (2010) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2010.5609672 · Cited by 117
Automated testing for SQL injection vulnerabilities: an input mutation approach
Appelt, D., Nguyen, C. D., Briand, L., & Alshahwan, N. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610403 · Cited by 114 · cited 1×
“Mutation testing has been extensively studied in traditional software systems [9, 33ś35, 50, 53, 54, 68].”
MethodBoosting operational DNN testing efficiency through conditioning
Li, Z., Ma, X., Xu, C., Cao, C., Xu, J., & Lü, J. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338930 · Cited by 108
Practical Accuracy Estimation for Efficient Deep Neural Network Testing
Chen, J., Wu, Z. J., Wang, Z., You, H., Zhang, L., & Yan, M. X. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3394112 · Cited by 107 · cited 1×
“Furthermore, there are a great deal of researches focusing on testing DL models in the literature [10, 20, 31, 45ś47, 55, 62, 65, 66, 69].”
BackgroundSelf-citationAdversarial sample detection for deep neural network through model mutation testingPreprint
Wang, J., DONG, G., Sun, J., Wang, X., & PEIXIN, Z. (2019) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 96
Continuous Incident Triage for Large-Scale Online Service Systems
Chen, J., He, X., Lin, Q., Zhang, H., Hao, D., Gao, F., Xu, Z., Dang, Y., & Zhang, D. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00042 · Cited by 92 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundSelf-citationThe Oxford dictionary of statistical terms
(2004) · Choice Reviews Online · doi:10.5860/choice.41-3801 · Cited by 89 · cited 1×
“Assuming the desired distribution to be equilibrium distribution [25], LEMON adopts”
BackgroundµDroid: an energy-aware mutation testing framework for Android
Jabbarvand, R., & Malek, S. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106244 · Cited by 84
Guided Mutation Testing for JavaScript Web Applications
Mirshokraie, S., Mesbah, A., & Pattabiraman, K. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2371458 · Cited by 80
Chen, J., Zhang, S., He, X., Lin, Q., Zhang, H., Hao, D., Kang, Y., Gao, F., Xu, Z., Dang, Y., et al. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416624 · Cited by 80 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundSelf-citationThung, K.-H., Yap, P.-T., & Shen, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67558-9_19 · Cited by 63 · cited 1×
“One of the most popular high-level libraries is Keras [2], which has been widely used in various critical domains [11, 41, 61].”
BackgroundCompiler bug isolation via effective witness test program generation
Chen, J., Han, J., Sun, P., Zhang, L., Hao, D., & Zhang, L. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338957 · Cited by 63 · cited 1×
“Following the setting of p in the MH algorithm [14, 23], we set p to be 0.”
MethodSelf-citationEnhanced compiler bug isolation via memoized search
Chen, J., Ma, H., & Zhang, L. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416570 · Cited by 59 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundSelf-citationTesting probabilistic programming systems
Dutta, S., Legunsen, O., Huang, Z., & Misailovíc, S. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236057 · Cited by 55 · cited 1×
“Besides, there are some work focusing on testing machine learning (ML) libraries [26, 27, 57, 58].”
BackgroundDeepCCI: End-to-end Deep Learning for Chemical-Chemical Interaction Prediction
Kwon, S., & Yoon, S. (2017) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3107411.3107451 · Cited by 53 · cited 1×
“One of the most popular high-level libraries is Keras [2], which has been widely used in various critical domains [11, 41, 61].”
BackgroundAutomatic testing of symbolic execution engines via program generation and differential testing
Kapus, T., & Cadar, C. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115669 · Cited by 52
Deep Validation: Toward Detecting Real-World Corner Cases for Deep Neural Networks
Wu, W., Xu, H. F., Zhong, S., Lyu, M. R.-T., & King, I. (2019) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2019.00026 · Cited by 47 · cited 1×
“Furthermore, there are a great deal of researches focusing on testing DL models in the literature [10, 20, 31, 45ś47, 55, 62, 65, 66, 69].”
BackgroundMultiple-Implementation Testing of Supervised Learning Software
Srisakaokul, S., Wu, Z., Astorga, A., Alebiosu, O., & Xie, T. (2016) · National Conference on Artificial Intelligence · Cited by 44
DeepEvolution: A Search-Based Testing Approach for Deep Neural Networks
Braiek, H. B., & Khomh, F. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00078 · Cited by 40
Testing Machine Learning Algorithms for Balanced Data Usage
Sharma, A., & Wehrheim, H. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00022 · Cited by 33
Coverage-directed differential testing of JVM implementations
Chen, Y., Su, T., Sun, C. P., Su, Z., & Zhao, J. (2016) · ACM SIGPLAN Notices · doi:10.1145/2980983.2908095 · Cited by 30
Supporting oracle construction via static analysis
Chen, J., Bai, Y., Hao, D., Zhang, L., Zhang, L., Xie, B., & Mei, H. (2016) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2970276.2970366 · Cited by 28 · cited 1×
“In traditional software systems, the test-oracle problem has been well studied and there are some ways of relieving the test-oracle problem [13, 17, 19, 22, 37, 49].”
MethodSelf-citation
Research works printed but not linked: 11
- Chen Junjie, 2019, Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 223-234
- Chen Junjie, 2020, Characterizing and Prioritizing Incidents for LargeScale Online Service Systems. In The 35th IEEE/ACM International Conference on Automated Software Engineering.
- Guo Qianyu, 2019, Proceedings of the 34th International Conference on Automated Software Engineering.
- Fitash Ul Haq Donghwan Shin Shiva Nejati and Lionel Briand. 2019. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. arXiv preprint arXiv:1912. 00805 ( 2019 ). Fitash Ul Haq Donghwan Shin Shiva Nejati and Lionel Briand. 2019. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. arXiv preprint arXiv:1912. 00805 ( 2019 ).
- Islam Md Johirul, 1906, A Comprehensive Study on Deep Learning Bug Characteristics. arXiv preprint arXiv
- 10.5555/3155562.3155636 · doi:10.5555/3155562.3155636
- Kass Robert E., 1998, Neal
- McKeeman William M, 1998, Diferential testing for software, Digital Technical Journal, 10
- Odena Augustus, 2018, Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv
- Yi Sun Yuheng Chen Xiaogang Wang and Xiaoou Tang. 2014. Deep learning face representation by joint identification-verification. In Advances in neural information processing systems. 1988-1996. Yi Sun Yuheng Chen Xiaogang Wang and Xiaoou Tang. 2014. Deep learning face representation by joint identification-verification. In Advances in neural information processing systems. 1988-1996.
- Zhang Jie M, 1906, Machine Learning Testing: Survey, Landscapes and Horizons. arXiv preprint arXiv
Cited by
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“Traditional techniques with API-level fuzzing [139], [140] or model-level fuzzing [141], [142] suffer from the following limitations: 1) lack of diverse API sequence thus cannot reveal bugs caused by chained API sequences; 2) cannot generate arbitrary code thus cannot explore the huge search space…”
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
“While such model-level fuzzers are able to nd bugs in DL libraries, due to the input/output constraints of DL APIs, model-level mutation/generation rules either are restrictive to certain shape-preserving APIs [71] or require manual annotation of the restrictions of all targeted APIs [25], leading to a limited number…”
BackgroundMethodResultInfluentialDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623343 · Cited by 200
Fairness Testing: A Comprehensive Survey and Analysis of Trends
Chen, Z., Zhang, J. M., Hort, M., Harman, M., & Sarro, F. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652155 · Cited by 198
“The ML testing literature [223, 228, 238, 282, 285] has consideredML frameworks as an important testing component and detected bugs inside ML frameworks that lead to accuracy problems in the final ML model.”
BackgroundMethodUnifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
Zhang, Z., Chen, C., Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., & Wang, R. (2023) · Transactions on machine learning research · doi:10.13140/rg.2.2.32710.69440/1 · Cited by 193
Metadata
- Authors
- Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D.
- Affiliations
- Tianjin University (CN)
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 788-799
- OpenAlex
- W3104663419
- Cited by
- 167
- References
- 61
- Citations by year
- 2026: 26 · 2025: 38 · 2024: 28 · 2023: 36 · 2022: 25 · 2021: 12 · 2020: 2
- FWCI
- 12.44
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- Funders
- National Natural Science Foundation of China (61872263,U1836214,61802275)
- Intelligent Manufacturing Special Fund of Tianjin (20191012,20193155)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- mutation testing, TensorFlow, Theano, CNTK, MXNet, lemon
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.