The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Deep learning library testing via effective model generation
Abstract
Deep learning (DL) techniques are rapidly developed and have been widely adopted in practice. However, similar to traditional software systems, DL systems also contain bugs, which could cause serious impacts especially in safety-critical domains. Recently, many research approaches have focused on testing DL models, while little attention has been paid for testing DL libraries, which is the basis of building DL models and directly affects the behavior of DL systems. In this work, we propose a novel approach, LEMON, to testing DL libraries. In particular, we (1) design a series of mutation rules for DL models, with the purpose of exploring different invoking sequences of library code and hard-to-trigger behaviors; and (2) propose a heuristic strategy to guide the model generation process towards the direction of amplifying the inconsistent degrees of the inconsistencies between different DL libraries caused by bugs, so as to mitigate the impact of potential noise introduced by uncertain factors in DL libraries. We conducted an empirical study to evaluate the effectiveness of LEMON with 20 release versions of 4 widely-used DL libraries, i.e., TensorFlow, Theano, CNTK, MXNet. The results demonstrate that LEMON detected 24 new bugs in the latest release versions of these libraries, where 7 bugs have been confirmed and one bug has been fixed by developers. Besides, the results confirm that the heuristic strategy for model generation indeed effectively guides LEMON in amplifying the inconsistent degrees for bugs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, model-level fuzzers can only cover a limited set of APIs with limited patterns [72], missing many diverse and interesting API sequences that can lead to bugs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [71]
Citations
Cites
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundAdversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,199
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundWhat every computer scientist should know about floating-point arithmetic
Goldberg, D. (1991) · ACM Computing Surveys · doi:10.1145/103162.103163 · Cited by 1,750 · cited 1×
“However, due to many uncertain factors in DL libraries, such as randomness and floating-point computing deviation [28], it is challenging to determine whether a found problem is a real bug in the library, or a problem caused by some uncertain factors.”
BackgroundAn Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110
Roulette-wheel selection via stochastic acceptance
Lipowski, A., & Lipowska, D. (2012) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2011.12.004 · Cited by 761
Research works printed but not linked: 11
- Chen Junjie, 2019, Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 223-234
- Chen Junjie, 2020, Characterizing and Prioritizing Incidents for LargeScale Online Service Systems. In The 35th IEEE/ACM International Conference on Automated Software Engineering.
- Guo Qianyu, 2019, Proceedings of the 34th International Conference on Automated Software Engineering.
- Fitash Ul Haq Donghwan Shin Shiva Nejati and Lionel Briand. 2019. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. arXiv preprint arXiv:1912. 00805 ( 2019 ). Fitash Ul Haq Donghwan Shin Shiva Nejati and Lionel Briand. 2019. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. arXiv preprint arXiv:1912. 00805 ( 2019 ).
- Islam Md Johirul, 1906, A Comprehensive Study on Deep Learning Bug Characteristics. arXiv preprint arXiv
- 10.5555/3155562.3155636 · doi:10.5555/3155562.3155636
- Kass Robert E., 1998, Neal
- McKeeman William M, 1998, Diferential testing for software, Digital Technical Journal, 10
- Odena Augustus, 2018, Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv
- Yi Sun Yuheng Chen Xiaogang Wang and Xiaoou Tang. 2014. Deep learning face representation by joint identification-verification. In Advances in neural information processing systems. 1988-1996. Yi Sun Yuheng Chen Xiaogang Wang and Xiaoou Tang. 2014. Deep learning face representation by joint identification-verification. In Advances in neural information processing systems. 1988-1996.
- Zhang Jie M, 1906, Machine Learning Testing: Survey, Landscapes and Horizons. arXiv preprint arXiv
Cited by
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“Traditional techniques with API-level fuzzing [139], [140] or model-level fuzzing [141], [142] suffer from the following limitations: 1) lack of diverse API sequence thus cannot reveal bugs caused by chained API sequences; 2) cannot generate arbitrary code thus cannot explore the huge search space…”
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
“While such model-level fuzzers are able to nd bugs in DL libraries, due to the input/output constraints of DL APIs, model-level mutation/generation rules either are restrictive to certain shape-preserving APIs [71] or require manual annotation of the restrictions of all targeted APIs [25], leading to a limited number…”
BackgroundMethodResultInfluentialDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623343 · Cited by 200
Fairness Testing: A Comprehensive Survey and Analysis of Trends
Chen, Z., Zhang, J. M., Hort, M., Harman, M., & Sarro, F. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652155 · Cited by 198
“The ML testing literature [223, 228, 238, 282, 285] has consideredML frameworks as an important testing component and detected bugs inside ML frameworks that lead to accuracy problems in the final ML model.”
BackgroundMethodUnifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
Zhang, Z., Chen, C., Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., & Wang, R. (2023) · Transactions on machine learning research · doi:10.13140/rg.2.2.32710.69440/1 · Cited by 193
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
A comprehensive study of deep learning compiler bugs
Shen, Q., Ma, H., Chen, J., Tian, Y., Cheung, S.-C., & Chen, X. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468591 · Cited by 179
“In addition, there are somework onDL testing, including DL model testing [19, 67], DL program testing [69, 77], and DL library testing [54, 66].”
MethodSelf-citationWhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Yang, C., Deng, Y., Lu, R., Yao, J., Liu, J., Jabbarvand, R., & Zhang, L. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689736 · Cited by 180
“In early work, DL models are either directly curated from existing open-source models [61] or created using simple shape-preserving operators [28, 82].”
MethodPrioritizing Test Inputs for Deep Neural Networks via Mutation Analysis
Wang, Z., You, H., Chen, J., Zhang, Y., Dong, X., & Zhang, W. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00046 · Cited by 171
“presented in the existing work [49]) on them as follows:”
BackgroundMethodSelf-citationNNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers
Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., & Zhang, L. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575707 · Cited by 167
“• LEMON [60] is a mutation-based model generator that mutates pre-trained Keras [19] models [61].”
InfluentialToward Understanding Deep Learning Framework Bugs
Chen, J., Liang, Y., Shen, Q., Jiang, J., & Li, S. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3587155 · Cited by 137
“The studied DL framework testing techniques share a significant common-ality in terms of test coverage and their coverage mainly depends on the used pre-trained models rather than mutated models by LEMON and Audee.”
BackgroundMethodInfluentialSelf-citationLarge Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTPreprint
Deng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02014 · Cited by 129
“Model-level fuzzers either reuse/mutate existing seed models [26, 50, 70], or generate DL models fromscratch [24, 35].”
BackgroundMethodInfluentialA survey on large language models for software engineering
Zhang, Q., Fang, C., Xie, Y., Zhang, Y., Yu, S., Sun, W., Yang, Y., & Chen, Z. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4670-0 · Cited by 124
Free lunch for testing: fuzzing deep-learning libraries from open source
Wei, A., Deng, Y., Yang, C., & Zhang, L. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510041 · Cited by 106
“Based on CRADLE, LEMON [65] advances testing DL libraries by proposing model-level mutation.”
BackgroundMethodResultInfluentialMetamorphic Testing of Deep Learning Compilers
Xiao, D., Liu, Z., Yuan, Y., Pang, Q., & Wang, S. (2022) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3508035 · Cited by 95
“Wang et al. [88] propose techniques to mutate DL models to better invoke corner execution paths of DL libraries.”
BackgroundSilent bugs in deep learning frameworks: an empirical study of Keras and TensorFlow
Tambon, F., Nikanjam, A., An, L., Khomh, F., & Antoniol, G. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10389-6 · Cited by 90
“– Differential testing for diagnosing silent bugs: Several techniques have been proposed to deal with bugs in DL libraries/frameworks [41, 48, 54] with some success, though it was shown that such tools generally achieve relatively low test coverage and tend to cover the same part of the code [12].”
BackgroundFuzzing deep-learning libraries via automated relational API inference
Deng, Y., Yang, C., Wei, A., & Zhang, L. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549085 · Cited by 86
“To reduce such threats, we adopt the number of covered APIs and detected bugs, following prior work on DL library testing [54, 65, 66].”
BackgroundMethodInfluentialSecurity for Machine Learning-based Software Systems: A Survey of Threats, Practices, and Challenges
Chen, H., & Babar, M. A. (2023) · ACM Computing Surveys · doi:10.1145/3638531 · Cited by 83
DocTer: documentation-guided fuzzing for testing deep learning API functions
Xie, D., Li, Y., Kim, M., Pham, H. V., Tan, L., Zhang, X., & Godfrey, M. W. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534220 · Cited by 81
“The DL library testing techniques focus on addressing the test oracle challenge, by using differential testing [18, 24, 42, 43, 50, 52] or oracle approximation [35, 60].”
MethodCoverage-guided tensor compiler fuzzing with joint IR-pass mutation
Liu, J., Wei, Y., Yang, S., Deng, Y., & Zhang, L. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3527317 · Cited by 80
History-driven test program synthesis for JVM testing
Zhao, Y., Wang, Z., Chen, J., Liu, M., Wu, M., Zhang, Y., & Zhang, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510059 · Cited by 79
“Since the first work on fuzzing [40], various techniques have been proposed for fuzzing software systems from different application domains [13, 32, 49, 51, 52, 55].”
BackgroundSelf-citationFuzzing Automatic Differentiation in Deep-Learning Libraries
Yang, C., Deng, Y., Yao, J., Tu, Y., Li, H., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00105 · Cited by 77
“While LEMON adopted advanced mutation rules (e.g., layer addition), it still only covers a small set of APIs [30] due to its strict mutation rules, e.g., an API cannot be added/removed in the model unless its input and output tensor shapes are identical.”
InfluentialMuffin: testing deep learning libraries via neural architecture fuzzing
Gu, J., Luo, X., Zhou, Y., & Wang, X. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510092 · Cited by 71
“Similar to existing work [44, 50], we focus on testing low-level libraries, e.g. , TensorFlow, CNTK, and Theano.”
BackgroundMethodInfluentialPredoo: precision testing of deep learning operators
Zhang, X., Sun, N., Fang, C., Liu, J., Liu, J., Chai, D., Jiang, W., & Chen, Z. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464843 · Cited by 68
“KEYWORDS precision testing, deep learning operators, floating-point operation ACM Reference Format: Xufan Zhang, Ning Sun, Chunrong Fang, Jiawei Liu, Jia Liu, Dong Chai, Jiang Wang, and Zhenyu Chen.”
BackgroundUnderstanding performance problems in deep learning systems
Cao, J., Chen, B., Sun, C., Hu, L., Wu, S., & Peng, X. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549123 · Cited by 66
EAGLE: creating equivalent graphs to test deep learning libraries
Wang, J., Lutellier, T., Qian, S., Pham, H. V., & Tan, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510165 · Cited by 65
“Differential testing of DL libraries: Previous work [6, 14, 28, 31, 34, 39, 42] uses differential testing to find inconsistencies between DL libraries.”
MethodResultExposing numerical bugs in deep learning via gradient back-propagation
Yan, M. X., Chen, J., Zhang, X., Tan, L., Wang, G., & Wang, Z. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468612 · Cited by 65
“Actually, both the program level and the infrastructure level are the basis of the production level since DL models are built based on DL programs by invoking DL libraries, and thus bugs in the former two levels could directly a � ect the performance of DL models [47, 56].”
MethodSelf-citationInteroperability in Deep Learning: A User Survey and Failure Analysis of ONNX Model Converters
Jajal, P., Jiang, W., Tewari, A., Kocinare, E., Woo, J., Sarraf, A., Lu, Y.-H., Thiruvathukal, G. K., & Davis, J. C. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680374 · Cited by 64
Fuzzing Deep Learning Compilers with HirGen
Ma, H., Shen, Q., Tian, Y., Chen, J., & Cheung, S.-C. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598053 · Cited by 59
“MT-DLComp and LEMON detected nothing.”
MethodInfluentialSelf-citationEnhanced compiler bug isolation via memoized search
Chen, J., Ma, H., & Zhang, L. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416570 · Cited by 59
Self-citationJITfuzz: Coverage-guided Fuzzing for JVM Just-in-Time Compilers
Wu, M., Lu, M., Cui, H., Chen, J., Zhang, Y., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00017 · Cited by 58
“Recently, a large group of researchers have been focusing on applying fuzzing/testing to the emerging AI/ML systems [77, 78, 79, 80, 81, 82, 83, 84].”
BackgroundSelf-citationNatural Test Generation for Precise Testing of Question Answering Software
Shen, Q., Chen, J., Zhang, J. M., Wang, H., Liu, S., & Tian, M. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556953 · Cited by 52
“ware [4, 5, 34, 40, 41, 43], QA software also contains bugs.”
Self-citationZhang, P., Ren, B., Dong, H., & Dai, Q. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3124006 · Cited by 50
“Recently, an increasing number of researchers have contributed to testing DL systems with a variety of approaches [9], [10], [11], [12], [13], [14].”
BackgroundNeuRI: Diversifying DNN Generation via Inductive Rule Inference
Liu, J., Peng, J., Wang, Y., & Zhang, L. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616337 · Cited by 47
“For example, LEMON [42] only operates shape-preserving operators without input constraints.”
BackgroundMethodInfluentialDuo: Differential Fuzzing for Deep Learning Operators
Zhang, X., Liu, J., Sun, N., Fang, C., Liu, J., Wang, J., Chai, D., & Chen, Z. (2021) · IEEE Transactions on Reliability · doi:10.1109/tr.2021.3107165 · Cited by 47
COMET: Coverage-guided Model Generation For Deep Learning Library Testing
Li, M., Cao, J., Tian, Y., Li, T. O., Wen, M., & Cheung, S.-C. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3583566 · Cited by 45
“Prior works [18, 19, 31, 43, 54] mainly follow the same testing paradigm, i.”
BackgroundMethodInfluentialVulnerability detection through machine learning-based fuzzing: A systematic review
Chafjiri, S. B., Legg, P., Hong, J. S., & Tsompanas, M. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103903 · Cited by 45
Regression Fuzzing for Deep Learning Systems
You, H., Wang, Z., Chen, J., Liu, S., & Li, S. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00019 · Cited by 44
“This is formally defined in formula (3), where p is the probability of being successful in one Bernoulli trial in geometric distribution used for approximating the desired distribution following the existing MCMC framework [18], [32], k a and k b are ranks of R A and R B , respectively.”
BackgroundMethodSelf-citationAttack as defense: characterizing adversarial examples using robustness
Zhao, Z., Chen, G., Wang, J., Yang, Y., Song, F., & Sun, J. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464822 · Cited by 43
“Some other works focus on testing the neural network at the architecture level [85] or the deep learning library itself [77].”
BackgroundACETest: Automated Constraint Extraction for Testing Deep Learning Operators
Shi, J., Xiao, Y., Li, Y., Li, Y., Yu, D., Yu, C., Su, H., Chen, Y., & Huo, W. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598088 · Cited by 42
“To detect bugs in DL libraries, multiple techniques [14, 15, 26, 28, 43, 45, 47, 51] have been proposed to test them by generating or mutating DL models.”
MethodOn the Challenges of Fuzzing Techniques via Large Language Models
Huang, L., Zhao, P., Ma, L., & Chen, H. (2025) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse67621.2025.00028 · Cited by 42
“By comparing TitanFuzz with current state-of-the-art API-level (such as FreeFuzz [55] and DeepREL [56]) and model-level (such as LEMON [57] and Muffin [58]) fuzzers, it was found that TitanFuzz’s API coverage in TensorFlow and PyTorch increased by 91.11% and 24.09% respectively.”
InfluentialGeneration-based Differential Fuzzing for Deep Learning Libraries
Liu, J., Huang, Y., Wang, Z., Ma, L., Fang, C., Gu, M., Zhang, X., & Chen, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3628159 · Cited by 40
Self-citationBoosting Compiler Testing via Compiler Optimization Exploration
Chen, J., & Suo, C. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3508362 · Cited by 40
“Software testing is one of the most important parts of the entire cycle of software development and is vital to guarantee the quality of software [20, 47, 60, 61].”
BackgroundSelf-citationDutta, S., Arunachalam, A., & Misailovíc, S. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00026 · Cited by 40
“debugging various non-deterministic systems such as ML frameworks [39, 41, 51, 70, 71, 73], Probabilistic Programming Systems [22, 23, 27, 47], and Approximation Algorithms [42].”
Wijaya, N. N., Setiadi, D. R. I. M., & Muslikh, A. R. (2024) · Journal of Computing Theories and Applications · doi:10.62411/jcta.9655 · Cited by 40
“This happens because the DL model will be designed with more neurons in each layer and even the layer itself to make the computation complex[24].”
MethodRevisiting deep neural network test coverage from the test effectiveness perspective
Yan, M. X., Chen, J., Cao, X., Wu, Z. J., Kang, Y., & Wang, Z. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2561 · Cited by 36
Self-citationFault Injection for TensorFlow Applications
Narayanan, N., Chen, Z., Fang, B., Li, G., Pattabiraman, K., & DeBardeleben, N. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3175930 · Cited by 34
“Mutation testing techniques have been used to create mutant ML models that are tested with certain inputs to improve the quality of test data [18], [19] and to localize bugs in the ML frameworks [20], [21].”
MethodA Comprehensive Study of Real-World Bugs in Machine Learning Model Optimization
Guan, H., Xiao, Y., Li, J., Liu, Y., & Bai, G. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00024 · Cited by 34
“Existing techniques [50], [52] generate inputs by mutating seed models.”
MethodDeep Learning Library Testing: Definition, Methods and Challenges
Zhang, X., Jiang, W., Shen, C. Q., Li, Q. L., Wang, Q., Lin, C., & Guan, X. (2025) · ACM Computing Surveys · doi:10.1145/3716497 · Cited by 33
“The model-level differential testing usually leverages the different results of a widely-used DL model (e.g., ResNet-50) on different platforms or frameworks to detect bugs [68, 129].”
InfluentialCost-Effective Testing of a Deep Learning Model through Input Reduction
Zhou, J., Li, F., Dong, J., Zhang, H., & Hao, D. (2020) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre5003.2020.00035 · Cited by 30
An Empirical Study of Fault Triggers in Deep Learning Frameworks
Du, X., Sui, Y., Liu, Z., & Ai, J. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3152239 · Cited by 30
“…deep learning frameworks for our study, we have investigated 14 popular deep learning frameworks, which have been widely used in recent years [13], [26], [27], [28], [29], including TensorFlow, MXNET, Paddle-Paddle, Keras, Pytorch, Caffe, CNTK, Torch, Deeplearning4j, Caffe2, Sonnet and Chainer.”
MethodGenCoG: A DSL-Based Approach to Generating Computation Graphs for TVM Testing
Wang, Z., Nie, P., Miao, X., Chen, Y., Wan, C., Bu, L., & Zhao, J. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598105 · Cited by 29
“For example, LEMON [35] suggests the idea of mutating realworld DL models to generate more complicated ones, while it restricts the mutation to operators whose input and output tensor shapes are identical.”
BackgroundMethodInfluentialSelf-citationEnhancing Differential Testing with LLMs for Testing Deep Learning Libraries
Li, M., Li, D., Liu, J., Cao, J., Tian, Y., & Cheung, S.-C. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735637 · Cited by 28
“The first threat is that the LLM’s hallucination issue may affect the quality of our synthesized counterparts, thus introducing false positive in-consistencies during bug detection.”
Towards mutation testing of Reinforcement Learning systems
Lu, Y., Sun, W., & Sun, M. (2022) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2022.102701 · Cited by 27
Towards Understanding the Faults of JavaScript-Based Deep Learning Systems
Quan, L., Guo, Q., Xie, X., Chen, S., Li, X., & Liu, Y. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3560427 · Cited by 27
“Since existing testing techniques [14, 48, 73, 82] suffer from low operator coverage, we emphasize this is a challenge for detecting API errors on JavaScript-based DL systems.”
Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 26
An Empirical Study on Common Bugs in Deep Learning Compilers
Du, X., Zheng, Z., Ma, L., & Zhao, J. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00030 · Cited by 26
“In addition, some recent progress has been made to analyze bug characteristics in DL frameworks [58]–[61].”
BackgroundAn Empirical Study on Numerical Bugs in Deep Learning Programs
Gan, W., Wang, Z., Chen, J., Chen, X., & Yan, M. X. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3559561 · Cited by 25
“Besides, there are also research projects on DL testing at other levels, such as model level [9, 17], library level [8, 16], and compiler level [15].”
BackgroundSelf-citationOneLog: towards end-to-end software log anomaly detection
Hashemi, S., & Mäntylä, M. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00428-x · Cited by 24
Smoke testing for machine learning: simple tests to discover severe bugs
Herbold, S., & Haar, T. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10073-7 · Cited by 23
“A different perspective on the test input generation was provided by Wang et al. (2020).”
MethodIvySyn: Automated Vulnerability Discovery in Deep Learning FrameworksPreprint
Christou, N., Jin, D., Atlidakis, V., Ray, B., & Kemerlis, V. P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.14921 · Cited by 23
“…bugs is to make use of pre-trained DL models and continuously: (1) perform alternations on them (e.g., by removing, duplicating, and shuffling parameters, or by modifying training data) [19], [52]–[55]; and (2) feed them back to the target DL frameworks hoping to trigger some erroneous behavior.”
MethodOn-the-fly Improving Performance of Deep Code Models via Input Denoising
Tian, Z., Chen, J., & Zhang, X. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00166 · Cited by 22
“There are also some work on ensuring the quality of deep learning systems from the testing perspective, which involve different levels such as model-level [61]–[64], program-level [65], [66], library-level [67], [68], and compiler-level [69], [70] testing.”
BackgroundSelf-citationHistory-Driven Fuzzing for Deep Learning Libraries
Harzevili, N. S., Mohajer, M. M., Wei, M., Pham, H. V., & Wang, S. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688838 · Cited by 20
Zhou, C., Qian, B., Go, G., Zhang, Q., Li, S., & Jiang, Y. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689757 · Cited by 20
Taxonomy of Aging-related Bugs in Deep Learning Libraries
Liu, Z., Zheng, Y., Du, X., Zheng, H., Ding, W., Miao, Y., & Zheng, Z. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00048 · Cited by 19
Reliability Assurance for Deep Neural Network Architectures Against Numerical Defects
Li, L., Zhang, Y., Ren, L., Xiong, Y., & Xie, T. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00156 · Cited by 18
“Approaches exist for understanding, detecting, and testing against these defects [50, 38, 43, 53].”
BackgroundA Tale of Two DL Cities: When Library Tests Meet Compiler
Shen, Q., Tian, Y., Ma, H., Chen, J., Huang, L., Fu, R., Cheung, S.-C., & Wang, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00025 · Cited by 18
“According to the test format during the generation of tests, they can be mainly divided into graph-level [19], [21], [30], [46] and API-level [15], [16], [47]–[49] test generation.”
Self-citationA Large-Scale Empirical Study on Improving the Fairness of Image Classification Models
Yang, J., Jiang, J., Sun, Z., & Chen, J. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652122 · Cited by 17
“Many testing [2, 19, 26, 31, 59, 73, 75, 78, 79, 87] and repair studies [56, 71, 77] aim to automatically identify and fix such issues.”
Self-citationGuan, H., Bai, G., & Liu, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680383 · Cited by 17
Li, Y., Shen, W., Wu, T., Chen, L., Wu, D., Zhou, Y., & Xu, B. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106954 · Cited by 17
A Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators
Chen, J., Jia, C., Yan, Y., Ge, J., Zheng, H., & Cheng, Y. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660796 · Cited by 16
“Several studies have been conducted to uncover errors in DL libraries [13, 15, 22, 28, 35], such as CRADLE [28], which tackled the oracle challenge of differentially testing multiple DL models on various backends in Keras [6].”
InfluentialSelf-citationDynamic Data Fault Localization for Deep Neural Networks
Yin, Y., Feng, Y., Weng, S., Liu, Z., Yao, Y., Zhang, Y., Zhao, Z., & Chen, Z. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616345 · Cited by 15
“Wang et al. [79] proposed LEMON to debug DL libraries by generating effective DL models via guided mutation.”
D3: Differential Testing of Distributed Deep Learning With Model Generation
Wang, J., Pham, H. V., Li, Q., Tan, L., Guo, Y., Aziz, A., & Meijer, E. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3461657 · Cited by 15
“Recentwork fuzzes [46], [69], [70] or generates [28], [71] DL models to test DL libraries.”
InfluentialOn Security Weaknesses and Vulnerabilities in Deep Learning Systems
Lai, Z., Chen, H., Sun, R., Zhang, Y., Xue, M., & Yuan, D. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3482707 · Cited by 15
“PreviousworkaddressingissuesinDLsystemshasfocused onmulti-foldaspects,includingtheusageofmachinelearning API[9],[10],determiningandcategorizingbugsinDLsoft-ware[11],[12],[13],[14],[15],andtestingtheDLlibrariesand programs[16],[17],[18].”
DevMuT: Testing Deep Learning Framework via Developer Expertise-Based Mutation
Mu, Y., Zhai, J., Fang, C., Chen, X., Cao, Z., Yang, P., Zou, Y., Zheng, T., & Chen, Z. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695523 · Cited by 15
“Some researchers [18, 19, 28, 29, 42, 56] adopt DL models as the test inputs and combine them with different mutation operators, such as modifying the model structure and editing layer parameters, to generate more diverse mutants and improve the test effectiveness.”
InfluentialCrossProbe: LLM-Empowered Cross-Project Bug Detection for Deep Learning Frameworks
Guan, H., Bai, G., & Liu, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728984 · Cited by 15
“We open-source CrossProbe and the associated knowledge database of API on [6], to facilitate further research of the code transfer between deep learning frameworks.”
Finding Deviated Behaviors of the Compressed DNN Models for Image Classifications
Tian, Y., Zhang, W., Wen, M., Cheung, S.-C., Sun, C. P., Ma, S., & Jiang, Y. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3583564 · Cited by 15
“Since the selection of ���+1 from all possible states only depends on ��� [11, 35, 73], this process is a typical Markov Chain.”
Deep learning framework testing via hierarchical and heuristic model generation
Zou, Y., Sun, H., Fang, C., Liu, J., & Zhang, Z. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111681 · Cited by 14
Unraveling Code Clone Dynamics in Deep Learning Frameworks
Assi, M., Hassan, S., & Zou, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3721125 · Cited by 13
“The main contributions of our work are as follows: (1) We analyze the characteristics of the evolution of code clones in DL frameworks throughout their entire lifespan.”
Predictive Mutation Analysis of Test Case Prioritization for Deep Neural Networks
Wei, Z., Wang, H., Ashraf, I., & Chan, W. K. (2022) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs57517.2022.00074 · Cited by 13
“Some works [25], [38] contain random mutation operators to construct a better test suite but lack fault-revealing abilities, which we do not include in the work.”
BackgroundSkipFuzz: Active Learning-based Input Selection for Fuzzing Deep Learning LibrariesPreprint
Kang, H. J., Rattanukul, P., Haryono, S. A., Nguyen, T. G., Ragkhitwetsagul, C., Păsăreanu, C. S., & Lo, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.04038 · Cited by 13
“Several approaches detect bugs through metamorphic and differential testing [17, 35, 46, 48, 51].”
BackgroundJiang, Y., Zhou, Z., Xu, B., Liu, B., Xu, R., & Huang, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.14813 · Cited by 13
“Testing Pipeline Code, Libraries, and Compilers CRA-DLE [33], AUDEE [18], LEMON [44], and NNSmith [27] employ differential testing to detect and diagnose failures across frameworks and compilers.”
Herbold, S., & Tunkel, S. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10273-9 · Cited by 12
“In recent years, this idea was adopted to define test oracles for deep learning (e.g., Pham et al., 2019; Wang et al., 2020; Guo et al., 2020; Asyrofi et al., 2020).”
MethodGo, G., Zhou, C., Zhang, Q., Zou, X., Shi, H., & Jiang, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680364 · Cited by 12
Improving Deep Learning Framework Testing with Model-Level Metamorphic Testing
Mu, Y., Zhai, J., Fang, C., Chen, X., Cao, Z., Yang, P., Zhao, K., Guo, A., & Chen, Z. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728972 · Cited by 12
“LEMON [59] introduces structure and weight mutations to create diverse models, conducting testing consistent with CRADLE.”
InfluentialA First Look at Bugs in LLM Inference Engines
Liu, M., Zhong, S., Bi, W., Zhang, Y., Chen, Z., Chen, Z., Liu, X., & Ma, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3788873 · Cited by 12
DeepDiffer: Find Deep Learning Compiler Bugs via Priority-guided Differential Fuzzing
Lin, K., Song, X., Zeng, Y., & Guo, S. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00066 · Cited by 12
“For instance, LEMON [41] tests DL frameworks by mutating Keras models.”
MethodEnhanced Fairness Testing via Generating Effective Initial Individual Discriminatory Instances
Tian, Z., Ma, M., Hort, M., Sarro, F., Zhang, H., & Chen, J. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3737697 · Cited by 12
“Artificial Intelligence (AI) systems have become increasingly popular over the years, and are now at the core of many datadriven software systems such as loan approval and risk assessments [39].”
Self-citationIterTestQ: Assembly-Level, Cross-Platform Testing of Quantum Computing Platforms
Paltenghi, M., & Pradel, M. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832211 · Cited by 10
“Grammar-based generators have been used to test compilers and similar systems, including the Java Virtual Machine [51], C compilers [15, 62], SMT solvers [59], database engines [49], and deep learning infrastructure [31, 34, 35, 57].”
Louloudakis, N., Gibson, P., Cano, J., & Rajan, A. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00054 · Cited by 10
“Under this context, CRADLE [37] applies output and model execution comparison, while LEMON [38] utilizes mutation testing to detect bugs in DL frameworks.”
MethodOBSan: An Out-Of-Bound Sanitizer to Harden DNN Executables
Chen, Y., Yuan, Y., & Wang, S. (2023) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2023.24103 · Cited by 10
“Recent works test DL frameworks or DL compilers [30, 42, 62, 63, 71, 83, 89, 91, 104].”
MethodMirrorFuzz: Leveraging LLM and Shared Bugs for Deep Learning Framework APIs Fuzzing
Ou, S., Li, Y., Yu, L., Wei, C., Wen, T., Chen, Q., Chen, Y., Tang, H., & Pan, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3619966 · Cited by 9
“Model-level fuzzing [19], [20], [21], [22], [23] builds DL models as test cases and uses differential testing to identify defects, but it can only cover a limited set of APIs.”
Skin Type Detection with Deep Learning: A Comparative Analysis
Ardaç, F. B. K., Kara, R., & ÇELİK, S. S. (2023) · Düzce Üniversitesi Bilim ve Teknoloji Dergisi · doi:10.29130/dubited.930096 · Cited by 9
TorchProbe: Fuzzing Dynamic Deep Learning Compilers
Su, Q., Geng, C., Pekhimenko, G. G., & Si, X. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8311-7_15 · Cited by 9
FetaFix: Automatic Fault Localization and Repair of Deep Learning Model Conversions
Louloudakis, N., Gibson, P., Cano, J., & Rajan, A. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756931 · Cited by 9
Fault Localization for Buggy Deep Learning Framework Conversions in Image Recognition
Louloudakis, N., Gibson, P., Cano, J., & Rajan, A. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00147 · Cited by 9
“LEMON [21] leverages the metrics used by CRADLE for its analysis to apply mutation testing.”
MethodA survey of coverage-guided greybox fuzzing with deep neural models
Qiu, J., Jiang, Y., Miao, Y., Luo, W., Pan, L., & Zheng, X. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107797 · Cited by 8
Testing Machine Learning and Deep Learning Systems: Achievements and Challenges
Albelali, S., & Ahmed, M. (2025) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-025-10276-w · Cited by 8
MMOS: Multi-Staged Mutation Operator Scheduling for Deep Learning Library Testing
Li, J., Li, S., Wu, J., Luo, L., Bai, Y., & Yu, H. (2022) · Globecom · doi:10.1109/globecom48099.2022.10001093 · Cited by 8
“Based on CRADLE, authors of [5] proposed a mutation based differential testing tool LEMON, which generated more DL models to completely explore codes of DL libraries.”
BackgroundMethodDeep Learning Framework Fuzzing Based on Model Mutation
Shen, X., Zhang, J., Wang, X., Yu, H., & Sun, G. (2021) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc53577.2021.00059 · Cited by 8
Metadata
- Authors
- Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D.
- Affiliations
- Tianjin University (CN)
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 788-799
- OpenAlex
- W3104663419
- Cited by
- 167
- References
- 61
- Citations by year
- 2026: 26 · 2025: 38 · 2024: 28 · 2023: 36 · 2022: 25 · 2021: 12 · 2020: 2
- FWCI
- 12.44
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- Funders
- National Natural Science Foundation of China (61872263,U1836214,61802275)
- Intelligent Manufacturing Special Fund of Tianjin (20191012,20193155)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- mutation testing, TensorFlow, Theano, CNTK, MXNet, lemon
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.