Deep learning library testing via effective model generation
The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Deep learning (DL) techniques are rapidly developed and have been widely adopted in practice. However, similar to traditional software systems, DL systems also contain bugs, which could cause serious impacts especially in safety-critical domains. Recently, many research approaches have focused on testing DL models, while little attention has been paid for testing DL libraries, which is the basis of building DL models and directly affects the behavior of DL systems. In this work, we propose a novel approach, LEMON, to testing DL libraries. In particular, we (1) design a series of mutation rules for DL models, with the purpose of exploring different invoking sequences of library code and hard-to-trigger behaviors; and (2) propose a heuristic strategy to guide the model generation process towards the direction of amplifying the inconsistent degrees of the inconsistencies between different DL libraries caused by bugs, so as to mitigate the impact of potential noise introduced by uncertain factors in DL libraries. We conducted an empirical study to evaluate the effectiveness of LEMON with 20 release versions of 4 widely-used DL libraries, i.e., TensorFlow, Theano, CNTK, MXNet. The results demonstrate that LEMON detected 24 new bugs in the latest release versions of these libraries, where 7 bugs have been confirmed and one bug has been fixed by developers. Besides, the results confirm that the heuristic strategy for model generation indeed effectively guides LEMON in amplifying the inconsistent degrees for bugs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, model-level fuzzers can only cover a limited set of APIs with limited patterns [72], missing many diverse and interesting API sequences that can lead to bugs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [71]
Citations
Cites
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundAdversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundWhat every computer scientist should know about floating-point arithmetic
Goldberg, D. (1991) · ACM Computing Surveys · doi:10.1145/103162.103163 · Cited by 1,750 · cited 1×
“However, due to many uncertain factors in DL libraries, such as randomness and floating-point computing deviation [28], it is challenging to determine whether a found problem is a real bug in the library, or a problem caused by some uncertain factors.”
BackgroundAn Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110
Roulette-wheel selection via stochastic acceptance
Lipowski, A., & Lipowska, D. (2012) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2011.12.004 · Cited by 761
Research works printed but not linked: 11
- Chen Junjie, 2019, Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 223-234
- Chen Junjie, 2020, Characterizing and Prioritizing Incidents for LargeScale Online Service Systems. In The 35th IEEE/ACM International Conference on Automated Software Engineering.
- Guo Qianyu, 2019, Proceedings of the 34th International Conference on Automated Software Engineering.
- Fitash Ul Haq Donghwan Shin Shiva Nejati and Lionel Briand. 2019. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. arXiv preprint arXiv:1912. 00805 ( 2019 ). Fitash Ul Haq Donghwan Shin Shiva Nejati and Lionel Briand. 2019. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. arXiv preprint arXiv:1912. 00805 ( 2019 ).
- Islam Md Johirul, 1906, A Comprehensive Study on Deep Learning Bug Characteristics. arXiv preprint arXiv
- 10.5555/3155562.3155636 · doi:10.5555/3155562.3155636
- Kass Robert E., 1998, Neal
- McKeeman William M, 1998, Diferential testing for software, Digital Technical Journal, 10
- Odena Augustus, 2018, Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv
- Yi Sun Yuheng Chen Xiaogang Wang and Xiaoou Tang. 2014. Deep learning face representation by joint identification-verification. In Advances in neural information processing systems. 1988-1996. Yi Sun Yuheng Chen Xiaogang Wang and Xiaoou Tang. 2014. Deep learning face representation by joint identification-verification. In Advances in neural information processing systems. 1988-1996.
- Zhang Jie M, 1906, Machine Learning Testing: Survey, Landscapes and Horizons. arXiv preprint arXiv
Cited by
Items 101–200 of 221
Semantic-Based Neural Network Repair
Schumi, R., & Sun, J. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598045 · Cited by 8
“[55] show a differential testing approach called LEMON that uses mutations to generate AI models.”
BackgroundMethodInfluentialWhere Agent Frameworks Fall Short: Examining Functional Challenges and Usability ConcernsPreprint
Zhu, X., Wu, J., Zhang, X., Li, T., Shen, C., Fang, C., Liu, Y., & Zhai, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.21806 · Cited by 10
Mutation-Based Deep Learning Framework Testing Method in JavaScript Environment
Zou, Y., Zhai, J., Fang, C., Liu, J., Zheng, T., & Chen, Z. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695478 · Cited by 7
Path analysis for effective fault localization in deep neural networks
Hashemifar, S., Parsa, S. A., & Kalaee, A. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.112805 · Cited by 7
“CRADLE [35] and LEMON [36] are tools designed to detect and localize bugs in deep learning libraries through differential testing and guided mutation, respectively.”
Lightweight Concolic Testing via Path-Condition Synthesis for Deep Learning Libraries
Kim, S., Kim, Y., Park, D., Jeon, Y., Yi, J., & Kim, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00202 · Cited by 7
Effective random test generation for deep learning compilers
Ren, L., Wang, Z., Zhang, L., Jiang, G., Xiong, Y., & Xie, T. (2025) · Science China Information Sciences · doi:10.1007/s11432-023-4301-6 · Cited by 7
“LEMON [36] and Muffin [15] are two recent work proposed to test deep learning libraries.”
Self-citationTowards effective metamorphic testing by algorithm stability for linear classification programs
Yang, Y., Li, Z., Wang, H., Xu, C., & Ma, X. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111012 · Cited by 7
“There were also many other works in testing machine learning framework [60, 67].”
ResultExAIS: executable AI semantics
Schumi, R., & Sun, J. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510112 · Cited by 7
“Recently, multiple researchers started working on AI framework testing [22, 23, 33, 38, 40, 41, 43].”
BackgroundMethodInfluentialCan Machine Learning Pipelines Be Better Configured?
Wang, Y., Wang, Y., Zhang, T., Yu, Y., Cheung, S.-C., Yu, H., & Zhu, Z. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616352 · Cited by 8
“Wang et al. [74], Guo et al. [42] and Phamet et al. [63] proposed novel mutation testing/differential techniques to detect behavioral inconsistencies across multiple ML frameworks.”
BackgroundA Systematic Survey on Debugging Techniques for Machine Learning SystemsPreprint
Nguyen, T.-D., Tian, H., Le, B., Thongtanunam, P., & McIntosh, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.03158 · Cited by 7
Evaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study
Harzevili, N. S., Wei, M., Mohajer, M. M., Pham, H. V., & Wang, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3729533 · Cited by 6
“Wang et al. [76] introduced LEMON, a novel approach for testing DL libraries.”
Automating Test Vector Validation for Silicon Verification at Scale
Nagaraj, V. (2025) · International Journal of Engineering and Architecture · doi:10.58425/ijea.v2i1.358 · Cited by 6
“On the other hand, DL models can be trained on historical test data to predict which areas in the design are more prone to faults and thus perform targeted test vector generation (Wang et al., 2020).”
Fuzz4Cuda: Fuzzing your NVIDIA GPU libraries through debug interface
Zhou, Y., Jia, P., Liu, J., & Fan, X. (2025) · Computers & Security · doi:10.1016/j.cose.2025.104754 · Cited by 6
Achieving Last-Mile Functional Coverage in Testing Chip Design Software Implementations
Yan, M. X., Chen, J., Mao, H., Jiang, J., Hao, J., Li, X., Tian, Z., Chen, Z., Li, D., Xian, Z., et al. (2023) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip58684.2023.00037 · Cited by 6
Self-citationS cuzer : A Scheduling Optimization Fuzzer for TVM
Chen, X., Lin, X., Wang, J., Sun, J., Wang, J., & Wang, W. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705308 · Cited by 6
“In response to this situation, the method we adopted was to conduct multiple repeated tests to improve the relevant information of the operator as much as possible.”
Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis
Mu, Y., Wang, R., Zhai, J., Fang, C., Chen, X., Wu, J., Guo, A., Shen, J., Li, B., & Chen, Z. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3821532 · Cited by 6
Assessing Robustness of Image Recognition Models to Changes in the Computational Environment
Louloudakis, N., Gibson, P., Cano, J., & Rajan, A. (2022) · Enlighten: Publications (University of Glasgow) · Cited by 6
DeepCov: Coverage Guided Deep Learning Framework Fuzzing
Wu, J., Li, S., Li, J., Luo, L., Yu, H., & Sun, G. (2022) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc55868.2022.00061 · Cited by 6
“LEMON [8] designs a bunch of mutation operators from intact-layer and inner-layer levels to generate models for differential testing and schedules them by the model inconsistency metric.”
BackgroundLiu, J., Zhang, X., Xu, L., Fang, C., Gu, M., Luo, W., Chai, D., Wang, J., Zhao, Z., & Chen, Z. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715104 · Cited by 5
“Furthermore, the proposed method has been employed in a real-world application of a large software technology company for months and solved quantities of FPPs in CNN operators, which indicates the elimination of the threats to construct validity.”
Testing Deep Learning Libraries via Neurosymbolic Constraint Learning
Naziri, M. M. A., Kim, S., Qin, F., Dutta, S., & d’Amorim, M. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787800 · Cited by 5
“Model-level testing [18, 19, 29, 30, 53] is more limited because the number of APIs that can construct a model by being used together is a small subset of all APIs available in the DL libraries.”
COSTELLO: Contrastive Testing for Embedding-Based Large Language Model as a Service Embeddings
Jiang, W., Zhai, J., Ma, S., Zhang, X., & Shen, C. Q. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643767 · Cited by 5
CEDAR: Continuous Testing of Deep Learning Libraries
Xie, D., Wang, J., Pham, H. V., Tan, L., Guo, Y., Aziz, A., & Meijer, E. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00044 · Cited by 5
Automatically debugging AutoML pipelines using maro: ML automated remediation oracle
Dolby, J., Tsay, J., & Hirzel, M. J. (2022) · doi:10.1145/3520312.3534868 · Cited by 5
“AI debugging tools such as LEMON [31] and Haq et al.”
MethodLeveraging Documentation to Test Deep Learning Library FunctionsPreprint
Xie, D., Li, Y., Kim, M., Pham, H. V., Tan, L., Zhang, X., & Godfrey, M. W. (2021) · arXiv (Cornell University) · Cited by 5
Yang, S., Fang, C., Lin, H., Chen, X., Liu, J., & Chen, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.17642 · Cited by 5
“Notably, such implementation significantly reduces the manual burden compared to traditional fuzzing techniques [35,58].”
InfluentialCross-Stack Validation of Language-Model Training: A Clinical Fine-Tuning Case StudyPreprint
Tran, T., & Dang, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.24267 · Cited by 5
“LEMON [18] and Audee [3] generate or mutate models to drive the comparison into rarely exercised code.”
Guo, H., Tao, C., Huang, Z., & Zou, W. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3736305 · Cited by 4
“Besides, recent studies [69, 71] also focus on debugging DL libraries.”
KLRAG: Deep Learning Library Vulnerability Detection via Knowledge-Level RAG
Liao, R., Yan, X., & Zhu, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9958-2_36 · Cited by 4
False-Positive Bug Reports in Deep Learning Compilers: Stages, Root Causes, and Mitigation
Huang, L., Shen, Q., Wang, D., Wu, Y., Wang, M., & Chen, J. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3774889 · Cited by 4
Self-citationNaziri, M. M. A., Singh, A. K., Wu, B., Qin, F., Dutta, S., & d’Amorim, M. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731739 · Cited by 4
“An impressive number of techniques have been recently proposed to find bugs in these libraries [1–3, 6, 7, 10, 11, 16, 17, 21, 22, 24], however we observe these techniques propose an independent evaluation methodology.”
Investigating the bugs in reinforcement learning programs: Insights from Stack Overflow and GitHub
Song, J., Li, Y., Tian, Y., Ma, H., Li, H., Zuo, J., Liu, J., & Niu, W. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00555-z · Cited by 4
“In addition, there is extensive research on testing deep learning models (Chen et al. 2020; Wang et al. 2021), testing deep learning programs (Yan et al. 2021; Zhang et al. 2020), and testing deep learning frameworks (Pham et al. 2019; Wang et al. 2020).”
MoCo: Fuzzing Deep Learning Libraries via Assembling Code
Ji, P., Feng, Y., Wu, D., Yan, L., Chen, P., Liu, J., & Zhao, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3509975 · Cited by 4
“Both LEMON [4] and Muffin [5] represent model-level methods that detect bugs by comparing the outputs or the computational processes of generated DL models.”
InfluentialExploring Effects of Computational Parameter Changes to Image Recognition SystemsPreprint
Louloudakis, N., Gibson, P., Cano, J., & Rajan, A. (2022) · Enlighten: Publications (University of Glasgow) · doi:10.48550/arxiv.2211.00471 · Cited by 4
Documentation-Guided Fuzzing for Testing Deep Learning API Functions
Li, Y. (2020) · UWSpace (University of Waterloo) · Cited by 4
TTrace: Lightweight Error Checking and Diagnosis for Distributed TrainingPreprint
Jiang, H., Zhu, S., Zhang, Z., Song, Z., Fu, X., Jia, Z., Wang, Y., & Li, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.09280 · Cited by 4
“TTrace detected 14 silent bugs, including 3 new ones, in the widely used Megatron-LM framework.”
Less Is More: Failing Test Generation with Large Language Models
Li, T.-O., Yan, X., Zong, W., Wang, Y., Tian, H., Xu, C., Wang, Y., Cheung, S.-C., & Kramer, J. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3793675 · Cited by 3
“It is because to avoid subjectivity bias, existing works on test generation [54, 97, 102, 109] often request project developers to confirm that the generated tests truly reproduce the faults residing in the project.”
Survey on Fuzzing Techniques in Deep Learning Libraries
Ji, J., Kong, W., Tian, J., Gu, T., Nie, Y., & Kuang, X. (2023) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc59305.2023.00073 · Cited by 3
“3) In 2020, Wang et al [50] proposed LEMON, to testing DL libraries via guided mutation.”
BackgroundPython Fuzzing for Trustworthy Machine Learning Frameworks
Yegorov, I., Kobrin, E., Parygina, D., Vishnyakov, A., & Fedotov, A. (2024) · Journal of Mathematical Sciences · doi:10.1007/s10958-024-07424-2 · Cited by 3
Directed Testing in MLIR: Unleashing Its Potential by Overcoming the Limitations of Random Fuzzing
Tong, W., Wang, Z., Tang, Z., Fang, J., Zhang, Y., & Ye, G. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729372 · Cited by 2
Self-citationChecker Bug Detection and Repair in Deep Learning LibrariesPreprint
Harzevili, N. S., Mohajer, M. M., Shin, J., Wei, M., Uddin, G., Yang, J., Wang, J., Wang, S., Ming, Z., Jiang, et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.06440 · Cited by 3
“2: Taxonomy of DL checker bugs. usage in the literature [23]–[26] and their widespread adoption in various application domains across academia and industry [27]–[36].”
BackgroundMultigranularity Coverage Criteria for Deep Learning Libraries
Shi, Y., Zheng, Z., Yin, B., & Xi, Z. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1903 · Cited by 2
“Recent approaches to DL library testing focus on model-level testing (e.g., CRADLE [11], LEMON [12], Audee [13] and Muffin) and API-level testing (e.g., Predoo [14], Duo [15], FreeFuzz [9], D2C [16] and Eagle [17]).”
DKFuzz: An Automated Approach for Testing the Underlying Kernels of Deep Learning Frameworks
Li, H., Li, X., Nie, Y., Du, S., & Wu, Q. (2025) · International Conference on Algorithms, Software Engineering and Network Security (ASENS) · doi:10.1109/asens64990.2025.11011223 · Cited by 2
WASIT: Deep and Continuous Differential Testing of WebAssembly System Interface Implementations
Hu, Y., Wen, Z., Xiao, B., Kong, Q., Yi, B., Ji, S., Wang, S., & Wang, W. (2025) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3731569.3764819 · Cited by 2
“Prior research work has proposed many API testing techniques [6, 17, 18, 55, 71, 76, 82].”
Scalpel: Automotive Deep Learning Framework Testing via Assembling Model Components
Zou, Y., Zhai, J., Fang, C., Guo, A., Liu, J., & Chen, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773242 · Cited by 2
Automating fuzz driver generation for deep learning libraries with large language models
Zheng, T., Meng, F., Yi, P., & Wu, Y. (2026) · Cybersecurity · doi:10.1186/s42400-025-00532-9 · Cited by 2
“Bug detection methods for DL libraries are usually categorized into two paradigms: model-level detection (e.g., CRADLE (Pham et al. 2019), Audee (Guo et al. 2020), LEMON (Wang et al. 2020 However, Deng et al. (2023) indicated that these methods failed to generate arbitrary tests with diverse API sequences and usage pa…”
InfluentialDeep Learning Framework Testing via Model Mutation: How Far Are We?
Mu, Y., Wang, R., Zhai, J., Fang, C., Chen, X., Yang, P., Cao, Z., Qian, R., Yang, S., & Chen, Z. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3651022 · Cited by 2
“Prior work mainly targets three defect types [9], [10], [19], [25]: (1) Crash defects, where exceptions or crashes occur in some frameworks but not others; (2) NaN defects, where outputs contain values such as “NaN” or “INF” inconsistently across frameworks; (3) Inconsistency defects, where output…”
InfluentialSubgraph-Oriented Testing for Deep Learning Libraries
Xie, X., Song, Y., Chen, S., & Chen, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3655712 · Cited by 2
“• LEMON [13] is a model mutation-based testing approach.”
Self-citationChu, R., Fu, X., Luo, B., Shi, J., & Zhou, X. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-50195-5 · Cited by 2
GPU-Fuzz: Finding Memory Errors in Deep Learning Frameworks
Li, Z., Lu, H., Guo, Y., Zhang, Z., Wang, S., & Zhang, F. (2026) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3770743.3804266 · Cited by 2
Wang, Z., Ding, D., Zhu, H., Lin, H., & Panda, A. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790178 · Cited by 2
“The use of iterative expression rewriting allows Entangle to scale to today’s models. was funded in part by the National Science Foundation under award CNS-2145471, and gifts from Google and the Stellar Foundation.”
Self-citationHow Do Large Language Models Perform in Deep Learning Code Generation? An Empirical Study
Ma, X., Du, X., Li, C., Meng, J., Fang, X., Ding, W., & Zheng, Z. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3816024 · Cited by 2
“As summarized in Table 2, we introduced mutations at both the layer and structure levels to modify the extracted substructures, inspired by prior work on DL library testing [50, 80], DL mutation [54, 79], and architecture transformation [24, 60].”
DyFuzz: Skeleton-based Fuzzing for Python Libraries
Xia, X., & Feng, Y. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00040 · Cited by 2
Python Coverage Guided Fuzzing for Deep Learning Framework
Nie, Y., Xiao, X., Yang, B., Li, H., Luo, L., Yu, H., & Sun, G. (2024) · International Conference on Electronic Engineering and Information Systems (EEISS) · doi:10.1109/eeiss62553.2024.00007 · Cited by 2
Evaluating Spectrum-Based Fault Localization on Deep Learning Libraries
Yan, M., Chen, J., Jiang, T., Jiang, J., & Wang, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3552622 · Cited by 2
“Existing DL library testing techniques can be categorized into two main streams, i.e., model-level testing approaches [1], [2], [57], [58], and API-level testing approaches [4], [6], [34], [35].”
Self-citationCitadel : Context Similarity Based Deep Learning Framework Bug Finding
Zhang, X., Zhai, J., Ma, S., Wang, S., & Shen, C. Q. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773993 · Cited by 2
“Therearetwo primary approaches to testing DL frameworks: model-level testing [39, 40, 66, 79] and API-level testing [33, 80].”
Exploring Robustness of Image Recognition Models on Hardware Accelerators
Louloudakis, N., Gibson, P., Cano, J., & Rajan, A. (2025) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw64639.2025.10962531 · Cited by 2
“LEMON [36] and CRADLE [30] explore mutation testing and fault localization from models sourced from different Deep Learning (DL) frameworks.”
Benchmarking Deep Learning FuzzersPreprint
Harzevili, N. S., Pham, H. V., & Wang, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.06912 · Cited by 2
“In this study, we chose TensorFlow [1] and PyTorch [39] which are both very popular DL frameworks that have been widely used in the literature [8, 30, 41, 54] for two main reasons.”
MethodEvaluating the Effectiveness of Coverage-Guided Fuzzing for Testing Deep Learning Library APIsPreprint
Qin, F., Naziri, M. M. A., Ai, H., Dutta, S., & d’Amorim, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.14626 · Cited by 2
Quality-Driven Selective Mutation for Deep LearningPreprint
Ahmed, Z., Dapaah, E. C., Makedonski, P., & Grabowski, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.22640 · Cited by 2
“…techniques [2, 24], generation and detection of adversarial examples [33, 38], test data generation [9, 34], differential testing of DL libraries [40], structural analysis of deep neural networks [12], oracle generation for autonomous vehicles [17], fault localization in deep neural networks [7,…”
A Survey of Security Testing Techniques for Deep Learning Frameworks
Li, H., Li, X., Nie, Y., & Tian, J. (2024) · doi:10.1109/icsip61881.2024.10671492 · Cited by 1
“Similar to AUDEE, LEMON [22] focuses on mutating existing DNN models by designing mutation strategies.”
MethodMagnifier: A Chiplet Feature-Aware Test Case Generation Method for Deep Learning Accelerators
Li, B., Zhu, Z., Liu, W., Cao, Q., Li, C., Ji, C., Li, X., & Zhou, X. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3528358 · Cited by 1
“These studies [5], [6], [7], [8] mainly employ fuzzing methods to mutate test cases, treating the system as a opaque system.”
RIMFuzz: real-time impact-aware mutation for library API fuzzing
Wang, X., & Zhao, L. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00050-1 · Cited by 1
“…(Pailoor et al. 2018; Sun et al. 2021; Bulekov et al. 2023), Rust libraries (Jiang et al. 2021; Takashima et al. 2021), Machine learning libraries (Wang et al. 2020; Wei et al. 2022; Xie et al. 2022; Deng et al. 2023), RESTful API (Atlidakis et al. 2019; Liu et al. 2022), and even arbitrary…”
XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries
Duan, B., Dong, R., Dong, N., Kim, D. S., & Yang, G. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00030 · Cited by 1
“Model-level fuzzing [17], [16], [34] focuses on generating or mutating entire DNN models and observing their behavior under various inputs.”
Deep Learning Framework Testing via Heuristic Guidance Based on Multiple Model Measurements
Zou, Y., Zhai, J., Fang, C., Mu, Y., Liu, J., & Chen, Z. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3830465 · Cited by 1
SpaceMutation : An LLM-assisted mutation testing framework for DNNs in distributed LEO satellites
Zeng, G., Luo, J., Qiao, Y., Teng, S. H., & Liu, A. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132431 · Cited by 1
How Effective Is Coverage-Guided Fuzzing to Test Deep Learning Library APIs?
Qin, F., Naziri, M. M. A., Ai, H., Dutta, S., & d’Amorim, M. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00023 · Cited by 1
“Model-based fuzzing [7], [8], [9], [10], [11], [12] and API-level fuzzing [5], [13], [14], [15], [16] are two complementary categories of bug finding techniques for DL library testing.”
Freeze-and-mutate: abnormal sample identification for DL applications through model core analysis
Wang, H., Chen, Z., & Xu, C. (2023) · Automated Software Engineering · doi:10.1007/s10515-022-00373-7 · Cited by 1
“Later, Wang et al. (2020) refined CRADLE for better performance by investigating and proposing ways to more effectively generate models for testing DL libraries via guided mutant generation.”
BackgroundMethodThe Curious Case of Randomness in Deep Learning Models for Heartbeat Classification
Gušev, M., Tudjarski, S., Stankovski, A., & Jovanov, M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-54321-0_5 · Cited by 1
A PSO-based Method to Test Deep Learning Library at API Level
Liao, S., & Shan, C. (2024) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3672758.3672777 · Cited by 1
“In contrast, Audee [ 19 ] and LEMON [ 20 ] offer a variety of mutation strategies to create seed models to improve CRADLE’s efficiency.”
Detecting Compatibility Issues for Machine Learning Libraries
Wang, Y., Cheung, S.-C., Yu, H., & Zhu, Z. (2025) · Springer eBooks · doi:10.1007/978-981-96-1797-5_4 · Cited by 1
Verify Distributed Deep Learning Model Implementation Refinement with Iterative Relation InferencePreprint
Wang, Z., Ding, D., Zhu, H., Lin, H., & Panda, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.09505 · Cited by 1
“Prior work has also proposed using fuzz testing [21, 22, 41, 42] to evaluate model equivalence.”
Self-citationTesting of machine learning algorithms and models
Sharma, A. (2023) · Carl von Ossiezky University of Oldenburg · Cited by 1
GPU Temperature Simulation-Based Testing for In-Vehicle Deep Learning FrameworksPreprint
Zou, Y., Zhai, J., Fang, C., & Chen, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.15815 · Cited by 1
“This method addresses a critical limitation in existing DL framework testing methods [56], which fail to consider the impact of temperature on DL framework quality.”
InfluentialFinding Compiler-Platform Interaction Bugs in Deep Learning Pipelines via Cross-Layer ConstraintsPreprint
Qiu, Y. L., Wang, J., Badhe, R., Limpanukorn, B., Kim, M., & Zhang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.18421 · Cited by 1
“The second category adopts transformation-based techniques, where new test inputs are derived by mutating or re-constructing existing programs or models [Pham et al.(2019), Guo et al.(2021), Wang et al.(2020), Gu et al.(2022), Luo et al.(2021), Zhou et al.(2024), Ma et al.(2023), Mu et al.(2025)].”
Demystifying Deep Learning Compiler Frontend Bugs: An LLM-Aided Empirical StudyPreprint
Yuan, X., Chen, W., Liu, J., Chen, P., Wei, J., Wu, G., Zhu, J., & Huang, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.25651 · Cited by 1
“…predominantly leverage fuzzing, ranging from API-and operator-level tests [25, 53, 63, 65], which target API behaviors and individual operator implementations, to model-level tests [27, 34, 37, 46, 61, 66], which synthesize diverse computation graphs to expose inconsistencies across frameworks.”
Crafting unusual programs for fuzzing deep learning libraries
Yang, S. (2023) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 1
Grounding Deep Learning Framework Testing in Data FlowPreprint
Ge, J., Mu, Y., Zou, Y., Hu, Y., Chen, Z., & Fang, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.15808 · Cited by 1
“Building on model-level differential testing, LEMON mutates model structures and weights and uses a heuristic strategy to favor mutants that amplify cross-library output inconsistencies [50].”
InfluentialPlanning and Designing The Development of Software Library for Deep LearningPreprint
Rachmawati, O. C. R., Barakbah, A. R., & Karlita, T. (2023) · Research Square · doi:10.21203/rs.3.rs-3148716/v1
Source File Tracking Localization: A Fault Localization Method for Deep Learning Frameworks
Ma, Z., Yang, B., & Zhang, Y. (2023) · Electronics · doi:10.3390/electronics12224579
NeuronSig: A general adversarial attack detector based on neuron signature of deep neural network
Jia, C., Chen, J., Li, X., Zheng, H., & Zhang, L. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.116054
Self-citationLightweight Code Coverage Analysis for Deep Learning Framework TestingPreprint
Li, S., Xiao, X., Yang, B., Li, J., Yu, H., Luo, L., Zhang, X., & Sun, G. (2024) · Research Square · doi:10.21203/rs.3.rs-4845487/v1
CGFuzz: A Dynamic Test Case Generation Method for DL Framework Based on Function Coverage
Qing, C., Yin, B., & Shi, J.-A. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00070
A Comprehensive Review on Deep Learning System Testing
Li, Y., Shan, C., Liu, Z., & Liao, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1548-3_12
Wu, Z., Wang, Z., Luo, C. W., Du, X., & Chen, J. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728948
“Besides the relevant problem of testing, search-based methods (including the ideas of adaptive sampling and local search employed in FairAS) have been widely used in many other SE tasks [9, 18, 29, 43, 48].”
Self-citationA Method for Detecting Numerical Defects in Deep Learning Programs Based on Abstract Interpretation
Xu, R., Dong, Y., Li, Y., Zeng, Q., & Dong, W. (2025) · International Conference on Computer Science, Engineering, and Education (CSEE) · doi:10.1109/csee64583.2025.00037
“These works aim either to test deep learning models by proposing various input generation techniques [23]–[25] or to test deep learning libraries and compilers [26]–[28].”
Test Script Repair of Deep Learning Library Testing
Fu, X. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728616
Testing deep neural networks across different computational configurations
Louloudakis, N. (2025) · ERA · doi:10.7488/era/6207
Improving the Reliability of AI Infrastructure Software with Data-Driven Software Analytics
Shiri Harzevili, N. (2025) · York University Digital Library (York University)
MGtest: Python Coverage-Guided Model-Level Fuzzing for DL Frameworks
Ji, J., Kong, W., & Liu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-3551-4_19
Enhancing Test Case Generation for Fuzzing Deep Learning Libraries Using Few-Shot Learning
Yang, B., Li, S., Li, J., Du, K., Yu, H., Sun, J. X., & Luo, L. (2025) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc67331.2025.00089
“Traditional fuzzers, including LEMON [12], MMOS [13], and Muffin [14], are among the early fuzzers, which typically generate test cases by applying mutations to the model.”
A Multi-Agent Fuzzing Framework for Deep Learning Library
Liao, R., Ou, S., Yan, X., & Zhu, K.-L. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343067
“DL libraries fuzzing is mainly done by generating many seeds, which can be classified into two categories depending on the seed type: model-level fuzzing[6], [14] and API-level fuzzing[15], [5].”
Testing Deep Learning Libraries with Semantic Equivalent API Patterns
Song, Y., Chen, S., Peng, H., & Xie, X. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00031
“Cross-hardware testing compares behavior of the same API on different devices (e.g., CPU vs. GPU) [21], while cross-library testing checks whether functionally equivalent APIs in different libraries produce equivalent outputs [12], [14], [23].”
BCI-Fuzz: Bug-triggering code innovated to fuzz deep learning libraries
Zhao, Z., Pan, L., Shao, S., & Luo, S. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112822
CIPIHunter: Detecting Configuration-Induced Prediction Instability in Deep Learning Frameworks
Anonymous, Meng, S., Kim, M., Chen, X., Fang, C., Chen, Z., & Zhai, J. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834345
MutDBD: Mutation-based training set diagnosis for backdoor defense in deep neural networks
Ma, M., Li, Y., Wang, J., Chen, L., & Zhou, Y. (2026) · Science of Computer Programming · doi:10.1016/j.scico.2026.103476
Prioritizing Test Cases Using Box Abstraction in Deep Neural Networks
Al-Qadasi, H. (2024) · theses.fr (ABES) · doi:10.70675/938590f5zd67az49c0zb793zd4c92c9a2890
Effective Python-frontend fuzzing for deep learning libraries with runtime coverage feedback
Chen, X., Cui, N., Chen, L., Shi, G., You, R., Xu, Q., & Yin, R. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108195
Ekka: Automated Diagnosis of Silent Errors in LLM InferencePreprint
Gu, Y., Zhang, Z., Zhu, S., Fu, X., (4002), J. W., Wang, Y., & Kasikci, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.04594
“Prior work generates diverse inputs to stress-test DL libraries (Wang et al., 2020; 2022) or performs differential testing across frameworks (Deng et al., 2023; Wang et al., 2025).”
Accelerating Deep Learning Compiler Testing via Message-Passing Neural Network With Attention
Ding, Y., Zhou, Z., Li, X., Zou, P., & Jiang, H. (2026) · IEEE Transactions on Reliability · doi:10.1109/tr.2026.3702727
Metadata
- Authors
- Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D.
- Affiliations
- Tianjin University (CN)
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 788-799
- OpenAlex
- W3104663419
- Cited by
- 167
- References
- 61
- Citations by year
- 2026: 26 · 2025: 38 · 2024: 28 · 2023: 36 · 2022: 25 · 2021: 12 · 2020: 2
- FWCI
- 12.44
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- Funders
- National Natural Science Foundation of China (61872263,U1836214,61802275)
- Intelligent Manufacturing Special Fund of Tianjin (20191012,20193155)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- mutation testing, TensorFlow, Theano, CNTK, MXNet, lemon
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.