Deep learning library testing via effective model generation
The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Deep learning (DL) techniques are rapidly developed and have been widely adopted in practice. However, similar to traditional software systems, DL systems also contain bugs, which could cause serious impacts especially in safety-critical domains. Recently, many research approaches have focused on testing DL models, while little attention has been paid for testing DL libraries, which is the basis of building DL models and directly affects the behavior of DL systems. In this work, we propose a novel approach, LEMON, to testing DL libraries. In particular, we (1) design a series of mutation rules for DL models, with the purpose of exploring different invoking sequences of library code and hard-to-trigger behaviors; and (2) propose a heuristic strategy to guide the model generation process towards the direction of amplifying the inconsistent degrees of the inconsistencies between different DL libraries caused by bugs, so as to mitigate the impact of potential noise introduced by uncertain factors in DL libraries. We conducted an empirical study to evaluate the effectiveness of LEMON with 20 release versions of 4 widely-used DL libraries, i.e., TensorFlow, Theano, CNTK, MXNet. The results demonstrate that LEMON detected 24 new bugs in the latest release versions of these libraries, where 7 bugs have been confirmed and one bug has been fixed by developers. Besides, the results confirm that the heuristic strategy for model generation indeed effectively guides LEMON in amplifying the inconsistent degrees for bugs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, model-level fuzzers can only cover a limited set of APIs with limited patterns [72], missing many diverse and interesting API sequences that can lead to bugs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [71]
Citations
Cites
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundAdversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · cited 1×
“Also, they are widely adopted in various domains in practice, such as autonomous driving cars [12], face recognition [59], speech recognition [29], aircraft collision avoidance systems [36], and software engineering [15, 16, 18, 21, 42, 70].”
BackgroundWhat every computer scientist should know about floating-point arithmetic
Goldberg, D. (1991) · ACM Computing Surveys · doi:10.1145/103162.103163 · Cited by 1,750 · cited 1×
“However, due to many uncertain factors in DL libraries, such as randomness and floating-point computing deviation [28], it is challenging to determine whether a found problem is a real bug in the library, or a problem caused by some uncertain factors.”
BackgroundAn Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110
Roulette-wheel selection via stochastic acceptance
Lipowski, A., & Lipowska, D. (2012) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2011.12.004 · Cited by 761
Research works printed but not linked: 11
- Chen Junjie, 2019, Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 223-234
- Chen Junjie, 2020, Characterizing and Prioritizing Incidents for LargeScale Online Service Systems. In The 35th IEEE/ACM International Conference on Automated Software Engineering.
- Guo Qianyu, 2019, Proceedings of the 34th International Conference on Automated Software Engineering.
- Fitash Ul Haq Donghwan Shin Shiva Nejati and Lionel Briand. 2019. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. arXiv preprint arXiv:1912. 00805 ( 2019 ). Fitash Ul Haq Donghwan Shin Shiva Nejati and Lionel Briand. 2019. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. arXiv preprint arXiv:1912. 00805 ( 2019 ).
- Islam Md Johirul, 1906, A Comprehensive Study on Deep Learning Bug Characteristics. arXiv preprint arXiv
- 10.5555/3155562.3155636 · doi:10.5555/3155562.3155636
- Kass Robert E., 1998, Neal
- McKeeman William M, 1998, Diferential testing for software, Digital Technical Journal, 10
- Odena Augustus, 2018, Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv
- Yi Sun Yuheng Chen Xiaogang Wang and Xiaoou Tang. 2014. Deep learning face representation by joint identification-verification. In Advances in neural information processing systems. 1988-1996. Yi Sun Yuheng Chen Xiaogang Wang and Xiaoou Tang. 2014. Deep learning face representation by joint identification-verification. In Advances in neural information processing systems. 1988-1996.
- Zhang Jie M, 1906, Machine Learning Testing: Survey, Landscapes and Horizons. arXiv preprint arXiv
Cited by
Items 201–221 of 221
EMERGE: Checking Implementation Equivalence for LLMs via Rule Synthesis
Zhan, Q., Hu, X., Xia, X., & Li, S. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834405
He, Z., & Han, X. (2026) · International Symposium on Robotics, Artificial Intelligence and Information Engineering (RAIIE) · doi:10.1109/raiie70320.2026.11567007
“Considering the inherent non-determinism in certain operations, a tolerance threshold is applied to determine whether observed differences are statistically significant[29].”
Improving Deep Learning Library Testing with Machine Learning
Molina, F., Naziri, M. M. A., Qin, F., Gorla, A., & d’Amorim, M. (2026) · doi:10.1145/3793654.3793744
LLM-Assisted Input-Requirement-Aware Differential Testing of Array Programming Frameworks
Zhou, Z., & He, J. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808162
Automated Knowledge-Aware Test Reuse
Zhang, Z., 橘高, 義., Hu, X., Xia, X., & Li, S. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808146
DECODE: Dynamic Exploration for Constraint-Guided Vulnerability Discovery in Deep Learning Operators
Liu, H., Wang, Z., Liu, Z., & Li, W. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808174
Self-citationRise From The Ashes: LLM-based Static Analysis for Deep Learning Framework BugsPreprint
Yang, S.-Y., Lin, H.-F., Fang, C.-R., Chen, X., Cheng, W., Liu, J.-W., Zhang, Y.-Y., Liu, H., & Chen, Z.-Y. (2026) · arXiv.org · doi:10.48550/arxiv.2607.00555
“LEMON [45] and AUDEE [15] are subsequently proposed to mutate these real-world models to detect more bugs.”
Wang, H., Yu, R., Pang, Y., Zhao, Y., Chen, J., Jiang, J., & Wang, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787792
“Traditionally used to assess test suite quality, mutation analysis has also been applied to detect bugs in compilers and other systems[29–31].”
Self-citationD3Fuzz: Dynamic, directed and decaying LLM-driven fuzzing for deep learning libraries
Sun, J., Dong, J., Zhang, H., Tian, Z., & Zhang, Y. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108289
IcFuzz: Fuzzing Isaac Sim with Semantic Stage Guidance and Multi-level Mutation
Chen, Z., Chen, Z., Jia, R., Liao, Z., Li, W., Liu, J., & Zheng, Z. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837550
“5 illustrates various classes under the Materials category, which can be instantiated as concrete objects and applied to prims to endow them with different physical properties.”
Harnessing LLMs for Document-Guided Fuzzing of Python LibrariesPreprint
Duan, B., Mahmud, T., Che, M., Yan, Y., Dong, N., Kim, D. S., & Yang, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.11744
Testing Deep Learning Library APIs via Cross-Framework Differential FuzzingPreprint
Duan, B., Dong, R., Dong, N., Kim, D. S., & Yang, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.11886
“Model-level fuzzing [9], [10], [11] generates or mutates complete neural networks and observes their behavior under different execution settings.”
Automated Testing for Data-Intensive Scalable Computing
Humayun, A. (2026) · VTechWorks (Virginia Tech)
He, Y., Yang, F., Su, T., Zhang, Y., & Pu, G. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00470
“: Code coverage is the primary metric for assessing testing effectiveness, as commonly adopted in prior TVM testing studies [10], [12], [27].”
Liu, S., Sun, C., Wu, F., & Chan, W. K. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832194
Self-citationUnderstanding and simplifying bug-inducing inputs for deep-learning compilers
Liu, J. (2024) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Randomness-aware testing of machine learning-based systems
Dutta, S. (2023) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Li, Z., Zhang, W., Li, L., Easwaran, A., & Chen, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.36499
“Library differential testing includes CRADLE [45], LEMON [46], Audee [47], and EAGLE [48].”
Demystifying Checker Bugs in Deep Learning Libraries
Harzevili, N. S., Shin, J., Uddin, G., Yang, J., Wang, J., Wang, S., Ming, J. Z., & Nagappan, N. (2026) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme67971.2026.00025
EMBERS: Empirically Motivated Robustness Testing of Deep Research Systems via Defect-Guided FuzzingPreprint
Li, S., Wang, Z., Wang, D., Jiang, Y., Cui, H., Zhang, C., Wang, G., & Chen, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.03723
Self-citationTensorMutation: Learning Weight-Space Mutations of Deep Learning Models on Neural GraphsPreprint
Ahmed, Z., Makedonski, P., & Grabowski, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.09650
Metadata
- Authors
- Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D.
- Affiliations
- Tianjin University (CN)
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 788-799
- OpenAlex
- W3104663419
- Cited by
- 167
- References
- 61
- Citations by year
- 2026: 26 · 2025: 38 · 2024: 28 · 2023: 36 · 2022: 25 · 2021: 12 · 2020: 2
- FWCI
- 12.44
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- Funders
- National Natural Science Foundation of China (61872263,U1836214,61802275)
- Intelligent Manufacturing Special Fund of Tianjin (20191012,20193155)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- mutation testing, TensorFlow, Theano, CNTK, MXNet, lemon
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.