Muffin: testing deep learning libraries via neural architecture fuzzing
The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Deep learning (DL) techniques are proven effective in many challenging tasks, and become widely-adopted in practice. However, previous work has shown that DL libraries, the basis of building and executing DL models, contain bugs and can cause severe consequences. Unfortunately, existing testing approaches still cannot comprehensively exercise DL libraries. They utilize existing trained models and only detect bugs in model inference phase. In this work we propose Muffin to address these issues. To this end, Muffin applies a specifically-designed model fuzzing approach, which allows it to generate diverse DL models to explore the target library, instead of relying only on existing trained models. Muffin makes differential testing feasible in the model training phase by tailoring a set of metrics to measure the inconsistencies between different DL libraries. In this way, Muffin can best exercise the library code to detect more bugs. To evaluate the effectiveness of Muffin, we conduct experiments on three widely-used DL libraries. The results demonstrate that Muffin can detect 39 new bugs in the latest release versions of popular DL libraries, including Tensorflow, CNTK, and Theano.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, model-level fuzzers can only cover a limited set of APIs with limited patterns [72], missing many diverse and interesting API sequences that can lead to bugs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [25]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“Deep learning (DL) techniques have been proven effective in many specific tasks, such as image recognition [29], video understanding [53] and machine translation [49].”
BackgroundDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“[16] propose a set of similarity metrics and coverage criteria to analyze stateful DL systems such as Recurrent Neural Networks (RNNs) [39].”
BackgroundEmpirical Evaluation of Rectified Activations in Convolutional Network
Xu, B., Wang, N., Chen, T., & Li, M. (2015) · arXiv · doi:10.48550/arxiv.1505.00853 · Cited by 2,264 · cited 1×
“Furthermore, many APIs in DL libraries have specific usage scenarios, e.g. , Convolution Layer for image processing tasks, Recurrent Layers for text processing and various activation functions ( e.g. , ReLU [41] and leaky-ReLU [57]).”
BackgroundNeural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
What every computer scientist should know about floating-point arithmetic
Goldberg, D. (1991) · ACM Computing Surveys · doi:10.1145/103162.103163 · Cited by 1,750 · cited 1×
“However, due to normal uncertain factors such as floating-point deviation [22], we cannot determine whether a value difference is caused by potential bugs or normal factors.”
BackgroundEvolutionary computation: Toward a new philosophy of machine intelligence
B�ck, T. (1997) · Complexity · doi:10.1002/(sici)1099-0526(199703/04)2:4<28::aid-cplx7>3.0.co;2-2 · Cited by 1,533
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217 · cited 4×
“[43] first propose neuron coverage as the criteria for testing DL models.”
BackgroundMethodInfluentialDeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 4×
“[48] implement DeepTest for detecting erroneous behaviors of DL-based self-driving cars via automatically generating test cases based on image transformations.”
BackgroundMethodInfluentialMachine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Research works printed but not linked: 9
- Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In Proc. of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI. USENIX Association, 265--283.
- 10.1002/(sici)1099-0526(199703/04)2:4 · doi:10.1002/(sici)1099-0526(199703/04)2:4
- Ian J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org/
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems. 2672--2680.
- Vinod Nair and Geoffrey E. Hinton. 2010. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proc. of the 27th International Conference on Machine Learning, ICML. Omnipress, 807--814.
- Augustus Odena, Catherine Olsson, David G. Andersen, and Ian J. Goodfellow. 2019. TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In Proc. of the 36th International Conference on Machine Learning, ICML (Proceedings of Machine Learning Research, Vol. 97). PMLR, 4901--4911.
- Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. of the 3rd International Conference on Learning Representations, ICLR, Yoshua Bengio and Yann LeCun (Eds.).
- The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, et al. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 (2016). · arXiv
- Martin Wistuba, Ambrish Rawat, and Tejaswini Pedapati. 2019. A Survey on Neural Architecture Search. CoRR abs/1905.01392 (2019). arXiv:1905.01392 http://arxiv.org/abs/1905.01392 · arXiv
Cited by
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
“While such model-level fuzzers are able to nd bugs in DL libraries, due to the input/output constraints of DL APIs, model-level mutation/generation rules either are restrictive to certain shape-preserving APIs [71] or require manual annotation of the restrictions of all targeted APIs [25], leading to a limited number…”
BackgroundMethodInfluentialDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623343 · Cited by 115
Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
Zhang, Z., Chen, C., Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., & Wang, R. (2024) · Transactions on machine learning research · doi:10.13140/rg.2.2.32710.69440/1 · Cited by 195
WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Yang, C., Deng, Y., Lu, R., Yao, J., Liu, J., Jabbarvand, R., & Zhang, L. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689736 · Cited by 82
“Therefore, they can be effectively triggered by TitanFuzz since TitanFuzz leverages LLMs to generate human-like programs by resembling the distribution of training data.”
BackgroundNNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers
Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., & Zhang, L. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575707 · Cited by 167
Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTPreprint
Deng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02014 · Cited by 131
“Due to the intricate tensor/operator constraints, such model-level fuzzers either only focus on manipulating shapepreserving APIs [70] or require manually-written specifications for each API [24, 35] to preserve model validity.”
BackgroundMethodInfluentialFuzzing Automatic Differentiation in Deep-Learning Libraries
Yang, C., Deng, Y., Yao, J., Tu, Y., Li, H., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00105 · Cited by 77
“To mitigate this, the recent Muffin work [29] is proposed to detect bugs in both inference and training phases by generating DL models via a top-down approach.”
BackgroundMethodInfluentialInteroperability in Deep Learning: A User Survey and Failure Analysis of ONNX Model Converters
Jajal, P., Jiang, W., Tewari, A., Kocinare, E., Woo, J., Sarraf, A., Lu, Y.-H., Thiruvathukal, G. K., & Davis, J. C. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680374 · Cited by 65
NeuRI: Diversifying DNN Generation via Inductive Rule Inference
Liu, J., Peng, J., Wang, Y., & Zhang, L. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616337 · Cited by 47
“In end-to-end benchmarks, we compare NeuRI with the state-of-the-art model-level fuzzers (namely NNSmith [25] and Muffin [15]) and the state-of-the-art operator-level fuzzer ( i.e., Deep-REL [12]).”
BackgroundMethodInfluentialCOMET: Coverage-guided Model Generation For Deep Learning Library Testing
Li, M., Cao, J., Tian, Y., Li, T. O., Wen, M., & Cheung, S.-C. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3583566 · Cited by 45
“Prior works [18, 19, 31, 43, 54] mainly follow the same testing paradigm, i.”
BackgroundMethodInfluentialOn the Challenges of Fuzzing Techniques via Large Language Models
Huang, L., Zhao, P., Ma, L., & Chen, H. (2025) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse67621.2025.00028 · Cited by 43
ACETest: Automated Constraint Extraction for Testing Deep Learning Operators
Shi, J., Xiao, Y., Li, Y., Li, Y., Yu, D., Yu, C., Su, H., Chen, Y., & Huo, W. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598088 · Cited by 42
“To detect bugs in DL libraries, multiple techniques [14, 15, 26, 28, 43, 45, 47, 51] have been proposed to test them by generating or mutating DL models.”
MethodGeneration-based Differential Fuzzing for Deep Learning Libraries
Liu, J., Huang, Y., Wang, Z., Ma, L., Fang, C., Gu, M., Zhang, X., & Chen, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3628159 · Cited by 40
“Our study is based on the results from using existing approaches [8], i.e., CRADLE [48], LEMON [58], AUDEE [24], and MUFFIN[22] in the same time period (i.e., one hour, as mentioned in their documents).”
BackgroundMethodInfluentialThe Human Side of Fuzzing: Challenges Faced by Developers during Fuzzing Activities
Nourry, O., Kashiwa, Y., Lin, B., Bavota, G., Lanza, M., & Kamei, Y. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3611668 · Cited by 29
Deep Learning Library Testing: Definition, Methods and Challenges
Zhang, X., Jiang, W., Shen, C. Q., Li, Q. L., Wang, Q., Lin, C., & Guan, X. (2025) · ACM Computing Surveys · doi:10.1145/3716497 · Cited by 34
“To fill the research gap, Gu et al. [32] proposed Muffin, which creatively generated structure information and layer information of the DL model to thoroughly explore possible abnormal behaviors of the model.”
GenCoG: A DSL-Based Approach to Generating Computation Graphs for TVM Testing
Wang, Z., Nie, P., Miao, X., Chen, Y., Wan, C., Bu, L., & Zhao, J. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598105 · Cited by 29
“• Mu n [12] is a DL library testing technique via neural architecture fuzzing.”
MethodEnhancing Differential Testing with LLMs for Testing Deep Learning Libraries
Li, M., Li, D., Liu, J., Cao, J., Tian, Y., & Cheung, S.-C. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735637 · Cited by 28
“The first threat is that the LLM’s hallucination issue may affect the quality of our synthesized counterparts, thus introducing false positive in-consistencies during bug detection.”
InfluentialYour Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 26
Machine Learning-Based Fuzz Testing Techniques: A Survey
Zhang, A., Zhang, Y., Xu, Y., Wang, C., & Li, S. (2023) · IEEE Access · doi:10.1109/access.2023.3347652 · Cited by 23
“An increasing number of scholars have worked on extending fuzzing techniques to machine learning frameworks and conducted experiments to verify their feasibility [65], [66], [67].”
BackgroundZhou, C., Qian, B., Go, G., Zhang, Q., Li, S., & Jiang, Y. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689757 · Cited by 20
Li, Z., Wu, J., Ling, X., Luo, T., Rui, Z., & Wu, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00132 · Cited by 18
Taxonomy of Aging-related Bugs in Deep Learning Libraries
Liu, Z., Zheng, Y., Du, X., Zheng, H., Ding, W., Miao, Y., & Zheng, Z. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00048 · Cited by 19
“References [60]–[64] used various testing technologies to test deep learning libraries.”
MethodA Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators
Chen, J., Jia, C., Yan, Y., Ge, J., Zheng, H., & Cheng, Y. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660796 · Cited by 16
“Several studies have been conducted to uncover errors in DL libraries [13, 15, 22, 28, 35], such as CRADLE [28], which tackled the oracle challenge of differentially testing multiple DL models on various backends in Keras [6].”
A Tale of Two DL Cities: When Library Tests Meet Compiler
Shen, Q., Tian, Y., Ma, H., Chen, J., Huang, L., Fu, R., Cheung, S.-C., & Wang, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00025 · Cited by 18
A First Look at Bugs in LLM Inference Engines
Liu, M., Zhong, S., Bi, W., Zhang, Y., Chen, Z., Chen, Z., Liu, X., & Ma, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3788873 · Cited by 12
Deep learning framework testing via hierarchical and heuristic model generation
Zou, Y., Sun, H., Fang, C., Liu, J., & Zhang, Z. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111681 · Cited by 14
A Usability Evaluation of AFL and libFuzzer with CS Students
Plöger, S., Meier, M., & Smith, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581178 · Cited by 10
Go, G., Zhou, C., Zhang, Q., Zou, X., Shi, H., & Jiang, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680364 · Cited by 12
DevMuT: Testing Deep Learning Framework via Developer Expertise-Based Mutation
Mu, Y., Zhai, J., Fang, C., Chen, X., Cao, Z., Yang, P., Zou, Y., Zheng, T., & Chen, Z. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695523 · Cited by 15
“Some researchers [18, 19, 28, 29, 42, 56] adopt DL models as the test inputs and combine them with different mutation operators, such as modifying the model structure and editing layer parameters, to generate more diverse mutants and improve the test effectiveness.”
InfluentialImproving Deep Learning Framework Testing with Model-Level Metamorphic Testing
Mu, Y., Zhai, J., Fang, C., Chen, X., Cao, Z., Yang, P., Zhao, K., Guo, A., & Chen, Z. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728972 · Cited by 12
“For the loss and gradient, we follow the settings of Gu et al. [23] to detect the accuracy bugs in them.”
InfluentialD3: Differential Testing of Distributed Deep Learning With Model Generation
Wang, J., Pham, H. V., Li, Q., Tan, L., Guo, Y., Aziz, A., & Meijer, E. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3461657 · Cited by 15
“We compare D 3 with fifteen existing tools EAGLE [26], DocTor [47], FreeFuzz [29], ∇ Fuzz [48], FuzzGPT [49], TitanFuzz [50], DeepREL [30], CRADLE [25], Audee [27], LEMON [28], NeuRI [51], Muffin [46], Ramos [52], GenCoG [53], and HirGen [54].”
InfluentialTest2Vec: An Execution Trace Embedding for Test Case PrioritizationPreprint
Jabbar, E., Zangeneh, S., Hemmati, H., & FELDT, R. H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.15428 · Cited by 9
“Other alternatives such as fuzzing [32, 84] and input data or test case diversication [13, 23] are also limited in covering software behaviour thoroughly, since they look at the software under test more like a black-box and do not leverage the underlying source code, execution traces, or specication artifacts.”
TorchProbe: Fuzzing Dynamic Deep Learning Compilers
Su, Q., Geng, C., Pekhimenko, G. G., & Si, X. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8311-7_15 · Cited by 9
Braberman, V. A., Bonomo, F., Charalambous, Y., Colonna, J. G., Cordeiro, L. C., & de Freitas, R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5624902 · Cited by 8
A survey of coverage-guided greybox fuzzing with deep neural models
Qiu, J., Jiang, Y., Miao, Y., Luo, W., Pan, L., & Zheng, X. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107797 · Cited by 8
MirrorFuzz: Leveraging LLM and Shared Bugs for Deep Learning Framework APIs Fuzzing
Ou, S., Li, Y., Yu, L., Wei, C., Wen, T., Chen, Q., Chen, Y., Tang, H., & Pan, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3619966 · Cited by 9
“Model-level fuzzing [19], [20], [21], [22], [23] builds DL models as test cases and uses differential testing to identify defects, but it can only cover a limited set of APIs.”
Mutation-Based Deep Learning Framework Testing Method in JavaScript Environment
Zou, Y., Zhai, J., Fang, C., Liu, J., Zheng, T., & Chen, Z. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695478 · Cited by 7
Evaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study
Harzevili, N. S., Wei, M., Mohajer, M. M., Pham, H. V., & Wang, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3729533 · Cited by 6
“Gu et al. [24] introduced Muffin, a model fuzzing technique aimed at generating diverse DL models to thoroughly test target libraries.”
Effective random test generation for deep learning compilers
Ren, L., Wang, Z., Zhang, L., Jiang, G., Xiong, Y., & Xie, T. (2025) · Science China Information Sciences · doi:10.1007/s11432-023-4301-6 · Cited by 7
Lightweight Concolic Testing via Path-Condition Synthesis for Deep Learning Libraries
Kim, S., Kim, Y., Park, D., Jeon, Y., Yi, J., & Kim, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00202 · Cited by 7
Detecting Numerical Deviations in Deep Learning Models Introduced by the TVM Compiler
Xia, Z.-C., Chen, Y., Nie, P., & Wang, Z. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00018 · Cited by 6
A Systematic Survey on Debugging Techniques for Machine Learning SystemsPreprint
Nguyen, T.-D., Tian, H., Le, B., Thongtanunam, P., & McIntosh, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.03158 · Cited by 7
Mind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators
Wen, E., Ma, S., Tempero, E., Sham, C., Sun, Y., Jia, H., Luo, D., Hua, J., Dietrich, J. B., Zhao, K., et al. (2025) · International Conference on Mobility, Sensing and Networking (MSN) · doi:10.1109/msn69125.2025.00054 · Cited by 4
“Building on this direction, LEMON [1], Muffin [6], and EAGLE [15] introduce various model mutation strategies to expand the pool of test models.”
Naziri, M. M. A., Singh, A. K., Wu, B., Qin, F., Dutta, S., & d’Amorim, M. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731739 · Cited by 4
MoCo: Fuzzing Deep Learning Libraries via Assembling Code
Ji, P., Feng, Y., Wu, D., Yan, L., Chen, P., Liu, J., & Zhao, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3509975 · Cited by 4
“The code files generated by MoCo are related to the process of constructing, training and evaluating DL models, which are the most common real-world user scenarios of DL libraries.”
InfluentialValidity-Preserving Delta Debugging via Generator Trace Reduction
Ren, L., Zhang, X., Hua, Z., Jiang, Y., He, X., Xiong, Y., & Xie, T. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705305 · Cited by 6
“The implementation cost depends on the number of kinds of APIs that produce random choices (instead of the generator’s size or the number of random choices in the generator).”
Testing Deep Learning Libraries via Neurosymbolic Constraint Learning
Naziri, M. M. A., Kim, S., Qin, F., Dutta, S., & d’Amorim, M. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787800 · Cited by 5
“Model-level testing [18, 19, 29, 30, 53] is more limited because the number of APIs that can construct a model by being used together is a small subset of all APIs available in the DL libraries.”
LLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer
Zhang, K., Xiao, D., Wu, D., Wang, S., Zhao, J., Lin, Y., Xu, T., & wang, S. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798258 · Cited by 3
Harnessing LLMs for Document-Guided Fuzzing of OpenCV Library
Duan, B., Mahmud, T., Che, M., Yan, Y., Dong, N., Kim, D. S., & Yang, G. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00017 · Cited by 4
Multigranularity Coverage Criteria for Deep Learning Libraries
Shi, Y., Zheng, Z., Yin, B., & Xi, Z. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1903 · Cited by 2
DKFuzz: An Automated Approach for Testing the Underlying Kernels of Deep Learning Frameworks
Li, H., Li, X., Nie, Y., Du, S., & Wu, Q. (2025) · International Conference on Algorithms, Software Engineering and Network Security (ASENS) · doi:10.1109/asens64990.2025.11011223 · Cited by 2
Scalpel: Automotive Deep Learning Framework Testing via Assembling Model Components
Zou, Y., Zhai, J., Fang, C., Guo, A., Liu, J., & Chen, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773242 · Cited by 2
Automating fuzz driver generation for deep learning libraries with large language models
Zheng, T., Meng, F., Yi, P., & Wu, Y. (2026) · Cybersecurity · doi:10.1186/s42400-025-00532-9 · Cited by 2
“In addition, we also compare our approach with other SOTA methods, i.e., Muffin (Gu et al. 2022), FreeFuzz (Wei et al. 2022), DeepREL (Deng et al. 2022), and ∇ Fuzz (Yang et al. 2023), that specifically designed for fuzz-ing DL library APIs to further evaluate the effectiveness of our approach in…”
Subgraph-Oriented Testing for Deep Learning Libraries
Xie, X., Song, Y., Chen, S., & Chen, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3655712 · Cited by 2
“While some approaches [18], [23], [24], [25], [26] consider combining APIs artificially, such arbitrary combinations may not match the popular API interaction patterns in the real-life usage.”
Survey on Fuzzing Techniques in Deep Learning Libraries
Ji, J., Kong, W., Tian, J., Gu, T., Nie, Y., & Kuang, X. (2023) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc59305.2023.00073 · Cited by 3
Directed Testing in MLIR: Unleashing Its Potential by Overcoming the Limitations of Random Fuzzing
Tong, W., Wang, Z., Tang, Z., Fang, J., Zhang, Y., & Ye, G. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729372 · Cited by 2
Citadel : Context Similarity Based Deep Learning Framework Bug Finding
Zhang, X., Zhai, J., Ma, S., Wang, S., & Shen, C. Q. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773993 · Cited by 2
Evaluating the Effectiveness of Coverage-Guided Fuzzing for Testing Deep Learning Library APIsPreprint
Qin, F., Naziri, M. M. A., Ai, H., Dutta, S., & d’Amorim, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.14626 · Cited by 2
A Survey of Security Testing Techniques for Deep Learning Frameworks
Li, H., Li, X., Nie, Y., & Tian, J. (2024) · doi:10.1109/icsip61881.2024.10671492 · Cited by 1
“Gu et al. [23] [24], CNTK [25], and Theano [26].”
BackgroundQuan, L., Xie, X., Guo, Q., Jiang, L., Chen, S., Wang, J., & Li, X. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714649 · Cited by 1
Deep Learning Framework Testing via Model Mutation: How Far Are We?
Mu, Y., Wang, R., Zhai, J., Fang, C., Chen, X., Yang, P., Cao, Z., Qian, R., Yang, S., & Chen, Z. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3651022 · Cited by 2
“Prior work mainly targets three defect types [9], [10], [19], [25]: (1) Crash defects, where exceptions or crashes occur in some frameworks but not others; (2) NaN defects, where outputs contain values such as “NaN” or “INF” inconsistently across frameworks; (3) Inconsistency defects, where output…”
How Effective Is Coverage-Guided Fuzzing to Test Deep Learning Library APIs?
Qin, F., Naziri, M. M. A., Ai, H., Dutta, S., & d’Amorim, M. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00023 · Cited by 1
Evaluating Spectrum-Based Fault Localization on Deep Learning Libraries
Yan, M., Chen, J., Jiang, T., Jiang, J., & Wang, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3552622 · Cited by 2
“Existing DL library testing techniques can be categorized into two main streams, i.e., model-level testing approaches [1], [2], [57], [58], and API-level testing approaches [4], [6], [34], [35].”
Finding Compiler-Platform Interaction Bugs in Deep Learning Pipelines via Cross-Layer ConstraintsPreprint
Qiu, Y. L., Wang, J., Badhe, R., Limpanukorn, B., Kim, M., & Zhang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.18421 · Cited by 1
“The second category adopts transformation-based techniques, where new test inputs are derived by mutating or re-constructing existing programs or models [Pham et al.(2019), Guo et al.(2021), Wang et al.(2020), Gu et al.(2022), Luo et al.(2021), Zhou et al.(2024), Ma et al.(2023), Mu et al.(2025)].”
Grounding Deep Learning Framework Testing in Data FlowPreprint
Ge, J., Mu, Y., Zou, Y., Hu, Y., Chen, Z., & Fang, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.15808 · Cited by 1
“Existing methods generate new test cases by applying mutation operators to the structures of seed models [12, 13, 20, 36, 50].”
InfluentialSource File Tracking Localization: A Fault Localization Method for Deep Learning Frameworks
Ma, Z., Yang, B., & Zhang, Y. (2023) · Electronics · doi:10.3390/electronics12224579
Python Coverage Guided Fuzzing for Deep Learning Framework
Nie, Y., Xiao, X., Yang, B., Li, H., Luo, L., Yu, H., & Sun, G. (2024) · International Conference on Electronic Engineering and Information Systems (EEISS) · doi:10.1109/eeiss62553.2024.00007 · Cited by 2
Lightweight Code Coverage Analysis for Deep Learning Framework TestingPreprint
Li, S., Xiao, X., Yang, B., Li, J., Yu, H., Luo, L., Zhang, X., & Sun, G. (2024) · Research Square · doi:10.21203/rs.3.rs-4845487/v1
JFuzzer: Detecting Optimization Errors in JavaScript Just-In-Time Compilers
Zhang, C., Chen, J., Cai, S., Chen, J., & Huang, C. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00054
“Some traditional fuzzers such as AFL [9] may not achieve good results in JavaScript. have gradually been employed in fuzzing and have shown significant effectiveness [13][14].”
BackgroundCGFuzz: A Dynamic Test Case Generation Method for DL Framework Based on Function Coverage
Qing, C., Yin, B., & Shi, J.-A. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00070
Magnifier: A Chiplet Feature-Aware Test Case Generation Method for Deep Learning Accelerators
Li, B., Zhu, Z., Liu, W., Cao, Q., Li, C., Ji, C., Li, X., & Zhou, X. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3528358 · Cited by 1
“Muffin [12] customizes a set of metrics to measure inconsistencies between different DL libraries, making differential testing more feasible.”
A Comprehensive Review on Deep Learning System Testing
Li, Y., Shan, C., Liu, Z., & Liao, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1548-3_12
A Comprehensive Security Testing Method for Intelligent System
Wu, Z., Zhang, M., & Li, H. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-2200-9_17
Test Script Repair of Deep Learning Library Testing
Fu, X. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728616
XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries
Duan, B., Dong, R., Dong, N., Kim, D. S., & Yang, G. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00030 · Cited by 1
Greybox Fuzzing and Its Applications
Rong, Y. (2024) · eScholarship (California Digital Library)
Enhancing Test Case Generation for Fuzzing Deep Learning Libraries Using Few-Shot Learning
Yang, B., Li, S., Li, J., Du, K., Yu, H., Sun, J. X., & Luo, L. (2025) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc67331.2025.00089
“Traditional fuzzers, including LEMON [12], MMOS [13], and Muffin [14], are among the early fuzzers, which typically generate test cases by applying mutations to the model.”
A Multi-Agent Fuzzing Framework for Deep Learning Library
Liao, R., Ou, S., Yan, X., & Zhu, K.-L. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343067
Testing Deep Learning Libraries with Semantic Equivalent API Patterns
Song, Y., Chen, S., Peng, H., & Xie, X. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00031
“Cross-hardware testing compares behavior of the same API on different devices (e.g., CPU vs. GPU) [21], while cross-library testing checks whether functionally equivalent APIs in different libraries produce equivalent outputs [12], [14], [23].”
Wang, T., Li, Y., Ma, M., Chen, L., & Zhou, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.06195
Self-citationCIPIHunter: Detecting Configuration-Induced Prediction Instability in Deep Learning Frameworks
Anonymous, Meng, S., Kim, M., Chen, X., Fang, C., Chen, Z., & Zhai, J. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834345
Accelerating Deep Learning Compiler Testing via Message-Passing Neural Network With Attention
Ding, Y., Zhou, Z., Li, X., Zou, P., & Jiang, H. (2026) · IEEE Transactions on Reliability · doi:10.1109/tr.2026.3702727
“One type of approach is graph-level testing techniques [17], [19], [20], [21], [22].”
Improving Deep Learning Library Testing with Machine Learning
Molina, F., Naziri, M. M. A., Qin, F., Gorla, A., & d’Amorim, M. (2026) · doi:10.1145/3793654.3793744
DECODE: Dynamic Exploration for Constraint-Guided Vulnerability Discovery in Deep Learning Operators
Liu, H., Wang, Z., Liu, Z., & Li, W. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808174
CuFuzz: An API-Knowledge-Graph Coverage-Driven Fuzzing Framework for CUDA Libraries
Fan, X., Fang, Y., Jia, P., Liu, Y., Xu, Y., Peng, X., & Zhou, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808170
Self-citationD3Fuzz: Dynamic, directed and decaying LLM-driven fuzzing for deep learning libraries
Sun, J., Dong, J., Zhang, H., Tian, Z., & Zhang, Y. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108289
Debugging Functionality-Twisting Translations by LLMs via Differential Testing with Bayesian Prior
Wu, S., Sun, X., Wang, X., & Zhou, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3820377
“Defects in deep learning libraries, which are essential to many intelligent software systems, can also be detected through inconsistencies in outputs [34].”
Self-citationLiu, S., Sun, C., Wu, F., & Chan, W. K. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832194
Crafting unusual programs for fuzzing deep learning libraries
Yang, S. (2023) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 1
Understanding and simplifying bug-inducing inputs for deep-learning compilers
Liu, J. (2024) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
From history to FUTURE and back again: bidirectional fuzzing for deep learning libraries
Li, Z., Wu, J., Ling, X., Luo, T., Cui, X., & Rui, Z. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10996-z
XLA from the Trenches: Correctness on TPU Workloads at ScalePreprint
Cano, I., Saadat, A., Bhatia, R. S., Ginzburg, S., Toulme, P. C., Zarkadas, I., Narayanan, P., Bhatu, P., Basile, N., Dietmüller, A., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.00624
Metadata
- Authors
- Gu, J., Luo, X., Zhou, Y., & Wang, X.
- Affiliations
- Fudan University (CN)
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 1418-1430
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4284708930
- Cited by
- 71
- References
- 47
- Citations by year
- 2026: 13 · 2025: 26 · 2024: 18 · 2023: 13 · 2022: 1
- FWCI
- 10.94
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Funders
- National Key Research and Development Program of China (2020YFA0711400)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- differential testing, muffins, TensorFlow, CNTK, Theano, software defect detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.