Free lunch for testing: fuzzing deep-learning libraries from open source
The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
Deep learning (DL) systems can make our life much easier, and thus are gaining more and more attention from both academia and industry. Meanwhile, bugs in DL systems can be disastrous, and can even threaten human lives in safety-critical applications. To date, a huge body of research efforts have been dedicated to testing DL models. However, interestingly, there is still limited work for testing the underlying DL libraries, which are the foundation for building, optimizing, and running DL models. One potential reason is that test generation for the underlying DL libraries can be rather challenging since their public APIs are mainly exposed in Python, making it even hard to automatically determine the API input parameter types due to dynamic typing. In this paper, we propose FreeFuzz, the first approach to fuzzing DL libraries via mining from open source. More specifically, FreeFuzz obtains code/models from three different sources: 1) code snippets from the library documentation, 2) library developer tests, and 3) DL models in the wild. Then, FreeFuzz automatically runs all the collected code/models with instrumentation to trace the dynamic information for each covered API, including the types and values of each parameter during invocation, and shapes of input/output tensors. Lastly, FreeFuzz will leverage the traced dynamic information to perform fuzz testing for each covered API. The extensive study of FreeFuzz on PyTorch and TensorFlow, two of the most popular DL libraries, shows that FreeFuzz is able to automatically trace valid dynamic information for fuzzing 1158 popular APIs, 9X more than state-of-the-art LEMON with 3.5X lower overhead than LEMON. To date, FreeFuzz has detected 49 bugs for PyTorch and TensorFlow (with 38 already confirmed by developers as previously unknown).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous API-level fuzzers [16,72] only focus on fuzzing each single DL library API in isolation.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“FreeFuzz [72], a state-of-the-art API-level fuzzer, will first collect the valid argument space (e.g., type and shape of the input tensor) for a target API by mining open-source code snippets.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“As a result, model-level fuzzers can only cover a limited set of APIs with limited patterns [72], missing many diverse and interesting API sequences that can lead to bugs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“As such, previous work cannot generate valid inputs to cover this API since they rely on known valid input/API pairs obtained from databases created by scraping open-source code [16,72].”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“We include both PyTorch (v1.12) [57] and TensorFlow (v2.10) [66], since they are two of the most popular DL libraries and are widely studied in prior DL library testing work [70,72,77].”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 1×
“In this section, we will give a brief introduction to the preliminaries of deep learning libraries based on PyTorch [51].”
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundIntriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“To mitigate such attacks, researchers have also proposed various adversarial defense techniques, including adversarial training [32, 46, 62], detection [35, 45, 76], and others [64].”
BackgroundGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“Through such applications, DL has substantially improved our daily life [19, 34, 56, 60, 67].”
BackgroundDeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 1×
“Various adversarial attack techniques [32, 47, 48, 73] have been proposed to generate the so-called “adversarial examples” by adding perturbations imperceptible to humans to intentionally fool the classification results given”
BackgroundDeep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968
Learning to forget: Continual prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 4,823 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundThe Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · cited 1×
“Through such applications, DL has substantially improved our daily life [19, 34, 56, 60, 67].”
Background
Research works printed but not linked: 12
- MongoDB: the application data platform, 2021. https://www.mongodb.com.
- N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and A. Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019. · arXiv
- S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014. · arXiv
- I. Dunn, H. Pouget, D. Kroening, and T. Melham. Exposing previously undetectable faults in deep neural networks. arXiv preprint arXiv:2106.00576, 2021. · arXiv
- I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. · arXiv
- J. Liu, Y. Wei, S. Yang, Y. Deng, and L. Zhang. Coverage-guided tensor compiler fuzzing with joint ir-pass mutation. arXiv preprint arXiv:2202.09947, 2022. · arXiv
- X. Ma, B. Li, Y. Wang, S. M. Erfani, S. Wijewickrema, G. Schoenebeck, D. Song, M. E. Houle, and J. Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint arXiv:1801.02613, 2018. · arXiv
- A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. · arXiv
- D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484--489, 2016.
- K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. · arXiv
- Y. Sun, D. Liang, X. Wang, and X. Tang. Deepid3: Face recognition with very deep neural networks. arXiv preprint arXiv:1502.00873, 2015. · arXiv
- Z. Zhao, G. Chen, J. Wang, Y. Yang, F. Song, and J. Sun. Attack as defense: Characterizing adversarial examples using robustness. arXiv preprint arXiv:2103.07633, 2021. · arXiv
Cited by
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“Traditional techniques with API-level fuzzing [139], [140] or model-level fuzzing [141], [142] suffer from the following limitations: 1) lack of diverse API sequence thus cannot reveal bugs caused by chained API sequences; 2) cannot generate arbitrary code thus cannot explore the huge search space…”
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
“FreeFuzz [72] is an API-level fuzzer that first learns the valid inputs for each target API through mining open-source code snippets and then applies simple mutations to generate diverse inputs to test a target API.”
BackgroundMethodInfluentialSelf-citationFuzz4ALL: Universal Fuzzing with Large Language Models
Xia, C. S., Paltenghi, M., Le Tian, J., Pradel, M., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639121 · Cited by 229
Self-citationDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623343 · Cited by 115
Self-citationWhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Yang, C., Deng, Y., Lu, R., Yao, J., Liu, J., Jabbarvand, R., & Zhang, L. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689736 · Cited by 82
“Therefore, they can be effectively triggered by TitanFuzz since TitanFuzz leverages LLMs to generate human-like programs by resembling the distribution of training data.”
BackgroundSelf-citationNNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers
Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., & Zhang, L. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575707 · Cited by 167
“For example, on line 4 in Figure 1, the Reshape operator takes %v1 as an input tensor and [62,62,2] as an attribute indicating the output shape.”
InfluentialSelf-citationFuzzing deep-learning libraries via automated relational API inference
Deng, Y., Yang, C., Wei, A., & Zhang, L. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549085 · Cited by 86
“We obtain the valid inputs traced from various input sources used by state-of-the-art FreeFuzz [65], which include documentations, developer tests, and 202 DL models in the wild,”
BackgroundMethodInfluentialSelf-citationOne fuzzing strategy to rule them all
Wu, M., Jiang, L. L., Xiang, J., Huang, Y., Cui, H., Zhang, L., & Zhang, Y. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510174 · Cited by 79
“In more recent years, researchers have also proposed various techniques for fuzzing different types of software systems [29, 36, 42, 55].”
BackgroundSelf-citationSoK: Prudent Evaluation Practices for Fuzzing
Schloegel, M., Bars, N., Schiller, N., Bernhard, L., Scharnowski, T., Crump, A., Ebrahim, A. A., Bissantz, N., Muench, M., & Holz, T. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00137 · Cited by 105
“…[109], [186], [143], [41], [111], [153], [2], [10], [161], [134], [96] 12/29 2022 ASE [58], [174] 2/4 FSE [66], [189] 2/6 ICSE [97], [124], [89], [163], [64], [115], [151], [52] 8/17 CCS [83], [12], [57], [144], [37], [29], [191] 7/8 NDSS [84], [169], [180] 3/6 S&P [74], [147], [102], [28],…”
History-driven test program synthesis for JVM testing
Zhao, Y., Wang, Z., Chen, J., Liu, M., Wu, M., Zhang, Y., & Zhang, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510059 · Cited by 79
“Since the first work on fuzzing [40], various techniques have been proposed for fuzzing software systems from different application domains [13, 32, 49, 51, 52, 55].”
BackgroundSelf-citationCoverage-guided tensor compiler fuzzing with joint IR-pass mutation
Liu, J., Wei, Y., Yang, S., Deng, Y., & Zhang, L. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3527317 · Cited by 51
“) Existing work on DL-library testing [Pham et al. 2019; Wang et al. 2020; Wei et al. 2022] mainly focuses on testing at the graph-operator level.”
BackgroundSelf-citationFuzzing Automatic Differentiation in Deep-Learning Libraries
Yang, C., Deng, Y., Yao, J., Tu, Y., Li, H., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00105 · Cited by 77
“Code coverage is one of the main criteria in software testing, and has also been recently adopted for testing DL libraries/compilers [29], [30], [57].”
InfluentialSelf-citationJITfuzz: Coverage-guided Fuzzing for JVM Just-in-Time Compilers
Wu, M., Lu, M., Cui, H., Chen, J., Zhang, Y., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00017 · Cited by 58
“Recently, a large group of researchers have been focusing on applying fuzzing/testing to the emerging AI/ML systems [77, 78, 79, 80, 81, 82, 83, 84].”
BackgroundSelf-citationVulnerability detection through machine learning-based fuzzing: A systematic review
Chafjiri, S. B., Legg, P., Hong, J. S., & Tsompanas, M. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103903 · Cited by 45
COMET: Coverage-guided Model Generation For Deep Learning Library Testing
Li, M., Cao, J., Tian, Y., Li, T. O., Wen, M., & Cheung, S.-C. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3583566 · Cited by 45
“FreeFuzz [56] also targets collecting DL library functions constraint.”
BackgroundACETest: Automated Constraint Extraction for Testing Deep Learning Operators
Shi, J., Xiao, Y., Li, Y., Li, Y., Yu, D., Yu, C., Su, H., Chen, Y., & Huo, W. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598088 · Cited by 42
“To save manual e ort, FreeFuzz [46] proposes to mine test cases from open source and DeepREL [10] leverages existing test cases to test more APIs by inferring relational APIs.”
BackgroundMethodInfluentialBoosting Compiler Testing by Injecting Real-World Code
Li, S., Theodoridis, T., & Su, Z. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3656386 · Cited by 42
Generation-based Differential Fuzzing for Deep Learning Libraries
Liu, J., Huang, Y., Wang, Z., Ma, L., Fang, C., Gu, M., Zhang, X., & Chen, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3628159 · Cited by 40
“Wei et al. [28, 57, 59] propose equivalence rules or mutations to test DL APIs.”
BackgroundDutta, S., Arunachalam, A., & Misailovíc, S. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00026 · Cited by 40
“debugging various non-deterministic systems such as ML frameworks [39, 41, 51, 70, 71, 73], Probabilistic Programming Systems [22, 23, 27, 47], and Approximation Algorithms [42].”
FuzzSlice: Pruning False Positives in Static Analysis Warnings through Function-Level Fuzzing
Murali, A., Mathews, N. S., Alfadel, M., Nagappan, M., & Xu, M. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623321 · Cited by 39
“• Technical Innovation (close-to-warning fuzzing): Unlike typical methods that reduce fuzzing cost by independently fuzzing modules and libraries [16–19], FuzzSlice focuses on generating compiled slices that encompass the warning location detected by a static analysis tool.”
BackgroundThe Human Side of Fuzzing: Challenges Faced by Developers during Fuzzing Activities
Nourry, O., Kashiwa, Y., Lin, B., Bavota, G., Lanza, M., & Kamei, Y. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3611668 · Cited by 29
On the Challenges of Fuzzing Techniques via Large Language Models
Huang, L., Zhao, P., Ma, L., & Chen, H. (2025) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse67621.2025.00028 · Cited by 43
A Roadmap for Software Testing in Open-Collaborative and AI-Powered Era
Wang, Q., Wang, J., Li, M., Wang, Y., & Liu, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709355 · Cited by 26
Deep Learning Library Testing: Definition, Methods and Challenges
Zhang, X., Jiang, W., Shen, C. Q., Li, Q. L., Wang, Q., Lin, C., & Guan, X. (2025) · ACM Computing Surveys · doi:10.1145/3716497 · Cited by 33
“Researchers have designed some DL library testing methods based on the metamorphic testing technique to verify the computation and optimization of DL libraries, and successfully detected real-world bugs in the DL framework and compiler [131, 136] (10 of collected papers).”
InfluentialGenCoG: A DSL-Based Approach to Generating Computation Graphs for TVM Testing
Wang, Z., Nie, P., Miao, X., Chen, Y., Wan, C., Bu, L., & Zhao, J. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598105 · Cited by 29
“FreeFuzz [37] and DocTer [40] infer speci cations by mining open-source code or documentation, which is likely to introduce false alarms.”
BackgroundSelfPiCo: Self-Guided Partial Code Execution with LLMs
Xue, Z., Gao, Z., Wang, S., Hu, X., Xia, X., & Li, S. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680368 · Cited by 31
Enhancing Differential Testing with LLMs for Testing Deep Learning Libraries
Li, M., Li, D., Liu, J., Cao, J., Tian, Y., & Cheung, S.-C. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735637 · Cited by 28
“We summarize these rules in Table 4.”
InfluentialA Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators
Chen, J., Jia, C., Yan, Y., Ge, J., Zheng, H., & Cheng, Y. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660796 · Cited by 16
“We compare Meta with the SOTA DL API testing methods DocTer [38], FreeFuzz [36], EAGLE [33], and the SOTA DL operator testing method Duo [43].”
Taxonomy of Aging-related Bugs in Deep Learning Libraries
Liu, Z., Zheng, Y., Du, X., Zheng, H., Ding, W., Miao, Y., & Zheng, Z. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00048 · Cited by 19
History-Driven Fuzzing for Deep Learning Libraries
Harzevili, N. S., Mohajer, M. M., Wei, M., Pham, H. V., & Wang, S. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688838 · Cited by 20
FuzzCoder: Code Large Language Model-Based Fuzz Testing for Industrial IoT Programs
Yang, L., Wei, C., Yang, J., Xia, W., Yang, Y., Luo, Y., Niyato, D. T., Sun, L., & Liu, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3577602 · Cited by 14
“Researchers further develop context-aware strategies such as energy-constrained evolutionary fuzzing and hybrid model-guided testing to address challenges like limited device resources and real-time operational dependencies [5], [6].”
Deep learning framework testing via hierarchical and heuristic model generation
Zou, Y., Sun, H., Fang, C., Liu, J., & Zhang, Z. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111681 · Cited by 14
Dynamic Data Fault Localization for Deep Neural Networks
Yin, Y., Feng, Y., Weng, S., Liu, Z., Yao, Y., Zhang, Y., Zhao, Z., & Chen, Z. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616345 · Cited by 15
“[82] proposed a fuzzing technique FreeFuzz to debug DL libraries based on open source APIs automatically.”
MethodYour Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 26
Type-aware LLM-based Test Generation for Python Programs
Liu, R., Zhang, Z., Hu, Y., Lin, Y., Gao, X., & Sun, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3830083 · Cited by 19
Guan, H., Bai, G., & Liu, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680383 · Cited by 17
CrossProbe: LLM-Empowered Cross-Project Bug Detection for Deep Learning Frameworks
Guan, H., Bai, G., & Liu, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728984 · Cited by 15
Li, Z., Wu, J., Ling, X., Luo, T., Rui, Z., & Wu, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00132 · Cited by 18
A Usability Evaluation of AFL and libFuzzer with CS Students
Plöger, S., Meier, M., & Smith, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581178 · Cited by 10
A Tale of Two DL Cities: When Library Tests Meet Compiler
Shen, Q., Tian, Y., Ma, H., Chen, J., Huang, L., Fu, R., Cheung, S.-C., & Wang, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00025 · Cited by 18
A survey of coverage-guided greybox fuzzing with deep neural models
Qiu, J., Jiang, Y., Miao, Y., Luo, W., Pan, L., & Zheng, X. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107797 · Cited by 8
Go, G., Zhou, C., Zhang, Q., Zou, X., Shi, H., & Jiang, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680364 · Cited by 12
D3: Differential Testing of Distributed Deep Learning With Model Generation
Wang, J., Pham, H. V., Li, Q., Tan, L., Guo, Y., Aziz, A., & Meijer, E. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3461657 · Cited by 15
“Existing techniques address this challenge by cross-checking different libraries or execution graphs to detect inconsistency bugs [25], [26], [27], [28], [29], [30].”
Balancing Effectiveness and Flakiness of Non-Deterministic Machine Learning Tests
Xia, C. S., Dutta, S., Misailovíc, S., Marinov, D., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00154 · Cited by 10
“Recently, researchers have proposed approaches for testing machine learning frameworks [92], [93], [94], [95], [96], [97], [98], [99], [100], [101], [102], probabilistic programming systems [37], [103], [104], and randomized algorithms [105].”
BackgroundSelf-citationTest & Evaluation Best Practices for Machine Learning-Enabled SystemsPreprint
Chandrasekaran, J., Cody, T., McCarthy, N., Lanus, E., & Freeman, L. E. B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4862589 · Cited by 14
DeepDiffer: Find Deep Learning Compiler Bugs via Priority-guided Differential Fuzzing
Lin, K., Song, X., Zeng, Y., & Guo, S. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00066 · Cited by 12
“Bug Counting: Following prior studies [38], [14], [4], we employ the count of distinct patches as an indicator of the number of identified bugs.”
MethodLiu, J., Zhang, X., Xu, L., Fang, C., Gu, M., Luo, W., Chai, D., Wang, J., Zhao, Z., & Chen, Z. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715104 · Cited by 5
“Furthermore, the proposed method has been employed in a real-world application of a large software technology company for months and solved quantities of FPPs in CNN operators, which indicates the elimination of the threats to construct validity.”
Mutation-Based Deep Learning Framework Testing Method in JavaScript Environment
Zou, Y., Zhai, J., Fang, C., Liu, J., Zheng, T., & Chen, Z. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695478 · Cited by 7
KLRAG: Deep Learning Library Vulnerability Detection via Knowledge-Level RAG
Liao, R., Yan, X., & Zhu, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9958-2_36 · Cited by 4
Foundation Models for Autonomous Driving Systems: An Initial Roadmap
Wu, X., Cheng, M., Ren, X., Hu, Q., Chen, J., Huang, Y., Cordy, M., Zhang, Y., Xie, X., Ma, L., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3807455 · Cited by 18
Metamong: Detecting Render-Update Bugs in Web Browsers through Fuzzing
Song, S., & Lee, B. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616336 · Cited by 5
“[ 51] leverages the open source to infer the API input parameter types for effective deep learning library fuzzing.”
BackgroundMoCo: Fuzzing Deep Learning Libraries via Assembling Code
Ji, P., Feng, Y., Wu, D., Yan, L., Chen, P., Liu, J., & Zhao, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3509975 · Cited by 4
“Moreover, DL libraries using the same front-end effectively share core logic code, which limits the performance of differential testing in detecting bugs [6].”
InfluentialEvaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study
Harzevili, N. S., Wei, M., Mohajer, M. M., Pham, H. V., & Wang, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3729533 · Cited by 6
“FreeFuzz [77]: We configured FreeFuzz to run 1,000 times per API with 60 seconds of fuzzing budget.”
InfluentialNaziri, M. M. A., Singh, A. K., Wu, B., Qin, F., Dutta, S., & d’Amorim, M. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731739 · Cited by 4
Improving Deep Learning Framework Testing with Model-Level Metamorphic Testing
Mu, Y., Zhai, J., Fang, C., Chen, X., Cao, Z., Yang, P., Zhao, K., Guo, A., & Chen, Z. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728972 · Cited by 12
“Until now, researchers have proposed different DL framework testing methods [14, 19, 24, 36, 37, 47, 51, 59–61 Researchers often use DL models [24, 36, 47, 59] or single framework interfaces [19, 60, 62, 63, 67] as inputs to test DL frameworks.”
S cuzer : A Scheduling Optimization Fuzzer for TVM
Chen, X., Lin, X., Wang, J., Sun, J., Wang, J., & Wang, W. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705308 · Cited by 6
“Though we use single-operator testing to help record operator-related information, not all operators are suitable for this method.”
Bridging Operator Semantic Inconsistencies: A Source-Level Cross-Framework Model Conversion Approach
Li, X., Lei, Y., Jia, Z., Zhang, Y., Liu, H., Chen, L., Dong, W., & Li, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729361 · Cited by 3
“As illustrated in Figure 2, DL libraries—key components of DL frameworks [59, 62]—are structured into three layers: a high-level API for model construction, a Python layer for operator and tensor management, and a C layer for low-level execution [59].”
Lightweight Concolic Testing via Path-Condition Synthesis for Deep Learning Libraries
Kim, S., Kim, Y., Park, D., Jeon, Y., Yi, J., & Kim, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00202 · Cited by 7
Fuzz4Cuda: Fuzzing your NVIDIA GPU libraries through debug interface
Zhou, Y., Jia, P., Liu, J., & Fan, X. (2025) · Computers & Security · doi:10.1016/j.cose.2025.104754 · Cited by 6
Reference-Based Retrieval-Augmented Unit Test Generation
Zhang, Z., Liu, X., Lin, Y., Gao, X., Sun, H., & Yuan, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3765758 · Cited by 10
“Prominent methods include random-based strategies [11, 41, 60], Search-Based Software Testing (SBST) [9, 14, 27, 30, 34, 52, 64, 73] and constraint-driven techniques [17, 35].”
Survey on Fuzzing Techniques in Deep Learning Libraries
Ji, J., Kong, W., Tian, J., Gu, T., Nie, Y., & Kuang, X. (2023) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc59305.2023.00073 · Cited by 3
“2) In 2022, Wei et al [51] proposed FreeFuzz, the first approach to fuzzing DL libraries via mining from actual model and API executions.FreeFuzz records the dynamic information for all the input parameters for each invoked API on the fly while running all the collected code/models.”
MethodTorchProbe: Fuzzing Dynamic Deep Learning Compilers
Su, Q., Geng, C., Pekhimenko, G. G., & Si, X. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8311-7_15 · Cited by 9
Feng, S. M., Liu, J., Lai, R., Ruan, C. F., Yu, Y. H., Zhang, L., & Chen, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728957 · Cited by 3
Self-citationCitadel : Context Similarity Based Deep Learning Framework Bug Finding
Zhang, X., Zhai, J., Ma, S., Wang, S., & Shen, C. Q. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773993 · Cited by 2
Harnessing LLMs for Document-Guided Fuzzing of OpenCV Library
Duan, B., Mahmud, T., Che, M., Yan, Y., Dong, N., Kim, D. S., & Yang, G. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00017 · Cited by 4
A Survey of Security Testing Techniques for Deep Learning Frameworks
Li, H., Li, X., Nie, Y., & Tian, J. (2024) · doi:10.1109/icsip61881.2024.10671492 · Cited by 1
“FreeFuzz [27] is the first fully automated tool for fuzz testing DL frameworks by executing a collection of covered APIs from open-source projects.”
MethodDKFuzz: An Automated Approach for Testing the Underlying Kernels of Deep Learning Frameworks
Li, H., Li, X., Nie, Y., Du, S., & Wu, Q. (2025) · International Conference on Algorithms, Software Engineering and Network Security (ASENS) · doi:10.1109/asens64990.2025.11011223 · Cited by 2
CEDAR: Continuous Testing of Deep Learning Libraries
Xie, D., Wang, J., Pham, H. V., Tan, L., Guo, Y., Aziz, A., & Meijer, E. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00044 · Cited by 5
QAQA-SS: An Improved Fuzzing Approach with Seed Scheduling Based on the UCB Algorithm for QA Systems
Chen, J., Wu, Y., Chen, H., Chen, J., & Akpaku, E. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00089 · Cited by 1
“A differential fuzzing framework called FreeFuzz [38] collects code and model differences from the open-source community to detect potential errors in deep-learning libraries.”
BackgroundMultigranularity Coverage Criteria for Deep Learning Libraries
Shi, Y., Zheng, Z., Yin, B., & Xi, Z. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1903 · Cited by 2
Liao, R., Yan, X., Pang, Z., & Zhu, K.-L. (2025) · Applied Sciences · doi:10.3390/app151910396 · Cited by 2
A PSO-based Method to Test Deep Learning Library at API Level
Liao, S., & Shan, C. (2024) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3672758.3672777 · Cited by 1
Characterizing, Detecting, and Correcting Comment Errors in Smart Contract Functions
Cheng, Y., Yang, H., Li, Z., & Tian, L. (2024) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse62657.2024.00050 · Cited by 1
“Deng et al. [52] proposes a new approach using large language models for fuzzing deep learning libraries [53], which can generate high-quality seed inputs and guide the generation towards a higher number of unique library API usages [54] and valid/diverse DL programs.”
BackgroundEvaluating Spectrum-Based Fault Localization on Deep Learning Libraries
Yan, M., Chen, J., Jiang, T., Jiang, J., & Wang, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3552622 · Cited by 2
“In this study, test cases are defined as code snippets that invoke DL library APIs to perform essential operations in the deep learning process, which are widely utilized in DL library testing [4], [5], [6], [34], [35].”
Directed Testing in MLIR: Unleashing Its Potential by Overcoming the Limitations of Random Fuzzing
Tong, W., Wang, Z., Tang, Z., Fang, J., Zhang, Y., & Ye, G. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729372 · Cited by 2
Source File Tracking Localization: A Fault Localization Method for Deep Learning Frameworks
Ma, Z., Yang, B., & Zhang, Y. (2023) · Electronics · doi:10.3390/electronics12224579
Optimizing asynchronous JavaScript applications
Turcotte, A. (2023) · doi:10.17760/d20560794
Python Coverage Guided Fuzzing for Deep Learning Framework
Nie, Y., Xiao, X., Yang, B., Li, H., Luo, L., Yu, H., & Sun, G. (2024) · International Conference on Electronic Engineering and Information Systems (EEISS) · doi:10.1109/eeiss62553.2024.00007 · Cited by 2
Lightweight Code Coverage Analysis for Deep Learning Framework TestingPreprint
Li, S., Xiao, X., Yang, B., Li, J., Yu, H., Luo, L., Zhang, X., & Sun, G. (2024) · Research Square · doi:10.21203/rs.3.rs-4845487/v1
JFuzzer: Detecting Optimization Errors in JavaScript Just-In-Time Compilers
Zhang, C., Chen, J., Cai, S., Chen, J., & Huang, C. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00054
“Some traditional fuzzers such as AFL [9] may not achieve good results in JavaScript. have gradually been employed in fuzzing and have shown significant effectiveness [13][14].”
BackgroundCGFuzz: A Dynamic Test Case Generation Method for DL Framework Based on Function Coverage
Qing, C., Yin, B., & Shi, J.-A. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00070
“Existing adequacy metrics of component testing generally include API coverage[15], code coverage[19], and so forth.”
BackgroundMethodPython Fuzzing for Trustworthy Machine Learning Frameworks
Yegorov, I., Kobrin, E., Parygina, D., Vishnyakov, A., & Fedotov, A. (2024) · Journal of Mathematical Sciences · doi:10.1007/s10958-024-07424-2 · Cited by 3
Magnifier: A Chiplet Feature-Aware Test Case Generation Method for Deep Learning Accelerators
Li, B., Zhu, Z., Liu, W., Cao, Q., Li, C., Ji, C., Li, X., & Zhou, X. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3528358 · Cited by 1
“FreeFuzz [15] uses fuzz testing for DL libraries by mining from open-source projects, utilizing tracked dynamic information.”
A Comprehensive Review on Deep Learning System Testing
Li, Y., Shan, C., Liu, Z., & Liao, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1548-3_12
Quan, L., Xie, X., Guo, Q., Jiang, L., Chen, S., Wang, J., & Li, X. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714649 · Cited by 1
FuMi: A Runtime Fuzz-based Machine Learning Precision Measurement and Testing Framework
Zhang, P., Papadakis, M., & Zhou, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3734866
Guo, H., Tao, C., Huang, Z., & Zou, W. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3736305 · Cited by 4
“Besides, recent studies [69, 71] also focus on debugging DL libraries.”
RIMFuzz: real-time impact-aware mutation for library API fuzzing
Wang, X., & Zhao, L. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00050-1 · Cited by 1
“…Sun et al. 2021; Bulekov et al. 2023), Rust libraries (Jiang et al. 2021; Takashima et al. 2021), Machine learning libraries (Wang et al. 2020; Wei et al. 2022; Xie et al. 2022; Deng et al. 2023), RESTful API (Atlidakis et al. 2019; Liu et al. 2022), and even arbitrary functions (Liu et al.…”
On the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations
Hundal, R. S., Xiao, Y., Cao, X., Dong, J. S., & Rigger, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00222 · Cited by 1
Testing False Recalls in E-Commerce Apps: a User-Perspective Blackbox Approach
Wu, S., Hu, Y., Gu, J., Mao, P., Meng, J., Fan, L., Luan, Z., Wang, X., & Zhou, Y. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00052
RunTyper: Enhancing Deep Type Inference Using Dynamic Analysis for Python
Yang, G., He, S., Song, F., & Chen, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3771544
XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries
Duan, B., Dong, R., Dong, N., Kim, D. S., & Yang, G. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00030 · Cited by 1
“In contrast, API-level fuzzing [35], [10] directly targets individual APIs within deep learning libraries, systematically exploring * Corresponding author. their parameter space to reveal abnormal behaviors.”
Scalpel: Automotive Deep Learning Framework Testing via Assembling Model Components
Zou, Y., Zhai, J., Fang, C., Guo, A., Liu, J., & Chen, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773242 · Cited by 2
Automating fuzz driver generation for deep learning libraries with large language models
Zheng, T., Meng, F., Yi, P., & Wu, Y. (2026) · Cybersecurity · doi:10.1186/s42400-025-00532-9 · Cited by 2
Parlak, İ. E., Çelik, M. T., Gürkan, I., & Yildiz, A. ̧. (2026) · Pamukkale University Journal of Engineering Sciences · doi:10.65206/pajes.57422
Mind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators
Wen, E., Ma, S., Tempero, E., Sham, C., Sun, Y., Jia, H., Luo, D., Hua, J., Dietrich, J. B., Zhao, K., et al. (2025) · International Conference on Mobility, Sensing and Networking (MSN) · doi:10.1109/msn69125.2025.00054 · Cited by 4
“To further enhance the diversity of our models, we also employ a well-tested operator mutation strategy from prior work [17].”
Subgraph-Oriented Testing for Deep Learning Libraries
Xie, X., Song, Y., Chen, S., & Chen, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3655712 · Cited by 2
“A performance degradation bug is identified when an API takes longer time to execute with lower-precision input (e.g., float16) than with higher-precision input (e.g., float32) [14].”
InfluentialTesting Deep Learning Libraries via Neurosymbolic Constraint Learning
Naziri, M. M. A., Kim, S., Qin, F., Dutta, S., & d’Amorim, M. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787800 · Cited by 5
“Hence, most existing techniques are either oblivious to the input constraints of the APIs under test (e.g., DeepREL [12] and Free-Fuzz [54]) or are imprecise in modeling input constraints (e.g., DocTer [55] and ACETest [47]).”
InfluentialEnhancing Test Case Generation for Fuzzing Deep Learning Libraries Using Few-Shot Learning
Yang, B., Li, S., Li, J., Du, K., Yu, H., Sun, J. X., & Luo, L. (2025) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc67331.2025.00089
“2) Code branch loss : Traditional fuzzers frequently utilize fixed-format code generation architectures, resulting in overly simplistic test cases, as seen with Freefuzz [15].”
Lightweight Code Coverage Analysis for Deep Learning Library Fuzzing
Li, S., Yang, B., Yu, H., & Luo, L. (2025) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc67331.2025.00090
Metadata
- Authors
- Wei, A., Deng, Y., Yang, C., & Zhang, L.
- Affiliations
- Stanford University (US)
- University of Illinois Urbana-Champaign (US)
- Nanjing University (CN)
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 995-1007
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4284686707
- Cited by
- 106
- References
- 55
- Citations by year
- 2026: 21 · 2025: 37 · 2024: 27 · 2023: 17 · 2022: 4
- FWCI
- 16.03
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Funders
- National Science Foundation (CCF-2131943 and CCF-2141474)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- fuzzing, deep learning libraries, PyTorch, TensorFlow, dynamic typing, instrumentation, bug detection, lemon
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.