Free lunch for testing: fuzzing deep-learning libraries from open source
The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
Deep learning (DL) systems can make our life much easier, and thus are gaining more and more attention from both academia and industry. Meanwhile, bugs in DL systems can be disastrous, and can even threaten human lives in safety-critical applications. To date, a huge body of research efforts have been dedicated to testing DL models. However, interestingly, there is still limited work for testing the underlying DL libraries, which are the foundation for building, optimizing, and running DL models. One potential reason is that test generation for the underlying DL libraries can be rather challenging since their public APIs are mainly exposed in Python, making it even hard to automatically determine the API input parameter types due to dynamic typing. In this paper, we propose FreeFuzz, the first approach to fuzzing DL libraries via mining from open source. More specifically, FreeFuzz obtains code/models from three different sources: 1) code snippets from the library documentation, 2) library developer tests, and 3) DL models in the wild. Then, FreeFuzz automatically runs all the collected code/models with instrumentation to trace the dynamic information for each covered API, including the types and values of each parameter during invocation, and shapes of input/output tensors. Lastly, FreeFuzz will leverage the traced dynamic information to perform fuzz testing for each covered API. The extensive study of FreeFuzz on PyTorch and TensorFlow, two of the most popular DL libraries, shows that FreeFuzz is able to automatically trace valid dynamic information for fuzzing 1158 popular APIs, 9X more than state-of-the-art LEMON with 3.5X lower overhead than LEMON. To date, FreeFuzz has detected 49 bugs for PyTorch and TensorFlow (with 38 already confirmed by developers as previously unknown).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous API-level fuzzers [16,72] only focus on fuzzing each single DL library API in isolation.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“FreeFuzz [72], a state-of-the-art API-level fuzzer, will first collect the valid argument space (e.g., type and shape of the input tensor) for a target API by mining open-source code snippets.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“As a result, model-level fuzzers can only cover a limited set of APIs with limited patterns [72], missing many diverse and interesting API sequences that can lead to bugs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“As such, previous work cannot generate valid inputs to cover this API since they rely on known valid input/API pairs obtained from databases created by scraping open-source code [16,72].”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“We include both PyTorch (v1.12) [57] and TensorFlow (v2.10) [66], since they are two of the most popular DL libraries and are widely studied in prior DL library testing work [70,72,77].”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 1×
“In this section, we will give a brief introduction to the preliminaries of deep learning libraries based on PyTorch [51].”
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundIntriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“To mitigate such attacks, researchers have also proposed various adversarial defense techniques, including adversarial training [32, 46, 62], detection [35, 45, 76], and others [64].”
BackgroundGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“Through such applications, DL has substantially improved our daily life [19, 34, 56, 60, 67].”
BackgroundDeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 1×
“Various adversarial attack techniques [32, 47, 48, 73] have been proposed to generate the so-called “adversarial examples” by adding perturbations imperceptible to humans to intentionally fool the classification results given”
BackgroundDeep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968
Learning to forget: Continual prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 4,823 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundThe Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · cited 1×
“Through such applications, DL has substantially improved our daily life [19, 34, 56, 60, 67].”
Background
Research works printed but not linked: 12
- MongoDB: the application data platform, 2021. https://www.mongodb.com.
- N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and A. Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019. · arXiv
- S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014. · arXiv
- I. Dunn, H. Pouget, D. Kroening, and T. Melham. Exposing previously undetectable faults in deep neural networks. arXiv preprint arXiv:2106.00576, 2021. · arXiv
- I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. · arXiv
- J. Liu, Y. Wei, S. Yang, Y. Deng, and L. Zhang. Coverage-guided tensor compiler fuzzing with joint ir-pass mutation. arXiv preprint arXiv:2202.09947, 2022. · arXiv
- X. Ma, B. Li, Y. Wang, S. M. Erfani, S. Wijewickrema, G. Schoenebeck, D. Song, M. E. Houle, and J. Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint arXiv:1801.02613, 2018. · arXiv
- A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. · arXiv
- D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484--489, 2016.
- K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. · arXiv
- Y. Sun, D. Liang, X. Wang, and X. Tang. Deepid3: Face recognition with very deep neural networks. arXiv preprint arXiv:1502.00873, 2015. · arXiv
- Z. Zhao, G. Chen, J. Wang, Y. Yang, F. Song, and J. Sun. Attack as defense: Characterizing adversarial examples using robustness. arXiv preprint arXiv:2103.07633, 2021. · arXiv
Cited by
Items 101–120 of 120
A Multi-Agent Fuzzing Framework for Deep Learning Library
Liao, R., Ou, S., Yan, X., & Zhu, K.-L. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343067
Testing Deep Learning Libraries with Semantic Equivalent API Patterns
Song, Y., Chen, S., Peng, H., & Xie, X. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00031
CIPIHunter: Detecting Configuration-Induced Prediction Instability in Deep Learning Frameworks
Anonymous, Meng, S., Kim, M., Chen, X., Fang, C., Chen, Z., & Zhai, J. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834345
RiskFuzz: Risk-Guided Fuzzing for Deep Learning Libraries
Pan, Z., Chen, X., Zhang, Z., Chen, L., & Shi, G. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11465052
SemuFuzz: An efficient fuzzing approach for deep learning libraries based on semantic representation
Chen, Z., Chen, J., Cai, S., Chen, J., & Gu, W. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112999
Accelerating Deep Learning Compiler Testing via Message-Passing Neural Network With Attention
Ding, Y., Zhou, Z., Li, X., Zou, P., & Jiang, H. (2026) · IEEE Transactions on Reliability · doi:10.1109/tr.2026.3702727
EMERGE: Checking Implementation Equivalence for LLMs via Rule Synthesis
Zhan, Q., Hu, X., Xia, X., & Li, S. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834405
He, Z., & Han, X. (2026) · International Symposium on Robotics, Artificial Intelligence and Information Engineering (RAIIE) · doi:10.1109/raiie70320.2026.11567007
Improving Deep Learning Library Testing with Machine Learning
Molina, F., Naziri, M. M. A., Qin, F., Gorla, A., & d’Amorim, M. (2026) · doi:10.1145/3793654.3793744
LLM-Assisted Input-Requirement-Aware Differential Testing of Array Programming Frameworks
Zhou, Z., & He, J. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808162
DECODE: Dynamic Exploration for Constraint-Guided Vulnerability Discovery in Deep Learning Operators
Liu, H., Wang, Z., Liu, Z., & Li, W. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808174
CuFuzz: An API-Knowledge-Graph Coverage-Driven Fuzzing Framework for CUDA Libraries
Fan, X., Fang, Y., Jia, P., Liu, Y., Xu, Y., Peng, X., & Zhou, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808170
How Effective Is Coverage-Guided Fuzzing to Test Deep Learning Library APIs?
Qin, F., Naziri, M. M. A., Ai, H., Dutta, S., & d’Amorim, M. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00023 · Cited by 1
Ardiansah, J. T., Shobiry, A. K. F., Handayani, A. N., Amrulloh, M. M., & Haruta, T. (2026) · SinkrOn · doi:10.33395/sinkron.v10i3.16531
D3Fuzz: Dynamic, directed and decaying LLM-driven fuzzing for deep learning libraries
Sun, J., Dong, J., Zhang, H., Tian, Z., & Zhang, Y. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108289
Liu, S., Sun, C., Wu, F., & Chan, W. K. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832194
Fuzzing the Boundary between Models and Code in Hybrid AI-Enabled Systems
Gao, X., Feng, Y., Lu, Y., Liu, Z., Chen, Z., & Xu, B. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832205
“Due to its scalability, fuzzing has been successfully applied across diverse domains [11, 12, 37, 66], such as browser engines [54] and network protocols [43].”
FPSieve: LLM-Powered Cascade for Filtering False Positives in OS Migration Testing
Dong, J., Fu, Y., Hu, C., Chen, Y., Xiao, Q., & Shi, H. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834460
From history to FUTURE and back again: bidirectional fuzzing for deep learning libraries
Li, Z., Wu, J., Ling, X., Luo, T., Cui, X., & Rui, Z. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10996-z
Demystifying Checker Bugs in Deep Learning Libraries
Harzevili, N. S., Shin, J., Uddin, G., Yang, J., Wang, J., Wang, S., Ming, J. Z., & Nagappan, N. (2026) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme67971.2026.00025
Metadata
- Authors
- Wei, A., Deng, Y., Yang, C., & Zhang, L.
- Affiliations
- Stanford University (US)
- University of Illinois Urbana-Champaign (US)
- Nanjing University (CN)
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 995-1007
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4284686707
- Cited by
- 106
- References
- 55
- Citations by year
- 2026: 21 · 2025: 37 · 2024: 27 · 2023: 17 · 2022: 4
- FWCI
- 16.03
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Funders
- National Science Foundation (CCF-2131943 and CCF-2141474)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- fuzzing, deep learning libraries, PyTorch, TensorFlow, dynamic typing, instrumentation, bug detection, lemon
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.