DocTer: documentation-guided fuzzing for testing deep learning API functions
Open accessThe DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index.
Abstract
Input constraints are useful for many software development tasks. For example, input constraints of a function enable the generation of valid inputs, i.e., inputs that follow these constraints, to test the function deeper. API functions of deep learning (DL) libraries have DL-specific input constraints, which are described informally in the free-form API documentation. Existing constraint-extraction techniques are ineffective for extracting DL-specific input constraints.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We include both PyTorch (v1.12) [57] and TensorFlow (v2.10) [66], since they are two of the most popular DL libraries and are widely studied in prior DL library testing work [70,72,77].”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 3×
“, TensorFlow [14] and PyTorch [39]) is crucial because these libraries are widely used and contain software bugs [26, 27, 42, 58, 59], which hurt not only the development but also the accuracy and speed of the DL models.”
BackgroundMethodTensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037 · cited 2×
“Dependency tree parsing: Once the sentences are normalized, they are fed to the dependency tree parser [33], which conducts POSTagging and builds tree structure relationships (i.”
BackgroundMethodNatural language processing with Python
Bird, S., Klein, E., & Loper, E. (2009) · CERN Document Server (European Organization for Nuclear Research) · Cited by 3,955
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.01274 · Cited by 1,819 · cited 1×
“• A document-analysis technique that extracts 16,035 constraints automatically from API documentation with the focus on four categories of input properties in DL APIs: structure, dtype, shape, and valid values for 2,415 API functions across the three widelyusedDL libraries, TensorFlow [14], PyTorch [39], andMXNet [17].”
MethodSelf-citationStatic Analysis of Shape in TensorFlow Programs
The Theano Development Team, Al‐Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2020.15 · Cited by 1,589
Fraser, G., & Arcuri, A. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.14 · Cited by 586 · cited 1×
“Existing test-generation tools [3, 5, 9, 19, 37, 44] such as AFL [3] and libFuzzer [5] have no knowledge of such input constraints, thus are very limited in testing DL API functions.”
MethodRandoop: feedback-directed random testing for Java
Pacheco, C., & Ernst, M. D. (2007) · Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion · doi:10.1145/1297846.1297902 · Cited by 558
Cited by
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“Traditional techniques with API-level fuzzing [139], [140] or model-level fuzzing [141], [142] suffer from the following limitations: 1) lack of diverse API sequence thus cannot reveal bugs caused by chained API sequences; 2) cannot generate arbitrary code thus cannot explore the huge search space…”
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
“While it is not explicitly stated in the documentation, the developers commented on the issue report that because the CPU and GPU use different lower-level shifting operators, the output result when shifting with negative values will be dependent on the implementation.”
MethodInfluentialDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623343 · Cited by 115
Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
Zhang, Z., Chen, C., Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., & Wang, R. (2024) · Transactions on machine learning research · doi:10.13140/rg.2.2.32710.69440/1 · Cited by 194
WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Yang, C., Deng, Y., Lu, R., Yao, J., Liu, J., Jabbarvand, R., & Zhang, L. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689736 · Cited by 82
Large Language Model Assisted Software Engineering: Prospects, Challenges, and a Case Study
Belzner, L., Gabor, T., & Wirsing, M. (2023) · AISoLA · doi:10.1007/978-3-031-46002-9_23 · Cited by 82
“According to a recent study [46], LLMs that are coached by few-shot learning achieve better extraction results than state-of-the-art techniques such as Jdoctor [4] and DocTer [45].”
BackgroundToward Understanding Deep Learning Framework Bugs
Chen, J., Liang, Y., Shen, Q., Jiang, J., & Li, S. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3587155 · Cited by 137
“For the second category, Predoo [78], FreeFuzz [67], and DocTer[68] are three representative techniques.”
MethodFuzzing deep-learning libraries via automated relational API inference
Deng, Y., Yang, C., Wei, A., & Zhang, L. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549085 · Cited by 86
“A very recent work DocTer [76] constructs rules to extract DL-specific input constraints from API documentation and uses them to generate valid/invalid inputs for testing DL libraries; meanwhile, it requires manual annotation for 30% of API parameters.”
MethodHow Effective are Large Language Models in Generating Software Specifications?
Xie, D., Yoo, B., Jiang, N., Kim, M., Tan, L., Zhang, X., & Lee, J. S. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00014 · Cited by 88
Self-citationFuzzing Automatic Differentiation in Deep-Learning Libraries
Yang, C., Deng, Y., Yao, J., Tu, Y., Li, H., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00105 · Cited by 77
“7K and 167K stats on GitHub, PyTorch and TensorFlow are the two most popular DL libraries, and they are also widely studied in prior DL library testing work [30], [31], [41].”
InfluentialFuzzing Deep Learning Compilers with HirGen
Ma, H., Shen, Q., Tian, Y., Chen, J., & Cheung, S.-C. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598053 · Cited by 59
“Specifically, HirGen provides an option of generating invalid computational graphs that violate type constraints and shape constraints [37].”
BackgroundAre LLMs Correctly Integrated into Software Systems?
Shao, Y.-C., Huang, Y.-H., Shen, J., Ma, L., Su, T., & Wan, C.-C. (2024) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00204 · Cited by 40
NeuRI: Diversifying DNN Generation via Inductive Rule Inference
Liu, J., Peng, J., Wang, Y., & Zhang, L. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616337 · Cited by 47
“Operator-level techniques [10, 43, 45] aim to test eachDLAPI in isolation.”
BackgroundMethodInfluentialCOMET: Coverage-guided Model Generation For Deep Learning Library Testing
Li, M., Cao, J., Tian, Y., Li, T. O., Wen, M., & Cheung, S.-C. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3583566 · Cited by 45
“DocTer [57] extracts the argument constraint of DL library functions from the 7See our implementation at https://github.”
BackgroundACETest: Automated Constraint Extraction for Testing Deep Learning Operators
Shi, J., Xiao, Y., Li, Y., Li, Y., Yu, D., Yu, C., Su, H., Chen, Y., & Huo, W. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598088 · Cited by 42
“• DocTer [50], a documentation-guided fuzzing technique for testing DL API functions.”
BackgroundMethodInfluentialSelf-citationDeep Learning Library Testing: Definition, Methods and Challenges
Zhang, X., Jiang, W., Shen, C. Q., Li, Q. L., Wang, Q., Lin, C., & Guan, X. (2025) · ACM Computing Surveys · doi:10.1145/3716497 · Cited by 33
“Most existing testing methods [20, 137] construct test inputs for the various API of the entire DL library to discover bugs.”
InfluentialSelf-citationGenCoG: A DSL-Based Approach to Generating Computation Graphs for TVM Testing
Wang, Z., Nie, P., Miao, X., Chen, Y., Wan, C., Bu, L., & Zhao, J. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598105 · Cited by 29
“FreeFuzz [37] and DocTer [40] infer specifications by mining open-source code or documentation, which is likely to introduce false alarms.”
BackgroundEnhancing Differential Testing with LLMs for Testing Deep Learning Libraries
Li, M., Li, D., Liu, J., Cao, J., Tian, Y., & Cheung, S.-C. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735637 · Cited by 28
“We summarize these rules in Table 4.”
InfluentialYour Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 26
History-Driven Fuzzing for Deep Learning Libraries
Harzevili, N. S., Mohajer, M. M., Wei, M., Pham, H. V., & Wang, S. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688838 · Cited by 20
Self-citationCommon challenges of deep reinforcement learning applications development: an empirical study
Morovati, M. M., Tambon, F., Taraghi, M., Nikanjam, A., & Khomh, F. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10500-5 · Cited by 19
“An example of such techniques focusing on the challenges of software API usage, is the work by Xie et al. [72] which proposes an approach to automatically extract the API parameter constraints of DL libraries/frameworks.”
Hu, X., Niu, F., Chen, J., Zhou, X., Zhang, J., He, J., Xia, X., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3786771 · Cited by 15
Li, Z., Wu, J., Ling, X., Luo, T., Rui, Z., & Wu, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00132 · Cited by 18
A Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators
Chen, J., Jia, C., Yan, Y., Ge, J., Zheng, H., & Cheng, Y. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660796 · Cited by 16
“We compare Meta with the SOTA DL API testing methods DocTer [38], FreeFuzz [36], EAGLE [33], and the SOTA DL operator testing method Duo [43].”
FuzzCoder: Code Large Language Model-Based Fuzz Testing for Industrial IoT Programs
Yang, L., Wei, C., Yang, J., Xia, W., Yang, Y., Luo, Y., Niyato, D. T., Sun, L., & Liu, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3577602 · Cited by 14
A Tale of Two DL Cities: When Library Tests Meet Compiler
Shen, Q., Tian, Y., Ma, H., Chen, J., Huang, L., Fu, R., Cheung, S.-C., & Wang, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00025 · Cited by 18
“We used the implementations of DocTer and DeepREL released by their works [15], [16].”
InfluentialTest & Evaluation Best Practices for Machine Learning-Enabled SystemsPreprint
Chandrasekaran, J., Cody, T., McCarthy, N., Lanus, E., & Freeman, L. E. B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4862589 · Cited by 14
Feng, X., Zhang, Y., Meng, M. H., Li, Y., Joe, C. E., Wang, Z., & Bai, G. (2023) · ISA Transactions · doi:10.1016/j.isatra.2023.04.025 · Cited by 14
“Dependency parsing is also studied in other work such as [33].”
Self-citationCrossProbe: LLM-Empowered Cross-Project Bug Detection for Deep Learning Frameworks
Guan, H., Bai, G., & Liu, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728984 · Cited by 15
Deep learning framework testing via hierarchical and heuristic model generation
Zou, Y., Sun, H., Fang, C., Liu, J., & Zhang, Z. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111681 · Cited by 14
Guan, H., Bai, G., & Liu, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680383 · Cited by 17
Hao, S., Nan, Y., Zheng, Z., & Liu, X. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00142 · Cited by 11
The Correctness Illusion in LLM-Generated GPU KernelsPreprint
Sarkar, D. (2026) · arXiv.org · doi:10.48550/arxiv.2606.20128 · Cited by 16
“FreeFuzz [14], DocTer [15], DeepREL [2], and NablaFuzz [16] fuzz the API layer of PyTorch, TensorFlow, JAX, and OneFlow.”
Go, G., Zhou, C., Zhang, Q., Zou, X., Shi, H., & Jiang, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680364 · Cited by 12
UPBEAT: Test Input Checks of Q# Quantum Libraries
Hu, T., Ye, G., Tang, Z., Tan, S. H., Wang, H., Li, M., & Wang, Z. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652120 · Cited by 12
Practitioners’ Expectations on Automated Test Generation
Yu, X., Liu, L., Hu, X., Keung, J., Xia, X., & Lo, D. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680386 · Cited by 16
Improving Deep Learning Framework Testing with Model-Level Metamorphic Testing
Mu, Y., Zhai, J., Fang, C., Chen, X., Cao, Z., Yang, P., Zhao, K., Guo, A., & Chen, Z. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728972 · Cited by 12
D3: Differential Testing of Distributed Deep Learning With Model Generation
Wang, J., Pham, H. V., Li, Q., Tan, L., Guo, Y., Aziz, A., & Meijer, E. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3461657 · Cited by 15
“We compare D 3 with fifteen existing tools EAGLE [26], DocTor [47], FreeFuzz [29], ∇ Fuzz [48], FuzzGPT [49], TitanFuzz [50], DeepREL [30], CRADLE [25], Audee [27], LEMON [28], NeuRI [51], Muffin [46], Ramos [52], GenCoG [53], and HirGen [54].”
Self-citationMirrorFuzz: Leveraging LLM and Shared Bugs for Deep Learning Framework APIs Fuzzing
Ou, S., Li, Y., Yu, L., Wei, C., Wen, T., Chen, Q., Chen, Y., Tang, H., & Pan, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3619966 · Cited by 9
“The keyword list is compiled using three strategies: (1) extracting common terms from prior DL framework testing studies [24], [25], [26], [29], [30], (2) adopting the custom keyword set from Orion [32], and (3) manually adding framework-specific terms, including Chinese terms found in bug reports…”
InfluentialSelf-citationEvaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study
Harzevili, N. S., Wei, M., Mohajer, M. M., Pham, H. V., & Wang, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3729533 · Cited by 6
“…extensive empirical study to benchmark seven State-of-the-Art (SOTA) DL fuzzers, including FreeFuzz [77], DeepRel [17], NablaFuzz [83], DocTer [81], ACETest [63], TitanFuzz [15], and FuzzGPT [16], against real-world bugs from the two most popular and open-source DL libraries ( TensorFlow (TF)…”
InfluentialSelf-citationAn Empirical Study of Bugs in Data Visualization Libraries
Lu, W., Tian, Y., Zhong, X., Ma, H., Xu, Z., Cheung, S.-C., & Sun, C. P. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729363 · Cited by 7
Lightweight Concolic Testing via Path-Condition Synthesis for Deep Learning Libraries
Kim, S., Kim, Y., Park, D., Jeon, Y., Yi, J., & Kim, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00202 · Cited by 7
Self-citationWhat kinds of contracts do ML APIs need?
Khairunnesa, S. S., Ahmed, S., Imtiaz, S. M., Rajan, H., & Leavens, G. T. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10320-z · Cited by 5
“Recently, Xie et al. (2022) proposed a technique to extract DL-specific input constraints from API documentation and to test APIs guided by such input constraints.”
MethodMind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators
Wen, E., Ma, S., Tempero, E., Sham, C., Sun, Y., Jia, H., Luo, D., Hua, J., Dietrich, J. B., Zhao, K., et al. (2025) · International Conference on Mobility, Sensing and Networking (MSN) · doi:10.1109/msn69125.2025.00054 · Cited by 4
“DocTer [18] and ACETest [14] extract and analyze input constraints from API documentation and C++ implementation to generate more valid and targeted parameter inputs.”
Naziri, M. M. A., Singh, A. K., Wu, B., Qin, F., Dutta, S., & d’Amorim, M. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731739 · Cited by 4
“An impressive number of techniques have been recently proposed to find bugs in these libraries [1–3, 6, 7, 10, 11, 16, 17, 21, 22, 24], however we observe these techniques propose an independent evaluation methodology.”
MoCo: Fuzzing Deep Learning Libraries via Assembling Code
Ji, P., Feng, Y., Wu, D., Yan, L., Chen, P., Liu, J., & Zhao, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3509975 · Cited by 4
“FreeFuzz [6] and DocTer [47] are API-level fuzz testing methods that extract information from official documents and open-source code.”
COSTELLO: Contrastive Testing for Embedding-Based Large Language Model as a Service Embeddings
Jiang, W., Zhai, J., Ma, S., Zhang, X., & Shen, C. Q. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643767 · Cited by 5
Self-citationConstraint-Guided Unit Test Generation for Machine Learning Libraries
Krodinger, L., Hajdari, A., Lukasczyk, S., & Fraser, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-24839-8_1 · Cited by 3
Testing Deep Learning Libraries via Neurosymbolic Constraint Learning
Naziri, M. M. A., Kim, S., Qin, F., Dutta, S., & d’Amorim, M. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787800 · Cited by 5
“DocTer [55] complements this by extracting API constraints (e.g., types, tensor shapes) directly from technical documentation.”
InfluentialLLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer
Zhang, K., Xiao, D., Wu, D., Wang, S., Zhao, J., Lin, Y., Xu, T., & wang, S. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798258 · Cited by 3
“These bugs are typically easier to detect due to their conspicuous symptoms [Christou et al. 2023; Deng et al. 2023, 2022; Wei et al. 2022; Xie et al. 2022; Yang et al. 2023].”
A novel combinatorial testing approach with fuzzing strategy
Chen, J., Chen, J., Cai, S., Chen, H., & Zhang, C. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2537 · Cited by 3
Harnessing LLMs for Document-Guided Fuzzing of OpenCV Library
Duan, B., Mahmud, T., Che, M., Yan, Y., Dong, N., Kim, D. S., & Yang, G. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00017 · Cited by 4
“Notably, previous document-guided fuzzing methods [22], [23] only focused on well-documented APIs, since they can not generate valid test cases that lack details of input parameters.”
InfluentialS cuzer : A Scheduling Optimization Fuzzer for TVM
Chen, X., Lin, X., Wang, J., Sun, J., Wang, J., & Wang, W. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705308 · Cited by 6
“Though we use single-operator testing to help record operator-related information, not all operators are suitable for this method.”
Feng, S. M., Liu, J., Lai, R., Ruan, C. F., Yu, Y. H., Zhang, L., & Chen, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728957 · Cited by 3
Multigranularity Coverage Criteria for Deep Learning Libraries
Shi, Y., Zheng, Z., Yin, B., & Xi, Z. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1903 · Cited by 2
“Bugs on APIs in this subsection were collected from XieData [31], which is a bug report dataset involving 62 APIs with faults on TensorFlow and 20 APIs with faults on DL libraries.”
History-driven Compiler Fuzzing via Assembling and Scheduling Bug-triggering Code Segments
Fan, Z., Ye, G., Hu, T., & Tang, Z. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00040 · Cited by 3
“Meanwhile, the researchers generated API inputs through careful design [56; 57; 58], extracting constraints and characterization strings to express richer semantics.”
Automating fuzz driver generation for deep learning libraries with large language models
Zheng, T., Meng, F., Yi, P., & Wu, Y. (2026) · Cybersecurity · doi:10.1186/s42400-025-00532-9 · Cited by 2
“…2019; Guo et al. 2020; Wang et al. 2020) and API usage code snippets (Wei et al. 2022; Deng et al. 2022) followed by simplistic constraint analysis (Xie et al. 2022; Wei et al. 2022) and mutation (Guo et al. 2020; Wang et al. 2020; Wei et al. 2022), LLM-based methods inherently capture latent…”
Subgraph-Oriented Testing for Deep Learning Libraries
Xie, X., Song, Y., Chen, S., & Chen, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3655712 · Cited by 2
“Suppose that in test execution, the actual input takes the output of conv2d, which is a float32 tensor in the shape of [16, 64, 224, 224].”
InfluentialHunting CUDA Bugs at Scale with cuFuzz
Ziad, M. T. I., & Kozyrakis, C. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798231 · Cited by 2
“DL library fuzzers [4, 5, 10, 46, 47] primarily depend on differential testing to expose bugs in the DL kernel and API implementations running on CPUs and/or GPUs.”
Detecting Contradictions from CoAP RFC Based on Knowledge Graph
Feng, X., Zhang, Y., Meng, M. H., & Teo, S. G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-23020-2_10 · Cited by 3
“Dependency parsing is also present in other work such as [33].”
Survey on Fuzzing Techniques in Deep Learning Libraries
Ji, J., Kong, W., Tian, J., Gu, T., Nie, Y., & Kuang, X. (2023) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc59305.2023.00073 · Cited by 3
“1) In 2022, Xie et al [54] proposed DocTer, constructs rules to extract DL-specific input constraints from API doc-464 umentation and uses the constraints to generate valid/invalid inputs for testing DL libraries.”
MethodDirected Testing in MLIR: Unleashing Its Potential by Overcoming the Limitations of Random Fuzzing
Tong, W., Wang, Z., Tang, Z., Fang, J., Zhang, Y., & Ye, G. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729372 · Cited by 3
Citadel : Context Similarity Based Deep Learning Framework Bug Finding
Zhang, X., Zhai, J., Ma, S., Wang, S., & Shen, C. Q. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773993 · Cited by 2
“For example, DocTer [81] conducts fuzzing for DL frameworks by extracting input constraints from API documentation and using these constraints to guide test case generation.”
InfluentialSelf-citationCEDAR: Continuous Testing of Deep Learning Libraries
Xie, D., Wang, J., Pham, H. V., Tan, L., Guo, Y., Aziz, A., & Meijer, E. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00044 · Cited by 5
“…by CEDAR would have remained undiscovered until today ( t 3 ) if CEDAR had not been applied—-considering the challenge and the effectiveness of employing such tools to detect bugs in DL libraries [6], [7]—-the possible bug latency reduction (indicated by the orange dashed line) amounts to 744 days.”
BackgroundMethodInfluentialSelf-citationMeijer, W. (2024) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3652620.3688201 · Cited by 2
“Previous work used API documentation already to build basic contracts, like valid input ranges [50, 72].”
Quan, L., Xie, X., Guo, Q., Jiang, L., Chen, S., Wang, J., & Li, X. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714649 · Cited by 1
Fuzz Driver Generation: A Survey and Outlook from the Perspective of Data Sources
Feng, X., Lu, S., Gu, T., Nie, Y., Yan, Q., Yang, M., Chen, J., & Kuang, X. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10040129 · Cited by 1
How Effective Is Coverage-Guided Fuzzing to Test Deep Learning Library APIs?
Qin, F., Naziri, M. M. A., Ai, H., Dutta, S., & d’Amorim, M. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00023 · Cited by 1
“Frontend: Since Python lacks explicit types, tools like FreeFuzz [5] and DocTer [4] mine code snippets or documentation to infer argument constraints.”
A PSO-based Method to Test Deep Learning Library at API Level
Liao, S., & Shan, C. (2024) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3672758.3672777 · Cited by 1
Evaluating Spectrum-Based Fault Localization on Deep Learning Libraries
Yan, M., Chen, J., Jiang, T., Jiang, J., & Wang, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3552622 · Cited by 2
“Regarding fuzzer-generated test cases, we chose two state-of-the-art DL library test generation tools as the representatives, namely DocTer [4] and DeepRel [6].”
InfluentialFinding Compiler-Platform Interaction Bugs in Deep Learning Pipelines via Cross-Layer ConstraintsPreprint
Qiu, Y. L., Wang, J., Badhe, R., Limpanukorn, B., Kim, M., & Zhang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.18421 · Cited by 1
“Prior work has shown that DNN operator constraints can be mined from model specifications and documentation [Liu et al.(2023a), Xie et al.(2022)].”
Self-citationTest-Input Generation for Tensor Programs: What Actually Finds Kernel BugsPreprint
Sarkar, D. (2026) · arXiv.org · doi:10.48550/arxiv.2606.27396 · Cited by 3
“FreeFuzz [11], DocTer [12], DeepREL [1], and Nabla-Fuzz [13] all fuzz the API layer of TensorFlow, PyTorch, and JAX.”
Operator-Aware Mixed-Precision Tolerance Calibration for Tensor KernelsPreprint
Sarkar, D. (2026) · arXiv.org · doi:10.48550/arxiv.2607.16228 · Cited by 5
“FreeFuzz [10], DocTer [11], DeepREL [2], and Nabla-Fuzz [12] use cross-backend or metamorphic oracles to find logical bugs without per-op atol.”
Grounding Deep Learning Framework Testing in Data FlowPreprint
Ge, J., Mu, Y., Zou, Y., Hu, Y., Chen, Z., & Fang, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.15808 · Cited by 1
“Other methods generate or refine API tests under different inputs and execution contexts [9, 11, 14, 17, 37, 43, 54].”
Python Coverage Guided Fuzzing for Deep Learning Framework
Nie, Y., Xiao, X., Yang, B., Li, H., Luo, L., Yu, H., & Sun, G. (2024) · International Conference on Electronic Engineering and Information Systems (EEISS) · doi:10.1109/eeiss62553.2024.00007 · Cited by 2
Lightweight Code Coverage Analysis for Deep Learning Framework TestingPreprint
Li, S., Xiao, X., Yang, B., Li, J., Yu, H., Luo, L., Zhang, X., & Sun, G. (2024) · Research Square · doi:10.21203/rs.3.rs-4845487/v1
Self-citationJFuzzer: Detecting Optimization Errors in JavaScript Just-In-Time Compilers
Zhang, C., Chen, J., Cai, S., Chen, J., & Huang, C. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00054
“Almost all types of software can benefit from fuzzing, including but not limited to autonomous driving systems [22][23], distributed systems [24], smart contracts [25][26], kernels [27][28], compilers [29][30], deep learning libraries [31][32], and others [33][34].”
BackgroundImproving Performance of Commercially Available AI Products in a Multi-Agent ConfigurationPreprint
Hymel, C., Peng, S., Xu, K., & Ranganathan, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.22129 · Cited by 1
“Recent studies [24] have shown that using just a few prompts, LLMs were able to generate better extraction results than existing techniques such as Jdoctor [25] and DocTer [26].”
A Comprehensive Review on Deep Learning System Testing
Li, Y., Shan, C., Liu, Z., & Liao, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1548-3_12
Self-citationRIMFuzz: real-time impact-aware mutation for library API fuzzing
Wang, X., & Zhao, L. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00050-1 · Cited by 1
“…et al. 2021; Bulekov et al. 2023), Rust libraries (Jiang et al. 2021; Takashima et al. 2021), Machine learning libraries (Wang et al. 2020; Wei et al. 2022; Xie et al. 2022; Deng et al. 2023), RESTful API (Atlidakis et al. 2019; Liu et al. 2022), and even arbitrary functions (Liu et al. 2023), etc.”
XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries
Duan, B., Dong, R., Dong, N., Kim, D. S., & Yang, G. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00030 · Cited by 1
“Following prior studies [38], [10], we count bugs identified as instances of three oracles.”
Greybox Fuzzing and Its Applications
Rong, Y. (2024) · eScholarship (California Digital Library)
Chen, T., Wang, Z., Li, L., Li, D., Li, Z., Chang, X., Bian, P., Liang, G., Wang, Q., & Xie, T. (2025) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2025.6
“API fuzzing [16, 4, 15, 44, 13, 19, 28, 5], as a special category of fuzzing approaches [53, 5, 25], is widely used to exploit API vulnerabilities.”
Enhancing Test Case Generation for Fuzzing Deep Learning Libraries Using Few-Shot Learning
Yang, B., Li, S., Li, J., Du, K., Yu, H., Sun, J. X., & Luo, L. (2025) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc67331.2025.00089
“Free-Fuzz [15], Docter [16], and DeepRel [17] are mutated based on the interface function provided to the DL model, resulting in test cases that can be utilized during the testing process.”
Lightweight Code Coverage Analysis for Deep Learning Library Fuzzing
Li, S., Yang, B., Yu, H., & Luo, L. (2025) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc67331.2025.00090
“Recent research highlights the importance of coverage analysis in enhancing deep learning (DL) library fuzzing [1–7], focusing on two main criteria: architectural coverage and code coverage [8, 9].”
DeepLancet: Effectively Detecting Deep Learning Library Bugs via LLM-assisted Testcase Generation
Luo, Z., & Hua, B. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00349
“We conduct a comparative evaluation of D EEP L ANCET against DocTer [16], regarding valid test case generation.”
InfluentialTesting Deep Learning Libraries with Semantic Equivalent API Patterns
Song, Y., Chen, S., Peng, H., & Xie, X. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00031
“The test inputs are generated using DocTer [15], a tool that automatically extracts parameter constraints from API documentation.”
CIPIHunter: Detecting Configuration-Induced Prediction Instability in Deep Learning Frameworks
Anonymous, Meng, S., Kim, M., Chen, X., Fang, C., Chen, Z., & Zhai, J. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834345
Self-citationRiskFuzz: Risk-Guided Fuzzing for Deep Learning Libraries
Pan, Z., Chen, X., Zhang, Z., Chen, L., & Shi, G. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11465052
NSGen: A Template-Based Framework to Find Bugs in Computer-Aided Engineering Tools
Li, X., Zou, P., Liu, H., Guo, S., Ren, Z., & Jiang, H. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3687867
“For DL API testing, Xie et al. [51] present DocTer, a documentation-guided fuzzer that extracts API constraints from natural-language documentation to generate valid yet diverse test inputs.”
Improving Deep Learning Library Testing with Machine Learning
Molina, F., Naziri, M. M. A., Qin, F., Gorla, A., & d’Amorim, M. (2026) · doi:10.1145/3793654.3793744
DECODE: Dynamic Exploration for Constraint-Guided Vulnerability Discovery in Deep Learning Operators
Liu, H., Wang, Z., Liu, Z., & Li, W. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808174
CuFuzz: An API-Knowledge-Graph Coverage-Driven Fuzzing Framework for CUDA Libraries
Fan, X., Fang, Y., Jia, P., Liu, Y., Xu, Y., Peng, X., & Zhou, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808170
D3Fuzz: Dynamic, directed and decaying LLM-driven fuzzing for deep learning libraries
Sun, J., Dong, J., Zhang, H., Tian, Z., & Zhang, Y. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108289
Generics-Aware Fuzz Target Generation for Rust Libraries via Structured API AnalysisPreprint
Chen, Y., Zhang, K., GuanJun, L. I. U., Li, X., & Dong, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.08637
“Several automated approaches instead generate fuzz targets by analyzing API signatures or extracting call sequences from documentation and examples [16, 35, 42].”
Liu, C., Song, Y., Fonseca, P., & Zhang, Y. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843891
“Fuzzing techniques have recently achieved great success in testing system software [9, 13, 19, 24, 25, 52, 66, 73, 80], leading to growing interest in adopting feedback-based fuzzing to test distributed systems [15, 57, 58, 86].”
Liu, S., Sun, C., Wu, F., & Chan, W. K. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832194
Fuzzing the Boundary between Models and Code in Hybrid AI-Enabled Systems
Gao, X., Feng, Y., Lu, Y., Liu, Z., Chen, Z., & Xu, B. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832205
“To improve the overall quality of complex software systems, researchers in software engineering have proposed various quality assurance techniques, such as testing [31, 63, 68] and debugging [14, 61].”
Acquiring and Verifying Repository Norms for Coding AgentsPreprint
He, K., Zhang, X., Chen, Z., Treude, C., & Liu, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.07757
“Other methods enable specification mining from abstracted symbolic traces [9] or extract input constraints from API documentation [31].”
Self-citation
Metadata
- Authors
- Xie, D., Li, Y., Kim, M., Pham, H. V., Tan, L., Zhang, X., & Godfrey, M. W.
- Affiliations
- Purdue University West Lafayette (US)
- University of Waterloo (CA)
- Ulsan National Institute of Science and Technology (KR)
- Venue
- Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2022-07-18
- Pages
- 176-188
- OpenAlex
- W4285490440
- Cited by
- 81
- References
- 54
- Citations by year
- 2026: 14 · 2025: 27 · 2024: 23 · 2023: 15 · 2022: 2
- FWCI
- 12.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Scholarworks@UNIST (Ulsan National Institute of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1901242)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- deep learning libraries, API testing, input constraints, constraint extraction, API documentation, software testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.