Fuzzing deep-learning libraries via automated relational API inference
Open accessThe DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Abstract
Deep Learning (DL) has gained wide attention in recent years. Meanwhile, bugs in DL systems can lead to serious consequences, and may even threaten human lives. As a result, a growing body of research has been dedicated to DL model testing. However, there is still limited work on testing DL libraries, e.g., PyTorch and TensorFlow, which serve as the foundations for building, training, and running DL models. Prior work on fuzzing DL libraries can only generate tests for APIs which have been invoked by documentation examples, developer tests, or DL models, leaving a large number of APIs untested. In this paper, we propose DeepREL, the first approach to automatically inferring relational APIs for more effective DL library fuzzing. Our basic hypothesis is that for a DL library under test, there may exist a number of APIs sharing similar input parameters and outputs; in this way, we can easily “borrow” test inputs from invoked APIs to test other relational APIs. Furthermore, we formalize the notion of value equivalence and status equivalence for relational APIs to serve as the oracle for effective bug finding. We have implemented DeepREL as a fully automated end-to-end relational API inference and fuzzing technique for DL libraries, which 1) automatically infers potential API relations based on API syntactic/semantic information, 2) synthesizes concrete test programs for invoking relational APIs, 3) validates the inferred relational APIs via representative test inputs, and finally 4) performs fuzzing on the verified relational APIs to find potential inconsistencies. Our evaluation on two of the most popular DL libraries, PyTorch and TensorFlow, demonstrates that DeepREL can cover 157% more APIs than state-of-the-art FreeFuzz. To date, DeepREL has detected 162 bugs in total, with 106 already confirmed by the developers as previously unknown bugs. Surprisingly, DeepREL has detected 13.5% of the high-priority bugs for the entire PyTorch issue-tracking system in a three-month period. Also, besides the 162 code bugs, we have also detected 14 documentation bugs (all confirmed).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous API-level fuzzers [16,72] only focus on fuzzing each single DL library API in isolation.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [16]
In the citing paper
“The inspiration comes from prior work [16], which shows that it is common in DL libraries for related APIs to share the same input, and borrowing inputs from one API can help trigger bugs in its relational APIs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [16]
In the citing paper
“As such, previous work cannot generate valid inputs to cover this API since they rely on known valid input/API pairs obtained from databases created by scraping open-source code [16,72].”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [16]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundSentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1×
“Next, we use SentenceBERT [58] to encode these informative description sentences into semantically meaningful sentence embeddings.”
MethodSpeech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundDeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 1×
“To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
Algorithms for the Assignment and Transportation Problems
Munkres, J. (1957) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0105003 · Cited by 3,986
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
Akhtar, N., & Mian, A. (2018) · IEEE Access · doi:10.1109/access.2018.2807385 · Cited by 2,021
Deep Learning for Health Informatics
Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G.-Z. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2636665 · Cited by 2,000 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundDeep learning for detecting robotic grasps
Lenz, I., Lee, H., & Saxena, A. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914549607 · Cited by 1,634 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundDeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 1×
“To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [15, 20, 28, 42, 46, 50], metrics for model testing [32, 35, 41, 52, 75], applicationspecific model testing [61, 83, 86], and verifying certain properties of models [16, 39].”
Method
Research works printed but not linked: 12
- Martín Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , and Michael Isard . 2016 . Tensorflow: A system for large-scale machine learning. In 12th $USENIX$ symposium on operating systems design and implementation ($OSDI$ 16). 265–283. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. 2016. Tensorflow: A system for large-scale machine learning. In 12th $USENIX$ symposium on operating systems design and implementation ($OSDI$ 16). 265–283.
- Nicholas Carlini Anish Athalye Nicolas Papernot Wieland Brendel Jonas Rauber Dimitris Tsipras Ian Goodfellow Aleksander Madry and Alexey Kurakin. 2019. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705. Nicholas Carlini Anish Athalye Nicolas Papernot Wieland Brendel Jonas Rauber Dimitris Tsipras Ian Goodfellow Aleksander Madry and Alexey Kurakin. 2019. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705. · arXiv
- Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543. Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543. · arXiv
- Felix A Gers , Jürgen Schmidhuber , and Fred Cummins . 2000. Learning to forget: Continual prediction with LSTM. Neural computation, 12, 10 ( 2000 ), 2451–2471. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural computation, 12, 10 (2000), 2451–2471.
- Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. · arXiv
- Changliu Liu Tomer Arnon Christopher Lazarus Christopher Strong Clark Barrett and Mykel J Kochenderfer. 2019. Algorithms for verifying deep neural networks. arXiv preprint arXiv:1903.06758. Changliu Liu Tomer Arnon Christopher Lazarus Christopher Strong Clark Barrett and Mykel J Kochenderfer. 2019. Algorithms for verifying deep neural networks. arXiv preprint arXiv:1903.06758. · arXiv
- Jiawei Liu Jinkun Lin Fabian Ruffy Cheng Tan Jinyang Li Aurojit Panda and Lingming Zhang. 2022. Finding Deep-Learning Compilation Bugs with NNSmith. arXiv preprint arXiv:2207.13066. Jiawei Liu Jinkun Lin Fabian Ruffy Cheng Tan Jinyang Li Aurojit Panda and Lingming Zhang. 2022. Finding Deep-Learning Compilation Bugs with NNSmith. arXiv preprint arXiv:2207.13066. · arXiv
- Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. · arXiv
- Seonwoo Min , Byunghan Lee , and Sungroh Yoon . 2017. Deep learning in bioinformatics. Briefings in bioinformatics, 18, 5 ( 2017 ), 851–869. Seonwoo Min, Byunghan Lee, and Sungroh Yoon. 2017. Deep learning in bioinformatics. Briefings in bioinformatics, 18, 5 (2017), 851–869.
- Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , and Luca Antiga . 2019 . Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32 (2019), 8026–8037. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca Antiga. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32 (2019), 8026–8037.
- Martin F Porter . 2001 . Snowball: A language for stemming algorithms. Martin F Porter. 2001. Snowball: A language for stemming algorithms.
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. · arXiv
Cited by
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
Self-citationDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623343 · Cited by 115
Self-citationUnifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
Zhang, Z., Chen, C., Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., & Wang, R. (2024) · Transactions on machine learning research · doi:10.13140/rg.2.2.32710.69440/1 · Cited by 195
NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers
Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., & Zhang, L. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575707 · Cited by 167
“More recently, DeepREL [15] aims to automatically infer relational APIs (e.g., APIs that return the same values/statuses when given the same inputs), and then leverages such API relations as test oracle for catching more bugs than FreeFuzz.”
Self-citationLarge Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTPreprint
Deng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02014 · Cited by 131
“On the other hand, API-level fuzzers focus on testing each individual API via effective input generation [71, 75] or oracle inference [15, 77].”
BackgroundMethodInfluentialSelf-citationLiu, Z., Chen, C., Wang, J., Chen, M., Wu, B., Tian, Z., Huang, Y., Hu, J., & Wang, Q. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639118 · Cited by 58
Fuzzing Automatic Differentiation in Deep-Learning Libraries
Yang, C., Deng, Y., Yao, J., Tu, Y., Li, H., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00105 · Cited by 77
“, DeepREL [53] automatically infers relational APIs (e.”
InfluentialSelf-citationNeuRI: Diversifying DNN Generation via Inductive Rule Inference
Liu, J., Peng, J., Wang, Y., & Zhang, L. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616337 · Cited by 47
“API relation [12] and gradient checking [50].”
BackgroundInfluentialSelf-citationOn the Challenges of Fuzzing Techniques via Large Language Models
Huang, L., Zhao, P., Ma, L., & Chen, H. (2025) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse67621.2025.00028 · Cited by 43
“In terms of overall code, TitanFuzz achieved 20.98% and 39.97% code coverage on PyTorch and TensorFlow respectively, significantly exceeding DeepREL [61] and Muffin.”
InfluentialACETest: Automated Constraint Extraction for Testing Deep Learning Operators
Shi, J., Xiao, Y., Li, Y., Li, Y., Yu, D., Yu, C., Su, H., Chen, Y., & Huo, W. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598088 · Cited by 42
“To save manual e ort, FreeFuzz [46] proposes to mine test cases from open source and DeepREL [10] leverages existing test cases to test more APIs by inferring relational APIs.”
BackgroundMethodInfluentialDeep Learning Library Testing: Definition, Methods and Challenges
Zhang, X., Jiang, W., Shen, C. Q., Li, Q. L., Wang, Q., Lin, C., & Guan, X. (2025) · ACM Computing Surveys · doi:10.1145/3716497 · Cited by 34
“Deng et al. [21] went one step further on the prior work and designed two elaborated equivalence (i.e., status equivalence and value equivalence) and matched thousands of API pairs based on these equivalence relations.”
InfluentialEnhancing Differential Testing with LLMs for Testing Deep Learning Libraries
Li, M., Li, D., Liu, J., Cao, J., Tian, Y., & Cheung, S.-C. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735637 · Cited by 28
“LLMs can make mistakes, while it is non-trivial to distinguish valid ones from invalid ones due to diverse implementations.”
InfluentialYour Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 26
Testing Graph Database Systems via Equivalent Query Rewriting
Mang, Q., Fang, A., Yu, B., Chen, H., & He, P. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639200 · Cited by 18
History-Driven Fuzzing for Deep Learning Libraries
Harzevili, N. S., Mohajer, M. M., Wei, M., Pham, H. V., & Wang, S. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688838 · Cited by 20
Li, Z., Wu, J., Ling, X., Luo, T., Rui, Z., & Wu, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00132 · Cited by 18
A Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators
Chen, J., Jia, C., Yan, Y., Ge, J., Zheng, H., & Cheng, Y. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660796 · Cited by 16
“These errors significantly impact the DL models built on top of them [9], making comprehensive testing of DL libraries essential before model deployment to reduce the risks.”
A Tale of Two DL Cities: When Library Tests Meet Compiler
Shen, Q., Tian, Y., Ma, H., Chen, J., Huang, L., Fu, R., Cheung, S.-C., & Wang, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00025 · Cited by 18
“We used the implementations of DocTer and DeepREL released by their works [15], [16].”
InfluentialTest & Evaluation Best Practices for Machine Learning-Enabled SystemsPreprint
Chandrasekaran, J., Cody, T., McCarthy, N., Lanus, E., & Freeman, L. E. B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4862589 · Cited by 14
CrossProbe: LLM-Empowered Cross-Project Bug Detection for Deep Learning Frameworks
Guan, H., Bai, G., & Liu, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728984 · Cited by 15
“Due to such challenges, existing approaches either restrict the searching within specific DL models [15, 16, 41, 48, 49] or focus on a partial subset of APIs [9, 11, 21].”
Deep learning framework testing via hierarchical and heuristic model generation
Zou, Y., Sun, H., Fang, C., Liu, J., & Zhang, Z. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111681 · Cited by 14
Guan, H., Bai, G., & Liu, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680383 · Cited by 17
“Indeed, existing approaches for testing DL libraries primarily focus on either fuzzing DL models [17, 19, 49, 57, 59] or evaluating the usability of library APIs [11, 13, 27, 42].”
AQUA: Analytics-driven quantum neural network (QNN) user assistance for software validation
Park, S., Baek, H., Yoon, J., Lee, Y. K., & Kim, J. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.05.047 · Cited by 15
Enhancing ROS System Fuzzing through Callback Tracing
Shen, Y., Liu, J., Xu, Y., Sun, H., Wang, M., Guan, N., Shi, H., & Jiang, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652111 · Cited by 11
“Specifically, fuzz testing (fuzzing) [3, 7, 10, 11, 16, 24, 27, 28, 31, 37, 38, 40–42], has been particularly effective in bug detection.”
The Correctness Illusion in LLM-Generated GPU KernelsPreprint
Sarkar, D. (2026) · arXiv.org · doi:10.48550/arxiv.2606.20128 · Cited by 16
“FreeFuzz [14], DocTer [15], DeepREL [2], and NablaFuzz [16] fuzz the API layer of PyTorch, TensorFlow, JAX, and OneFlow.”
Go, G., Zhou, C., Zhang, Q., Zou, X., Shi, H., & Jiang, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680364 · Cited by 12
Practitioners’ Expectations on Automated Test Generation
Yu, X., Liu, L., Hu, X., Keung, J., Xia, X., & Lo, D. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680386 · Cited by 16
Improving Deep Learning Framework Testing with Model-Level Metamorphic Testing
Mu, Y., Zhai, J., Fang, C., Chen, X., Cao, Z., Yang, P., Zhao, K., Guo, A., & Chen, Z. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728972 · Cited by 12
“Until now, researchers have proposed different DL framework testing methods [14, 19, 24, 36, 37, 47, 51, 59–61 Researchers often use DL models [24, 36, 47, 59] or single framework interfaces [19, 60, 62, 63, 67] as inputs to test DL frameworks.”
D3: Differential Testing of Distributed Deep Learning With Model Generation
Wang, J., Pham, H. V., Li, Q., Tan, L., Guo, Y., Aziz, A., & Meijer, E. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3461657 · Cited by 15
“Existing techniques address this challenge by cross-checking different libraries or execution graphs to detect inconsistency bugs [25], [26], [27], [28], [29], [30].”
TorchProbe: Fuzzing Dynamic Deep Learning Compilers
Su, Q., Geng, C., Pekhimenko, G. G., & Si, X. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8311-7_15 · Cited by 9
Balancing Effectiveness and Flakiness of Non-Deterministic Machine Learning Tests
Xia, C. S., Dutta, S., Misailovíc, S., Marinov, D., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00154 · Cited by 10
“Recently, researchers have proposed approaches for testing machine learning frameworks [92], [93], [94], [95], [96], [97], [98], [99], [100], [101], [102], probabilistic programming systems [37], [103], [104], and randomized algorithms [105].”
BackgroundSelf-citationA survey of coverage-guided greybox fuzzing with deep neural models
Qiu, J., Jiang, Y., Miao, Y., Luo, W., Pan, L., & Zheng, X. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107797 · Cited by 8
MirrorFuzz: Leveraging LLM and Shared Bugs for Deep Learning Framework APIs Fuzzing
Ou, S., Li, Y., Yu, L., Wei, C., Wen, T., Chen, Q., Chen, Y., Tang, H., & Pan, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3619966 · Cited by 9
“Following prior work on fuzzing DL Frameworks [24], [25], [30], [31], we report the number of detected bugs.”
InfluentialDeepDiffer: Find Deep Learning Compiler Bugs via Priority-guided Differential Fuzzing
Lin, K., Song, X., Zeng, Y., & Guo, S. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00066 · Cited by 12
“Some techniques target fuzzing DL operators but they may struggle to detect bugs in high-level optimizations [40], such as FreeFuzz [42], ACETest [43], etc.”
MethodMutation-Based Deep Learning Framework Testing Method in JavaScript Environment
Zou, Y., Zhai, J., Fang, C., Liu, J., Zheng, T., & Chen, Z. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695478 · Cited by 7
Evaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study
Harzevili, N. S., Wei, M., Mohajer, M. M., Pham, H. V., & Wang, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3729533 · Cited by 6
“The focus of our study is to evaluate recent SOTA API-level DL fuzzers, i.e., FreeFuzz [77], DeepRel [17], NablaFuzz [83], DocTer [81], ACETest [63], TitanFuzz [15], and FuzzGPT [16], which are publicly available.”
InfluentialCan Machine Learning Pipelines Be Better Configured?
Wang, Y., Wang, Y., Zhang, T., Yu, Y., Cheung, S.-C., Yu, H., & Zhu, Z. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616352 · Cited by 8
“Existing and new testing techniques [33, 37, 39, 45, 45, 48, 51, 52, 59, 60, 62, 65, 69– 71, 75, 77, 78, 81, 82, 86] were also proposed for and adapted to machine learning applications to expose real bug.”
MethodMetamong: Detecting Render-Update Bugs in Web Browsers through Fuzzing
Song, S., & Lee, B. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616336 · Cited by 5
“Some studies leveraged the fuzzing to find the semantic bugs from the deep learning libraries [32, 51].”
BackgroundJanus: Detecting Rendering Bugs in Web Browsers via Visual Delta Consistency
Zhou, C., Zhang, Q., Qian, B., & Jiang, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00013 · Cited by 8
Lightweight Concolic Testing via Path-Condition Synthesis for Deep Learning Libraries
Kim, S., Kim, Y., Park, D., Jeon, Y., Yi, J., & Kim, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00202 · Cited by 7
A Systematic Survey on Debugging Techniques for Machine Learning SystemsPreprint
Nguyen, T.-D., Tian, H., Le, B., Thongtanunam, P., & McIntosh, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.03158 · Cited by 7
“DeepREL [94], constructs a large dataset of corresponding API pairs between major deep learning frameworks like PyTorch and TensorFlow.”
Fuzz4Cuda: Fuzzing your NVIDIA GPU libraries through debug interface
Zhou, Y., Jia, P., Liu, J., & Fan, X. (2025) · Computers & Security · doi:10.1016/j.cose.2025.104754 · Cited by 6
Mind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators
Wen, E., Ma, S., Tempero, E., Sham, C., Sun, Y., Jia, H., Luo, D., Hua, J., Dietrich, J. B., Zhao, K., et al. (2025) · International Conference on Mobility, Sensing and Networking (MSN) · doi:10.1109/msn69125.2025.00054 · Cited by 4
“DeepREL [3] and TENSORSCOPE [4] identify related APIs within a library and corresponding APIs across different libraries.”
A lightweight adaptive random testing method for deep learning systems
Mao, C., Song, Y., & Chen, J. (2023) · Software Practice and Experience · doi:10.1002/spe.3256 · Cited by 4
“However, a number of studies have shown that there are a number of vulnerabilities in some mainstream frameworks of DL or their applications, 30-32 and these potential faults bring a potential threat to the reliability of DL systems.”
BackgroundNaziri, M. M. A., Singh, A. K., Wu, B., Qin, F., Dutta, S., & d’Amorim, M. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731739 · Cited by 4
MoCo: Fuzzing Deep Learning Libraries via Assembling Code
Ji, P., Feng, Y., Wu, D., Yan, L., Chen, P., Liu, J., & Zhao, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3509975 · Cited by 4
“Deng et al. propose DeepREL, which infers relational APIs and compares their output values or execution states in order to identify inconsistencies [7].”
InfluentialKLRAG: Deep Learning Library Vulnerability Detection via Knowledge-Level RAG
Liao, R., Yan, X., & Zhu, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9958-2_36 · Cited by 4
LMFuzz: Program repair fuzzing based on large language models
Lin, R., Wang, R., Hu, G., & Xu, X. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00568-8 · Cited by 3
“Compared to the traditional method DeepREL (Deng et al. 2022), TitanFuzz improves coverage by 50.84% and 30.38% for PyTorch and TensorFlow, respectively.”
InfluentialTesting Deep Learning Libraries via Neurosymbolic Constraint Learning
Naziri, M. M. A., Kim, S., Qin, F., Dutta, S., & d’Amorim, M. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787800 · Cited by 5
“To improve testing efficiency, DeepREL [12] identifies APIs with similar behaviors to share test inputs across them, increasing coverage.”
InfluentialLLM-Powered Silent Bug Fuzzing in Deep Learning Libraries via Versatile and Controlled Bug Transfer
Zhang, K., Xiao, D., Wu, D., Wang, S., Zhao, J., Lin, Y., Xu, T., & wang, S. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798258 · Cited by 3
“These bugs are typically easier to detect due to their conspicuous symptoms [Christou et al. 2023; Deng et al. 2023, 2022; Wei et al. 2022; Xie et al. 2022; Yang et al. 2023].”
Harnessing LLMs for Document-Guided Fuzzing of OpenCV Library
Duan, B., Mahmud, T., Che, M., Yan, Y., Dong, N., Kim, D. S., & Yang, G. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00017 · Cited by 4
“It has been widely used in various domains, including operating systems [60], network protocols [61], web applications [62], and APIs [63].”
Why Do Deep Learning Projects Differ in Compatible Framework Versions? An Exploratory Study
Lei, H., Zhang, S., Wang, J., Xiao, G., Liu, Y., & Sui, Y. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00076 · Cited by 3
“For example, according to the Python-PyPI repository [11], TensorFlow [9] and PyTorch [10], two of the most popular DL frameworks [12], release an average of 13 and 5 versions per year, respectively (as of December 2022).”
BackgroundMethodS cuzer : A Scheduling Optimization Fuzzer for TVM
Chen, X., Lin, X., Wang, J., Sun, J., Wang, J., & Wang, W. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705308 · Cited by 6
“Though we use single-operator testing to help record operator-related information, not all operators are suitable for this method.”
Feng, S. M., Liu, J., Lai, R., Ruan, C. F., Yu, Y. H., Zhang, L., & Chen, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728957 · Cited by 3
Self-citationMultigranularity Coverage Criteria for Deep Learning Libraries
Shi, Y., Zheng, Z., Yin, B., & Xi, Z. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1903 · Cited by 2
Scalpel: Automotive Deep Learning Framework Testing via Assembling Model Components
Zou, Y., Zhai, J., Fang, C., Guo, A., Liu, J., & Chen, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773242 · Cited by 2
Automating fuzz driver generation for deep learning libraries with large language models
Zheng, T., Meng, F., Yi, P., & Wu, Y. (2026) · Cybersecurity · doi:10.1186/s42400-025-00532-9 · Cited by 2
“In addition, we also compare our approach with other SOTA methods, i.e., Muffin (Gu et al. 2022), FreeFuzz (Wei et al. 2022), DeepREL (Deng et al. 2022), and ∇ Fuzz (Yang et al. 2023), that specifically designed for fuzz-ing DL library APIs to further evaluate the effectiveness of our approach in…”
InfluentialSubgraph-Oriented Testing for Deep Learning Libraries
Xie, X., Song, Y., Chen, S., & Chen, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3655712 · Cited by 2
“Following the common practice in DL testing studies [14], [16], [17], [18], [19], we execute multiple test runs, aggregate the results, and then deduplicate them, as different test cases may correspond to the same underlying bug.”
InfluentialSurvey on Fuzzing Techniques in Deep Learning Libraries
Ji, J., Kong, W., Tian, J., Gu, T., Nie, Y., & Kuang, X. (2023) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc59305.2023.00073 · Cited by 3
“3) In 2022, Deng et al [17] in order to test relational APIs to further overcome limitations of Freefuzz, build DeepREL upon FreeFuzz to automatically infer relational APIs and leverage them to fuzz DL libraries.DeepREL first automatically infers all possible candidate matched API pairs based on API…”
BackgroundDirected Testing in MLIR: Unleashing Its Potential by Overcoming the Limitations of Random Fuzzing
Tong, W., Wang, Z., Tang, Z., Fang, J., Zhang, Y., & Ye, G. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729372 · Cited by 2
Citadel : Context Similarity Based Deep Learning Framework Bug Finding
Zhang, X., Zhai, J., Ma, S., Wang, S., & Shen, C. Q. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773993 · Cited by 2
“DeepREL [33] identifies relational API functions of DL frameworks and ‘borrows’ test inputs from invoked API functions to test other relational API functions.”
InfluentialEvaluating the Effectiveness of Coverage-Guided Fuzzing for Testing Deep Learning Library APIsPreprint
Qin, F., Naziri, M. M. A., Ai, H., Dutta, S., & d’Amorim, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.14626 · Cited by 2
“Model-based fuzzing [23, 24, 35, 36, 42, 49] and API-level fuzzing [11, 13, 16, 28, 50] are two complementary categories of bug finding techniques for DL library testing.”
InfluentialFast Constraint Synthesis for C++ Function Templates
Ding, S., & Zhang, Q. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3720422 · Cited by 1
Deep Learning Framework Testing via Model Mutation: How Far Are We?
Mu, Y., Wang, R., Zhai, J., Fang, C., Chen, X., Yang, P., Cao, Z., Qian, R., Yang, S., & Chen, Z. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3651022 · Cited by 2
“Deng et al. [12] test interfaces with similar input and output parameters by inferring semantic relationships and comparing execution states.”
TDiFf: Detecting Bugs in DataFrame Systems via Transferred DBMS Test Cases
Hu, J., Chen, S., & Wu, R. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837548 · Cited by 1
How Effective Is Coverage-Guided Fuzzing to Test Deep Learning Library APIs?
Qin, F., Naziri, M. M. A., Ai, H., Dutta, S., & d’Amorim, M. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00023 · Cited by 1
“DeepREL [13] leverages API signature similarities to maximize coverage.”
A PSO-based Method to Test Deep Learning Library at API Level
Liao, S., & Shan, C. (2024) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3672758.3672777 · Cited by 1
“Code coverage is a widely used test adequacy metric in current testing methods [ 21 - 23 ] .”
InfluentialDetecting Compatibility Issues for Machine Learning Libraries
Wang, Y., Cheung, S.-C., Yu, H., & Zhu, Z. (2025) · Springer eBooks · doi:10.1007/978-981-96-1797-5_4 · Cited by 1
Evaluating Spectrum-Based Fault Localization on Deep Learning Libraries
Yan, M., Chen, J., Jiang, T., Jiang, J., & Wang, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3552622 · Cited by 2
“Regarding fuzzer-generated test cases, we chose two state-of-the-art DL library test generation tools as the representatives, namely DocTer [4] and DeepRel [6].”
InfluentialOn the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations
Hundal, R. S., Xiao, Y., Cao, X., Dong, J. S., & Rigger, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00222 · Cited by 2
“Differential testing is primarily used as a black-box testing technique and thus, widely applicable for systems with neural networks [37]–[41].”
Tensor Algebraic Property Skeletons: Amplifying Property-Based Testing for AI CompilersPreprint
Qiu, Y., Limpanukorn, B., Lee, S., Wang, J., Zhang, Q., & Kim, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.06747 · Cited by 1
“Existing work has therefore made substantial progress on this well-formedness problem Liu et al. (2023a;b); Wang et al. (2023); Deng et al. (2022); Ma et al. (2023); Mu et al. (2025); Limpanukorn et al. (2025).”
Finding Compiler-Platform Interaction Bugs in Deep Learning Pipelines via Cross-Layer ConstraintsPreprint
Qiu, Y. L., Wang, J., Badhe, R., Limpanukorn, B., Kim, M., & Zhang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.18421 · Cited by 1
“Next, they detect issues via coarse pass/fail signals [Liu et al.(2023a), Liu et al.(2023b), Deng et al.(2022)] or differential mismatches [Ma et al.(2023), Mu et al.(2025), Zhou et al.(2024)].”
InfluentialTest-Input Generation for Tensor Programs: What Actually Finds Kernel BugsPreprint
Sarkar, D. (2026) · arXiv.org · doi:10.48550/arxiv.2606.27396 · Cited by 3
Operator-Aware Mixed-Precision Tolerance Calibration for Tensor KernelsPreprint
Sarkar, D. (2026) · arXiv.org · doi:10.48550/arxiv.2607.16228 · Cited by 5
Source File Tracking Localization: A Fault Localization Method for Deep Learning Frameworks
Ma, Z., Yang, B., & Zhang, Y. (2023) · Electronics · doi:10.3390/electronics12224579
Python Coverage Guided Fuzzing for Deep Learning Framework
Nie, Y., Xiao, X., Yang, B., Li, H., Luo, L., Yu, H., & Sun, G. (2024) · International Conference on Electronic Engineering and Information Systems (EEISS) · doi:10.1109/eeiss62553.2024.00007 · Cited by 2
Lightweight Code Coverage Analysis for Deep Learning Framework TestingPreprint
Li, S., Xiao, X., Yang, B., Li, J., Yu, H., Luo, L., Zhang, X., & Sun, G. (2024) · Research Square · doi:10.21203/rs.3.rs-4845487/v1
CGFuzz: A Dynamic Test Case Generation Method for DL Framework Based on Function Coverage
Qing, C., Yin, B., & Shi, J.-A. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00070
“Component testing, on the other hand, directly tests the framework as a test unit using the provided API[13][14][15] or operator[16][17][18].”
MethodMagnifier: A Chiplet Feature-Aware Test Case Generation Method for Deep Learning Accelerators
Li, B., Zhu, Z., Liu, W., Cao, Q., Li, C., Ji, C., Li, X., & Zhou, X. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3528358 · Cited by 1
“DeepREL [14] targets APIs with many shared similar input parameters and outputs, using automatic inference relations for end-to-end testing of APIs.”
A Comprehensive Review on Deep Learning System Testing
Li, Y., Shan, C., Liu, Z., & Liao, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1548-3_12
FuMi: A Runtime Fuzz-based Machine Learning Precision Measurement and Testing Framework
Zhang, P., Papadakis, M., & Zhou, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3734866
XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries
Duan, B., Dong, R., Dong, N., Kim, D. S., & Yang, G. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00030 · Cited by 1
“Existing approaches consist of fuzzing for deep learning frameworks[8], compilers[37], computational graphs[33], and APIs[12].”
Greybox Fuzzing and Its Applications
Rong, Y. (2024) · eScholarship (California Digital Library)
Pool++: Enhancing Unit Test Coverage Through Audio Data and Mock Generation
Ferreira, P. P. (2025) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
DT4LM: Differential Testing for Reliable Language Model Updates in Classification Tasks
Zuo, X., Xiao, Y., Cao, X., Wang, W., & Dong, J. S. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3622251 · Cited by 1
“Differential testing is a well-established technique in deep neural network testing [35], [36], [37], aiming to identify and generate differential inputs that expose discrepancies among multiple models.”
Chen, T., Wang, Z., Li, L., Li, D., Li, Z., Chang, X., Bian, P., Liang, G., Wang, Q., & Xie, T. (2025) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2025.6
“API fuzzing [16, 4, 15, 44, 13, 19, 28, 5], as a special category of fuzzing approaches [53, 5, 25], is widely used to exploit API vulnerabilities.”
Enhancing Test Case Generation for Fuzzing Deep Learning Libraries Using Few-Shot Learning
Yang, B., Li, S., Li, J., Du, K., Yu, H., Sun, J. X., & Luo, L. (2025) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc67331.2025.00089
“Free-Fuzz [15], Docter [16], and DeepRel [17] are mutated based on the interface function provided to the DL model, resulting in test cases that can be utilized during the testing process.”
A Multi-Agent Fuzzing Framework for Deep Learning Library
Liao, R., Ou, S., Yan, X., & Zhu, K.-L. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11343067
“DL libraries fuzzing is mainly done by generating many seeds, which can be classified into two categories depending on the seed type: model-level fuzzing[6], [14] and API-level fuzzing[15], [5].”
DeepLancet: Effectively Detecting Deep Learning Library Bugs via LLM-assisted Testcase Generation
Luo, Z., & Hua, B. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00349
“Third, PyTorch has undergone extensive testing and quality assurance [13], [29], [30] , including unit tests, integration tests, and contributions from a large open-source community.”
Testing Deep Learning Libraries with Semantic Equivalent API Patterns
Song, Y., Chen, S., Peng, H., & Xie, X. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00031
“To mitigate these limitations, Deng et al. [33] proposed DeepREL, which can only test APIs for which there are one-to-one mapped equivalent counterparts in the same DL library.”
InfluentialRiskFuzz: Risk-Guided Fuzzing for Deep Learning Libraries
Pan, Z., Chen, X., Zhang, Z., Chen, L., & Shi, G. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11465052
SemuFuzz: An efficient fuzzing approach for deep learning libraries based on semantic representation
Chen, Z., Chen, J., Cai, S., Chen, J., & Gu, W. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112999
He, Z., & Han, X. (2026) · International Symposium on Robotics, Artificial Intelligence and Information Engineering (RAIIE) · doi:10.1109/raiie70320.2026.11567007
“This metric quantifies how many library APIs (particularly externally exposed interfaces) are exercised during testing, reflecting the depth of functional exploration[36,37].”
Improving Deep Learning Library Testing with Machine Learning
Molina, F., Naziri, M. M. A., Qin, F., Gorla, A., & d’Amorim, M. (2026) · doi:10.1145/3793654.3793744
LLM-Assisted Input-Requirement-Aware Differential Testing of Array Programming Frameworks
Zhou, Z., & He, J. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808162
DECODE: Dynamic Exploration for Constraint-Guided Vulnerability Discovery in Deep Learning Operators
Liu, H., Wang, Z., Liu, Z., & Li, W. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808174
Testing IFrame Rendering in Web Browsers
Park, D., Jeon, Y., Wi, S., & Kim, M. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834417
Directed Neuro-Symbolic Stochastic Execution for Verification of Distributed Parallel AI ProgramsPreprint
Koorma, G., Sharma, V., Edwards, G., & Eslamimehr, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.07947
“DeepXplore [8] introduced coverage-guided whitebox testing of neural networks, and DeepREL fuzzes deep-learning libraries through automatically inferred relational APIs [9].”
Liang, Z., Yao, C., Xie, Y., Zhang, J., & Zhu, J. (2026) · Sustainable Computing Informatics and Systems · doi:10.1016/j.suscom.2026.101464
Metadata
- Authors
- Deng, Y., Yang, C., Wei, A., & Zhang, L.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Stanford University (US)
- Venue
- Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2022-11-07
- Pages
- 44-56
- OpenAlex
- W4308643012
- Cited by
- 86
- References
- 57
- Citations by year
- 2026: 17 · 2025: 34 · 2024: 20 · 2023: 15
- FWCI
- 13.01
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (CCF-2131943,CCF-2141474)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- fuzzing, deep learning libraries, API testing, PyTorch, TensorFlow, bug detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.