Fuzzing deep-learning libraries via automated relational API inference
Open accessThe DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Abstract
Deep Learning (DL) has gained wide attention in recent years. Meanwhile, bugs in DL systems can lead to serious consequences, and may even threaten human lives. As a result, a growing body of research has been dedicated to DL model testing. However, there is still limited work on testing DL libraries, e.g., PyTorch and TensorFlow, which serve as the foundations for building, training, and running DL models. Prior work on fuzzing DL libraries can only generate tests for APIs which have been invoked by documentation examples, developer tests, or DL models, leaving a large number of APIs untested. In this paper, we propose DeepREL, the first approach to automatically inferring relational APIs for more effective DL library fuzzing. Our basic hypothesis is that for a DL library under test, there may exist a number of APIs sharing similar input parameters and outputs; in this way, we can easily “borrow” test inputs from invoked APIs to test other relational APIs. Furthermore, we formalize the notion of value equivalence and status equivalence for relational APIs to serve as the oracle for effective bug finding. We have implemented DeepREL as a fully automated end-to-end relational API inference and fuzzing technique for DL libraries, which 1) automatically infers potential API relations based on API syntactic/semantic information, 2) synthesizes concrete test programs for invoking relational APIs, 3) validates the inferred relational APIs via representative test inputs, and finally 4) performs fuzzing on the verified relational APIs to find potential inconsistencies. Our evaluation on two of the most popular DL libraries, PyTorch and TensorFlow, demonstrates that DeepREL can cover 157% more APIs than state-of-the-art FreeFuzz. To date, DeepREL has detected 162 bugs in total, with 106 already confirmed by the developers as previously unknown bugs. Surprisingly, DeepREL has detected 13.5% of the high-priority bugs for the entire PyTorch issue-tracking system in a three-month period. Also, besides the 162 code bugs, we have also detected 14 documentation bugs (all confirmed).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous API-level fuzzers [16,72] only focus on fuzzing each single DL library API in isolation.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [16]
In the citing paper
“The inspiration comes from prior work [16], which shows that it is common in DL libraries for related APIs to share the same input, and borrowing inputs from one API can help trigger bugs in its relational APIs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [16]
In the citing paper
“As such, previous work cannot generate valid inputs to cover this API since they rely on known valid input/API pairs obtained from databases created by scraping open-source code [16,72].”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [16]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundSentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1×
“Next, we use SentenceBERT [58] to encode these informative description sentences into semantically meaningful sentence embeddings.”
MethodSpeech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundDeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 1×
“To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
Algorithms for the Assignment and Transportation Problems
Munkres, J. (1957) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0105003 · Cited by 3,986
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
Akhtar, N., & Mian, A. (2018) · IEEE Access · doi:10.1109/access.2018.2807385 · Cited by 2,021
Deep Learning for Health Informatics
Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G.-Z. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2636665 · Cited by 2,000 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundDeep learning for detecting robotic grasps
Lenz, I., Lee, H., & Saxena, A. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914549607 · Cited by 1,634 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundDeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 1×
“To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [15, 20, 28, 42, 46, 50], metrics for model testing [32, 35, 41, 52, 75], applicationspecific model testing [61, 83, 86], and verifying certain properties of models [16, 39].”
MethodCoverage-based Greybox Fuzzing as Markov Chain
Böhme, M., Pham, V.-T., & Roychoudhury, A. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978428 · Cited by 649 · cited 1×
“, various fuzzing techniques have been applied for 24h or even more [18, 40, 58, 66, 78] (including the recent LEMON work [64]).”
MethodMultimodal deep learning for robust RGB-D object recognition
Eitel, A., Springenberg, J. T., Spinello, L., Riedmiller, M., & Burgard, W. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353446 · Cited by 631 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundZhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238187 · Cited by 611 · cited 1×
“specific model testing [62, 85, 88], and verifying certain properties of models [15, 38].”
BackgroundSelf-citationDeepGauge: multi-granularity testing criteria for deep learning systems
Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238202 · Cited by 588 · cited 1×
“To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
Deep learning code fragments for code clone detection
White, M., Tufano, M., Vendome, C., & Poshyvanyk, D. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970326 · Cited by 558 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundGu, X., Zhang, H., & Kim, S. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180167 · Cited by 547 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
Background(1998) · Cited by 469
Guiding Deep Learning System Testing Using Surprise Adequacy
Kim, J., Feldt, R., & Yoo, S. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00108 · Cited by 430 · cited 1×
“To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundSelf-citationA survey on deep learning for software engineering
Yang, Y., Xia, X., Lo, D., & Grundy, J. (2022) · ACM Computing Surveys · doi:10.1145/3505243 · Cited by 223 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundDLFix: context-based code transformation learning for automated program repair
Li, Y., Wang, S., & Nguyen, T. N. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380345 · Cited by 209 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundCRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries
Pham, H. V., Lutellier, T., Qi, W., & Tan, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00107 · Cited by 190 · cited 11×
“To reduce such threats, we adopt the number of covered APIs and detected bugs, following prior work on DL library testing [54, 65, 66].”
BackgroundMethodInfluentialDeep learning library testing via effective model generation
Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409761 · Cited by 167 · cited 7×
“To reduce such threats, we adopt the number of covered APIs and detected bugs, following prior work on DL library testing [54, 65, 66].”
BackgroundMethodInfluentialAutomated testing of graphics shader compilers
Donaldson, A. F., Evrard, H., Lascu, A., & Thomson, P. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133917 · Cited by 160 · cited 1×
“It is commonly used in prior work on fuzzing or testing [22, 61, 80].”
MethodMemLock: memory usage guided fuzzing
Wen, C., Wang, H., Li, Y., Qin, S., Liu, Y., Xu, Z., Chen, H., Xie, X., Pu, G., & Liu, T. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380396 · Cited by 160 · cited 1×
“, various fuzzing techniques have been applied for 24h or even more [17, 40, 59, 67, 80] (including the recent LEMON work [65]).”
MethodExploring API Embedding for API Usages and Applications
Nguyen, T. D., Nguyen, A. T., Phan, H. D., & Nguyen, T. N. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.47 · Cited by 159 · cited 1×
“Prior work [49, 50] proposed to train neural networks to mine API mappings by learning transformations between the vector spaces of APIs from different languages (e.”
BackgroundDeepBillboard: systematic physical-world testing of autonomous driving systems
Zhou, H., Li, W., Kong, Z., Guo, J., Zhang, Y., Yu, B., Zhang, L., & Liu, C. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380422 · Cited by 153 · cited 1×
“specific model testing [62, 85, 88], and verifying certain properties of models [15, 38].”
BackgroundSelf-citationAn extensive study on pre-trained models for program understanding and generation
Zeng, Z., Tan, H., Zhang, H., Li, J., Zhang, Y., & Zhang, L. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534390 · Cited by 153 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundSelf-citationDeepXplore: automated whitebox testing of deep learning systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2019) · Communications of the ACM · doi:10.1145/3361566 · Cited by 150
Is neuron coverage a meaningful measure for testing deep neural networks?
Harel-Canada, F., Wang, L., Gulzar, M. A., Gu, Q., & Kim, M. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409754 · Cited by 145 · cited 1×
“To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
ProFuzzer: On-the-fly Input Type Probing for Better Zero-Day Vulnerability Discovery
You, W., Wang, X., Ma, S., Huang, J., Zhang, X., Wang, X., & Liang, B. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00057 · Cited by 131 · cited 2×
“, various fuzzing techniques have been applied for 24h or even more [17, 40, 59, 67, 80] (including the recent LEMON work [65]).”
MethodPractical, Low-Effort Equivalence Verification of Real Code
Ramos, D. A., & Engler, D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22110-1_55 · Cited by 126 · cited 2×
“UC-KLEE [56] leverages symbolic execution to check two different implemen-”
MethodSearch-based inference of polynomial metamorphic relations
Zhang, J., Chen, J., Hao, D., Xiong, Y., Xie, B., Zhang, L., & Mei, H. (2014) · Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/2642937.2642994 · Cited by 115
Self-citationAutomatic generation of oracles for exceptional behaviors
Goffi, A., Gorla, A., Ernst, M. D., & Pezzè, M. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931061 · Cited by 111 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundDeepDelta: learning to repair compilation errors
Mesbah, A., Rice, A., Johnston, E., Glorioso, N., & Aftandilian, E. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3340455 · Cited by 107 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundFree lunch for testing: fuzzing deep-learning libraries from open source
Wei, A., Deng, Y., Yang, C., & Zhang, L. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510041 · Cited by 106 · cited 10×
“We obtain the valid inputs traced from various input sources used by state-of-the-art FreeFuzz [65], which include documentations, developer tests, and 202 DL models in the wild,”
BackgroundMethodInfluentialSelf-citationDeep just-in-time defect prediction: how far are we?
Zeng, Z., Zhang, Y., Zhang, H., & Zhang, L. (2021) · Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3460319.3464819 · Cited by 102 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundSelf-citationMTFuzz: fuzzing with a multi-task neural network
She, D., Krishna, R., Yan, L., Jana, S., & Ray, B. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409723 · Cited by 93 · cited 1×
“, various fuzzing techniques have been applied for 24h or even more [17, 40, 59, 67, 80] (including the recent LEMON work [65]).”
MethodAudee: automated testing for deep learning frameworks
Guo, Q., Xie, X., Li, Y., Zhang, X., Liu, Y., Li, X., & Shen, C. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416571 · Cited by 92 · cited 9×
“Among those 106 confirmed bugs, only 9 can be found by FreeFuzz, and none of them can be detected by CRADLE, AUDEE, or LEMON.”
BackgroundMethodInfluentialAn SMT-Based Approach for Verifying Binarized Neural Networks
Amir, G., Wu, H., Barrett, C., & Katz, G. (2021) · Lecture notes in computer science · doi:10.26226/morressier.604907f41a80aac83ca25cda · Cited by 84
DocTer: documentation-guided fuzzing for testing deep learning API functions
Xie, D., Li, Y., Kim, M., Pham, H. V., Tan, L., Zhang, X., & Godfrey, M. W. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534220 · Cited by 81 · cited 1×
“A very recent work DocTer [76] constructs rules to extract DL-specific input constraints from API documentation and uses them to generate valid/invalid inputs for testing DL libraries; meanwhile, it requires manual annotation for 30% of API parameters.”
MethodAutomated conformance testing for JavaScript engines via deep compiler fuzzing
Ye, G., Tang, Z., Tan, S. H., Huang, S., Fang, D., Sun, X., Bian, L., Wang, H., & Wang, Z. (2021) · Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation · doi:10.1145/3453483.3454054 · Cited by 77 · cited 1×
“Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
BackgroundCorrelations between deep neural network model coverage criteria and model quality
Yan, S., Tao, G., Liu, X., Zhai, J., Ma, S., Xu, L., & Zhang, X. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409671 · Cited by 74 · cited 1×
“To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation
Blasi, A., Gorla, A., Ernst, M. D., Pezzè, M., & Carzaniga, A. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111041 · Cited by 69 · cited 10×
“We substituted the first three phases of DeepREL with MeMo, and found that it can infer only 17/28 API relations for TensorFlow/PyTorch (compared to 8808/4290 relations found by DeepREL), and can at most detect 8 (including 6 found by MeMo and 2 found by FreeFuzz on the APIs involved in the MeMo inferred relations) of…”
BackgroundInfluentialPredoo: precision testing of deep learning operators
Zhang, X., Sun, N., Fang, C., Liu, J., Liu, J., Chai, D., Jiang, W., & Chen, Z. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464843 · Cited by 68 · cited 1×
“, Predoo [86] can only test 7 APIs for TensorFlow.”
BackgroundPath-based function embedding and its application to error-handling specification mining
DeFreez, D., Thakur, A. V., & Rubio-González, C. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236059 · Cited by 61 · cited 2×
“Func2vec [21] is a technique for finding function synonyms in Linux file systems and drivers via learning function embeddings to improve the quality of error handling specifications for Linux code.”
BackgroundMethodFully automated functional fuzzing of Android apps for detecting non-crashing logic bugs
Su, T., Yan, Y., Wang, J., Sun, J., Xiong, Y., Pu, G., Wang, K., & Su, Z. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485533 · Cited by 60 · cited 1×
“It is commonly used in prior work on fuzzing or testing [22, 61, 80].”
MethodCoverage-guided tensor compiler fuzzing with joint IR-pass mutation
Liu, J., Wei, Y., Yang, S., Deng, Y., & Zhang, L. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3527317 · Cited by 51 · cited 1×
“, various fuzzing techniques have been applied for 24h or even more [17, 40, 59, 67, 80] (including the recent LEMON work [65]).”
MethodSelf-citationAutomated inference of likely metamorphic relations for model transformations
Troya, J., Segura, S., & Ruiz–Cortés, A. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.05.043 · Cited by 49 · cited 1×
“, TensorFlow or PyTorch) as test objects, this work can also be viewed as an instance/extension of the traditional metamorphic oracle generation work, which aims to automatically infer metamorphic relations as test oracle [16, 26, 33, 43, 63, 75, 83, 84].”
BackgroundMapping API elements for code migration with vector representations
Nguyen, T. D., Nguyen, A. T., & Nguyen, T. N. (2016) · Proceedings of the 38th International Conference on Software Engineering Companion · doi:10.1145/2889160.2892661 · Cited by 46 · cited 1×
“Prior work [49, 50] proposed to train neural networks to mine API mappings by learning transformations between the vector spaces of APIs from different languages (e.”
BackgroundAutomatic Discovery and Cleansing of Numerical Metamorphic Relations
Zhang, B., Zhang, H., Chen, J., Hao, D., & Moscato, P. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00035 · Cited by 42 · cited 1×
“, TensorFlow or PyTorch) as test objects, this work can also be viewed as an instance/extension of the traditional metamorphic oracle generation work, which aims to automatically infer metamorphic relations as test oracle [16, 26, 33, 43, 63, 75, 83, 84].”
BackgroundSearch-based synthesis of equivalent method sequences
Goffi, A., Gorla, A., Mattavelli, A., Pezzè, M., & Tonella, P. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635888 · Cited by 36 · cited 6×
“For example, SBES [27, 43] synthesizes sequences of method invocations that are equivalent to a target method based on the observed dynamic program behaviors.”
BackgroundMethodInfluentialTechniques for Automatic Detection of Metamorphic Relations
Kanewala, U. (2014) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2014.62 · Cited by 31 · cited 1×
“…if we treat the entire systems under test (e.g., TensorFlow or PyTorch) as test objects, this work can also be viewed as an instance/extension of the metamorphic oracle generation work, which aims to automatically infer metamorphic relations as test oracle [17, 27, 34, 43, 62, 73, 81, 82].”
Discovering discrepancies in numerical libraries
Vanover, J., Deng, X., & Rubio-González, C. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397380 · Cited by 18 · cited 8×
“Also, FPDiff can only target numerical library APIs taking double or int variables as input, whereas the input of APIs in DL libraries can be much more complex (e.g., tensors of different dimensions/types, objects, strings, etc.).”
BackgroundInfluentialNeoFlow: A Flexible Framework for Enabling Efficient Compilation for High Performance DNN Training
Zheng, S., Chen, R., Jin, Y., Wei, A., Wu, B., Li, X., Yan, S., & Liang, Y. E. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3138862 · Cited by 16 · cited 1×
“Actually, the backward part needs to construct a very complex graph [87], but we omit it in Figure 1 for simplicity.”
BackgroundSelf-citationA Genetic Algorithm-Based Approach for Composite Metamorphic Relations Construction
Xiang, Z., Wu, H., & Yu, F. (2019) · Information · doi:10.3390/info10120392 · Cited by 12 · cited 1×
“, TensorFlow or PyTorch) as test objects, this work can also be viewed as an instance/extension of the traditional metamorphic oracle generation work, which aims to automatically infer metamorphic relations as test oracle [16, 26, 33, 43, 63, 75, 83, 84].”
BackgroundTowards Zero Knowledge Learning for Cross Language API Mappings
Bui, N. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00054 · Cited by 9 · cited 1×
“Another work [18] leveraged an unsupervised domain adaptation approach to automatically construct and align vector spaces for identifying API mappings with much less human effort.”
BackgroundSynthesis of Equivalent Method Calls in Guava
Mattavelli, A., Goffi, A., & Gorla, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-22183-0_19 · Cited by 6 · cited 6×
“For example, SBES [27, 43] synthesizes sequences of method invocations that are equivalent to a target method based on the observed dynamic program behaviors.”
BackgroundMethodInfluential
Research works printed but not linked: 12
- Martín Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , and Michael Isard . 2016 . Tensorflow: A system for large-scale machine learning. In 12th $USENIX$ symposium on operating systems design and implementation ($OSDI$ 16). 265–283. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. 2016. Tensorflow: A system for large-scale machine learning. In 12th $USENIX$ symposium on operating systems design and implementation ($OSDI$ 16). 265–283.
- Nicholas Carlini Anish Athalye Nicolas Papernot Wieland Brendel Jonas Rauber Dimitris Tsipras Ian Goodfellow Aleksander Madry and Alexey Kurakin. 2019. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705. Nicholas Carlini Anish Athalye Nicolas Papernot Wieland Brendel Jonas Rauber Dimitris Tsipras Ian Goodfellow Aleksander Madry and Alexey Kurakin. 2019. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705. · arXiv
- Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543. Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543. · arXiv
- Felix A Gers , Jürgen Schmidhuber , and Fred Cummins . 2000. Learning to forget: Continual prediction with LSTM. Neural computation, 12, 10 ( 2000 ), 2451–2471. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural computation, 12, 10 (2000), 2451–2471.
- Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. · arXiv
- Changliu Liu Tomer Arnon Christopher Lazarus Christopher Strong Clark Barrett and Mykel J Kochenderfer. 2019. Algorithms for verifying deep neural networks. arXiv preprint arXiv:1903.06758. Changliu Liu Tomer Arnon Christopher Lazarus Christopher Strong Clark Barrett and Mykel J Kochenderfer. 2019. Algorithms for verifying deep neural networks. arXiv preprint arXiv:1903.06758. · arXiv
- Jiawei Liu Jinkun Lin Fabian Ruffy Cheng Tan Jinyang Li Aurojit Panda and Lingming Zhang. 2022. Finding Deep-Learning Compilation Bugs with NNSmith. arXiv preprint arXiv:2207.13066. Jiawei Liu Jinkun Lin Fabian Ruffy Cheng Tan Jinyang Li Aurojit Panda and Lingming Zhang. 2022. Finding Deep-Learning Compilation Bugs with NNSmith. arXiv preprint arXiv:2207.13066. · arXiv
- Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. · arXiv
- Seonwoo Min , Byunghan Lee , and Sungroh Yoon . 2017. Deep learning in bioinformatics. Briefings in bioinformatics, 18, 5 ( 2017 ), 851–869. Seonwoo Min, Byunghan Lee, and Sungroh Yoon. 2017. Deep learning in bioinformatics. Briefings in bioinformatics, 18, 5 (2017), 851–869.
- Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , and Luca Antiga . 2019 . Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32 (2019), 8026–8037. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca Antiga. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32 (2019), 8026–8037.
- Martin F Porter . 2001 . Snowball: A language for stemming algorithms. Martin F Porter. 2001. Snowball: A language for stemming algorithms.
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. · arXiv
Cited by
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
Self-citationDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623343 · Cited by 115
Self-citationUnifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
Zhang, Z., Chen, C., Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., & Wang, R. (2024) · Transactions on machine learning research · doi:10.13140/rg.2.2.32710.69440/1 · Cited by 195
NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers
Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., & Zhang, L. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575707 · Cited by 167
“More recently, DeepREL [15] aims to automatically infer relational APIs (e.g., APIs that return the same values/statuses when given the same inputs), and then leverages such API relations as test oracle for catching more bugs than FreeFuzz.”
Self-citationLarge Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTPreprint
Deng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02014 · Cited by 131
“On the other hand, API-level fuzzers focus on testing each individual API via effective input generation [71, 75] or oracle inference [15, 77].”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Deng, Y., Yang, C., Wei, A., & Zhang, L.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Stanford University (US)
- Venue
- Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2022-11-07
- Pages
- 44-56
- OpenAlex
- W4308643012
- Cited by
- 86
- References
- 57
- Citations by year
- 2026: 17 · 2025: 34 · 2024: 20 · 2023: 15
- FWCI
- 13.01
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (CCF-2131943,CCF-2141474)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- fuzzing, deep learning libraries, API testing, PyTorch, TensorFlow, bug detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.