Free lunch for testing: fuzzing deep-learning libraries from open source
The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
Deep learning (DL) systems can make our life much easier, and thus are gaining more and more attention from both academia and industry. Meanwhile, bugs in DL systems can be disastrous, and can even threaten human lives in safety-critical applications. To date, a huge body of research efforts have been dedicated to testing DL models. However, interestingly, there is still limited work for testing the underlying DL libraries, which are the foundation for building, optimizing, and running DL models. One potential reason is that test generation for the underlying DL libraries can be rather challenging since their public APIs are mainly exposed in Python, making it even hard to automatically determine the API input parameter types due to dynamic typing. In this paper, we propose FreeFuzz, the first approach to fuzzing DL libraries via mining from open source. More specifically, FreeFuzz obtains code/models from three different sources: 1) code snippets from the library documentation, 2) library developer tests, and 3) DL models in the wild. Then, FreeFuzz automatically runs all the collected code/models with instrumentation to trace the dynamic information for each covered API, including the types and values of each parameter during invocation, and shapes of input/output tensors. Lastly, FreeFuzz will leverage the traced dynamic information to perform fuzz testing for each covered API. The extensive study of FreeFuzz on PyTorch and TensorFlow, two of the most popular DL libraries, shows that FreeFuzz is able to automatically trace valid dynamic information for fuzzing 1158 popular APIs, 9X more than state-of-the-art LEMON with 3.5X lower overhead than LEMON. To date, FreeFuzz has detected 49 bugs for PyTorch and TensorFlow (with 38 already confirmed by developers as previously unknown).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous API-level fuzzers [16,72] only focus on fuzzing each single DL library API in isolation.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“FreeFuzz [72], a state-of-the-art API-level fuzzer, will first collect the valid argument space (e.g., type and shape of the input tensor) for a target API by mining open-source code snippets.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“As a result, model-level fuzzers can only cover a limited set of APIs with limited patterns [72], missing many diverse and interesting API sequences that can lead to bugs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“As such, previous work cannot generate valid inputs to cover this API since they rely on known valid input/API pairs obtained from databases created by scraping open-source code [16,72].”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
In the citing paper
“We include both PyTorch (v1.12) [57] and TensorFlow (v2.10) [66], since they are two of the most popular DL libraries and are widely studied in prior DL library testing work [70,72,77].”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [72]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 1×
“In this section, we will give a brief introduction to the preliminaries of deep learning libraries based on PyTorch [51].”
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundIntriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“To mitigate such attacks, researchers have also proposed various adversarial defense techniques, including adversarial training [32, 46, 62], detection [35, 45, 76], and others [64].”
BackgroundGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“Through such applications, DL has substantially improved our daily life [19, 34, 56, 60, 67].”
BackgroundDeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 1×
“Various adversarial attack techniques [32, 47, 48, 73] have been proposed to generate the so-called “adversarial examples” by adding perturbations imperceptible to humans to intentionally fool the classification results given”
BackgroundDeep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968
Learning to forget: Continual prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 4,823 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundThe Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · cited 1×
“Through such applications, DL has substantially improved our daily life [19, 34, 56, 60, 67].”
BackgroundThreat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
Akhtar, N., & Mian, A. (2018) · IEEE Access · doi:10.1109/access.2018.2807385 · Cited by 2,021
A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809 · cited 1×
“Through such applications, DL has substantially improved our daily life [19, 34, 56, 60, 67].”
BackgroundDeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217 · cited 2×
“plore [52] proposed a metric called neuron coverage for whitebox testing of DL models and leveraged gradient-based techniques to search for more effective tests.”
MethodDeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 2×
“Meanwhile, there are also a series of work targeting specific applications, such as autonomous driving, including DeepTest [63], DeepRoad [71], and DeepBillboard [78].”
BackgroundMethodArcuri, A., & Briand, L. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985795 · Cited by 968 · cited 1×
“Furthermore, although FreeFuzz is fuzzing against 1158 APIs (each with 1000 times) and the randomness can be largely mitigated by such a large number of APIs, it is still possible that the nondeterminism in FreeFuzz can affect the effectiveness of FreeFuzz in different runs [17, 40].”
BackgroundMetamorphic Testing: A Review of Challenges and Opportunities
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2019) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 665
Klees, G., Ruef, A., Cooper, B., Wei, S., & Hicks, M. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243804 · Cited by 633 · cited 1×
“Furthermore, although FreeFuzz is fuzzing against 1158 APIs (each with 1000 times) and the randomness can be largely mitigated by such a large number of APIs, it is still possible that the nondeterminism in FreeFuzz can affect the effectiveness of FreeFuzz in different runs [17, 40].”
BackgroundZhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238187 · Cited by 611 · cited 2×
“Meanwhile, there are also a series of work targeting specific applications, such as autonomous driving, including DeepTest [63], DeepRoad [71], and DeepBillboard [78].”
BackgroundMethodSelf-citationDeepGauge: multi-granularity testing criteria for deep learning systems
Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238202 · Cited by 588 · cited 2×
“While various other metrics [39, 43] have also been proposed recently, the correlation between such”
BackgroundMethodGuiding Deep Learning System Testing Using Surprise Adequacy
Kim, J., Feldt, R., & Yoo, S. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00108 · Cited by 430 · cited 2×
“While various other metrics [39, 43] have also been proposed recently, the correlation between such”
BackgroundMethodDeepMutation: Mutation Testing of Deep Learning Systems
Ma, L., Zhang, F., Sun, J., Xue, M., Li, B., Juefei-Xu, F., Xie, C., Li, L., Liu, Y., Zhao, J., et al. (2018) · 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2018.00021 · Cited by 366
Coverage-Based Greybox Fuzzing as Markov Chain
Bohme, M., Pham, V.-T., & Roychoudhury, A. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2785841 · Cited by 340
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundSelf-citationConcolic testing for deep neural networks
Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., & Kroening, D. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238172 · Cited by 332 · cited 1×
“Lastly, researchers have also explored concolic testing [61] to achieve higher coverage for DNN models, mutation testing [38, 44] to simulate real faults in DL models, and test input generation for DNNs [26] by exploiting features”
BackgroundParry, O., Kapfhammer, G. M., Hilton, M. L., & McMinn, P. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3476105 · Cited by 238 · cited 1×
“This type of tests are called implementation-dependent flaky tests, described in prior work on test flakiness [41, 50, 72].”
MethodA survey on deep learning for software engineering
Yang, Y., Xia, X., Lo, D., & Grundy, J. (2022) · ACM Computing Surveys · doi:10.1145/3505243 · Cited by 223 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundCRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries
Pham, H. V., Lutellier, T., Qi, W., & Tan, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00107 · Cited by 190 · cited 9×
“CRADLE [53] is the trailing-blazing work for testing DL libraries.”
BackgroundMethodResultInfluentialDeep learning library testing via effective model generation
Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409761 · Cited by 167 · cited 10×
“Based on CRADLE, LEMON [65] advances testing DL libraries by proposing model-level mutation.”
BackgroundMethodResultInfluentialDeepBillboard: systematic physical-world testing of autonomous driving systems
Zhou, H., Li, W., Kong, Z., Guo, J., Zhang, Y., Yu, B., Zhang, L., & Liu, C. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380422 · Cited by 153 · cited 2×
“Meanwhile, there are also a series of work targeting specific applications, such as autonomous driving, including DeepTest [63], DeepRoad [71], and DeepBillboard [78].”
BackgroundMethodSelf-citationAdversarial Sample Detection for Deep Neural Network through Model Mutation Testing
Wang, J., Dong, G., Sun, J., Wang, X., & Zhang, P. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00126 · Cited by 151 · cited 1×
“To mitigate such attacks, researchers have also proposed various adversarial defense techniques, including adversarial training [32, 46, 62], detection [35, 45, 76], and others [64].”
BackgroundIs neuron coverage a meaningful measure for testing deep neural networks?
Harel-Canada, F., Wang, L., Gulzar, M. A., Gu, Q., & Kim, M. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409754 · Cited by 145 · cited 2×
“metrics and the robustness of models is still unclear [25, 36, 69].”
BackgroundMethodAn extensive study of static regression test selection in modern software evolution
Legunsen, O., Hariri, F., Shi, A., Lu, Y., Zhang, L., & Marinov, D. (2016) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2950290.2950361 · Cited by 131 · cited 1×
“Code coverage is a widely adopted test adequacy criterion for testing traditional software systems [21, 44, 76] and even the recent tensor compilers [46].”
BackgroundSelf-citationHybrid regression test selection
Zhang, L. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180198 · Cited by 128 · cited 1×
“Code coverage is a widely adopted test adequacy criterion for testing traditional software systems [21, 44, 76] and even the recent tensor compilers [46].”
BackgroundSelf-citationDeepCrime: mutation testing of deep learning systems based on real faults
Humbatova, N., Jahangirova, G., & Tonella, P. (2021) · Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3460319.3464825 · Cited by 123 · cited 1×
“Lastly, researchers have also explored concolic testing [61] to achieve higher coverage for DNN models, mutation testing [38, 44] to simulate real faults in DL models, and test input generation for DNNs [26] by exploiting features”
BackgroundWen, C., Wang, H., Li, Y., Qin, S., Liu, Y., Xu, Z., Chen, H., Xie, X., Pu, G., & Liu, T. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380396 · Cited by 122 · cited 1×
“Therefore, following existing fuzzing work [55, 66, 77],”
SQUIRREL: Testing Database Management Systems with Language Validity and Coverage Feedback
Zhong, R., Chen, Y., Hu, H. P., Zhang, H., Lee, W., & Wu, D. (2020) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3372297.3417260 · Cited by 111 · cited 1×
“Therefore, following existing fuzzing work [55, 66, 77],”
Deep just-in-time defect prediction: how far are we?
Zeng, Z., Zhang, Y., Zhang, H., & Zhang, L. (2021) · Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3460319.3464819 · Cited by 102 · cited 1×
“Deep Learning (DL) has been playing a significant role in various application domains, including image classification [39, 62], natural language processing [33, 35], game playing [61], and software engineering [23, 45, 74, 75].”
BackgroundSelf-citationPractical Accuracy Estimation for Efficient Deep Neural Network Testing
Chen, J., Wu, Z., Wang, Z., You, H., Zhang, L., & Yan, M. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3394112 · Cited by 93 · cited 1×
“Following prior work on software testing in general and DL library testing [24, 57, 69], we also report the number of actual bugs detected for the studied DL libraries.”
MethodSelf-citationAdvDoor: adversarial backdoor attack of deep learning system
Zhang, Q., Ding, Y., Tian, Y., Guo, J., Yuan, M., & Jiang, Y. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464809 · Cited by 90 · cited 1×
“Various adversarial attack techniques [32, 47, 48, 73] have been proposed to generate the so-called “adversarial examples” by adding perturbations imperceptible to humans to intentionally fool the classification results given”
BackgroundDetecting flaky tests in probabilistic and machine learning applications
Dutta, S., Shi, A., Choudhary, R., Zhang, Z., Jain, A., & Misailovic, S. (2020) · Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3395363.3397366 · Cited by 78
MTFuzz: fuzzing with a multi-task neural network
She, D., Krishna, R., Yan, L., Jana, S., & Ray, B. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409723 · Cited by 74
Correlations between deep neural network model coverage criteria and model quality
Yan, S., Tao, G., Liu, X., Zhai, J., Ma, S., Xu, L., & Zhang, X. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409671 · Cited by 74 · cited 2×
“metrics and the robustness of models is still unclear [25, 36, 69].”
BackgroundMethodAn Empirical Study on Correlation between Coverage and Robustness for Deep Neural Networks
Dong, Y., Zhang, P., Wang, J., Liu, S., Sun, J., Hao, J., Wang, X., Wang, L., Dong, J. S., & Dai, T. (2020) · IEEE International Conference on Engineering of Complex Computer Systems (ICECCS) · doi:10.1109/iceccs51672.2020.00016 · Cited by 74 · cited 1×
“metrics and the robustness of models is still unclear [25, 36, 69].”
BackgroundDetecting numerical bugs in neural network architectures
Zhang, Y., Ren, L., Chen, L., Xiong, Y., Cheung, S.-C., & Xie, T. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409720 · Cited by 70 · cited 1×
“Various techniques have also been proposed to detect numerical bugs introduced when building a DNN model at the architecture level with static analysis [75],”
Predoo: precision testing of deep learning operators
Zhang, X., Sun, N., Fang, C., Liu, J., Liu, J., Chai, D., Jiang, W., & Chen, Z. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464843 · Cited by 68 · cited 2×
“Predoo: precision testing of deep learning operators.”
MethodExposing numerical bugs in deep learning via gradient back-propagation
Yan, M. X., Chen, J., Zhang, X., Tan, L., Wang, G., & Wang, Z. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468612 · Cited by 65 · cited 1×
“Various techniques have also been proposed to detect numerical bugs introduced when building a DNN model at the architecture level with static analysis [81], and via gradient propagation [72].”
MethodA large-scale longitudinal study of flaky tests
Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., & Bell, J. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428270 · Cited by 64 · cited 1×
“This type of tests are called implementation-dependent flaky tests, described in prior work on test flakiness [41, 50, 72].”
MethodSelf-citationDynamic inference of static types for ruby
An, J.-H., Chaudhuri, A., Foster, J. S., & Hicks, M. (2011) · ACM SIGPLAN Notices · doi:10.1145/1925844.1926437 · Cited by 63
Fully automated functional fuzzing of Android apps for detecting non-crashing logic bugs
Su, T., Yan, Y., Wang, J., Sun, J., Xiong, Y., Pu, G., Wang, K., & Su, Z. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485533 · Cited by 60 · cited 1×
“We leverage metamorphic relations [23, 59] to detect performance bugs with FreeFuzz.”
MethodTypeDevil: Dynamic Type Inconsistency Analysis for JavaScript
Pradel, M., Schuh, P., & Sen, K. (2015) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2015.51 · Cited by 54 · cited 1×
“explored in the literature for traditional applications [15, 16, 54].”
BackgroundGenerative type-aware mutation for testing SMT solvers
Park, J., Winterer, D., Zhang, C., & Su, Z. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485529 · Cited by 51 · cited 1×
“Our inputmining and type-aware [49]/dbbased mutations can all help generate more valid inputs.”
BackgroundTesting probabilistic programming systems
Dutta, S., Legunsen, O., Huang, Z., & Misailovic, S. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236057 · Cited by 49
FLEX: fixing flaky tests in machine learning projects by updating assertion bounds
Dutta, S., Shi, A., & Misailovíc, S. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468615 · Cited by 47
Enhanced compiler bug isolation via memoized search
Chen, J., Ma, H., & Zhang, L. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416570 · Cited by 37 · cited 1×
“application domains, including image classification [37, 58], natural language processing [31, 33], game playing [57], and software engineering [22, 42, 70].”
BackgroundSelf-citationStorm: program reduction for testing and debugging probabilistic programming systems
Dutta, S., Zhang, W., Huang, Z., & Misailovíc, S. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338972 · Cited by 35
Domain-Specific Fixes for Flaky Tests with Wrong Assumptions on Underdetermined Specifications
Zhang, P., Jiang, Y., Wei, A., Stodden, V., Marinov, D., & Shi, A. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00018 · Cited by 29 · cited 1×
“This type of tests are called implementation-dependent flaky tests, described in prior work on test flakiness [41, 50, 72].”
MethodSelf-citationDetecting Adversarial Examples in Deep Neural Networks using Normalizing Filters
Gu, S., Yi, P., Zhu, T., Yao, Y., & Wang, W. (2019) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0007370301640173 · Cited by 16 · cited 1×
“To mitigate such attacks, researchers have also proposed various adversarial defense techniques, including adversarial training [32, 46, 62], detection [35, 45, 76], and others [64].”
BackgroundBrosgol, B. M. (2011) · ACM SIGAda Ada Letters · doi:10.1145/2070336.2070341 · Cited by 10
Trace Typing: An Approach for Evaluating Retrofitted Type SystemsPreprint
Andreasen, E., Gordon, C. S., Chandra, S., Sridharan, M., Tip, F., & Sen, K. (2016) · doi:10.4230/lipics.ecoop.2016.1 · Cited by 4 · cited 1×
“explored in the literature for traditional applications [15, 16, 54].”
Background
Research works printed but not linked: 12
- MongoDB: the application data platform, 2021. https://www.mongodb.com.
- N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and A. Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019. · arXiv
- S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014. · arXiv
- I. Dunn, H. Pouget, D. Kroening, and T. Melham. Exposing previously undetectable faults in deep neural networks. arXiv preprint arXiv:2106.00576, 2021. · arXiv
- I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. · arXiv
- J. Liu, Y. Wei, S. Yang, Y. Deng, and L. Zhang. Coverage-guided tensor compiler fuzzing with joint ir-pass mutation. arXiv preprint arXiv:2202.09947, 2022. · arXiv
- X. Ma, B. Li, Y. Wang, S. M. Erfani, S. Wijewickrema, G. Schoenebeck, D. Song, M. E. Houle, and J. Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint arXiv:1801.02613, 2018. · arXiv
- A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. · arXiv
- D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484--489, 2016.
- K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. · arXiv
- Y. Sun, D. Liang, X. Wang, and X. Tang. Deepid3: Face recognition with very deep neural networks. arXiv preprint arXiv:1502.00873, 2015. · arXiv
- Z. Zhao, G. Chen, J. Wang, Y. Yang, F. Song, and J. Sun. Attack as defense: Characterizing adversarial examples using robustness. arXiv preprint arXiv:2103.07633, 2021. · arXiv
Cited by
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“Traditional techniques with API-level fuzzing [139], [140] or model-level fuzzing [141], [142] suffer from the following limitations: 1) lack of diverse API sequence thus cannot reveal bugs caused by chained API sequences; 2) cannot generate arbitrary code thus cannot explore the huge search space…”
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
“FreeFuzz [72] is an API-level fuzzer that first learns the valid inputs for each target API through mining open-source code snippets and then applies simple mutations to generate diverse inputs to test a target API.”
BackgroundMethodInfluentialSelf-citationFuzz4ALL: Universal Fuzzing with Large Language Models
Xia, C. S., Paltenghi, M., Le Tian, J., Pradel, M., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639121 · Cited by 229
Self-citationDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623343 · Cited by 115
Self-citationWhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Yang, C., Deng, Y., Lu, R., Yao, J., Liu, J., Jabbarvand, R., & Zhang, L. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689736 · Cited by 82
“Therefore, they can be effectively triggered by TitanFuzz since TitanFuzz leverages LLMs to generate human-like programs by resembling the distribution of training data.”
BackgroundSelf-citation
Metadata
- Authors
- Wei, A., Deng, Y., Yang, C., & Zhang, L.
- Affiliations
- Stanford University (US)
- University of Illinois Urbana-Champaign (US)
- Nanjing University (CN)
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 995-1007
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4284686707
- Cited by
- 106
- References
- 55
- Citations by year
- 2026: 21 · 2025: 37 · 2024: 27 · 2023: 17 · 2022: 4
- FWCI
- 16.03
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Funders
- National Science Foundation (CCF-2131943 and CCF-2141474)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- fuzzing, deep learning libraries, PyTorch, TensorFlow, dynamic typing, instrumentation, bug detection, lemon
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.