Muffin: testing deep learning libraries via neural architecture fuzzing
The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Deep learning (DL) techniques are proven effective in many challenging tasks, and become widely-adopted in practice. However, previous work has shown that DL libraries, the basis of building and executing DL models, contain bugs and can cause severe consequences. Unfortunately, existing testing approaches still cannot comprehensively exercise DL libraries. They utilize existing trained models and only detect bugs in model inference phase. In this work we propose Muffin to address these issues. To this end, Muffin applies a specifically-designed model fuzzing approach, which allows it to generate diverse DL models to explore the target library, instead of relying only on existing trained models. Muffin makes differential testing feasible in the model training phase by tailoring a set of metrics to measure the inconsistencies between different DL libraries. In this way, Muffin can best exercise the library code to detect more bugs. To evaluate the effectiveness of Muffin, we conduct experiments on three widely-used DL libraries. The results demonstrate that Muffin can detect 39 new bugs in the latest release versions of popular DL libraries, including Tensorflow, CNTK, and Theano.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a result, model-level fuzzers can only cover a limited set of APIs with limited patterns [72], missing many diverse and interesting API sequences that can lead to bugs.”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [25]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“Deep learning (DL) techniques have been proven effective in many specific tasks, such as image recognition [29], video understanding [53] and machine translation [49].”
BackgroundDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“[16] propose a set of similarity metrics and coverage criteria to analyze stateful DL systems such as Recurrent Neural Networks (RNNs) [39].”
BackgroundEmpirical Evaluation of Rectified Activations in Convolutional Network
Xu, B., Wang, N., Chen, T., & Li, M. (2015) · arXiv · doi:10.48550/arxiv.1505.00853 · Cited by 2,264 · cited 1×
“Furthermore, many APIs in DL libraries have specific usage scenarios, e.g. , Convolution Layer for image processing tasks, Recurrent Layers for text processing and various activation functions ( e.g. , ReLU [41] and leaky-ReLU [57]).”
BackgroundNeural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
What every computer scientist should know about floating-point arithmetic
Goldberg, D. (1991) · ACM Computing Surveys · doi:10.1145/103162.103163 · Cited by 1,750 · cited 1×
“However, due to normal uncertain factors such as floating-point deviation [22], we cannot determine whether a value difference is caused by potential bugs or normal factors.”
BackgroundEvolutionary computation: Toward a new philosophy of machine intelligence
B�ck, T. (1997) · Complexity · doi:10.1002/(sici)1099-0526(199703/04)2:4<28::aid-cplx7>3.0.co;2-2 · Cited by 1,533
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217 · cited 4×
“[43] first propose neuron coverage as the criteria for testing DL models.”
BackgroundMethodInfluentialDeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 4×
“[48] implement DeepTest for detecting erroneous behaviors of DL-based self-driving cars via automatically generating test cases based on image transformations.”
BackgroundMethodInfluentialMachine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Learning Deep Transformer Models for Machine Translation
Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., & Chao, L. S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1176 · Cited by 673 · cited 1×
“Deep learning (DL) techniques have been proven effective in many specific tasks, such as image recognition [29], video understanding [53] andmachine translation [49].”
BackgroundZhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238187 · Cited by 611 · cited 3×
“Similarly, Zhang et al. [60] implement DeepRoad, which applies Generative Adversarial Networks (GANs) [24] to test DL-based self-driving cars.”
BackgroundDeepGauge: multi-granularity testing criteria for deep learning systems
Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238202 · Cited by 588
Finding and understanding bugs in C compilers
Yang, X., Chen, Y., Eide, E., & Regehr, J. (2011) · ACM SIGPLAN Notices · doi:10.1145/1993316.1993532 · Cited by 586
Gupta, A., Anpalagan, A., Guan, L., & Khwaja, A. S. (2021) · Array · doi:10.1016/j.array.2021.100057 · Cited by 565 · cited 1×
“, self-driving cars [28], virtual assistants [34], and software operations [11, 12, 25].”
BackgroundConvolutional recurrent neural networks for music classification
Choi, K., Fazekas, G., Sandler, M., & Cho, K. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952585 · Cited by 517 · cited 1×
“Keras [4] is one of the most popular high-level DL libraries that has been widely used in various domains [15, 32, 40].”
BackgroundTheory of the backpropagation neural network
Hecht-Nielsen, R. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90469-8 · Cited by 472 · cited 1×
“, the library codes for back propagation [30].”
BackgroundLong-Term Feature Banks for Detailed Video Understanding
Wu, C.-Y., Feichtenhofer, C., Fan, H., He, K., Krahenbuhl, P., & Girshick, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00037 · Cited by 441 · cited 1×
“Deep learning (DL) techniques have been proven effective in many specific tasks, such as image recognition [29], video understanding [53] and machine translation [49].”
BackgroundGuiding Deep Learning System Testing Using Surprise Adequacy
Kim, J., Feldt, R., & Yoo, S. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00108 · Cited by 430 · cited 4×
“On the other hand, there is a great deal of researches focusing on the testing of DL models [35–38, 42, 43, 48, 54, 55].”
BackgroundInfluentialDeepHunter: a coverage-guided fuzz testing framework for deep neural networks
Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., Zhao, J., Li, B., Yin, J., & See, S. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330579 · Cited by 426 · cited 1×
“Moreover, there are a lot of studies intend to reveal defects in DL models via generating adversarial inputs or finding corner cases [48, 55, 60].”
BackgroundKepuska, V., & Bohouta, G. (2018) · 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc.2018.8301638 · Cited by 418 · cited 1×
“DL systems ( i.e. , software systems based on DL techniques) have been widely adopted in various domains in practice, e.g. , self-driving cars [28], virtual assistants [34], and software operations [11, 12, 25].”
MethodDeepMutation: Mutation Testing of Deep Learning Systems
Ma, L., Zhang, F., Sun, J., Xue, M., Li, B., Juefei-Xu, F., Xie, C., Li, L., Liu, Y., Zhao, J., et al. (2018) · 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2018.00021 · Cited by 366
A comprehensive study on deep learning bug characteristics
Islam, M. J., Nguyen, G., Pan, R., & Rajan, H. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338955 · Cited by 305 · cited 1×
“Bugs in DL libraries can inevitably cause unexpected outputs, or even fatal failure of DL systems [33].”
BackgroundDLFuzz: differential fuzzing testing of deep learning systems
Guo, J., Jiang, Y., Zhao, Y., Chen, Q., & Sun, J. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264835 · Cited by 281 · cited 1×
“Guo et al. [27] propose DLFuzz, a differential fuzzing testing framework that exposes DL model errors through mutating inputs to maximize model output difference.”
BackgroundMODE: automated neural network model debugging via state differential analysis and input selection
Ma, S., Liu, Y., Lee, W.-C., Zhang, X., & Grama, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236082 · Cited by 211 · cited 1×
“On the other hand, there is a great deal of researches focusing on the testing of DL models [35–38, 42, 43, 48, 54, 55].”
BackgroundModern Mathematical Methods for Physicists and Engineers
Cantrell, C. D. (2000) · Cambridge University Press eBooks · doi:10.1017/9780511811487 · Cited by 203
CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries
Pham, H. V., Lutellier, T., Qi, W., & Tan, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00107 · Cited by 190 · cited 7×
“Existing approaches [44, 50] utilize differential testing based on the model outputs with different DL libraries.”
BackgroundMethodResultInfluentialDwarakanath, A., Ahuja, M., Sikand, S., Rao, R. M., Bose, R. P. J. C., Dubash, N., & Podder, S. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213858 · Cited by 183 · cited 1×
“[19] adopt metamorphic testing to test image classification applications through mutating the training and testing data.”
MethodCoverage-directed differential testing of JVM implementations
Chen, Y., Su, T., Sun, C., Su, Z., & Zhao, J. (2016) · Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2908080.2908095 · Cited by 169 · cited 1×
“Differential testing has been successful in uncovering bugs across various types of programs, such as compilers [58], Java Virtual Machine (JVM) implementations [13, 14], web applications [10], and security-related APIs [46].”
BackgroundDeep learning library testing via effective model generation
Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409761 · Cited by 167 · cited 5×
“Similar to existing work [44, 50], we focus on testing low-level libraries, e.g. , TensorFlow, CNTK, and Theano.”
BackgroundMethodInfluentialDeepStellar: model-based quantitative analysis of stateful deep learning systems
Du, X., Xie, X., Li, Y., Ma, L., Liu, Y., & Zhao, J. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338954 · Cited by 163 · cited 2×
“[16] propose a set of similarity metrics and coverage criteria to analyze stateful DL systems such as Recurrent Neural Networks (RNNs) [39].”
BackgroundDeepLocalize: Fault Localization for Deep Neural Networks
Wardat, M., Le, W., & Rajan, H. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00034 · Cited by 118 · cited 2×
“Besides, there are also many researches focus on detecting different kinds of bugs in model structures or training parameter settings [51, 62].”
BackgroundMethodDeep Differential Testing of JVM Implementations
Chen, Y., Su, T., & Su, Z. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00127 · Cited by 114 · cited 1×
“Differential testing has been successful in uncovering bugs across various types of programs, such as compilers [58], Java Virtual Machine (JVM) implementations [13, 14], web applications [10], and security-related APIs [46].”
BackgroundAUTOTRAINER: An Automatic DNN Training Problem Detection and Repair System
Zhang, X., Zhai, J., Ma, S., & Shen, C. Q. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00043 · Cited by 103 · cited 1×
“Similarly, Zhang et al. [61] propose AUTOTRAINER, a tool that detects and auto-repairs commonly-seen model training problems such as vanishing gradient, exploding gradients and slow convergence.”
BackgroundXie, X., Zhang, Z., Chen, T. Y., Liu, Y., Poon, P.-L., & Xu, B. (2020) · IEEE Transactions on Reliability · doi:10.1109/tr.2020.2972266 · Cited by 96 · cited 1×
“In the literature, there is a body of work focusing on testing machine learning (ML) libraries as well [17–19, 56, 59].”
BackgroundKeras R-CNN: library for cell detection in biological images using deep neural networks
Hung, J., Goodman, A., Ravel, D., Lopes, S. C. P., Rangel, G. W., Nery, O. A., Malleret, B., Nosten, F. H., Lacerda, M., Ferreira, M. U., et al. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-03635-x · Cited by 88 · cited 1×
“Keras [4] is one of the most popular high-level DL libraries that has been widely used in various domains [15, 32, 40].”
BackgroundContinuous Incident Triage for Large-Scale Online Service Systems
Chen, J., He, X., Lin, Q., Zhang, H., Hao, D., Gao, F., Xu, Z., Dang, Y., & Zhang, D. (2019) · 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00042 · Cited by 80 · cited 1×
“DL systems ( i.e. , software systems based on DL techniques) have been widely adopted in various domains in practice, e.g. , self-driving cars [28], virtual assistants [34], and software operations [11, 12, 25].”
MethodAutomated black-box detection of side-channel vulnerabilities in web applications
Chapman, P., & Evans, D. (2011) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2046707.2046737 · Cited by 70 · cited 1×
“Differential testing has been successful in uncovering bugs across various types of programs, such as compilers [58], Java Virtual Machine (JVM) implementations [13, 14], web applications [10], and security-related APIs [46].”
BackgroundDetecting numerical bugs in neural network architectures
Zhang, Y., Ren, L., Chen, L., Xiong, Y., Cheung, S.-C., & Xie, T. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409720 · Cited by 70 · cited 2×
“For instance, Zhang et al. [62] propose DEBAR, a static analysis approach for detecting numerical bugs in DL models.”
BackgroundPerception and Practices of Differential Testing
Gulzar, M. A., Zhu, Y., & Han, X. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00016 · Cited by 58 · cited 1×
“Our work is also related to differential testing, an effective method that use similar programs as cross referencing oracles to detect bugs [26].”
Method(7794023), X. X., (7695560), Z. Z., (9865748), T. C., (4702669), Y. L., (9827297), P. P., & (7787783), B. X. (2020) · Figshare · Cited by 57
Testing probabilistic programming systems
Dutta, S., Legunsen, O., Huang, Z., & Misailovic, S. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236057 · Cited by 49 · cited 1×
“[17] propose ProbFuzz to test probabilistic programming systems via generating programs based on pre-defined templates.”
BackgroundDeep Validation: Toward Detecting Real-World Corner Cases for Deep Neural Networks
Wu, W., Xu, H., Zhong, S., Lyu, M. R., & King, I. (2019) · 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) · doi:10.1109/dsn.2019.00026 · Cited by 48 · cited 1×
“quite challenging to find specific inputs that can trigger corner cases [54].”
BackgroundA security policy oracle: detecting security holes using multiple API implementations
Srivastava, V. K. L., Bond, M. D., McKinley, K. S., & Shmatikov, V. (2011) · ACM SIGPLAN Notices · doi:10.1145/1993316.1993539 · Cited by 44
Chen, J., Zhang, S., He, X., Lin, Q., Zhang, H., Hao, D., Kang, Y., Gao, F., Xu, Z., Dang, Y., et al. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416624 · Cited by 37 · cited 1×
“DL systems ( i.e. , software systems based on DL techniques) have been widely adopted in various domains in practice, e.g. , self-driving cars [28], virtual assistants [34], and software operations [11, 12, 25].”
MethodStorm: program reduction for testing and debugging probabilistic programming systems
Dutta, S., Zhang, W., Huang, Z., & Misailovíc, S. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338972 · Cited by 35
Efficient customer incident triage via linking with system incidents
Gu, J., Wen, J., Wang, Z., Zhao, P., Luo, C. W., Kang, Y., Zhou, Y., Li, Y., Sun, J. C., Xu, Z., et al. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417061 · Cited by 20 · cited 1×
“DL systems ( i.e. , software systems based on DL techniques) have been widely adopted in various domains in practice, e.g. , self-driving cars [28], virtual assistants [34], and software operations [11, 12, 25].”
MethodSelf-citationTotal Solar Irradiance Forecasting with Keras Recurrent Neural Networks
Muralikrishna, A., Vieira, L. E. A., Santos, R., & Almeida, A. P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58814-4_18 · Cited by 3 · cited 1×
“Keras [4] is one of the most popular high-level DL libraries that has been widely used in various domains [15, 32, 40].”
Background
Research works printed but not linked: 9
- Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In Proc. of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI. USENIX Association, 265--283.
- 10.1002/(sici)1099-0526(199703/04)2:4 · doi:10.1002/(sici)1099-0526(199703/04)2:4
- Ian J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org/
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems. 2672--2680.
- Vinod Nair and Geoffrey E. Hinton. 2010. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proc. of the 27th International Conference on Machine Learning, ICML. Omnipress, 807--814.
- Augustus Odena, Catherine Olsson, David G. Andersen, and Ian J. Goodfellow. 2019. TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In Proc. of the 36th International Conference on Machine Learning, ICML (Proceedings of Machine Learning Research, Vol. 97). PMLR, 4901--4911.
- Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. of the 3rd International Conference on Learning Representations, ICLR, Yoshua Bengio and Yann LeCun (Eds.).
- The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, et al. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 (2016). · arXiv
- Martin Wistuba, Ambrish Rawat, and Tejaswini Pedapati. 2019. A Survey on Neural Architecture Search. CoRR abs/1905.01392 (2019). arXiv:1905.01392 http://arxiv.org/abs/1905.01392 · arXiv
Cited by
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
“While such model-level fuzzers are able to nd bugs in DL libraries, due to the input/output constraints of DL APIs, model-level mutation/generation rules either are restrictive to certain shape-preserving APIs [71] or require manual annotation of the restrictions of all targeted APIs [25], leading to a limited number…”
BackgroundMethodInfluentialDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623343 · Cited by 115
Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
Zhang, Z., Chen, C., Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., & Wang, R. (2024) · Transactions on machine learning research · doi:10.13140/rg.2.2.32710.69440/1 · Cited by 194
WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Yang, C., Deng, Y., Lu, R., Yao, J., Liu, J., Jabbarvand, R., & Zhang, L. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689736 · Cited by 82
“Therefore, they can be effectively triggered by TitanFuzz since TitanFuzz leverages LLMs to generate human-like programs by resembling the distribution of training data.”
BackgroundNNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers
Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., & Zhang, L. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575707 · Cited by 167
Metadata
- Authors
- Gu, J., Luo, X., Zhou, Y., & Wang, X.
- Affiliations
- Fudan University (CN)
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 1418-1430
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4284708930
- Cited by
- 71
- References
- 47
- Citations by year
- 2026: 13 · 2025: 26 · 2024: 18 · 2023: 13 · 2022: 1
- FWCI
- 10.94
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Funders
- National Key Research and Development Program of China (2020YFA0711400)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- differential testing, muffins, TensorFlow, CNTK, Theano, software defect detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.