Finding and understanding bugs in C compilers
The DOI matches its Crossref record; 22 of the 23 references it cites were found in a registry or an index.
Abstract
Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to compiler developers. Every compiler we tested was found to crash and also to silently generate wrong code when presented with valid input. In this paper we present our compiler-testing tool and the results of our bug-hunting study. Our first contribution is to advance the state of the art in compiler testing. Unlike previous tools, Csmith generates programs that cover a large subset of C while avoiding the undefined and unspecified behaviors that would destroy its ability to automatically find wrong-code bugs. Our second contribution is a collection of qualitative and quantitative results about the bugs we have found in open-source C compilers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This paper demonstrates that modern titanic LLMs can be leveraged to directly perform both generation-based and mutation-based fuzzing studied for decades, while being fully automated, generalizable, and applicable to domains challenging for traditional approaches (such as DL systems).”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [80]
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Klein, G., Elphinstone, K., Heiser, G., Andronick, J., Cock, D., Derrin, P., Elkaduwe, D., Engelhardt, K., Kolanski, R., Norrish, M., et al. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629596 · Cited by 1,633 · cited 1×
“As measured by the responses to our bug reports, the defects discovered by Csmith are important.”
MethodAn empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,281
Formal verification of a realistic compiler
Leroy, X. (2009) · Communications of the ACM · doi:10.1145/1538788.1538814 · Cited by 1,187 · cited 1×
“Wolfe [30], talk ing about independent software vendors (ISVs) says: An ISV with a complex code can work around correctness, turn off the optimizer in one or two .les, and usually they have to do that for any of the compilers they use (emphasis ours).”
BackgroundEncyclopedia of Software Engineering
(2002) · Wiley eBooks · doi:10.1002/0471028959 · Cited by 1,161
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079
An experimental evaluation of the assumption of independence in multiversion programming
Knight, J. C., & Leveson, N. G. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312924 · Cited by 774 · cited 3×
“In summary, despite the fact that Knight and Leveson [13] found a substantial number of correlated errors in an experiment on Nversion programming, Csmith has yielded no evidence of correlated failures among unrelated C compilers.”
BackgroundInfluential(1998) · Cited by 469
Grammar-based whitebox fuzzing
Godefroid, P., Kiezun, A., & Levin, M. Y. (2008) · Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1375581.1375607 · Cited by 462 · cited 1×
“[8] showed a way to solve this problem by integrating a constraint solver with a grammar for the language being generated.”
BackgroundHDD: hierarchical delta debugging
Misherghi, G., & Su, Z. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134307 · Cited by 274 · cited 1×
“Delta debugging [31] automates test-case reduction, but all existing variants that are intended for reducing C programs—such as hierarchical delta debugging [18] and Wilkerson’s implementation [29]—introduce undefined behavior.”
Background
Research works printed but not linked: 7
- F. Bellard . TCC: Tiny C compiler, ver. 0.9.25 , May 2009 . http://bellard.org/tcc/. F. Bellard. TCC: Tiny C compiler, ver. 0.9.25, May 2009. http://bellard.org/tcc/.
- C. L. Biffle . Undefined behavior in Google NaCl , Jan. 2010 . http://code.google.com/p/nativeclient/issues/detail?id=245. C. L. Biffle. Undefined behavior in Google NaCl, Jan. 2010. http://code.google.com/p/nativeclient/issues/detail?id=245.
- 10.1002/spe.v37:14 · doi:10.1002/spe.v37:14
- B. Turner . Random Program Generator , Jan. 2007 . http://sites.google.com/site/brturn2/randomcprogramgenerator. B. Turner. Random Program Generator, Jan. 2007. http://sites.google.com/site/brturn2/randomcprogramgenerator.
- 10.5555/1060289.1060313 · doi:10.5555/1060289.1060313
- D. S. Wilkerson . Delta ver . 2006 .08.03, Aug. 2006. http://delta.tigris.org/. D. S. Wilkerson. Delta ver. 2006.08.03, Aug. 2006. http://delta.tigris.org/.
- C. Zhao Automated test program generation for an industrial optimizing compiler . In Proc. ICSE Workshop on Automation of Software Test , pages 36 -- 43 , May 2009 . C. Zhao et al. Automated test program generation for an industrial optimizing compiler. In Proc. ICSE Workshop on Automation of Software Test, pages 36--43, May 2009.
Cited by
FairFuzz: a targeted mutation strategy for increasing greybox fuzz testing coverage
Lemieux, C., & Sen, K. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238176 · Cited by 477
“In practice, this trade-off has paid off, and fuzz testers have found numerous correctness bugs and security vulnerabilities in widely used software [9, 14, 25, 26, 28, 34, 39, 41, 43].”
BackgroundMethodCompiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388
“Others, however, can cause compilers to silently miscompile a program and produce wrong code, subverting the programmer’s intent.”
BackgroundInfluentialSkyfire: Data-Driven Seed Generation for Fuzzing
Wang, J., Chen, B., Wei, L., & Liu, Y. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.23 · Cited by 315
“CSmith [19], LangFuzz [20], IFuzzer [21], Radamsa [41], and the approach in [18] use the context-free grammar as the specification to generate inputs.”
BackgroundMethodDeng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
“Additionally, TitanFuzz is the rst work to demonstrate that LLMs can perform both generation-based [30, 80] and mutation-based [39, 50] fuzzing studied for decades [84], while being fully automated, generalizable, and applicable to domains challenging for traditional approaches (such as DL systems).”
BackgroundTest-case reduction for C compiler bugs
Regehr, J., Chen, Y., Cuoq, P., Eide, E., Ellison, C., & Yang, X. (2012) · Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2254064.2254104 · Cited by 310
“However, we have created two reducers that rely on the fact that Csmith already knows how to create conforming C programs [17].”
BackgroundSelf-citationSuperion: Grammar-Aware Greybox Fuzzing
Wang, J., Chen, B., Wei, L., & Liu, Y. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00081 · Cited by 260
“A fuzzer can be classified as generation-based (e.g., [32, 62, 64, 69]) or mutation-based (e.g., [9, 43, 55, 59]), depending on whether test inputs are generated by the knowledge of the input format or grammar or by modifying well-formed test inputs.”
Learning to Fuzz from Symbolic Execution with Application to Smart Contracts
He, J., Balunović, M., Ambroladze, N., Tsankov, P., & Vechev, M. (2019) · Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3319535.3363230 · Cited by 269
Verdi: a framework for implementing and formally verifying distributed systems
Wilcox, J. R., Woos, D., Panchekha, P., Tatlock, Z., Wang, X., Ernst, M. D., & Anderson, T. (2015) · Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2737924.2737958 · Cited by 270
“Previous work has shown that formal verification can help produce extremely reliable systems, including compilers [41] and operating systems [18, 39].”
BackgroundMethodLiang, H., Pei, X., Jia, X., Shen, W., & Zhang, J. (2018) · IEEE Transactions on Reliability · doi:10.1109/tr.2018.2834476 · Cited by 231
“[117] in 2011.”
Fuzz4ALL: Universal Fuzzing with Large Language Models
Xia, C. S., Paltenghi, M., Le Tian, J., Pradel, M., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639121 · Cited by 229
Program-Adaptive Mutational Fuzzing
Cha, S. K., Woo, M., & Brumley, D. (2015) · 2015 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2015.50 · Cited by 231
“generation-based fuzzing) [21, 24, 40, 45, 56] is another class of fuzzing, which differs from mutational fuzzing since it does not require a seed file.”
BackgroundCompiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · ACM SIGPLAN Notices · doi:10.1145/2666356.2594334 · Cited by 218
Chen, J., Patra, J., Pradel, M., Xiong, Y., Zhang, H., Hao, D., & Zhang, L. (2021) · ACM Computing Surveys · doi:10.1145/3363562 · Cited by 215
“In test program generation, to generate only valid programs without undefined behaviors, tools like Csmith [123] rely on static analysis to approximate the boundary of a valid program, and thus cannot generate all valid programs.”
BackgroundMethodInfluentialBrubaker, C., Jana, S., Ray, B., Khurshid, S., & Shmatikov, V. (2014) · 2014 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2014.15 · Cited by 218
“Unlike [29, 31], we developed an automated technique for discovering certificate validation vulnerabilities.”
MethodAn executable formal semantics of C with applications
Ellison, C., & Rosu, G. (2012) · ACM SIGPLAN Notices · doi:10.1145/2103621.2103719 · Cited by 188
CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries
Pham, H. V., Lutellier, T., Qi, W., & Tan, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00107 · Cited by 190
Coverage-directed differential testing of JVM implementations
Chen, Y., Su, T., Sun, C., Su, Z., & Zhao, J. (2016) · Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2908080.2908095 · Cited by 169
“…there do not exist any models or automated means for identifying the root cause of the discrepancy, although many general purposed test case reducers ( e.g. , AST-Based reducer and the modular reducer [31] for CSmith [41]) do exist for pruning the large irrelevant portions from test inputs.”
BackgroundSelf-citationFinding compiler bugs via live code mutation
Sun, C., Le, V., & Su, Z. (2016) · Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications · doi:10.1145/2983990.2984038 · Cited by 168
“Csmith [35] and all these efforts [2, 20, 21, 36] generate test programs from scratch.”
BackgroundMethodInfluentialAn executable formal semantics of C with applications
Ellison, C., & Rosu, G. (2012) · Proceedings of the 39th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages · doi:10.1145/2103656.2103719 · Cited by 169
Padhye, R., Lemieux, C., Sen, K., Papadakis, M., & Traon, Y. L. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330576 · Cited by 223
PerfFuzz: automatically generating pathological inputs
Lemieux, C., Padhye, R., Sen, K., & Song, D. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213874 · Cited by 158
“Fuzz testing , in which a program is bombarded with many randomly generated inputs, has been very successful in finding security vulnerabilities and correctness bugs [15, 27, 37, 38, 47, 51, 58, 60].”
BackgroundCompiler fuzzing through deep learning
Cummins, C., Petoumenos, P., Murray, A., & Leather, H. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213848 · Cited by 147
“While CSmith has been successfully used to identify hundreds of bugs in compilers, it and similar approaches have a significant drawback.”
BackgroundMethodInfluentialLidbury, C., Lascu, A., Chong, N., & Donaldson, A. F. (2015) · Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2737924.2737986 · Cited by 143
“To avoid undefined behaviours arising from vector computations we designed a set of “safe math” vector macros, following the approach used by Csmith for scalar operations [20].”
BackgroundMethodInfluentialChen, Y., Groce, A., Zhang, C., Wong, W.-K., Fern, X., Eide, E., & Regehr, J. (2013) · Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2491956.2462173 · Cited by 145
“” For C compilers, Csmith [44] has identified more than 450 previously unknown bugs.”
BackgroundMethodInfluentialSelf-citationFinding deep compiler bugs via guided stochastic program mutation
Le, V., Sun, C. P., & Su, Z. (2015) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2814270.2814319 · Cited by 152
“Both bugs start from seed programs generated by Csmith [30], and trigger by a sequence of mutations, i.e. , inserting and deleting statements derived from the seeds.”
BackgroundMethodInfluentialRandom testing for C and C++ compilers with YARPGen
Livinskii, V., Babokin, D., & Regehr, J. D. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428264 · Cited by 171
“The Csmith [Yang et al. 2011] program generator avoided some undefined behaviors using static analysis, but used wrapper functions to prevent unsafe arithmetic operations.”
BackgroundMethodResultInfluentialSelf-citationTranslation validation for a verified OS kernel
Sewell, T. A. L., Myreen, M. O., & Klein, G. (2013) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2491956.2462183 · Cited by 157
“In their study on compiler testing, Yang et al reported 325 previously unknown defects in 11 different C compilers [34].”
BackgroundCodeAlchemist: Semantics-Aware Code Generation to Find Vulnerabilities in JavaScript Engines
Han, H.-S., Oh, D., & Cha, S. K. (2019) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23263 · Cited by 135
Synthesizing program input grammars
Bastani, O., Sharma, R., Aiken, A., & Liang, P. (2017) · Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/3062341.3062349 · Cited by 131
Effective Program Debloating via Reinforcement Learning
Heo, K., Lee, W., Pashakhanloo, P., & Naik, M. (2018) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3243734.3243838 · Cited by 150
“We can mitigate this issue by using grammar-based fuzzers [25, 44].”
MethodProvably correct peephole optimizations with alive
Lopes, N. P., Menendez, D., Nagarakatte, S., & Regehr, J. D. (2015) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2737924.2737965 · Cited by 150
“…and Subject Descriptors D.2.4 [Programming Languages]: Software/Program Verification; D.3.4 [Programming Languages]: Processors—Compilers; F.3.1 [Logics and Meanings of Programs]: Specifying and Verifying and Reasoning about Programs Keywords Compiler Verification, Peephole Optimization, Alive”
BackgroundInfluentialSelf-citationMorrisett, G., Tan, G., Tassarotti, J., Tristan, J.-B., & Gan, E. (2012) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2254064.2254111 · Cited by 131
A comprehensive study of deep learning compiler bugs
Shen, Q., Ma, H., Chen, J., Tian, Y., Cheung, S.-C., & Chen, X. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468591 · Cited by 179
“[71] developed Csmith, the most widely-used C test program generation tool, to test C compilers.”
MethodToward understanding compiler bugs in GCC and LLVM
Sun, C. P., Le, V., Zhang, Q., & Su, Z. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931074 · Cited by 151
“It is surprising that most of the research and engineering efforts have been devoted to testing or validating C compilers [3, 11, 12, 13, 14, 23, 29, 34, 35] but little on C++, although C++ is one of the most popular programming languages in industry (among top three according to the TIOBE index [33]).”
BackgroundMethodInfluentialAutomated testing of graphics shader compilers
Donaldson, A. F., Evrard, H., Lascu, A., & Thomson, P. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133917 · Cited by 160
“A popular method for circumventing the oracle problem in compiler validation is differential testing [McKeeman 1998; Yang et al. 2011], whereby multiple compilers for the same language are cross-checked against a set of (usually randomly-generated) programs.”
MethodFinding bugs in database systems via query partitioning
Rigger, M., & Su, Z. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428279 · Cited by 176
“It has proven to be effective in many domains [Brummayer and Biere 2009; Kapus and Cadar 2017; McKeeman 1998; Yang et al. 2011].”
BackgroundDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623343 · Cited by 115
“More importantly, compared with traditional fuzzing techniques that require intensive human efforts for building generation/mutation strategies [16, 28, 31, 32, 60, 76, 82], TitanFuzz is fully automated, and can be easily generalized to different application domains and program-minglanguages.”
BackgroundPerses: syntax-guided program reduction
Sun, C. P., Li, Y., Zhang, Q., Gu, T., & Su, Z. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180236 · Cited by 135
“In contrast, the original bug-triggering test programs may have hundreds or thousands of lines of code [9, 11, 22, 25].”
BackgroundAlive2: bounded translation validation for LLVM
Lopes, N. P., Lee, J., Hur, C.-K., Liu, Z., & Regehr, J. D. (2021) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3453483.3454030 · Cited by 168
“Tools like Csmith [48], EMI [20], and SPE [50] have found hundreds of bugs in commercial compilers, including GCC, LLVM, and MSVC.”
MethodSelf-citationNEZHA: Efficient Domain-Independent Differential Testing
Petsios, T., Tang, A., Stolfo, S. J., Keromytis, A. D., & Jana, S. (2017) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2017.27 · Cited by 161
“Similarly, existing differential testing tools for C compilers, Java virtual machines, and malware detectors, all include grammars for the respective input format and use domain-specific mutations [31], [41], [65].”
BackgroundMethodEfficient sampling of SAT solutions for testing
Dutra, R., Laeufer, K., Bachrach, J. R., & Sen, K. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180248 · Cited by 114
“If multiple solutions could be generated efficiently, this would significantly speedup symbolic execution and reap the benefits of random testing [6, 17, 21, 22, 34, 45, 46].”
BackgroundAn empirical comparison of compiler testing techniques
Chen, J., Hu, W., Hao, D., Xiong, Y., Zhang, H., Zhang, L., & Xie, B. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884878 · Cited by 107
“To compare the effectiveness of the three compiler testing techniques, we use CSmith [29] to generate a set of test programs and use this set of test programs as the inputs to the compilers.”
BackgroundMethodInfluentialDetecting optimization bugs in database engines via non-optimizing reference engine construction
Rigger, M., & Su, Z. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409710 · Cited by 134
“Similarly to metamorphic testing, differential testing has proven to be effective in many domains [6, 16, 31, 33, 36, 61].”
BackgroundMethodFioraldi, A., Maier, D., Zhang, D., & Balzarotti, D. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560602 · Cited by 106
“[74]); (2)Mutation-based generation uses instead previous testcases stored in a corpus to derive new inputs.”
MethodFuzzing JavaScript Engines with Aspect-preserving Mutation
Park, S., Xu, W. X., Yun, I., Jang, D., & Kim, T. (2020) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40000.2020.00067 · Cited by 111
DeepFuzz: Automatic Generation of Syntax Valid C Programs for Fuzz Testing
Liu, X., Li, X., Prajapati, R., & Wu, D. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011044 · Cited by 95
“Csmith (Yang et al. 2011) is a tool that can generate random C programs.”
BackgroundMethodDifFuzz: Differential Fuzzing for Side-Channel Analysis
Nilizadeh, S., Noller, Y., & Păsăreanu, C. S. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00034 · Cited by 100
“l fuzzing has been successfully applied before for finding bugs and vulnerabilities in a variety of applications, such as LF and XZ parsers, PDF viewers, SSL/TLS libraries, and C compilers [36], [38], [41]. However, to the best of our knowledge, we are the first to explore differential fuzzing for side-channel analysis.…”
MethodWhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Yang, C., Deng, Y., Lu, R., Yao, J., Liu, J., Jabbarvand, R., & Zhang, L. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689736 · Cited by 82
“We then discuss the recent advances of LLM-based testing.”
MethodNNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers
Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., & Zhang, L. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575707 · Cited by 167
“As one of the most widely studied compiler fuzzing approaches in the literature [39], grammar-based techniques (such as Csmith [67], jsfunfuzz [56], and LangFuzz [24]) aim to generate syntactically valid input programs acceptable by the underlying compilers.”
Groce, A., Zhang, C., Eide, E. N., Chen, Y., & Regehr, J. D. (2012) · International Symposium on Software Testing and Analysis · doi:10.1145/2338965.2336763 · Cited by 119
“…coverage and fault detection, it has been assumed that a good random test configuration includes as many API calls or other input domain features as possible, and this has been the guiding principle in largescale efforts to test C compilers [36], file systems [17], and utility libraries [29].”
BackgroundSelf-citationSkeletal program enumeration for rigorous compiler testing
Zhang, Q., Sun, C. P., & Su, Z. (2017) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3062341.3062379 · Cited by 126
Baldur: Whole-Proof Generation and Repair with Large Language Models
First, E., Rabe, M. N., Ringer, T., & Brun, Y. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616243 · Cited by 94
“For example, CompCert, a C compiler verified using the Coq interactive theorem prover [88], was the only compiler on a list including the ubiquitous GCC and LLVM, in which a comprehensive study found no bugs [106].”
MethodSteering symbolic execution to less traveled paths
Li, Y., Su, Z., Wang, L., & Li, X. (2013) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2509136.2509553 · Cited by 94
“The recent work on Csmith [33] gives more concrete evidence.”
BackgroundMethodFormal verification of SSA-based optimizations for LLVM
Zhao, J., Nagarakatte, S., Martin, M. M. K., & Zdancewic, S. (2013) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2491956.2462164 · Cited by 93
“A recent study by Yang et al. [20] used random test-case generation to expose serious bugs in mainstream compilers including GCC, LLVM, and commercial compilers.”
BackgroundEfficient Compiler Autotuning via Bayesian Optimization
Chen, J., Xu, N., Chen, P., & Zhang, H. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00110 · Cited by 93
“Csmith is a state-of-the-art C program generator, and its produced programs tend to invoke a large amount of compiler-optimization code, which facilitates the evaluation of compiler autotuning.”
BackgroundInfluentialPolygeist: Raising C to Polyhedral MLIR
Moses, W. S., Chelini, L., Zhao, R., & Zinenko, O. (2021) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1109/pact52795.2021.00011 · Cited by 109
“Beyond affecting affine schedules, this technique enables easy rematerialization and local transposition buffers, crucial on GPUs [55], as well as software pipelining; all without having to produce C source which is known to be complex [56].”
BackgroundA systematic review of fuzzing based on machine learning techniques
Wang, Y., Jia, P., Liu, L., Huang, C., & Liu, Z. (2020) · PLoS ONE · doi:10.1371/journal.pone.0237749 · Cited by 100
Synthesizing benchmarks for predictive modeling
Cummins, C., Petoumenos, P., Wang, Z., & Leather, H. (2017) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo.2017.7863731 · Cited by 114
“Grammar-based fuzz testers have been developed for C [37] and OpenCL [27].”
Into the depths of C: elaborating the de facto standards
Memarian, K., Matthiesen, J., Lingard, J., Nienhuis, K., Chisnall, D., Watson, R. N. M., & Sewell, P. (2016) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2908080.2908081 · Cited by 89
“Of their 561 Csmith tests [55], Cerberus currently gives the same result as GCC for 556; the other 5 time-out after 5min.”
MethodLearning to Prioritize Test Programs for Compiler Testing
Chen, J., Bai, Y., Hao, D., Xiong, Y., Zhang, H., & Xie, B. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.70 · Cited by 90
“[2], [33] proposed and implemented a tool, called CSmith, to randomly generate C programs without undefined behaviors for testing C compilers, and Regehr et al.”
BackgroundMethodInfluentialEverCrypt: A Fast, Verified, Cross-Platform Cryptographic Provider
Protzenko, J., Parno, B. J., Fromherz, A., Hawblitzel, C., Polubelova, M., Bhargavan, K., Beurdouche, B., Choi, J., Delignat-Lavaud, A., Fournet, C., et al. (2020) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40000.2020.00114 · Cited by 150
“use of such complex verification toolchains, the empirical result is qualitatively better code compared with traditional software development practices [37, 38, 79].”
MethodFinding deep compiler bugs via guided stochastic program mutation
Le, V., Sun, C. P., & Su, Z. (2015) · ACM SIGPLAN Notices · doi:10.1145/2858965.2814319 · Cited by 96
Frigate: A Validated, Extensible, and Efficient Compiler and Interpreter for Secure Computation
Mood, B., Gupta, D., Carter, H., Butler, K. R. B., & Traynor, P. (2016) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp.2016.20 · Cited by 83
“When tested with Csmith [54], there were still errors found that demonstrated the limitations of formal verification.”
Automated conformance testing for JavaScript engines via deep compiler fuzzing
Ye, G., Tang, Z., Tan, S. H., Huang, S., Fang, D., Sun, X., Bian, L., Wang, H., & Wang, Z. (2021) · Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation · doi:10.1145/3453483.3454054 · Cited by 77
“CSmith [65] is an effective program generator for generating C test programs using predefined grammar rules.”
MethodInfluentialICS Protocol Fuzzing: Coverage Guided Packet Crack and Generation
Luo, Z., Zuo, F., Shen, Y., Jiao, X., Chang, W., & Jiang, Y. (2020) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac18072.2020.9218603 · Cited by 77
“Grammar-based fuzzers generate inputs by leveraging the given context-free grammar [16], [17], [18].”
BackgroundCompilerGym: Robust, Performant Compiler Optimization Environments for AI Research
Cummins, C., Wasti, B., Guo, J., Cui, B., Ansel, J., Gomez, S., Jain, S., Liu, J., Teytaud, O., Steiner, B., et al. (2022) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo53902.2022.9741258 · Cited by 120
“TABLE VI: Comparison of four reinforcement learning algorithms, trained for 100k episodes on Csmith [8] programs, when evaluated on datasets from a range of program domains.”
BackgroundMethodInfluentialTesting Static Analyzers with Randomly Generated Programs
Cuoq, P., Monate, B., Pacalet, A., Prévosto, V., Regehr, J. D., Yakobowski, B., & Yang, X. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-28891-3_12 · Cited by 76
“Csmith [9] is an automatic generator of random C programs, originally intended for differential testing [6] of compilers.”
BackgroundSelf-citationFinding and analyzing compiler warning defects
Sun, C. P., Le, V., & Su, Z. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884879 · Cited by 81
“Among them, two notable efforts are Csmith [13,33,37] and Orion [21].”
BackgroundMethodInfluentialGrayC: Greybox Fuzzing of Compilers and Analysers for C
Even-Mendoza, K., Sharma, A., Donaldson, A. F., & Cadar, C. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598130 · Cited by 87
Optimizing test prioritization via test distribution analysis
Chen, J., Lou, Y., Zhang, L., Zhou, J., Wang, X., Hao, D., & Zhang, L. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236053 · Cited by 87
“Here following prior work [56, 77], we regard each failure as each fault in industrial subjects, since it is hard to distinguish whether different failures are caused by the same fault [77] and many faulty versions in the study have only one failure.”
BackgroundCompiler testing via a theory of sound optimisations in the C11/C++11 memory model
Morisset, R., Pawan, P., & Nardelli, F. Z. (2013) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2491956.2491967 · Cited by 73
“Test-case generation We rely on a small extension of Csmith [24] to generate random programs that cover a large subset of C while avoiding undefined and unspecified behaviours.”
BackgroundMethodMechanising and verifying the WebAssembly specification
Watt, C. (2017) · ACM SIGPLAN International Conference on Certified Programs and Proofs (CPP) · doi:10.1145/3167082 · Cited by 73
An Empirical Study on the Correctness of Formally Verified Distributed Systems
Fonseca, P., Zhang, K., Wang, X., & Krishnamurthy, A. (2017) · European Conference on Computer Systems · doi:10.1145/3064176.3064183 · Cited by 84
“Nevertheless, our study analyzed the largest number of verified implementations and the largest number of bugs, relative to all previous studies of which we are aware [55].”
BackgroundResultFuzzing the Rust Typechecker Using CLP (T)
Dewey, K. T., Roesch, J., & Hardekopf, B. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.65 · Cited by 91
“This idea is well-established as a confidence-building and bug-finding technique for compilers and interpreters; for example, thousands of bugs have collectively been found by the jsfunfuzz [1], LangFuzz [2], and CSmith [3] language fuzzers.”
KernelGPT: Enhanced Kernel Fuzzing via Large Language Models
Yang, C., Zhao, Z., & Zhang, L. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716022 · Cited by 130
Discipline Matters: Refactoring of Preprocessor Directives in the #ifdef Hell
Medeiros, F., Ribeiro, M., Gheyi, R., Apel, S., Kästner, C., Ferreira, B. G., Carvalho, L. C. D. S., & Fonseca, B. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2688333 · Cited by 72
How far we have come: testing decompilation correctness of C decompilers
Liu, Z., & Wang, S. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397370 · Cited by 78
“2.3.0) [68] to generate seed C programs. gcov (ver.”
BackgroundMethodInfluentialLidbury, C., Lascu, A., Chong, N., & Donaldson, A. F. (2015) · ACM SIGPLAN Notices · doi:10.1145/2813885.2737986 · Cited by 70
Zhao, L., Li, G., Sutter, B. D., & Regehr, J. D. (2011) · ACM International Conference on Embedded Software (EMSOFT) · doi:10.1145/2038642.2038687 · Cited by 70
“compiler bugs are not uncommon [25] and can undermine the safety policy.”
BackgroundSelf-citationTest Case Prioritization for Compilers: A Text-Vector Based Approach
Chen, J., Bai, Y., Hao, D., Xiong, Y., Zhang, H., Zhang, L., & Xie, B. (2016) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2016.19 · Cited by 64
“The two subjects cover all C compilers used in the literature of C compiler testing [20], [21], [22].”
BackgroundMethodResultInfluentialAn empirical study of optimization bugs in GCC and LLVM
Zhou, Z., Ren, Z., Gao, G., & Jiang, H. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110884 · Cited by 69
QDiff: Differential Testing of Quantum Software Stacks
Wang, J., Zhang, Q., Xu, G. H., & Kim, M. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678792 · Cited by 126
“Csmith [55] randomly generates C programs and checks for inconsistent behaviors via differential testing.”
BackgroundHistory-driven test program synthesis for JVM testing
Zhao, Y., Wang, Z., Chen, J., Liu, M., Wu, M., Zhang, Y., & Zhang, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510059 · Cited by 79
“[53] proposed Csmith, which generates C programs based on the grammar of the C language.”
Verdi: a framework for implementing and formally verifying distributed systems
Wilcox, J. R., Woos, D., Panchekha, P., Tatlock, Z., Wang, X., Ernst, M. D., & Anderson, T. E. (2015) · ACM SIGPLAN Notices · doi:10.1145/2813885.2737958 · Cited by 70
Kargén, U., & Shahmehri, N. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2786844 · Cited by 60
“use generation-based fuzzing to find bugs in C compilers [33], and Holler et al.”
MethodCompiler bug isolation via effective witness test program generation
Chen, J., Han, J., Sun, P., Zhang, L., Hao, D., & Zhang, L. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338957 · Cited by 63
“Besides, randomly generated test programs via test-program generation tools like Csmith [82] have been demonstrated to be quite eective for detecting compiler bugs [15, 82].”
BackgroundMethodInfluentialHistory-Guided Configuration Diversification for Compiler Test-Program Generation
Chen, J., Wang, G., Hao, D., Xiong, Y., Zhang, H., & Zhang, L. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00037 · Cited by 64
“For example, Csmith [1], the most widely-used C program generator, randomly generates C programs according to a test configuration with 71 options.”
BackgroundMethodInfluentialDifferential regression testing for REST APIs
Godefroid, P., Lehmann, D., & Polishchuk, M. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397374 · Cited by 85
“, clang and gcc), and then comparing the runtime behavior of the binaries produced by each compiler against each other [48].”
BackgroundJIT-Ppcking: differential fuzzing of JavaScript engines
Bernhard, L., Scharnowski, T., Schlögel, M., Blazytko, T., & Holz, T. (2022) · Dokumentenrepositorium der RUB (Ruhr University Bochum) · doi:10.1145/3548606.3560624
“Previously, differential testing was used to uncover bugs in CPUs [24], x509 certificate validation [7, 42], C compilers [29, 63] and the JVM [10].”
MethodTest-case reduction for C compiler bugs
Regehr, J. D., Chen, Y., Cuoq, P., Eide, E. N., Ellison, C., & Yang, X. (2012) · ACM SIGPLAN Notices · doi:10.1145/2345156.2254104 · Cited by 58
Self-citationOn the unusual effectiveness of type-aware operator mutations for testing SMT solvers
Winterer, D., Zhang, C., & Su, Z. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428261 · Cited by 85
“, for testing compilers [Cummins et al. 2018; Le et al. 2014; Lidbury et al. 2015; Yang et al. 2011; Zhang et al. 2017], testing database management systems [Jung et al.”
MethodANGHABENCH: A Suite with One Million Compilable C Benchmarks for Code-Size Reduction
da Silva, A. F., Kind, B. C., Magalhães, J. W. D. S., Rocha, J. N., Guimarães, B. C. F., & Pereira, F. M. Q. (2021) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo51591.2021.9370322 · Cited by 55
“Random program generators, such as CS MITH [31], LDRG EN [32] and Orange3 [52], [53] have been successfully used to produce C programs for stress-testing compilers, often finding correctness bugs in industrial implementations [54].”
MethodInfluentialOne Engine to Fuzz ’em All: Generic Language Processor Testing with Semantic Validation
Chen, Y., Zhong, R., Hu, H. P., Zhang, H., Yang, Y., Wu, D., & Lee, W. (2021) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40001.2021.00071 · Cited by 55
“To further improve the semantic correctness, recent fuzzers adopt highly specialized analyses for specific languages [53, 74, 78].”
BackgroundInfluentialSelf-citationRandomized stress-testing of link-time optimizers
Le, V., Sun, C. P., & Su, Z. (2015) · International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771785 · Cited by 57
“Our evaluation also benefited significantly from Berkeley Delta [13], and University of Utah’s Csmith [25] and C-Reduce [19] tools.”
BackgroundMethodInfluentialA new verified compiler backend for CakeML
Tan, Y. K., Myreen, M. O., Kumar, R., Fox, A. C. J., Owens, S., & Norrish, M. (2016) · ACM SIGPLAN International Conference on Functional Programming (ICFP) · doi:10.1145/2951913.2951924 · Cited by 72
“Optimising compilers are complex pieces of software and, as such, errors are almost inevitable in their implementations, as Yang et al. (2011) showed with systematic experiments.”
BackgroundCoverage-guided tensor compiler fuzzing with joint IR-pass mutation
Liu, J., Wei, Y., Yang, S., Deng, Y., & Zhang, L. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3527317 · Cited by 51
“However, it suffers from the common limitations of generation-based fuzzing techniques [Holler et al. 2012; Yang et al. 2011], e.”
BackgroundMethodThe verified CakeML compiler backend
Tan, Y. K., Myreen, M. O., Kumar, R., Fox, A. C. J., Owens, S., & Norrish, M. (2019) · Journal of Functional Programming · doi:10.1017/s0956796818000229 · Cited by 53
“Optimising compilers are complex pieces of software and, as such, errors are almost inevitable in their implementations, as Yang et al. (2011) showed with systematic experiments.”
BackgroundLuo, Z., Zuo, F., Jiang, Y., Gao, J., Jiao, X., & Sun, J. (2019) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3358227 · Cited by 53
“CSmith [42] is a fuzzer designed for C programming language and generates C programs based on randomly selected production rule in the grammar.”
BackgroundDetecting critical bugs in SMT solvers using blackbox mutational fuzzing
Mansur, M. N., Christakis, M., Wüstholz, V., & Zhang, F. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409763 · Cited by 53
End-to-end verification of stack-space bounds for C programs
Carbonneaux, Q., Hoffmann, J., Ramananandro, T., & Shao, Z. (2014) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2594291.2594301 · Cited by 52
“This article describes a framework that enables the formal verification of stack-space bounds of compiled machine code at the C level.”
Background
Metadata
- Authors
- Yang, X., Chen, Y., Eide, E., & Regehr, J.
- Affiliations
- University of Utah (US)
- Venue
- Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation
- Publisher
- ACM
- Published
- 2011-06-04
- Pages
- 283-294
- ISSN
- 1531-7102 (ISSN-L)
- OpenAlex
- W4238083723
- Cited by
- 825
- References
- 23
- Citations by year
- 2026: 71 · 2025: 115 · 2024: 93 · 2023: 88 · 2022: 59 · 2021: 50 · 2020: 58 · 2019: 45 · 2018: 50 · 2017: 40 · 2016: 42 · 2015: 45 · 2014: 24 · 2013: 23 · 2012: 19
- FWCI
- 15.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Keywords
- compiler bugs, C compiler, compiler testing, undefined behavior, bug reports
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.