Finding and understanding bugs in C compilers
The DOI matches its Crossref record; 22 of the 23 references it cites were found in a registry or an index.
Abstract
Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to compiler developers. Every compiler we tested was found to crash and also to silently generate wrong code when presented with valid input. In this paper we present our compiler-testing tool and the results of our bug-hunting study. Our first contribution is to advance the state of the art in compiler testing. Unlike previous tools, Csmith generates programs that cover a large subset of C while avoiding the undefined and unspecified behaviors that would destroy its ability to automatically find wrong-code bugs. Our second contribution is a collection of qualitative and quantitative results about the bugs we have found in open-source C compilers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This paper demonstrates that modern titanic LLMs can be leveraged to directly perform both generation-based and mutation-based fuzzing studied for decades, while being fully automated, generalizable, and applicable to domains challenging for traditional approaches (such as DL systems).”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [80]
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Klein, G., Elphinstone, K., Heiser, G., Andronick, J., Cock, D., Derrin, P., Elkaduwe, D., Engelhardt, K., Kolanski, R., Norrish, M., et al. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629596 · Cited by 1,633 · cited 1×
“As measured by the responses to our bug reports, the defects discovered by Csmith are important.”
MethodAn empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,281
Formal verification of a realistic compiler
Leroy, X. (2009) · Communications of the ACM · doi:10.1145/1538788.1538814 · Cited by 1,187 · cited 1×
“Wolfe [30], talk ing about independent software vendors (ISVs) says: An ISV with a complex code can work around correctness, turn off the optimizer in one or two .les, and usually they have to do that for any of the compilers they use (emphasis ours).”
BackgroundEncyclopedia of Software Engineering
(2002) · Wiley eBooks · doi:10.1002/0471028959 · Cited by 1,161
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079
An experimental evaluation of the assumption of independence in multiversion programming
Knight, J. C., & Leveson, N. G. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312924 · Cited by 774 · cited 3×
“In summary, despite the fact that Knight and Leveson [13] found a substantial number of correlated errors in an experiment on Nversion programming, Csmith has yielded no evidence of correlated failures among unrelated C compilers.”
BackgroundInfluential(1998) · Cited by 469
Grammar-based whitebox fuzzing
Godefroid, P., Kiezun, A., & Levin, M. Y. (2008) · Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1375581.1375607 · Cited by 462 · cited 1×
“[8] showed a way to solve this problem by integrating a constraint solver with a grammar for the language being generated.”
BackgroundHDD: hierarchical delta debugging
Misherghi, G., & Su, Z. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134307 · Cited by 274 · cited 1×
“Delta debugging [31] automates test-case reduction, but all existing variants that are intended for reducing C programs—such as hierarchical delta debugging [18] and Wilkerson’s implementation [29]—introduce undefined behavior.”
Background
Research works printed but not linked: 7
- F. Bellard . TCC: Tiny C compiler, ver. 0.9.25 , May 2009 . http://bellard.org/tcc/. F. Bellard. TCC: Tiny C compiler, ver. 0.9.25, May 2009. http://bellard.org/tcc/.
- C. L. Biffle . Undefined behavior in Google NaCl , Jan. 2010 . http://code.google.com/p/nativeclient/issues/detail?id=245. C. L. Biffle. Undefined behavior in Google NaCl, Jan. 2010. http://code.google.com/p/nativeclient/issues/detail?id=245.
- 10.1002/spe.v37:14 · doi:10.1002/spe.v37:14
- B. Turner . Random Program Generator , Jan. 2007 . http://sites.google.com/site/brturn2/randomcprogramgenerator. B. Turner. Random Program Generator, Jan. 2007. http://sites.google.com/site/brturn2/randomcprogramgenerator.
- 10.5555/1060289.1060313 · doi:10.5555/1060289.1060313
- D. S. Wilkerson . Delta ver . 2006 .08.03, Aug. 2006. http://delta.tigris.org/. D. S. Wilkerson. Delta ver. 2006.08.03, Aug. 2006. http://delta.tigris.org/.
- C. Zhao Automated test program generation for an industrial optimizing compiler . In Proc. ICSE Workshop on Automation of Software Test , pages 36 -- 43 , May 2009 . C. Zhao et al. Automated test program generation for an industrial optimizing compiler. In Proc. ICSE Workshop on Automation of Software Test, pages 36--43, May 2009.
Cited by
Items 101–200 of 833
Language fuzzing using constraint logic programming
Dewey, K. T., Roesch, J., & Hardekopf, B. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642963 · Cited by 52
Coverage Prediction for Accelerating Compiler Testing
Chen, J., Wang, G., Hao, D., Xiong, Y., Zhang, H., Zhang, L., & Xie, B. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2889771 · Cited by 52
“Therefore, automated compiler testing techniques (those based on randomly generated programs by automatic test-generation tools like Csmith [3]) tend to run a large number of”
BackgroundMethodInfluentialQED at Large: A Survey of Engineering of Formally Verified Software
Ringer, T., Palmskog, K., Sergey, I., Milos, G., & Tatlock, Z. (2019) · Foundations and Trends in Programming Languages · doi:10.1561/2500000045 · Cited by 49
“For example, the formally verified optimizing C compiler CompCert (Leroy, 2006) is empirically more reliable than GCC and LLVM: the test generation tool Csmith (Yang et al., 2011) found 79 bugs in GCC and 202 bugs in LLVM, but was unable to find any bugs in the verified parts of CompCert.”
MethodFinding missed optimizations through the lens of dead code elimination
Theodoridis, T., Rigger, M., & Su, Z. (2022) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3503222.3507764 · Cited by 58
“Examples include among others the development of Csmith which heavily is used for differential testing on compilers [37], and EMI is an approach that derives programs that are equivalent given a certain input used for compiler testing [19, 20].”
BackgroundMethodResultInfluentialSFADiff: Automated Evasion Attacks and Fingerprinting Using Black-box Differential Automata Learning
Argyros, G., Stais, I., Jana, S., Keromytis, A. D., & Kiayias, A. (2016) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2976749.2978383 · Cited by 67
“Differential testing has been used successfully for testing a diverse set of systems including C compilers [32], Java virtual machine implementations [11], SSL/TLS implementations [8], mobile applications for privacy leaks [20], PDF malware detectors [31], and space flight software [18].”
MethodModular, compositional, and executable formal semantics for LLVM IR
Zakowski, Y., Beck, C., Yoon, I., Zaichuk, I., Zaliva, V., & Zdancewic, S. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3473572 · Cited by 66
Cause Reduction for Quick Testing
Groce, A., Alipour, M. A., Zhang, C., Chen, Y., & Regehr, J. D. (2014) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2014.37 · Cited by 55
“VI. GCC: THE POTENTIALLY HIGH COST OF REDUCTION Finally, we attempted to apply cause reduction to test cases produced by Csmith [8] using the GCC 4.3.0 compiler (released 3/5/2008), using C-Reduce [32] modified to attempt only line-level reduction, since we hypothesized that reducing C programs would be more expensive…”
BackgroundMethodSelf-citationCompiler testing: a systematic literature analysis
Tang, Y., Ren, Z., Kong, W., & Jiang, H. (2019) · Frontiers of Computer Science · doi:10.1007/s11704-019-8231-0 · Cited by 58
“Furthermore, CLsmith [16] has been proposed for many core compiler testing based on Csmith.”
BackgroundMethodInfluentialTesting probabilistic programming systems
Dutta, S., Legunsen, O., Huang, Z., & Misailovic, S. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236057 · Cited by 49
“, CSmith [122]) often generates large programs to maximize bug-finding capability, small programs seem sufficient for successful detection and debugging in the PP system domain.”
BackgroundMethodInfluentialSTADS: Software Testing as Species Discovery
Böhme, M. (2018) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3210309 · Cited by 70
“For instance, the search space for CSmith [129] is a subset of all C programs (rather than a random sequence of UTF8-characters).”
BackgroundA design and verification methodology for secure isolated regions
Sinha, R., Costa, M., Lal, A., Lopes, N. P., Rajamani, S. K., Seshia, S. A., & Vaswani, K. (2016) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2908080.2908113 · Cited by 58
“However, compilers are large code bases and have been shown to have bugs and produce wrong code at times [6, 23, 26, 29, 46].”
BackgroundFinding and Understanding Bugs in FPGA Synthesis Tools
Herklotz, Y., & Wickerson, J. (2020) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3373087.3375310 · Cited by 49
“This is similar to the safe math wrappers that are used in an existing C fuzzer called Csmith [26] to avoid undefined behaviour like signed overflow.”
MethodInfluentialSteinhöfel, D., & Zeller, A. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549139 · Cited by 75
“CSmith [31] and LangFuzz [14] use CFGs as a basis to randomly create syntactically valid C and JavaScript programs, respectively; Grammarinator [13] produces inputs from ANTLR grammars.”
MethodLLM4VV: Developing LLM-driven testsuite for compiler validation
Munley, C., Jarmusch, A., & Chandrasekaran, S. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.05.034 · Cited by 68
The HACMS program: using formal methods to eliminate exploitable bugs
Fisher, K. S., Launchbury, J., & Richards, R. J. (2017) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2015.0401 · Cited by 52
“The existence of two of these artefacts were crucial to DARPA’s decision to fund the HACMS program, serving as a basis of confidence that the program had some chance of succeeding: the seL4 microkernel and the CompCert verifying C compiler.”
BackgroundInfluentialSynthesizing program input grammars
Bastani, O., Sharma, R., Aiken, A., & Liang, P. (2017) · ACM SIGPLAN Notices · doi:10.1145/3140587.3062349 · Cited by 59
Generating correctness proofs with neural networks
Sanchez-Stern, A., Alhessi, Y., Saul, L., & Lerner, S. (2020) · Proceedings of the 4th ACM SIGPLAN International Workshop on Machine Learning and Programming Languages · doi:10.1145/3394450.3397466 · Cited by 47
“As an example of this high-level of assurance, a study of compilers [39] has shown that CompCert [26], a compiler proved correct in the Coq proof assistant, is signi fi cantly more robust than its non-veri fi ed counterparts.”
Systematically Covering Input Structure
Havrikov, N., & Zeller, A. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00027 · Cited by 48
“Now as then, such producers are mainly used for testing compilers and interpreters: CSmith [43] produces syntactically correct C programs, and LANGFUZZ [1] uses a JavaScript grammar to parse, recombine, and mutate existing inputs while maintaining syntactic validity.”
MethodCounting and generating lambda terms
Grygiel, K., & Lescanne, P. (2013) · Journal of Functional Programming · doi:10.1017/s0956796813000178 · Cited by 48
“However, the generation is not based on unranking, therefore Csmith lacks the ability to construct test case of a specific size on demand, but Csmith can generate large terms, which reveals to be useful, since the greatest number of distinct crash errors is found by programs containing 8K–16K tokens.”
BackgroundInfluentialMetamorphic testing for (graphics) compilers
Donaldson, A. F., & Lascu, A. (2016) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/2896971.2896978 · Cited by 53
“A notable method for testing compilers is random differential testing, popularised by the Csmith tool [8], whereby a program is randomly generated (a process known as fuzzing) and then compiled by different compilers at multiple optimisation levels.”
MethodCedar: A New Language for Expressive, Fast, Safe, and Analyzable Authorization
Cutler, J. W., Disselkoen, C., Eline, A., He, S., Headley, K., Hicks, M., Hietala, K., Ioannidis, E., Kastner, J., Mamat, A., et al. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3649835 · Cited by 136
Automatic testing of symbolic execution engines via program generation and differential testing
Kapus, T., & Cadar, C. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115669 · Cited by 52
“The first step of our approach is to generate small programs using the Csmith [34] tool used in compiler testing.”
BackgroundMethodInfluentialAutomatically finding bugs in a commercial cyber-physical system development tool chain with SLforge
Chowdhury, S. A., Mohian, S., Mehra, S., Gawsane, S., Johnson, T. T., & Csallner, C. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180231 · Cited by 48
“, GCC and LLVM) that are part of CPS development tool chains [17, 25, 30, 45, 58].”
BackgroundCausal testing: understanding defects' root causes
Johnson, B., Brun, Y., & Meliou, A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380377 · Cited by 78
“Differential testing can also produce oracles by comparing the executions of the same inputs on multiple implementations of the same specification [16, 19, 24, 61, 63, 72].”
BackgroundAutomatically Generating Precise Oracles from Structured Natural Language Specifications
Motwani, M., & Brun, Y. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00035 · Cited by 44
“Differential testing can also produce oracles by comparing behavior of multiple implementations of the same specification [10], [12], [19], [59], [63], [78] (e.g., comparing the behavior of Node.js to that of Rhino), but requires multiple implementations, whereas Swami requires none.”
Fuzzing Loop Optimizations in Compilers for C++ and Data-Parallel Languages
Livinskii, V., Babokin, D., & Regehr, J. D. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591295 · Cited by 45
“However, these results are in line with previously reported code coverage due to generative random fuzzers (for example, Table 3 from the Csmith paper [Yang et al. 2011] and Tables 8 and 9 from the YARPGen v.1 paper [Livinskii et al. 2020]).”
BackgroundMethodResultInfluentialSelf-citationA formal C memory model supporting integer-pointer casts
Kang, J., Hur, C.-K., Mansky, W., Garbuzov, D., Zdancewic, S., & Vafeiadis, V. (2015) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2737924.2738005 · Cited by 45
“2738005 surprising non-local changes in program behavior and difficult-tofind bugs [13, 14].”
BackgroundA Framework for Checking Regression Test Selection Tools
Zhu, C., Legunsen, O., Shi, A., & Gligoric, M. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00056 · Cited by 56
“Csmith [36] is a randomized compiler-testing tool for C. JDolly is the most recent work on testing refactoring engines [27].”
MethodGraphFuzz: library API fuzzing with lifetime-aware dataflow graphs
Green, H., & Avgerinos, T. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510228 · Cited by 79
“Generation-based fuzzers such as jsfunfuzz [9] andCSmith [6] implement only the generation function while mutation-based fuzzers also implement the mutation and/or crossover functions.”
BackgroundMethodInfluentialSteering symbolic execution to less traveled paths
Li, Y., Su, Z., Wang, L., & Li, X. (2013) · ACM SIGPLAN Notices · doi:10.1145/2544173.2509553 · Cited by 44
Enhanced compiler bug isolation via memoized search
Chen, J., Ma, H., & Zhang, L. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416570 · Cited by 37
“In the study, we used both GCC and LLVM as subjects to investigate the effectiveness of RecBi, covering almost all popular open-source C compilers used in the existing work [13, 16, 21, 41, 70].”
BackgroundMethodInfluentialTest-case reduction and deduplication almost for free with transformation-based compiler testing
Donaldson, A. F., Thomson, P., Teliman, V., Milizia, S., Maselco, A. P., & Karpiński, A. (2021) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3453483.3454092 · Cited by 52
JITfuzz: Coverage-guided Fuzzing for JVM Just-in-Time Compilers
Wu, M., Lu, M., Cui, H., Chen, J., Zhang, Y., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00017 · Cited by 58
“[30] proposed Csmith, which generates the seeding C programs to explore C compiler while preventing the undefined and unspecified behaviors that might cause testing insufficiency.”
BackgroundTesting static analyses for precision and soundness
Taneja, J., Liu, Z., & Regehr, J. D. (2020) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1145/3368826.3377927 · Cited by 43
“Given these facts, it is perhaps unsurprising that the miscompilation error in LLVM that this bug caused had been originally discovered by Csmith.”
MethodSelf-citationCompiler fuzzing: how much does it matter?
Marcozzi, M., Tang, Q., Donaldson, A. F., & Cadar, C. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360581 · Cited by 53
“Regarding the fuzzers of our study, Orange3 takes the approach of generating programs with known results [Nagai et al. 2014]; Csmith [Yang et al. 2011] and Yarpgen [Yarpgen 2018] are intended to be applied for differential testing; while the equivalence modulo inputs family of tools [Le et al. 2014,…”
BackgroundMethodResultInfluentialGroce, A., Havelund, K., Holzmann, G. J., Joshi, R., & Xu, R.-G. (2014) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-014-9408-8 · Cited by 47
“Recently, the merits of random testing have been more widely acknowledged [66], resulting in a large emerging body of work on sophisticated random testing tools [13, 15, 35, 36, 41, 114] and key factors in the success or failure of random testing efforts [44, 88].”
BackgroundWill you still compile me tomorrow? static cross-version compiler validation
Hawblitzel, C., Lahiri, S. K., Pawar, K., Hashmi, H., Gokbulut, S., Fernando, L., Detlefs, D., & Wadsworth, S. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2491442 · Cited by 45
“[17] found a large number of compiler bugs by comparing the output of different compilers to each other, including gcc and LLVM.”
BackgroundA Language Designer's Workbench
Eelco Visser, B., Wachsmuth, G. H., Tolmach, A. P., Néron, P., Vergu, V. A., Passalaqua, A., & Konat, G. D. P. (2014) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/2661136.2661149 · Cited by 42
“Recent advances in executable definitions of dynamic semantics [17, 44], DSLs for semantics specification [46], model checking of type systems [43], random test generation for compiler correctness [11, 28, 55], and verified compilers [32], promise a semantics engineering discipline with automated testing and verificat…”
BackgroundFinding missed compiler optimizations by differential testing
Barany, G. (2018) · doi:10.1145/3178372.3179521 · Cited by 42
“To compare results, Csmith [Yang et al. 2011] generates programs that compute a checksum over the final values of all global variables and prints it to the standard output before exiting; Orange3 [Nagai et al.”
BackgroundMethodResultInfluentialA Formally-Verified C Compiler Supporting Floating-Point Arithmetic
Boldo, S., Jourdan, J.-H., Leroy, X., & Melquiond, G. (2013) · IEEE Symposium on Computer Arithmetic (ARITH) · doi:10.1109/arith.2013.30 · Cited by 41
“Miscompilation happens more often than one may think: Yang et al [12] tested many production-quality C compilers using differential random testing, and found hundreds of cases where the compiler either crashes at compile-time or—much worse— silently generates an incorrect executable from a correct source program.”
BackgroundVerified peephole optimizations for CompCert
Mullen, E., Zuniga, D., Tatlock, Z., & Grossman, D. (2016) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2908080.2908109 · Cited by 46
“[13] applied Csmith [29] to LLVM and found that peephole passes were the most bug-dense component of LLVM.”
BackgroundMethodHow functional programming mattered
Hu, Z., Hughes, J., & Wang, M. (2015) · National Science Review · doi:10.1093/nsr/nwv042 · Cited by 59
Quickly generating diverse valid test inputs with reinforcement learning
Reddy, S., Lemieux, C., Padhye, R., & Sen, K. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380399 · Cited by 62
“Some researchers have attempted to write generators that produce diverse valid inputs by construction, such as for testing compilers [35, 47], but solutions turn out to be highly complex domain-specific implementations.”
BackgroundTowards AI-Assisted Synthesis of Verified Dafny Methods
Misu, M. R. H., Lopes, C. V., Ma, I., & Noble, J. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643763 · Cited by 38
“For instance, CompCert C [CompCert 2023] is a formally verified C compiler that won the ACM Software Systems award in 2021; 1 in a comprehensive study [Yang et al. 2011], researchers found no bugs in the CompCert C compiler compared to GCC [GCC 2023] and LLVM [LLVM 2023] toolchains.”
Chowdhury, S. A., Shrestha, S. L., Johnson, T. T., & Csallner, C. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380381 · Cited by 40
“Random program generators have significantly improved the bug-finding capability of both types of differential testing instantiations [8, 28, 62].”
BackgroundMethodChaliasos, S., Sotiropoulos, T., Spinellis, D. D., Gervais, A., Livshits, B., & Mitropoulos, D. (2022) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3519939.3523427 · Cited by 43
“Relying on Csmith, several other program generators have emerged for (1) testing other compilers (e.g., OpenCL) [26], or link-time optimizers [23], and (2) generating more expressive programs [18].”
BackgroundMethodResultInfluentialSupporting systems of systems hazard analysis using multi-agent simulation
Alexander, R., & Kelly, T. (2012) · Safety Science · doi:10.1016/j.ssci.2012.07.006 · Cited by 43
“As with software testing, simple coverage metrics may not be very useful (Yang et al., 2011) – what matters is how good the technique is at finding hazards and their causes.”
MethodTesting noninterference, quickly
Hriţcu, C., Hughes, J., Pierce, B. C., Spector-Zabusky, A., Vytiniotis, D., de Amorim, A. A., & Lampropoulos, L. (2013) · ACM SIGPLAN International Conference on Functional Programming (ICFP) · doi:10.1145/2500365.2500574 · Cited by 48
“CSmith (Yang et al., 2011) is a C compiler testing tool that generates random C programs, avoiding ones whose behavior is undefined by the C99 standard.”
MethodThe next 700 compiler correctness theorems (functional pearl)
PATTERSON, D. C., & Ahmed, A. J. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3341689 · Cited by 46
Verified Compilation of Floating-Point Computations
Boldo, S., Jourdan, J.-H., Leroy, X., & Melquiond, G. (2014) · Journal of Automated Reasoning · doi:10.1007/s10817-014-9317-x · Cited by 36
“[36] tested many production-quality C compilers using differential random testing, and found hundreds of cases where the compiler either crashes at compile-time or—much worse—silently generates an incorrect executable from a correct source program.”
GDsmith: Detecting Bugs in Cypher Graph Database Engines
Hua, Z., Lin, W. T., Ren, L., Li, Z., Zhang, L., Jiao, W., & Xie, T. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598046 · Cited by 36
RustSmith: Random Differential Compiler Testing for Rust
Sharma, M., Yu, P., & Donaldson, A. F. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3604919 · Cited by 44
“The name, and some of the design of RustSmith is inspired by Csmith [20], a widely-used generator of C programs.”
Position paper: the science of deep specification
Appel, A. W., Beringer, L., Chlipala, A. J., Pierce, B. C., Shao, Z., Weirich, S., & Zdancewic, S. (2017) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2016.0331 · Cited by 50
“For example, John Regehr’s Csmith project [6] used random testing to assess all popular C compilers, and reported: ‘The striking thing about our CompCert results is that the middle-end bugs we found in all other compilers are absent.”
Feedback-directed differential testing of interactive debuggers
Lehmann, D., & Pradel, M. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236037 · Cited by 36
“More recent approaches include CSmith [39], which randomly creates C programs, and “equivalence modulo inputs” [20], which creates variants of such programs with supposedly equivalent behavior.”
BackgroundMethodInfluentialProperty-driven benchmark generation: synthesizing programs of realistic structure
Steffen, B., Isberner, M., Naujokat, S., Margaria, T., & Geske, M. (2014) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-014-0336-z · Cited by 34
“Closest seems the Csmith’s test-generation techniques [58] for C compilers that generates random programs that comprise C code with complex features, while it at the same time ensures that every generated program has a unique interpretation by avoiding unspecified facets of C as described in the C99 standard.”
BackgroundSyRust: automatic testing of Rust libraries with semantic-aware program synthesis
Takashima, Y., Martins, R., Jia, L., & Păsăreanu, C. S. (2021) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3453483.3454084 · Cited by 43
“Some mutate and piece together existing code segment [16, 19], some generate code from scratch [13, 39], and some like ours, generate API call chains [7, 12, 27].”
BackgroundEF↯CF: High Performance Smart Contract Fuzzing for Exploit Generation
Rodler, M., Paaßen, D., Li, W., Bernhard, L., Holz, T., Karame, G., & Davi, L. (2023) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp57164.2023.00034 · Cited by 54
“Most smart contract fuzzers [25, 27, 39] use an approach akin to grammar-fuzzing [7, 62] to generate inputs with the ABI.”
MethodSpeculative disassembly of binary code
Khadra, M. A. B., Stoffel, D., & Kunz, W. (2016) · International Conference on Compilers, Architecture, and Synthesis for Embedded Systems (CASES) · doi:10.1145/2968455.2968505 · Cited by 37
Detecting Logic Bugs of Join Optimizations in DBMS
Tang, X., Wu, S., Zhang, D., Li, F., & Chen, G. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3588909 · Cited by 40
“Differential testing has been shown to be effective in many areas [12, 16, 30, 33, 38, 59].”
FREEDOM: Engineering a State-of-the-Art DOM Fuzzer
Xu, W. X., Park, S., & Kim, T. (2020) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3372297.3423340 · Cited by 33
“Csmith [56] generates random C source text for stress-testing compilers.”
BackgroundDetecting Compiler Warning Defects Via Diversity-Guided Program Mutation
Tang, Y., Jiang, H., Zhou, Z., Li, X., Ren, Z., & Kong, W. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3119186 · Cited by 36
“Although there exist several test program generators [2], [10], DIPROM employs Epiphron-generated test programs as the seeds.”
BackgroundMethodInfluentialGenerative type-aware mutation for testing SMT solvers
Park, J., Winterer, D., Zhang, C., & Su, Z. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485529 · Cited by 51
“Csmith [Yang et al. 2011] generates random C programs through repeated application of rules from the C grammar.”
MethodBlack-Box Equivalence Checking Across Compiler Optimizations
Dahiya, M., & Bansal, S. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-71237-6_7 · Cited by 37
“Our work also overlaps with previous work on verified compilation [21, 42, 43], compiler testing tools [17, 18, 40], undefined behaviour detection [39], and domain specific languages for coding and verifying compiler optimizations [19,20].”
ResultValidating optimizations of concurrent C/C++ programs
Chakraborty, S. S., & Vafeiadis, V. (2016) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1145/2854038.2854051 · Cited by 33
“Although testing does not ensure correctness, it has been extremely effective at finding bugs (Yang et al. 2011; Le et al. 2014).”
BackgroundJEST: N+1-Version Differential Testing of Both JavaScript Engines and Specification
Park, J., An, S., Youn, D., Kim, G., & Ryu, S. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00015 · Cited by 59
Who’s debugging the debuggers? exposing debug information bugs in optimized binaries
Luna, G. A. D., Italiano, D., Massarelli, L., Österlund, S., Giuffrida, C., & Querzoni, L. (2021) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3445814.3446695 · Cited by 33
“, don’t take any input), and single-threaded. There are several strategies that can be used when implementing this module. We can follow a purely generative approach, e.g., by using tools like Csmith [25] or Yarpgen [4]; or a more refined mutational approache guided by some feedback loop (e.g., using the coverage of th…”
MethodInfluentialWell-typed programs can go wrong: a study of typing-related bugs in JVM compilers
Chaliasos, S., Sotiropoulos, T., Drosos, G.-P., Mitropoulos, C., Mitropoulos, D., & Spinellis, D. D. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485500 · Cited by 38
“This observation comes in contrast to existing compiler testing techniques, which feed compilers with compilable programs [Le et al. 2014; Livinskii et al. 2020; Yang et al. 2011].”
BackgroundMethodResultInfluentialTesting Database Systems via Differential Query Execution
Song, J., Dou, W., Cui, Z., Dai, Q., Wang, W., Wei, J., Zhong, H., & Huang, T. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00175 · Cited by 32
“SQLsmith [29] is an open source random SQL query generator, which is inspired by Csmith [58].”
MethodMaking refactoring safer through impact analysis
Mongiovi, M., Gheyi, R., Soares, G., Teixeira, L., & Borba, P. H. M. (2013) · Science of Computer Programming · doi:10.1016/j.scico.2013.11.001 · Cited by 32
“Finally, compilersmay have introduced behavioral changes during the optimization process [36].”
MethodCrellvm: verified credible compilation for LLVM
Kang, J., Kim, Y., Song, Y., Lee, J., Park, S., Shin, M. D., Kim, Y., Cho, S., Choi, J., Hur, C.-K., et al. (2018) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3192366.3192377 · Cited by 39
“Compiler Testing Random testing tools such as CSmith [17, 42, 53] and EMI [24] have been very successful.”
MethodInfluentialFormal verification of high-level synthesis
Herklotz, Y., Pollard, J. D., Ramanathan, N., & Wickerson, J. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485494 · Cited by 31
“This section also reports on our campaign to fuzz-test Vericert using over a hundred thousand random C programs generated by Csmith [Yang et al. 2011] in order to confirm that its correctness theorem is watertight.”
MethodInfluentialBoosting Compiler Testing via Compiler Optimization Exploration
Chen, J., & Suo, C. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3508362 · Cited by 40
“In particular, some GCC and LLVM versions have been tested by Csmith under the optimization levels [62], which could cause that these versions may be immune to Csmith under the optimization levels and thus may affect the comparison between optimization levels and optimization settings in our empirical study.”
BackgroundMethodResultInfluentialGenerating focused random tests using directed swarm testing
Alipour, M. A., Groce, A., Gopinath, R., & Christi, A. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931056 · Cited by 35
“It is seldom practical to implement different random testers for all the potential focuses that might be needed, and many powerful random testers [16, 31, 37] tend to be based on generating complete inputs (e.”
BackgroundMethodInfluentialEffect-driven QuickChecking of compilers
Midtgaard, J., Justesen, M. N., Kasting, P., Nielson, F., & Nielson, H. R. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3110259 · Cited by 38
“By doing so it can be used to test C compilers against each other (or different optimization levels of same C compiler against each other), and it has been used to find hundreds of bugs in GCC, LLVM, etc. CSmith supports an impressive range of language features compared to our implementation.”
BackgroundInfluentialFuzzing Deep Learning Compilers with HirGen
Ma, H., Shen, Q., Tian, Y., Chen, J., & Cheung, S.-C. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598053 · Cited by 59
A Typed C11 Semantics for Interactive Theorem Proving
Krebbers, R., & Wiedijk, F. (2015) · ACM SIGPLAN International Conference on Certified Programs and Proofs (CPP) · doi:10.1145/2676724.2693571 · Cited by 30
“The execution order in e1 + e2 is unspecified, but this does not mean that either e1 or e2 will be executed entirely before the other.”
BackgroundSoremekun, E., Pavese, E., Havrikov, N., Grunske, L., & Zeller, A. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3013716 · Cited by 35
“Such producers are mainly used for testing compilers and interpreters: CSmith [39] produces syntactically correct C programs, and LANGFUZZ [40] uses a JavaScript grammar to parse, recombine, and mutate existing inputs while maintaining most of the syntactic validity.”
MethodEnriching Compiler Testing with Real Program from Bug Report
Zhong, H. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556894 · Cited by 41
“As a test oracle, differential testing has been introduced to detect bugs in refactoring [49], cross-language APIs [94], web browsers [46], virtual machines [67], crashes [57], JVMs [92], SSL/TLS [45], and compilers [88].”
BackgroundMethodResultInfluentialAutomatic equivalence checking of programs with uninterpreted functions and integer arithmetic
Lopes, N. P., & Monteiro, J. C. (2015) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-015-0366-1 · Cited by 32
“This is an important topic, since all mainstream compilers were shown recently to have several bugs in the optimization passes [43,72].”
BackgroundTowards a verified range analysis for JavaScript JITs
Brown, F., Renner, J., Nötzli, A., Lerner, S., Shacham, H., & Stefan, D. (2020) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3385412.3385968 · Cited by 32
“For example, the CSmith tool automatically generates useful —well-formed, undefined-behavior-free—inputs for testing C compilers [120].”
MethodBoosting Compiler Testing by Injecting Real-World Code
Li, S., Theodoridis, T., & Su, Z. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3656386 · Cited by 42
Finding compiler bugs via live code mutation
Sun, C. P., Le, V., & Su, Z. (2016) · ACM SIGPLAN Notices · doi:10.1145/3022671.2984038 · Cited by 33
Debug information validation for optimized code
Li, Y., Ding, S., Zhang, Q., & Italiano, D. (2020) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3385412.3386020 · Cited by 29
“Moreover, the random input programs used in our work are generated by Csmith [35].”
BackgroundMethodInfluentialWhen does my program do this? learning circumstances of software behavior
Kampmann, A., Havrikov, N., Soremekun, E., & Zeller, A. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409687 · Cited by 29
PAVFuzz: State-Sensitive Fuzz Testing of Protocols in Autonomous Vehicles
Zuo, F., Luo, Z., Yu, J., Liu, Z., & Jiang, Y. (2021) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac18074.2021.9586321 · Cited by 29
“For example, CSmith [23], a fuzzer designed for C programming language, generates C programs based on randomly selected production rules in the grammar.”
BackgroundCsmithEdge: more effective compiler testing by handling undefined behaviour less conservatively
Even-Mendoza, K., Cadar, C., & Donaldson, A. F. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10146-1 · Cited by 37
“In particular, compiler fuzzers have proven successful at finding bugs in mature compilers via randomised testing (Yang et al. 2011; Le et al. 2014; Nakamura and Ishiura 2016; Livinskii et al. 2020; Lidbury et al. 2015; Donaldson et al. 2017; Marcozzi et al. 2019).”
BackgroundMethodInfluentialQuickFuzz: an automatic random fuzzer for common file formats
Grieco, G., Ceresa, M., & Buiras, P. (2016) · doi:10.1145/2976002.2976017 · Cited by 29
“This is a well-known issue that has been studied extensively by Pałka et al. [14] and Yang et al. [19] in the context of testing a compiler.”
BackgroundThe Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators
Ou, X., Li, C., Jiang, Y., & Xu, C. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3622781.3674171 · Cited by 58
“…lacking advanced optimizations such as Havoc, mopt, fork-based execution, or seed pool culling found in mature fuzzers, 𝜇 CFuzz significantly outperforms AFL++ [20], GrayC [18], Csmith [53], and YARPGen [35, 36] in terms of both code coverage and unique crashes, as demonstrated in Section 5.2.”
InfluentialEVMFuzz: Differential fuzz testing of Ethereum virtual machine
Fu, Y., Ren, M., Ma, F., Yang, X., Shi, H., Li, S., & Liao, X. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2556 · Cited by 28
“In the future, we can even automatically generate contracts for testing, similar to the the C program generation tool Csmith [38] and the SQL generation tool SQLsmith [20].”
MethodSelf-citationCoverage-directed differential testing of JVM implementations
Chen, Y., Su, T., Sun, C. P., Su, Z., & Zhao, J. (2016) · ACM SIGPLAN Notices · doi:10.1145/2980983.2908095 · Cited by 30
Self-citationValidating JIT Compilers via Compilation Space Exploration
Li, C., Jiang, Y., Xu, C., & Su, Z. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613140 · Cited by 37
“Program generators like Csmith [59] and YARP-Gen [32, 33] can produce random C programs.”
BackgroundMethodAnalysing the program analyser
Cadar, C., & Donaldson, A. F. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2889206 · Cited by 30
“…it is capable of compiling, so that the compiler can be used to compile itself, (B) formal verification of compiler source code [10] or translation validation with respect to generated code [14] and (C) the application of automated testing methods, such as fuzzing [19] and metamorphic testing [9].”
BackgroundMethodInfluentialFinding and understanding bugs in software model checkers
Zhang, C., Su, T., Yan, Y., Zhang, F., Pu, G., & Su, Z. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338932 · Cited by 42
“Yang et al. [53] propose Csmith, which uses differential testing to hunt bugs in C compilers.”
BackgroundTranslation validation for a verified OS kernel
Sewell, T. A. L., Myreen, M. O., & Klein, G. (2013) · ACM SIGPLAN Notices · doi:10.1145/2499370.2462183 · Cited by 28
BlendFuzz: A Model-Based Framework for Fuzz Testing Programs with Grammatical Inputs
Yang, D., Zhang, Y., & Liu, Q. (2012) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom.2012.99 · Cited by 27
“Code coverage provides a quantitative measurement for the effectiveness of an approach, but they may be too shallow to capture all the effects of random testing, as is noted in [13].”
BackgroundStorm: program reduction for testing and debugging probabilistic programming systems
Dutta, S., Zhang, W., Huang, Z., & Misailovíc, S. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338972 · Cited by 35
“C-Reduce [45] reduces test cases for C programs (often generated using CSmith [56] test generator).”
MethodAnalysis of OpenMP 4.5 Offloading in Implementations: Correctness and Overhead
Diaz, J. M. M., Friedline, K., Pophale, S., Hernández, Ó., Bernholdt, D. E., & Chandrasekaran, S. (2019) · Parallel Computing · doi:10.1016/j.parco.2019.102546 · Cited by 27
NeuRI: Diversifying DNN Generation via Inductive Rule Inference
Liu, J., Peng, J., Wang, Y., & Zhang, L. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616337 · Cited by 47
Grapple: A Graph System for Static Finite-State Property Checking of Large-Scale Systems Code
Zuo, Z., Thorpe, J., Wang, Y., Pan, Q., Lu, S., Wang, K., Xu, G. H., Wang, L., & Li, X. (2019) · European Conference on Computer Systems · doi:10.1145/3302424.3303972 · Cited by 31
“While dynamic analysis [7, 9, 22, 28, 42, 74] is often the weapon of choice for precisely capturing bugs in large programs, it needs representative workloads to achieve coverage, which is often a daunting task for distributed systems.”
BackgroundRFC-directed differential testing of certificate validation in SSL/TLS implementations
Chen, C., Tian, C., Duan, Z., & Zhao, L. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180226 · Cited by 30
“Grammar-based testing techniques [33, 43, 59, 64] generate programs based on grammars. example, Csmith [64] employs generated programs to find bugs in open-source C compilers.”
Background
Metadata
- Authors
- Yang, X., Chen, Y., Eide, E., & Regehr, J.
- Affiliations
- University of Utah (US)
- Venue
- Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation
- Publisher
- ACM
- Published
- 2011-06-04
- Pages
- 283-294
- ISSN
- 1531-7102 (ISSN-L)
- OpenAlex
- W4238083723
- Cited by
- 825
- References
- 23
- Citations by year
- 2026: 71 · 2025: 115 · 2024: 93 · 2023: 88 · 2022: 59 · 2021: 50 · 2020: 58 · 2019: 45 · 2018: 50 · 2017: 40 · 2016: 42 · 2015: 45 · 2014: 24 · 2013: 23 · 2012: 19
- FWCI
- 15.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Keywords
- compiler bugs, C compiler, compiler testing, undefined behavior, bug reports
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.