Finding and understanding bugs in C compilers
The DOI matches its Crossref record; 22 of the 23 references it cites were found in a registry or an index.
Abstract
Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to compiler developers. Every compiler we tested was found to crash and also to silently generate wrong code when presented with valid input. In this paper we present our compiler-testing tool and the results of our bug-hunting study. Our first contribution is to advance the state of the art in compiler testing. Unlike previous tools, Csmith generates programs that cover a large subset of C while avoiding the undefined and unspecified behaviors that would destroy its ability to automatically find wrong-code bugs. Our second contribution is a collection of qualitative and quantitative results about the bugs we have found in open-source C compilers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This paper demonstrates that modern titanic LLMs can be leveraged to directly perform both generation-based and mutation-based fuzzing studied for decades, while being fully automated, generalizable, and applicable to domains challenging for traditional approaches (such as DL systems).”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [80]
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Klein, G., Elphinstone, K., Heiser, G., Andronick, J., Cock, D., Derrin, P., Elkaduwe, D., Engelhardt, K., Kolanski, R., Norrish, M., et al. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629596 · Cited by 1,633 · cited 1×
“As measured by the responses to our bug reports, the defects discovered by Csmith are important.”
MethodAn empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,281
Formal verification of a realistic compiler
Leroy, X. (2009) · Communications of the ACM · doi:10.1145/1538788.1538814 · Cited by 1,187 · cited 1×
“Wolfe [30], talk ing about independent software vendors (ISVs) says: An ISV with a complex code can work around correctness, turn off the optimizer in one or two .les, and usually they have to do that for any of the compilers they use (emphasis ours).”
BackgroundEncyclopedia of Software Engineering
(2002) · Wiley eBooks · doi:10.1002/0471028959 · Cited by 1,161
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079
An experimental evaluation of the assumption of independence in multiversion programming
Knight, J. C., & Leveson, N. G. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312924 · Cited by 774 · cited 3×
“In summary, despite the fact that Knight and Leveson [13] found a substantial number of correlated errors in an experiment on Nversion programming, Csmith has yielded no evidence of correlated failures among unrelated C compilers.”
BackgroundInfluential(1998) · Cited by 469
Grammar-based whitebox fuzzing
Godefroid, P., Kiezun, A., & Levin, M. Y. (2008) · Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1375581.1375607 · Cited by 462 · cited 1×
“[8] showed a way to solve this problem by integrating a constraint solver with a grammar for the language being generated.”
BackgroundHDD: hierarchical delta debugging
Misherghi, G., & Su, Z. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134307 · Cited by 274 · cited 1×
“Delta debugging [31] automates test-case reduction, but all existing variants that are intended for reducing C programs—such as hierarchical delta debugging [18] and Wilkerson’s implementation [29]—introduce undefined behavior.”
Background
Research works printed but not linked: 7
- F. Bellard . TCC: Tiny C compiler, ver. 0.9.25 , May 2009 . http://bellard.org/tcc/. F. Bellard. TCC: Tiny C compiler, ver. 0.9.25, May 2009. http://bellard.org/tcc/.
- C. L. Biffle . Undefined behavior in Google NaCl , Jan. 2010 . http://code.google.com/p/nativeclient/issues/detail?id=245. C. L. Biffle. Undefined behavior in Google NaCl, Jan. 2010. http://code.google.com/p/nativeclient/issues/detail?id=245.
- 10.1002/spe.v37:14 · doi:10.1002/spe.v37:14
- B. Turner . Random Program Generator , Jan. 2007 . http://sites.google.com/site/brturn2/randomcprogramgenerator. B. Turner. Random Program Generator, Jan. 2007. http://sites.google.com/site/brturn2/randomcprogramgenerator.
- 10.5555/1060289.1060313 · doi:10.5555/1060289.1060313
- D. S. Wilkerson . Delta ver . 2006 .08.03, Aug. 2006. http://delta.tigris.org/. D. S. Wilkerson. Delta ver. 2006.08.03, Aug. 2006. http://delta.tigris.org/.
- C. Zhao Automated test program generation for an industrial optimizing compiler . In Proc. ICSE Workshop on Automation of Software Test , pages 36 -- 43 , May 2009 . C. Zhao et al. Automated test program generation for an industrial optimizing compiler. In Proc. ICSE Workshop on Automation of Software Test, pages 36--43, May 2009.
Cited by
Items 301–400 of 833
Into the depths of C: elaborating the de facto standards
Memarian, K., Matthiesen, J., Lingard, J., Nienhuis, K., Chisnall, D., Watson, R. N. M., & Sewell, P. (2016) · ACM SIGPLAN Notices · doi:10.1145/2980983.2908081 · Cited by 15
Alive-Infer: data-driven precondition inference for peephole optimizations in LLVM
Menendez, D., & Nagarakatte, S. (2017) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3062341.3062372 · Cited by 20
“They are also a persistent source of LLVM bugs [25, 30, 51].”
BackgroundOn the Caching Schemes to Speed Up Program Reduction
Tian, Y., Zhang, X., Dong, Y., Xu, Z., Zhang, M., Jiang, Y., Cheung, S.-C., & Sun, C. P. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3617172 · Cited by 15
“In total, our benchmark suite contains 31 C compiler bugs, which consists of programs that have the common size of automatically generated iles via fuzzing techniques such as CSmith [51] and EMI [26].”
BackgroundMethodInfluentialCLFuzz: Vulnerability Detection of Cryptographic Algorithm Implementation via Semantic-aware Fuzzing
Zhou, Y., Ma, F., Chen, Y., Ren, M., & Jiang, Y. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3628160 · Cited by 20
“Besides, CSmith [67] and LangFuzz [36] use CFGs (control low graph) to generate valid test inputs.”
MethodSelf-citationRefined Input, Degraded Output: The Counterintuitive World of Compiler Behavior
Theodoridis, T., & Su, Z. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3656404 · Cited by 19
“Csmith programs are self-contained and do not require inputs, thus we can compute precise information about them ( e.g. , dead code and variable value ranges).”
MethodInfluentialFuzzing MLIR Compiler Infrastructure via Operation Dependency Analysis
Suo, C., Chen, J., Liu, S., Jiang, J., Zhao, Y., & Wang, J. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680360 · Cited by 23
Rango: Adaptive Retrieval-Augmented Proving for Automated Software Verification
Thompson, K., Saavedra, N., Carrott, P., Fisher, K. D., Sanchez-Stern, A., Brun, Y., Ferreira, J. F., Lerner, S., & First, E. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00161 · Cited by 57
“Notably, a study investigating C compilers [92], among them CompCert [47], LLVM, and GCC, observed that the only compiler for which no bugs were found was CompCert [92], which is formally verified using the Coq proof assistant.”
OpenACC 2.5 Validation Testsuite Targeting Multiple Architectures
Friedline, K., Chandrasekaran, S., Lopez, M. G., & Hernández, Ó. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67630-2_39 · Cited by 14
Using Large-Scale Anomaly Detection on Code to Improve Kotlin Compiler
Bryksin, T., Petukhov, V., Alexin, I., Prikhodko, S., Shpilman, A., Kovalenko, V., & Povarov, N. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387447 · Cited by 15
“In the domain of programming language development, code anomalies, thanks to being atypical code, while having been actually implemented by someone, can be used as a source of data for fuzz testing techniques, which have proven to be valuable for finding tricky issues in compilers [42].”
BackgroundCompiler testing via a theory of sound optimisations in the C11/C++11 memory model
Morisset, R., Pawan, P., & Nardelli, F. Z. (2013) · ACM SIGPLAN Notices · doi:10.1145/2499370.2491967 · Cited by 14
Metamorphic Shader Fusion for Testing Graphics Shader Compilers
Xiao, D., Liu, Z., & Wang, S. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00201 · Cited by 19
“tional compilers [12, 26, 33, 40, 45, 48, 57, 60] (by detecting the inconsistency of compiled outputs), we do not deem DT suitable for testing shader compilers based on two reasons.”
MethodHelp, help, i'm being suppressed! The significance of suppressors in software testing
Groce, A., Zhang, C., Alipour, M. A., Eide, E. N., Chen, Y., & Regehr, J. D. (2013) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2013.6698892 · Cited by 14
“2) C Compilers: Csmith [3] uses a customized grammar for a subset of the C language to generate random C programs.”
BackgroundMethodInfluentialSelf-citationGroce, A., Holmes, J., & Kellar, K. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092704 · Cited by 14
“, our algorithms do not apply to complex custom test generators such as Csmith [49] or jsfunfuzz [46]), but it is oen highly eective.”
BackgroundMethod$$\textsc {Wys}^\star $$ : A DSL for Verified Secure Multi-party Computations
Rastogi, A., Swamy, N., & Hicks, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-17138-4_5 · Cited by 18
“The hope is that formal verification will reduce the occurrence of security threatening bugs, as it has in prior work [15, 36, 50, 63,64].”
BackgroundQuickFuzz: an automatic random fuzzer for common file formats
Grieco, G., Ceresa, M., & Buiras, P. (2016) · ACM SIGPLAN Notices · doi:10.1145/3241625.2976017 · Cited by 14
Do Judge a Test by its Cover: Combining Combinatorial and Property-Based Testing
Goldstein, H., Hughes, J., Lampropoulos, L., & Pierce, B. C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72019-3_10 · Cited by 24
“For instance, a generator for well-typed simply typed lambda-terms has been used to reveal bugs in GHC [6,20,32], while a generator for C programs that avoid “undefined behaviors” has been used to find many bugs in production compilers [24, 34, 41] The cited studies are all examples of differential testing, where diff…”
WasmRef-Isabelle: A Verified Monadic Interpreter and Industrial Fuzzing Oracle for WebAssembly
Watt, C., Trela, M., Lammich, P., & Märkl, F. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591224 · Cited by 15
Defect Categorization in Compilers: A Multi-vocal Literature Review
Rahman, A., Bose, D. B., Barsha, F. L., & Pandita, R. (2023) · ACM Computing Surveys · doi:10.1145/3626313 · Cited by 14
PPR: Pairwise Program Reduction
Zhang, M., Xu, Z., Tian, Y., Jiang, Y., & Sun, C. P. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616275 · Cited by 13
Zhou, X., Xu, Z., Zhang, M., Tian, Y., & Sun, C. P. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00071 · Cited by 13
“Many minimization techniques [1], [2], [3], [4], [5], [6], [7], [8], [9], [10] have been proposed and widely used in various scenarios [11], [12], [13], [14], [15], especially in facilitating software testing and debugging [16], [17], [18].”
Suzzer: A Vulnerability-Guided Fuzzer Based on Deep Learning
Zhao, Y., Li, Y., Yang, T., & Xie, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-42921-8_8 · Cited by 13
P4Fuzz: Compiler Fuzzer forDependable Programmable Dataplanes
Agape, A.-A., Danceanu, M. C., Hansen, R. R., & Schmid, S. (2020) · International Conference on Distributed Computing and Networking (ICDCN) · doi:10.1145/3427796.3427798 · Cited by 13
“, Csmith [27] for the C language), less is known about fuzzers for compilers: potentially an effective tool to find weaknesses and bugs in the compiler.”
BackgroundSpecTest: Specification-Based Compiler Testing
Schumi, R., & Sun, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-71500-7_14 · Cited by 18
“Various compiler testing methods work without any coverage by just randomly generating test cases according to a grammar, which defines valid programs [52,60].”
BackgroundMethodFind bugs in static bug finders
Wang, J., Huang, Y., Wang, S., & Wang, Q. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527899 · Cited by 16
“Randomized differential testing is a widely-used black-box differential testing technique in which the inputs are randomly generated [12, 17, 31, 35, 48, 49, 57, 58, 60, 66, 68].”
MethodGoldstein, H., & Pierce, B. C. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3563291 · Cited by 21
Where Did My Variable Go? Poking Holes in Incomplete Debug Information
Assaiante, C., D’Elia, D. C., Luna, G. A. D., & Querzoni, L. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575720 · Cited by 13
“We work with the popular Csmith fuzzer [43] to generate programs to be checked for our conjectures.”
MethodStructurized grammar‐based fuzz testing for programs with highly structured inputs
Yan, J., Zhang, Y., & Yang, D. (2013) · Security and Communication Networks · doi:10.1002/sec.714 · Cited by 13
Fully Automated Compiler Testing of a Reasoning Engine via Mutated Grammar Fuzzing
Köroglu, Y., & Wotawa, F. (2019) · doi:10.1109/ast.2019.00010 · Cited by 13
“Since then, many automated grammar-fuzzing tools were implemented [21]–[24].”
MethodNguyen, H. L., Nassar, N., Kehrer, T., & Grunske, L. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416668 · Cited by 13
“To tackle this problem, generator-based fuzzers such as Peach [62] or Csmith [88] leverage domain-specific knowledge of the input format to generate input files that are syntactically valid by con-”
BackgroundAutomatic Test Case Reduction for OpenCL
Pflanzer, M., Donaldson, A. F., & Lascu, A. (2016) · International Workshop on OpenCL · doi:10.1145/2909437.2909439 · Cited by 14
“The CLsmith tool that we use to generate random kernels [8] is built on top of the Csmith generator for C programs [15].”
BackgroundMethodResultInfluentialFedDefender: Backdoor Attack Defense in Federated Learning
Gill, W., Anwar, A., & Gulzar, M. A. (2023) · SE4SafeML@SIGSOFT FSE · doi:10.1145/3617574.3617858 · Cited by 13
Varity: Quantifying Floating-Point Variations in HPC Systems Through Randomized Testing
Laguna, I. (2020) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps47924.2020.00070 · Cited by 12
“A notable instance is Csmith [13], which detects compiler bugs in C compilers.”
BackgroundMethodДоверяй, но проверяй: SFI safety for native-compiled Wasm
Johnson, E., Thien, D., Alhessi, Y., Narayan, S., Brown, F., Lerner, S., McMullen, T., Savage, S., & Stefan, D. (2021) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.24078 · Cited by 12
“Specifically, we (1) use Csmith [82] to randomly generate valid C/C++ programs, then compile them to Wasm with Clang and (2) directly generate random Wasm modules using Binaryen [2].”
BackgroundMethodInfluentialEvaluating and Improving Static Analysis Tools Via Differential Mutation Analysis
Groce, A., Ahmed, I., Feist, J., Grieco, G., Gesi, J., Meidani, M., & Chen, Q. (2021) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs54544.2021.00032 · Cited by 13
“Cuoq et al. [14] proposed the generation of random programs ( ´a la Csmith [56]) to test analysis tools aiming for soundness, in limited circumstances, but noted that na¨ıve differential testing of analysis tools was not possible.”
BackgroundMethodFormally Verified Loop-Invariant Code Motion and Assorted Optimizations
Monniaux, D. P., & Six, C. (2022) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3529507 · Cited by 12
“In particular, CompCert does not have the middle-end bugs usually found in compilers [21], notably in optimization passes.”
BackgroundChecking correctness of code generator architecture specifications
Hasabnis, N., Qiao, R., & Sekar, R. (2015) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo.2015.7054197 · Cited by 12
ECP SOLLVE: Validation and Verification Testsuite Status Update and Compiler Insight for OpenMP
Huber, T., Pophale, S., Baker, N., Carr, M., Rao, N., Reap, J., Holsapple, K., Davis, J. H., Burnus, T., Lee, S., et al. (2022) · IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC) · doi:10.1109/p3hpc56579.2022.00017 · Cited by 12
“Other related work includes Csmith [27], a comprehensive, well-cited work where the authors perform a randomized test-case generator exposing compiler bugs using differential testing.”
BackgroundInfluentialTaintFuzzer: SoC Security Verification using Taint Inference-enabled Fuzzing
Hossain, M. M., Dipu, N. F., Azar, K. Z., Rahman, F., Farahmandi, F., & Tehranipoor, M. M. (2023) · doi:10.1109/iccad57390.2023.10323726 · Cited by 12
“Fuzzing was originally used by software (SW) engineers to identify SW’s weaknesses and vulnerabilities by exposing it to unexpected inputs [17]–[20].”
MethodAdvances in Property-Based Testing for $$\alpha $$ Prolog
Cheney, J., Momigliano, A., & Pessina, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-41135-4_3 · Cited by 18
Termination-checking for LLVM peephole optimizations
Menendez, D., & Nagarakatte, S. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884809 · Cited by 12
“Unsurprisingly, peephole optimizations are a common source of compiler bugs [19, 26, 38].”
BackgroundMethodC-to-Verilog translation validation
Leung, A., Bounov, D., & Lerner, S. (2015) · ACM/IEEE International Conference on Formal Methods and Models for Co-Design (MEMOCODE) · doi:10.1109/memcod.2015.7340466 · Cited by 12
Practical Validation of Bytecode to Bytecode JIT Compiler Dynamic Deoptimization.
Béra, C., Miranda, E., Denker, M., & Ducasse, S. (2016) · The Journal of Object Technology · doi:10.5381/jot.2016.15.2.a1 · Cited by 18
Generating Well-Typed Terms That Are Not “Useless”
Frank, J., Quiring, B., & Lampropoulos, L. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3632919 · Cited by 11
“the other hand, the CSmith project, the crown jewel of imperative compiler testing (Yang et al. 2011), follows a different, more ad-hoc, approach developing a fine-tuned hand-written generator to generate C programs that don’t exhibit undefined behavior.”
BackgroundMethodInfluentialRust-twins: Automatic Rust Compiler Testing through Program Mutation and Dual Macros Generation
Yang, W., Gao, C., Liu, X., Li, Y., & Xue, Y. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695059 · Cited by 15
Evaluating program analysis and testing tools with the RUGRAT random benchmark application generator
Hussain, I., Csallner, C., Grechanik, M., Fu, C., Xie, Q., Park, S., Taneja, K., & Hossain, B. M. M. (2012) · International Workshop on Dynamic Analysis · doi:10.1145/2338966.2336798 · Cited by 11
“Csmith has been used to test compilers [28] and static analyzers [6].”
MethodTest input generation using dynamic programming
Zaeem, R. N., & Khurshid, S. (2012) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2393596.2393635 · Cited by 11
“Most recently, Yang et al. used random test generation to find numerous bugs in mainstream C compilers [49].”
BackgroundReliable and fast DWARF-based stack unwinding
Bastian, T., Kell, S., & Nardelli, F. Z. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360572 · Cited by 11
“By leveraging the CSmith fuzzer [Yang et al. 2011] and CReduce test-case reducer [Regehr et al.”
MethodUncovering Smart Contract VM Bugs Via Differential Fuzzing
Maier, D., Fäßler, F., & Seifert, J. (2021) · Reversing and Offensive-Oriented Trends Symposium · doi:10.1145/3503921.3503923 · Cited by 11
Zhao, Y., Chen, J., Fu, R., Ye, H., & Wang, Z. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598077 · Cited by 12
“Like other software, compilers also contain bugs [8, 9, 28, 34, 43], which could make any software processed by the buggy compiler produce unexpected behaviors (even disasters in safety-critical domains).”
BackgroundMethodResultInfluentialRevealing Compiler Heuristics Through Automated Discovery and Optimization
Seeker, V., Cummins, C., Cole, M., Franke, B., Hazelwood, K., & Leather, H. (2024) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo57630.2024.10444847 · Cited by 14
“It has been successfully used by multiple research studies on exposing bugs in compilers to drive large testing efforts [38]–[41].”
MethodTSTL: a language and tool for testing (demo)
Groce, A., Pinto, J., Azimi, P., & Mittal, P. (2015) · International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2784769 · Cited by 15
“Differential testing is a powerful method for finding bugs in software ranging from simple container classes to optimizing compilers [16], but its implementation in a testing system often requires writing a large amount of repetitive, and sometimes subtle, code.”
MethodFinding common ground: choose, assert, and assume
Groce, A., & Erwig, M. (2012) · International Workshop on Dynamic Analysis · doi:10.1145/2338966.2336800 · Cited by 11
“As an example, the Csmith compiler testing system [33] is a standalone tool that takes as input parameters a set of configuration options determining which C language features to use, and a random seed, and outputs a C program.”
MethodBounded exhaustive test-input generation on GPUs
Çelik, A., Pai, S., Khurshid, S., & Gligoric, M. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133918 · Cited by 13
Different is Good: Detecting the Use of Uninitialized Variables through Differential Replay
Cao, M., Hou, X., Wang, T., Qu, H., Zhou, Y., Bai, X., & Wang, F. (2019) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3319535.3345654 · Cited by 16
“For instance, differential testing was used to test the compiler of C language [38, 52, 65], SSL/TLS implementations [10, 13, 49, 60] and complex software systems [5, 12, 26, 29, 61].”
MethodThe Trusted Computing Base of the CompCert Verified Compiler
Monniaux, D. P., & Boulmé, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99336-8_8 · Cited by 11
“com/csmith-project/csmith and [57] (29)https://github.”
InfluentialA dynamic stochastic model for automatic grammar‐based test generation
Guo, H., & Qiu, Z. (2014) · Software Practice and Experience · doi:10.1002/spe.2278 · Cited by 11
“It has been argued in [35, 49] that the reduced test case is either still too large or involves undefined behavior which prohibits reproducing failures.”
BackgroundMethodBesson, F., Blazy, S., & Wilke, P. (2017) · Journal of Automated Reasoning · doi:10.1007/s10817-017-9439-z · Cited by 11
“With this respect, the CompCert C semantics successfully run hundreds of random test programs generated by CSmith [33].”
MethodAutomatic annotation of tasks in structured code
Ramos, P., de Souza, G., Soares, D., Araujo, G. C. S. D., & Pereira, F. M. Q. (2018) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/3243176.3243200 · Cited by 11
“4 we use random programs produced with CSmith [61].”
Method: Priority Aware Test Case Reduction
Gharachorlu, G., & Sumner, W. N. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-16722-6_24 · Cited by 11
Finding and Understanding Defects in Static Analyzers by Constructing Automated Oracles
He, W., Di, P., Ming, M., Zhang, C., Su, T., Li, S., & Sui, Y. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660781 · Cited by 25
Differential Optimization Testing of Gremlin-Based Graph Database Systems
Zheng, Y., Dou, W., Tang, L., Cui, Z., Song, J., Cheng, Z., Wang, W., Wei, J., Zhong, H., & Tao, H. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00012 · Cited by 12
Migrating Software from x86 to ARM Architecture: An Instruction Prediction Approach
Ford, B. W., Qasem, A., Tešić, J., & Zong, Z. (2021) · IEEE International Conference on Networking, Architecture and Storage (NAS) · doi:10.1109/nas51552.2021.9605443 · Cited by 13
Advanced Grammar-Based Fuzzing
Sargsyan, S., Hakobyan, J., Mehrabyan, M., Mkoyan, R., Sahakyan, V., Melkonyan, V., Arutunian, M., Fahradyan, A., & Avetisyan, A. I. (2022) · Ivannikov Memorial Workshop (IVMEM) · doi:10.1109/ivmem57067.2022.9983953 · Cited by 10
SSA Translation Is an Abstract Interpretation
Lemerre, M. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3571258 · Cited by 21
“To avoid the problem of addressable variables in C that cannot be directly translated to SSA, we have used the Csmith [Yang et al. 2011] C generator, using options to generate only non-addressable integer variables and complex unstructured control flow, to generate 100 large C files.”
Methoddewolf: Improving Decompilation by leveraging User Surveys
Enders, S., Behner, E.-M. C., Bergmann, N., Rybalka, M., Padilla, E., Hui, E. X., Low, H., & Sim, N. (2023) · Workshop on Binary Analysis Research (BAR) · doi:10.14722/bar.2023.23001 · Cited by 10
“We fuzzed dewolf with Csmith-generated functions [48], to find bugs that did not occur in the above set.”
MethodAutomating formal proofs for reactive systems
Ricketts, D., Robert, V., Jang, D., Tatlock, Z., & Lerner, S. (2014) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2594291.2594338 · Cited by 13
“…Descriptors F.3.1 [Logics and meanings of programs]: Mechanical verification; D.2.4 [Software Engineering]: Correctness proofs, formal methods General Terms Languages, Verification Keywords interactive proof assistants, proof automation, domainspecific languages, reactive systems, dependent types”
BackgroundInfluentialA trustworthy mechanized formalization of R
Bodin, M., Díaz, T., & Tanter, É. (2018) · Symposium on Dynamic Languages (DLS) · doi:10.1145/3276945.3276946 · Cited by 10
“This additional code is called the shim, and unfortunately, it usually concentrates most bugs in certified software [36].”
BackgroundUsing mutants to help developers distinguish and debug (compiler) faults
Holmes, J., & Groce, A. (2020) · Software Testing Verification and Reliability · doi:10.1002/stvr.1727 · Cited by 10
“Important compilers, whether JITs where security is paramount or C compilers used to compile core systems code, also tend to already be the subjects of extensive automated testing [5, 6].”
BackgroundInfluentialType Inference for C: Applications to the Static Analysis of Incomplete Programs
Melo, L. T. C., Ribeiro, R. G., Guimarães, B. C. F., & Pereira, F. M. Q. (2020) · ACM Transactions on Programming Languages and Systems · doi:10.1145/3421472 · Cited by 24
“As a comparison, when CSmith [Yang et al. 2011] was used to generate a training set, code reduction was of 5.6%; and when LDRGen [Barany 2017] was used with the same purpose, this improvement was of 5.5%.”
MethodInfluentialC4: the C compiler concurrency checker
Windsor, M., Donaldson, A. F., & Wickerson, J. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3469079 · Cited by 10
“We have measured the differential code coverage thatC4 achieves on a version of the LLVM compiler, versus bothCsmith [26] (a leading C program generator that focuses on sequential C) and the LLVM test suite.”
MethodTesting noninterference, quickly
Hriţcu, C., Hughes, J., Pierce, B. C., Spector-Zabusky, A., Vytiniotis, D., de Amorim, A. A., & Lampropoulos, L. (2013) · ACM SIGPLAN Notices · doi:10.1145/2544174.2500574 · Cited by 15
Evaluating Test Suite Effectiveness and Assessing Student Code via Constraint Logic Programming
Dewey, K. T., Conrad, P. T., Craig, M., & Morozova, E. (2017) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3059009.3059051 · Cited by 10
Deng, C., & Namjoshi, K. S. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66706-5_5 · Cited by 10
“[20,11] and related work) and from the viewpoint of formally guaranteeing correct compilation (cf.”
BackgroundSynEva: Evaluating ML Programs by Mirror Program Synthesis
Qin, Y., Wang, H., Xu, C., Ma, X., & Lü, J. (2018) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs.2018.00031 · Cited by 10
“Chen et al. [9] combined differential testing with mutation testing to focus testing efforts on representative classfiles of JVMs. Yang et al. [10] provided a tool, CSmith, to automatically generate C programs that cover a large subset of C features, and used such generated programs’ executions on…”
MethodAvoiding the Familiar to Speed Up Test Case Reduction
Gharachorlu, G., & Sumner, W. N. (2018) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs.2018.00056 · Cited by 10
“c), we used Csmith [24], a tool for generating random C programs that statically and dynamically conform to the C99 standard [25] and can help to find and understand bugs in C compilers [4].”
Information-Flow Preservation in Compiler Optimisations
Besson, F., Dang, A., & Jensen, T. W. (2019) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf.2019.00023 · Cited by 10
“They may perform aggressive optimisations which may exploit undefined behaviours [30] or they may have subtle bugs and generate invalid code [31].”
A formal C memory model supporting integer-pointer casts
Kang, J., Hur, C.-K., Mansky, W., Garbuzov, D., Zdancewic, S., & Vafeiadis, V. (2015) · ACM SIGPLAN Notices · doi:10.1145/2813885.2738005 · Cited by 10
An SMT Encoding of LLVM’s Memory Model for Bounded Translation Validation
Lee, J., Kim, D., Hur, C.-K., & Lopes, N. P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-81688-9_35 · Cited by 17
“There has also been work on automatic verification of classes of optimizations, such as peephole optimizations [25, 31], semi-automated proofs [48], translation validation [20, 35, 42, 44], and fuzzing [23,46].”
BackgroundWalz, A., & Sikora, A. (2018) · IEEE International Carnahan Conference on Security Technology (ICCST) · doi:10.1109/ccst.2018.8585565 · Cited by 14
Kitten: A Simple Yet Effective Baseline for Evaluating LLM-Based Compiler Testing Techniques
Xie, Y., Xu, Z., Tian, Y., Zhou, M. Q., Zhou, X., & Sun, C. P. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731731 · Cited by 10
Verified peephole optimizations for CompCert
Mullen, E., Zuniga, D., Tatlock, Z., & Grossman, D. (2016) · ACM SIGPLAN Notices · doi:10.1145/2980983.2908109 · Cited by 10
Interpreter-guided differential JIT compiler unit testing
Polito, G., Ducasse, S., & Tesone, P. (2022) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3519939.3523457 · Cited by 16
“Remarkable existing work cover random test generation solutions such as CSmith [32], grammar-based solutions such as the work originated by Purdom [26] and mutation-based test generation such as equivalence-modulo-input (EMI) [17].”
UBFuzz: Finding Bugs in Sanitizer Implementations
Li, S., & Su, Z. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3617232.3624874 · Cited by 17
A Comprehensive Study of Open-Source Printed Circuit Board (PCB) Design Software Bugs
Zhao, X. G., Jiang, H., Guo, S., Liu, D., Liu, H., Shi, C., & Li, H. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3450918 · Cited by 9
“Existing methods like Csmith [15] and SLforge [48] could be adapted with circuit-related domain knowledge and physical constraints of PCB design in mind.”
MethodT-Rec: Fine-Grained Language-Agnostic Program Reduction Guided by Lexical Syntax
Xu, Z., Tian, Y., Zhang, M., Zhang, J., Liu, P., Jiang, Y., & Sun, C. P. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3690631 · Cited by 15
“For the 7 SMT-LIBv2 benchmarks, the differences are all due to that T - Rec considers simple symbols as keywords (according to the lexical syntax) and does not reduce them, while the passes of C-Reduce treat them as identifiers and replace them with smaller ones.”
A change-centric approach to compile configurable systems with #ifdefs
Braz, L., Gheyi, R., Mongiovi, M., Ribeiro, M., Medeiros, F., & Teixeira, L. (2016) · ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE) · doi:10.1145/2993236.2993250 · Cited by 9
“The implementation of compilers may vary according to the operating system [22].”
BackgroundPointer disambiguation via strict inequalities
Maalej, M., Paisante, V. M., Ramos, P., Gonnord, L., & Pereira, F. M. Q. (2017) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo.2017.7863735 · Cited by 9
Namjoshi, K. S., & Tabajara, L. M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-39322-9_1 · Cited by 10
Grammar-based testing for little languages: an experience report with student compilers
van Heerden, P., Raselimo, M., Sagonas, K., & Fischer, B. (2020) · ACM SIGPLAN International Conference on Software Language Engineering (SLE) · doi:10.1145/3426425.3426946 · Cited by 10
“In the area of random testing of compilers, of special note is Csmith [29], a well-engineered, highly effective tool for generating C programs that avoid undefined or unspecified behavior by construction.”
MethodFuzzing With Optimized Grammar-Aware Mutation Strategies
Deng, J., Zhu, X., Xiao, X., Wen, S., Li, Q., & Xia, S. (2021) · IEEE Access · doi:10.1109/access.2021.3093904 · Cited by 9
“Inorder to perform fuzz testing on C compilers, CSmith [25] generates C source code according to C language grammar.”
MethodIndustrial Deployment of Compiler Fuzzing Techniques for Two GPU Shading Languages
Donaldson, A. F., Clayton, B., Harrison, R., Mohsin, H., Neto, D., Teliman, V., & Watson, H. (2023) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst57152.2023.00042 · Cited by 13
“Differential testing of compilers was popularised by the Csmith project [49], which generates random C programs that are, by construction, free from undefined behaviour.”
MethodWarpAttack: Bypassing CFI through Compiler-Introduced Double-Fetches
Xu, J., Bartolomeo, L. D., Toffalini, F., Mao, B., & Payer, M. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179433 · Cited by 9
“The correctness of compilers has been extensively tested [58, 59], verified, and certified [60, 61].”
BackgroundType-safe runtime code generation: accelerate to LLVM
McDonell, T. L., Chakravarty, M. M. T., Grover, V., & Newton, R. (2015) · ACM SIGPLAN Notices · doi:10.1145/2887747.2804313 · Cited by 11
A design and verification methodology for secure isolated regions
Sinha, R., Costa, M., Lal, A., Lopes, N. P., Rajamani, S. K., Seshia, S. A., & Vaswani, K. (2016) · ACM SIGPLAN Notices · doi:10.1145/2980983.2908113 · Cited by 11
ALPHAPROG: Reinforcement Generation of Valid Programs for Compiler Fuzzing
Li, X., Liu, X., Chen, L., Prajapati, R., & Wu, D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21527 · Cited by 9
“To generate high-quality programs in the context of compiler testing, researchers propose to use rigorous generation engines that encodes formal language grammars for whole program generation (Yang et al. 2011).”
BackgroundBonsai: synthesis-based reasoning for type systems
Chandra, K., & Bodík, R. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3158150 · Cited by 13
“is likely contributes to its eectiveness, since larger programs are likelier to uncover soundness bugs [62].”
BackgroundLightweight, Modular Verification for WebAssembly-to-Native Instruction Selection
VanHattum, A., Pardeshi, M., Fallin, C., Sampson, A., & Brown, F. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3617232.3624862 · Cited by 10
“The most subtle reasoning—and many bugs—occur when there is a large semantic gap between the IR and ISA, and when producing efficient machine code is a first-order priority [76, 53].”
BackgroundFuzzing JavaScript engines with a syntax-aware neural program model
Xu, H., Wang, Y., Jiang, Z., Fan, S., Fu, S., & Xie, P. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103947 · Cited by 13
Compiling with Abstract Interpretation
Lesbre, D., & Lemerre, M. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3656392 · Cited by 8
“We have run TAI on some C programs generated using CSmith [Yang et al. 2011] (limited to the constructs that Imp can represent: integer variables and non-recursive function calls only).”
MethodPeng, C., Jiang, M., Wu, L., & Zhou, Y. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690237 · Cited by 8
Metadata
- Authors
- Yang, X., Chen, Y., Eide, E., & Regehr, J.
- Affiliations
- University of Utah (US)
- Venue
- Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation
- Publisher
- ACM
- Published
- 2011-06-04
- Pages
- 283-294
- ISSN
- 1531-7102 (ISSN-L)
- OpenAlex
- W4238083723
- Cited by
- 825
- References
- 23
- Citations by year
- 2026: 71 · 2025: 115 · 2024: 93 · 2023: 88 · 2022: 59 · 2021: 50 · 2020: 58 · 2019: 45 · 2018: 50 · 2017: 40 · 2016: 42 · 2015: 45 · 2014: 24 · 2013: 23 · 2012: 19
- FWCI
- 15.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Keywords
- compiler bugs, C compiler, compiler testing, undefined behavior, bug reports
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.