Finding and understanding bugs in C compilers
The DOI matches its Crossref record; 22 of the 23 references it cites were found in a registry or an index.
Abstract
Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to compiler developers. Every compiler we tested was found to crash and also to silently generate wrong code when presented with valid input. In this paper we present our compiler-testing tool and the results of our bug-hunting study. Our first contribution is to advance the state of the art in compiler testing. Unlike previous tools, Csmith generates programs that cover a large subset of C while avoiding the undefined and unspecified behaviors that would destroy its ability to automatically find wrong-code bugs. Our second contribution is a collection of qualitative and quantitative results about the bugs we have found in open-source C compilers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This paper demonstrates that modern titanic LLMs can be leveraged to directly perform both generation-based and mutation-based fuzzing studied for decades, while being fully automated, generalizable, and applicable to domains challenging for traditional approaches (such as DL systems).”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [80]
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Klein, G., Elphinstone, K., Heiser, G., Andronick, J., Cock, D., Derrin, P., Elkaduwe, D., Engelhardt, K., Kolanski, R., Norrish, M., et al. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629596 · Cited by 1,633 · cited 1×
“As measured by the responses to our bug reports, the defects discovered by Csmith are important.”
MethodAn empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,281
Formal verification of a realistic compiler
Leroy, X. (2009) · Communications of the ACM · doi:10.1145/1538788.1538814 · Cited by 1,187 · cited 1×
“Wolfe [30], talk ing about independent software vendors (ISVs) says: An ISV with a complex code can work around correctness, turn off the optimizer in one or two .les, and usually they have to do that for any of the compilers they use (emphasis ours).”
BackgroundEncyclopedia of Software Engineering
(2002) · Wiley eBooks · doi:10.1002/0471028959 · Cited by 1,161
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079
An experimental evaluation of the assumption of independence in multiversion programming
Knight, J. C., & Leveson, N. G. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312924 · Cited by 774 · cited 3×
“In summary, despite the fact that Knight and Leveson [13] found a substantial number of correlated errors in an experiment on Nversion programming, Csmith has yielded no evidence of correlated failures among unrelated C compilers.”
BackgroundInfluential(1998) · Cited by 469
Grammar-based whitebox fuzzing
Godefroid, P., Kiezun, A., & Levin, M. Y. (2008) · Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1375581.1375607 · Cited by 462 · cited 1×
“[8] showed a way to solve this problem by integrating a constraint solver with a grammar for the language being generated.”
BackgroundHDD: hierarchical delta debugging
Misherghi, G., & Su, Z. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134307 · Cited by 274 · cited 1×
“Delta debugging [31] automates test-case reduction, but all existing variants that are intended for reducing C programs—such as hierarchical delta debugging [18] and Wilkerson’s implementation [29]—introduce undefined behavior.”
Background
Research works printed but not linked: 7
- F. Bellard . TCC: Tiny C compiler, ver. 0.9.25 , May 2009 . http://bellard.org/tcc/. F. Bellard. TCC: Tiny C compiler, ver. 0.9.25, May 2009. http://bellard.org/tcc/.
- C. L. Biffle . Undefined behavior in Google NaCl , Jan. 2010 . http://code.google.com/p/nativeclient/issues/detail?id=245. C. L. Biffle. Undefined behavior in Google NaCl, Jan. 2010. http://code.google.com/p/nativeclient/issues/detail?id=245.
- 10.1002/spe.v37:14 · doi:10.1002/spe.v37:14
- B. Turner . Random Program Generator , Jan. 2007 . http://sites.google.com/site/brturn2/randomcprogramgenerator. B. Turner. Random Program Generator, Jan. 2007. http://sites.google.com/site/brturn2/randomcprogramgenerator.
- 10.5555/1060289.1060313 · doi:10.5555/1060289.1060313
- D. S. Wilkerson . Delta ver . 2006 .08.03, Aug. 2006. http://delta.tigris.org/. D. S. Wilkerson. Delta ver. 2006.08.03, Aug. 2006. http://delta.tigris.org/.
- C. Zhao Automated test program generation for an industrial optimizing compiler . In Proc. ICSE Workshop on Automation of Software Test , pages 36 -- 43 , May 2009 . C. Zhao et al. Automated test program generation for an industrial optimizing compiler. In Proc. ICSE Workshop on Automation of Software Test, pages 36--43, May 2009.
Cited by
Items 501–600 of 833
Compiler Testing with Relaxed Memory Models
Geeson, L., & Smith, L. (2024) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo57630.2024.10444836 · Cited by 9
ESFuzzer: An Efficient Way to Fuzz WebAssembly Interpreter
Han, J., Zhang, Z., Du, Y., Wang, W. J., & Chen, X. (2024) · Electronics · doi:10.3390/electronics13081498 · Cited by 4
Fuzzing API Error Handling Behaviors using Coverage Guided Fault Injection
Sharma, S., Tanksalkar, S. R., Cherupattamoolayil, S., & Machiry, A. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3637650 · Cited by 4
RGFuzz: Rule-Guided Fuzzer for WebAssembly Runtimes
Park, J., Kim, Y., & Yun, I. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00003 · Cited by 9
“Csmith [51] is a notable compiler fuzzer that performs differential testing for C compilers, randomly generating test cases while avoiding undefined and un-specified C behaviors.”
QEDCartographer: Automating Formal Verification Using Reward-Free Reinforcement Learning
Sanchez-Stern, A., Varghese, A., Kaufman, Z., Zhang, D., Ringer, T., & Brun, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00033 · Cited by 13
“For example, a study [90] of C compilers found that every tested compiler, including LLVM [43] and GCC [75], had bugs, except the portions of CompCert [47] formally verified in Coq.”
vanHelsing: A Fast Proof Checker for Debuggable Compiler Verification
Lezuo, R., Drăgan, I., Barany, G., & Krall, A. (2015) · International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) · doi:10.1109/synasc.2015.34 · Cited by 4
“Compilers themselves are a large and complex piece of software and recent research [1], [2] shows that they are far from bug-free.”
BackgroundA Practical, Principled Measure of Fuzzer Appeal: A Preliminary Study
Gavrilov, M., Dewey, K. T., Groce, A., Zamanzadeh, D. J., & Hardekopf, B. (2020) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs51102.2020.00071 · Cited by 9
“Manually grouping bugs is impractical, given that fuzzers often find hundreds of thousands of triggers [2], [7], [13], [14].”
BackgroundMethodInfluentialResource Sharing for Verified High-Level Synthesis
Pardalos, M., Herklotz, Y., & Wickerson, J. (2022) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm53951.2022.9786208 · Cited by 4
“That the Csmith compiler testing tool [25] has found hundreds of bugs in GCC and LLVM but none in (the verified parts of) CompCert is a testament to the reliability of this development approach.”
MUTAGEN: Reliable Coverage-Guided, Property-Based Testing using Exhaustive Mutations
Mista, A., & Russo, A. (2023) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst57152.2023.00025 · Cited by 4
“Generating random valid programs to test compilers is a clear example of this limitation [8], where developers are forced to write specialized generators by hand [39, 41, 47].”
BackgroundImproved Models for Policy-Agent Learning of Compiler Directives in HLS
Munafo, R., Shahzad, H., Sanaullah, A., Arora, S., Drepper, U., & Herbordt, M. (2023) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec58863.2023.10363530 · Cited by 5
“Exploration of auto-generated orderings brings up several risks: compilers are not designed for arbitrary nonstandard orderings and often emit semantically incorrect output programs [27]; flawless detection of such output is a “halting problem”.”
BackgroundC-CORE: Clustering by Code Representation to Prioritize Test Cases in Compiler Testing
Zhou, W. J., Jiang, X., & Qin, C. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.043248 · Cited by 5
Accurate String Constraints Solution Counting with Weighted Automata
Sherman, E., & Harris, A. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00049 · Cited by 4
“For this reason, we systematically generated synthetic benchmarks, which have been proven to be effective in testing compiler correctness [40] and finding defects in symbolic execution engines [41].”
MethodApplication of Domain-aware Binary Fuzzing to Aid Android Virtual Machine Testing
Kyle, S. C., Leather, H., Franke, B., Butcher, D., & Monteith, S. (2015) · ACM SIGPLAN Notices · doi:10.1145/2817817.2731198 · Cited by 5
Testing compilers for programmable switches through switch hardware simulation
Wong, M. D., Varma, A. K., & Sivaraman, A. (2020) · ACM International Conference on emerging Networking EXperiments and Technologies (CoNEXT) · doi:10.1145/3386367.3431309 · Cited by 4
“, [25, 33]) that aren’t available to switch compiler developers.”
BackgroundMiMIs: Simple, Efficient, and Fast Bounded-Exhaustive Test Case Generators
Dewey, K. T., Hairapetian, S., & Gavrilov, M. (2020) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst46399.2020.00016 · Cited by 4
“, [1]–[20]) are responsible for collectively finding tens of thousands of bugs in popular software, including GCC and Clang [1], [4], [5], [21], Firefox [22]–[24], Eclipse [12], NetBeans [12], and the Rust compiler [8].”
BackgroundProgramming and execution models for parallel bounded exhaustive testing
Awar, N. A., Jain, K., Rossbach, C. J., & Gligoric, M. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485543 · Cited by 4
Dfuzzer: A D-Bus Service Fuzzing Tool
Marhefka, M., & Müller, P. (2014) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2014.51 · Cited by 4
“Applications communicating via D-Bus either provide interfaces which contain callable methods with input arguments, or This work was supported by the Brno University of Technology projects FIT-S-12-1 and FIT-S-14-2486 call such methods over the message bus, providing arguments for them.”
MethodHow We Built Cedar: A Verification-Guided Approach
Disselkoen, C., Eline, A., He, S., Headley, K., Hicks, M., Hietala, K., Kastner, J., Mamat, A., McCutchen, M., Rungta, N., et al. (2024) · Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3663529.3663854 · Cited by 3
SafeRace: Assessing and Addressing WebGPU Memory Safety in the Presence of Data Races
Levine, R., Lee, A., Abbas, N., Little, K., & Sorensen, T. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763075 · Cited by 4
SEDiff: scope-aware differential fuzzing to test internal function models in symbolic execution
Li, P., Meng, W., & Lu, K. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549080 · Cited by 5
“In particular, Kapus and Cadar [23] checked whether the results of symbolic execution conform to concrete execution for programs generated by Csmith [49].”
ResultDifferential Testing of Cryptographic Libraries with Hybrid Fuzzing
Jin, H., An, D., & Kwon, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-29371-9_7 · Cited by 3
Jiang, H., Wang, Z., Zhou, Z., Li, H., Guo, S., Sun, W., & Zhang, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368553 · Cited by 3
“TEPACS first employs Csmith [11], a widely used code generator that conforms to the C99 standard, to generate pragma-uninserted testing programs.”
BackgroundMethodInfluentialTacoma: Enhanced Browser Fuzzing with Fine-Grained Semantic Alignment
Wang, J., Qian, P., Huang, X., Ying, X., Chen, Y., Ji, S., Chen, J., Xie, J., & Liu, L. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680351 · Cited by 6
Self-citationMorphQ++: A Reproducibility Study of Metamorphic Testing on Quantum Compilers
Kitt, L., & Cohen, M. B. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694959 · Cited by 8
Ratte: Fuzzing for Miscompilations in Multi-Level Compilers Using Composable Semantics
Yu, P., Wu, N., & Donaldson, A. F. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716270 · Cited by 7
A Comprehensive Study of OOP-Related Bugs in C++ Compilers
Wang, B., Chen, C., Chen, J., Xu, B., Ye, C., Lin, Y., Dong, G., & Sun, J. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3566490 · Cited by 8
Stelle, G., & Stefanović, D. (2018) · Implementation and Application of Functional Languages · doi:10.1145/3310232.3310236 · Cited by 3
“Other approaches, such as fuzz testing, have confirmed that verified programs remove effectively all bugs [18].”
BackgroundTest Case Reduction: A Framework, Benchmark, and Comparative Study
Kreutzer, P., Kunze, T., & Philippsen, M. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00012 · Cited by 3
FuzzyFlow: Leveraging Dataflow To Find and Squash Program Optimization Bugs
Schaad, P., Schneider, T., Ben‐Nun, T., Calotoiu, A., Ziogas, A. N., & Hoefler, T. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3581784.3613214 · Cited by 6
Detecting Optimizing Compiler Bugs via History-Driven Test Program Mutation
Zeng, J., Tang, Y., Zhang, J., Li, W., & Huang, Z. (2024) · Asia-Pacific Symposium on Internetware · doi:10.1145/3671016.3671387 · Cited by 3
CF-GKAT: Efficient Validation of Control-Flow Transformations
Zhang, C., Kappé, T., Narváez, D. E., & Naus, N. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3704857 · Cited by 6
S cuzer : A Scheduling Optimization Fuzzer for TVM
Chen, X., Lin, X., Wang, J., Sun, J., Wang, J., & Wang, W. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705308 · Cited by 6
“What distinguishes the DL compiler from the DL library is the introduction of scheduling optimization, which allows the model to be optimized for different chip structures.”
DESIL: Detecting Silent Bugs in MLIR Compiler Infrastructure
Suo, C., Wang, J., Wang, Y. F., Jiang, J., Shen, Q., & Chen, J. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763161 · Cited by 3
“Compiler bugs are generally categorized into two main types: crash bugs and silent bugs (also known as wrong code bugs) [21].”
InfluentialTrustable virtual machine scheduling in a cloud
Hermenier, F., & Henrio, L. (2017) · Symposium on Cloud Computing · doi:10.1145/3127479.3128608 · Cited by 3
“The approach is not exhaustive but the random picking already proved its effectiveness [28].”
BackgroundDCT: Differential Combination Testing of Deep Learning Systems
Wang, C., Ge, W., Li, X., & Feng, Z. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30508-6_55 · Cited by 3
Static duplicate bug-report identification for compilers
Hao, D., Junjie, C. H. E. N., ZHANG, L., Wenxiang, H. U., Zhang, H., & Xiong, Y. (2019) · Scientia Sinica Informationis · doi:10.1360/n112019-00001 · Cited by 3
“比如 , Yang 等 [16] 提出了 针对 C 语言编译器的测试程序自动生成工具 Csmith, 这也是当前最广泛使用的 C 程序自动生成工 具 .”
A trustworthy mechanized formalization of R
Bodin, M., Díaz, T., & Tanter, É. (2020) · ACM SIGPLAN Notices · doi:10.1145/3393673.3276946 · Cited by 3
Detecting Inconsistent Thrown Exceptions
Xu, L., & Zhong, H. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00045 · Cited by 6
“For example, although differential testing has been used to detect bugs [14], [15], [26], one of its limitations lies in undefined behaviors [17].”
BackgroundCrellvm: verified credible compilation for LLVM
Kang, J., Kim, Y., Song, Y., Lee, J., Park, S., Shin, M. D., Kim, Y., Cho, S., Choi, J., Hur, C.-K., et al. (2018) · ACM SIGPLAN Notices · doi:10.1145/3296979.3192377 · Cited by 4
Detecting Compiler Bugs Via a Deep Learning-Based Framework
Tang, Y., Ren, Z., Jiang, H., Qiao, L., Liu, D., Zhou, Z., & Kong, W. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500206 · Cited by 3
Benchmark Fuzzing for Android Taint Analyses
Schott, S., & Pauck, F. (2022) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam55253.2022.00007 · Cited by 6
“Yang et. al have developed C SMITH [40], a generator that aims at testing C compilers for correctness.”
BackgroundPrinz, J., & Lampropoulos, L. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591292 · Cited by 5
“…in the form of handwritten random generators for particularly challenging constraints [Hritcu et al. 2016; Midtgaard et al. 2017; Pałka et al. 2011; Yang et al. 2011] and as a general problem for automatically deriving such generators from a language of constraints [Bulwahn 2012b; Claessen et al.…”
BackgroundLazy Code Transformations in a Formally Verified Compiler
Gourdin, L. (2023) · doi:10.1145/3605158.3605848 · Cited by 3
An Empirical Study of JVMs’ Behaviors on Erroneous JNI Interoperations
Hwang, S., Lee, S., & Ryu, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3373239 · Cited by 3
“Yang et al. [44] introduced Csmith, a tool for generating test programs to verify C compilers.”
MethodMonniaux, D. P. (2024) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-024-00760-3 · Cited by 4
“Formally verified compilation Mainstream compilers have bugs [46, 48].”
TLS-DeepDiffer: Message Tuples-Based Deep Differential Fuzzing for TLS Protocol Implementations
Zhao, Z. M., Song, X., Zhong, Q., Zeng, Y., Hu, C., & Guo, S. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00100 · Cited by 4
“…have applied differential fuzzing to many fields of software testing, such as finding side-channel vulnerabilities [34], Java Virtual Machine (JVM) implementations [35], Web application firewalls [36], C compilers [37], detection of HTTP Request Smuggling [38], and TLS certificate validation [15] [39].”
Formal Proofs of Code Generation and Verification Tools
Leroy, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10431-7_1 · Cited by 3
“An executable version of the CompCert C semantics was built to enable testing of the semantics, in particular random testing using Csmith [13].”
Nguyen, L. V., Schilling, C., Bogomolov, S., & Johnson, T. T. (2015) · International Conference on Hybrid Systems: computation and control · doi:10.1145/2728606.2728650 · Cited by 3
“1 There are many tools and meth-ods for random program generation in various languages (C, Java, etc.) [4].”
Improving Fitness Function for Language Fuzzing with PCFG Model
Sun, X., Fu, Y. V., Dong, Y., Liu, Z., & Zhang, Y. (2018) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2018.00098 · Cited by 4
“Early work in language fuzzing, such as including LangFuzz [8], JsFunFuzz [9] and CSmith [10], mainly used manually crafted rules to guide the generation of test case programs.”
BackgroundAutomating formal proofs for reactive systems
Ricketts, D., Robert, V., Jang, D., Tatlock, Z., & Lerner, S. (2014) · ACM SIGPLAN Notices · doi:10.1145/2666356.2594338 · Cited by 3
Generating Highly Structured Inputs: A Survey
Li, Y., Qin, Y., Tian, J., Gu, T., Nie, Y., & Kuang, X. (2021) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc53577.2021.00075 · Cited by 3
Indexed Labels for Loop Iteration Dependent Costs
Tranquilli, P. (2013) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.117.2 · Cited by 3
“…Loop Iteration Dependent Costs Paolo Tranquilli∗ DISI (Dipartimento di Informatica – Scienza e Ingegneria) Università di Bologna Alma Mater tranquil@cs.unibo.it We present an extension to the labelling approach, a technique for lifting resource consumption information from compiled to source code.”
MethodAbstract interpretation of microcontroller code: Intervals meet congruences
Bräuer, J., King, A., & Kowalewski, S. (2012) · Science of Computer Programming · doi:10.1016/j.scico.2012.06.001 · Cited by 3
“It is also not unknown for compilation itself to introduce errors, particularly at higher levels of optimisation [29, 78].”
BackgroundDon’t Go Down the Rabbit Hole: Reprioritizing Enumeration for Property-Based Testing
Mittelman, S. E., Resnick, A., Pérez, I., Goodloe, A. E., & Lampropoulos, L. (2023) · International Symposium on Haskell · doi:10.1145/3609026.3609730 · Cited by 5
MorphQ++: A Reproducibility Study of Metamorphic Testing on Quantum Compilers
Kitt, L., & Cohen, M. B. (2024) · doi:10.1145/3695750.3695823 · Cited by 5
Mutation-Based Fuzzing of the Swift Compiler with Incomplete Type Information
Hyatt, S. C., & Dewey, K. T. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989032 · Cited by 2
“CSmith [3] generates well-typed C programs devoid of undefined behavior from scratch, via the help of a conservative program analysis.”
Distinguishability-Guided Test Program Generation for WebAssembly Runtime Performance Testing
Jiang, S., Zeng, R., Zhou, Y., & Lyu, M. R.-T. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00078 · Cited by 4
“Csmith [20] and YARPGen [61] are two typical program generators designed for generating C programs free of undefined behaviors.”
InfluentialExploiting Undefined Behavior in C/C++ Programs for Optimization: A Study on the Performance Impact
Popescu, L., & Lopes, N. P. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3729260 · Cited by 2
Leveraging Compilation Statistics for Compiler Phase Ordering
Zhao, J., Xia, C., & Wang, Z. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00054 · Cited by 2
“First, we trained a proximal policy optimization (PPO) model on 100 randomly generated programs from Csmith [48], following the approach in Autophase.”
SUNFLOWER: Enhancing Linux Kernel Fuzzing via Exploit-Driven Seed Generation
Zhang, Q., Shen, Y., Liu, J., Xu, Y., Shi, H., Jiang, Y., & Chang, W. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00057 · Cited by 2
“In the realm of compiler fuzzing, Csmith [39] stands out as a notable tool that generates random C programs, which are utilized to test compilers.”
Finding Compiler Bugs through Cross-Language Code Generator and Differential Testing
Feng, Q., Ma, X., Feng, Z., Akhin, M., Song, W., & Liang, P. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763152 · Cited by 2
“While these approaches [7, 11, 12, 24, 28, 38] have been extensively studied for verifying single-language compilers, the correctness of compilers in cross-language systems remains largely unexplored.”
LMFuzz: Program repair fuzzing based on large language models
Lin, R., Wang, R., Hu, G., & Xu, X. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00568-8 · Cited by 3
“We compared LMFuzz with traditional fuzzers (such as Csmith (Yang et al. 2011), GrayC (Even-Mendoza et al. 2023), YarpGen (Livinskii et al. 2020), TypeFuzz (Park et al. 2021), Go-fuzz, and MorphQ (Paltenghi and Pradel 2023)) as well as with the latest fuzzer, Fuzz4All (Xia et al. 2024).”
Zou, P.-Y., Li, X.-C., Zhao, X., Guo, S.-K., Zhou, Z.-D., Ma, Y., & Jiang, H. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3634318
“Following established practices in fuzzing-based synthesis and compiler testing [7], [48], we define a fault-free test case as a Verilog design that can be successfully synthesized by the tool, meaning that the synthesis process completes without errors and produces a valid netlist.”
RVISmith: Fuzzing Compilers for RVV Intrinsics
He, Y., Huang, C., Qu, X., Chen, H., Yang, W. M., & Xie, T. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744790 · Cited by 3
Accelerating Delta Debugging through Probabilistic Monotonicity Assessment
Tao, Y., & Xue, J. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756940 · Cited by 2
Finding and Understanding Bugs in FPGA Place-and-Route Engines
Cosgrove, O., Donaldson, A. F., & Wickerson, J. (2026) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3748173.3779185 · Cited by 2
Bento: Fine-Grained Memory Isolation for COTS WebAssembly Binaries
Draissi, O., & Davi, L. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792439 · Cited by 2
“Our base evaluation corpus consists of our 52 core benchmarks, 300 diverse programs generated with csmith [68], and the three real-world applications with known vulnerabilities from our case studies (Section 7), for a total of 355 unique applications.”
Testing the Unknown: A Framework for OpenMP Testing via Random Program Generation
Laguna, I., Chapman, P., Parasyris, K., Georgakoudis, G., & Rubio-González, C. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00080 · Cited by 4
“We use the same approach that is used in previous work [7] to construct a random program, i.e., uniform distributions are used to choose elements of the program.”
Validity-Preserving Delta Debugging via Generator Trace Reduction
Ren, L., Zhang, X., Hua, Z., Jiang, Y., He, X., Xiong, Y., & Xie, T. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705305 · Cited by 6
“For example, Csmith [55] can generate C programs that meet the standard of C language and do not contain undefined behaviors.”
Cornucopia : A Framework for Feedback Guided Generation of Binaries
Singhal, V., Pillai, A. A., Saumya, C., Kulkarni, M., & Machiry, A. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3561152 · Cited by 2
“W can also use systematic approaches such as Csmith [74] to generate C rograms with the desired features and then use them in Cornucopia to generate a complete binary dataset.”
InfluentialTsimpourlas, F., Petoumenos, P., Xu, M., Cummins, C., Hazelwood, K., Rajan, A., & Leather, H. (2022) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/3559009.3569644 · Cited by 4
“For each Ro-dinia benchmark’s target feature vector, we measure the minimum Euclidean distance to it achieved between BenchPress , code from GitHub, CLgen and CLSmith [1, 37].”
BackgroundMethodVerification of the ROS NavFn planner using executable specification languages
Martin-Martin, E., Montenegro, M., Riesco, A., Rodríguez-Hortalá, J., & Rubio, R. (2023) · Journal of Logical and Algebraic Methods in Programming · doi:10.1016/j.jlamp.2023.100860 · Cited by 4
Automated Ambiguity Detection in Layout-Sensitive Grammars
Liu, J., Zhu, F., & He, F. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3622838 · Cited by 2
“This makes sense because a language with an empty alphabet is trivially empty and thus not interesting at all, though, in theory, we could avoid this axiom (see our artifact for more details).”
BackgroundTorchProbe: Fuzzing Dynamic Deep Learning Compilers
Su, Q., Geng, C., Pekhimenko, G. G., & Si, X. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8311-7_15 · Cited by 9
Enumerating Valid Non-Alpha-Equivalent Programs for Interpreter Testing
Xia, X., Feng, Y., Shi, Q., Jones, J. A., Zhang, X., & Xu, B. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3647994 · Cited by 2
TR-Fuzz: A syntax valid tool for fuzzing C compilers
Zhang, C., Chen, J., Cai, S., Wen, Z., Sosu, R. N. A., & Chen, H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103155 · Cited by 2
Randomised Testing of the Compiler for a Verification-Aware Programming Language
Donaldson, A. F., Sheth, D., Tristan, J.-B., & Usher, A. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00044 · Cited by 6
“Both tools take a grammar-aided, generative approach to producing programs, inspired by the well-known Csmith tool for C compiler testing [52] (and also used by other “-smith” tools, including XDsmith).”
MethodFully Verified Instruction Scheduling
Yang, Z., Shirako, J., & Sarkar, V. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689739 · Cited by 2
CoqPilot, a plugin for LLM-based generation of proofs
Kozyrev, A., Solovev, G., Khramov, N., & Podkopaev, A. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695357 · Cited by 23
“The formal verification approach has proved to be fruitful: for instance, Com-pCert [11], a C compiler written in Coq, was the only C compiler in which an extensive study found no bugs [20].”
Divergence-Aware Testing of Graphics Shader Compiler Back-Ends
Xiao, D., Wang, S., Liu, Z., Peng, Y., Wu, D., & Su, Z. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3729305 · Cited by 4
“In line with other testing works [Le et al. 2014, 2015; Yang et al. 2011], ShaDiv cannot guarantee the absence of bugs in the back-ends of shader compilers.”
An Empirical Study of Bugs in Data Visualization Libraries
Lu, W., Tian, Y., Zhong, X., Ma, H., Xu, Z., Cheung, S.-C., & Sun, C. P. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729363 · Cited by 7
Fuzzing JavaScript JIT compilers with a high-quality differential test oracle
Li, J., Xu, H., Wang, Y., Jiang, Z., Chun, H., Xie, P., Chen, Y., & Xia, T. (2025) · Computers & Security · doi:10.1016/j.cose.2025.104660 · Cited by 7
Self-citationProgramming Language Theoretic Security in the Real World: A Mirage or the Future?
Ruef, A., & Rohlf, C. (2015) · Advances in information security · doi:10.1007/978-3-319-14039-1_15 · Cited by 2
Compiler verification for fun and profit
Leroy, X. (2014) · Formal Methods in Computer-Aided Design · doi:10.1109/fmcad.2014.6987587 · Cited by 2
“Recent experimental studies [1] show that many widely-used production-quality compilers suffer from miscompilation.”
BackgroundImproved semantics and implementation through property-based testing with QuickCheck
Li, H., & Thompson, S. (2014) · International Workshop on Automation of Software Test (AST) · doi:10.1145/2593501.2593509 · Cited by 2
“For instance, In [10], Yang, Chen, et.”
Lightweight runtime checking of C programs with RTC
Milewicz, R. M., Vanka, R., Tuck, J. M., Quinlan, D. J., & Pirkelbauer, P. (2016) · Computer Languages · doi:10.1016/j.cl.2016.01.001 · Cited by 2
“5, we give an arbitrary program derived from one generated by Csmith, a randomized test-case generation tool for C compilers [35].”
MethodTaming a Fuzzer Using Delta Debugging Trails
Pei, Y., Christi, A., Fern, X. Z., Groce, A., & Wong, W. (2014) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2014.58 · Cited by 2
Automated Vulnerable Codes Mutation through Deep Learning for Variability Detection
Cheng, K., Du, G., Wu, T., Chen, L., & Shi, G. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892444 · Cited by 2
“The traditional way is to generate new code according to grammar rules [12]– [15].”
BackgroundAuto Off-Target: Enabling Thorough and Scalable Testing for Complex Software Systems
Kuchta, T., & Zator, B. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556915 · Cited by 2
“CSmith [41] synthesizes correct programs in C with the aim of testing compilers.”
BackgroundAchieving Formal Parallel Program Debugging by Incentivizing CS/HPC Collaborative Tool Development
Gopalakrishnan, G., & Sawaya, J. (2015) · doi:10.1145/2753524.2753531 · Cited by 2
“Unfortunately, if sequential compiler bugs are any indication [45], there is room for considerable caution in going forward without a principled approach toward concurrency bug detection and elimination.”
BackgroundSieh, V., Burlacu, R., Hönig, T., Janker, H., Raffeck, P., Wägemann, P., & Schröder‐Preikschat, W. (2018) · IEEE Embedded Systems Letters · doi:10.1109/les.2018.2868823 · Cited by 2
“The currently employed benchmark generator CSMITH creates many programs, but as it is targeted on finding bugs in compilers, it does not cover all scenarios presented above.”
MethodFuzzing channel-based concurrency runtimes using types and effects
Stiévenart, Q., & Madsen, M. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428254 · Cited by 2
“present Csmith, a program generation technique to discover bugs in C compilers [Yang et al. 2011].”
MethodSymbolic Execution: Foundations, Techniques, Applications, and Future Perspectives
Steinhöfel, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08166-8_22 · Cited by 3
Towards Verified Scalable Parallel Computing with Coq and Spark
Loulergue, F., & Philippe, J. (2023) · Formal Techniques for Java-like Programs · doi:10.1145/3605156.3606450 · Cited by 2
“Although it may not be trivial to understand very precisely what are the guarantees provided by such formal developments [34], they indeed offer code without bugs [47].”
BackgroundCnerator: A Python application for the controlled stochastic generation of standard C source code
Ortín, F., & Escalada, J. (2021) · SoftwareX · doi:10.1016/j.softx.2021.100711 · Cited by 2
Proceedings of the International Conference on Embedded Software
(2023) · doi:10.1145/3607890 · Cited by 2
Mechanized semantics and verified compilation for a dataflow synchronous language with reset
Brun, L. (2020) · theses.fr (ABES) · doi:10.70675/0d242e5czfe16z49f6za04ezf51f4dabb08e · Cited by 9
Theory of Software Testing With Persistent State
Hamlet, D. (2015) · IEEE Transactions on Reliability · doi:10.1109/tr.2015.2436443 · Cited by 4
“29 Whatever the implementor's intentions, they may not be realized.”
BackgroundCompiling Files in Parallel: A Study with GCC
Belinassi, G., Biener, R., Hubička, J., Cordeiro, D. D. A., & Goldman, A. (2022) · International Symposium on Computer Architecture and High Performance Computing Workshops (SBAC-PADW) · doi:10.1109/sbac-padw56527.2022.00011 · Cited by 1
“…the correctness of our changes by (1) boot-strapping GCC; (2) guaranteeing that the GCC test suite was passing; and (3) compiling random programs generated with csmith [28] (a tool that can generate random C programs that conform to the C99 standard) and ensuring that the correct result was found.”
MethodInteractive Source-to-Source Optimizations Validated using Static Resource Analysis
Bertholon, G., Charguéraud, A., Kœhler, T., Bytyqi, B., & Rouhling, D. (2024) · State Of the Art in Java Program Analysis · doi:10.1145/3652588.3663320 · Cited by 6
Metadata
- Authors
- Yang, X., Chen, Y., Eide, E., & Regehr, J.
- Affiliations
- University of Utah (US)
- Venue
- Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation
- Publisher
- ACM
- Published
- 2011-06-04
- Pages
- 283-294
- ISSN
- 1531-7102 (ISSN-L)
- OpenAlex
- W4238083723
- Cited by
- 825
- References
- 23
- Citations by year
- 2026: 71 · 2025: 115 · 2024: 93 · 2023: 88 · 2022: 59 · 2021: 50 · 2020: 58 · 2019: 45 · 2018: 50 · 2017: 40 · 2016: 42 · 2015: 45 · 2014: 24 · 2013: 23 · 2012: 19
- FWCI
- 15.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Keywords
- compiler bugs, C compiler, compiler testing, undefined behavior, bug reports
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.