Finding and understanding bugs in C compilers
The DOI matches its Crossref record; 22 of the 23 references it cites were found in a registry or an index.
Abstract
Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to compiler developers. Every compiler we tested was found to crash and also to silently generate wrong code when presented with valid input. In this paper we present our compiler-testing tool and the results of our bug-hunting study. Our first contribution is to advance the state of the art in compiler testing. Unlike previous tools, Csmith generates programs that cover a large subset of C while avoiding the undefined and unspecified behaviors that would destroy its ability to automatically find wrong-code bugs. Our second contribution is a collection of qualitative and quantitative results about the bugs we have found in open-source C compilers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This paper demonstrates that modern titanic LLMs can be leveraged to directly perform both generation-based and mutation-based fuzzing studied for decades, while being fully automated, generalizable, and applicable to domains challenging for traditional approaches (such as DL systems).”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [80]
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Klein, G., Elphinstone, K., Heiser, G., Andronick, J., Cock, D., Derrin, P., Elkaduwe, D., Engelhardt, K., Kolanski, R., Norrish, M., et al. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629596 · Cited by 1,633 · cited 1×
“As measured by the responses to our bug reports, the defects discovered by Csmith are important.”
MethodAn empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,281
Formal verification of a realistic compiler
Leroy, X. (2009) · Communications of the ACM · doi:10.1145/1538788.1538814 · Cited by 1,187 · cited 1×
“Wolfe [30], talk ing about independent software vendors (ISVs) says: An ISV with a complex code can work around correctness, turn off the optimizer in one or two .les, and usually they have to do that for any of the compilers they use (emphasis ours).”
BackgroundEncyclopedia of Software Engineering
(2002) · Wiley eBooks · doi:10.1002/0471028959 · Cited by 1,161
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079
An experimental evaluation of the assumption of independence in multiversion programming
Knight, J. C., & Leveson, N. G. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312924 · Cited by 774 · cited 3×
“In summary, despite the fact that Knight and Leveson [13] found a substantial number of correlated errors in an experiment on Nversion programming, Csmith has yielded no evidence of correlated failures among unrelated C compilers.”
BackgroundInfluential(1998) · Cited by 469
Grammar-based whitebox fuzzing
Godefroid, P., Kiezun, A., & Levin, M. Y. (2008) · Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1375581.1375607 · Cited by 462 · cited 1×
“[8] showed a way to solve this problem by integrating a constraint solver with a grammar for the language being generated.”
BackgroundHDD: hierarchical delta debugging
Misherghi, G., & Su, Z. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134307 · Cited by 274 · cited 1×
“Delta debugging [31] automates test-case reduction, but all existing variants that are intended for reducing C programs—such as hierarchical delta debugging [18] and Wilkerson’s implementation [29]—introduce undefined behavior.”
Background
Research works printed but not linked: 7
- F. Bellard . TCC: Tiny C compiler, ver. 0.9.25 , May 2009 . http://bellard.org/tcc/. F. Bellard. TCC: Tiny C compiler, ver. 0.9.25, May 2009. http://bellard.org/tcc/.
- C. L. Biffle . Undefined behavior in Google NaCl , Jan. 2010 . http://code.google.com/p/nativeclient/issues/detail?id=245. C. L. Biffle. Undefined behavior in Google NaCl, Jan. 2010. http://code.google.com/p/nativeclient/issues/detail?id=245.
- 10.1002/spe.v37:14 · doi:10.1002/spe.v37:14
- B. Turner . Random Program Generator , Jan. 2007 . http://sites.google.com/site/brturn2/randomcprogramgenerator. B. Turner. Random Program Generator, Jan. 2007. http://sites.google.com/site/brturn2/randomcprogramgenerator.
- 10.5555/1060289.1060313 · doi:10.5555/1060289.1060313
- D. S. Wilkerson . Delta ver . 2006 .08.03, Aug. 2006. http://delta.tigris.org/. D. S. Wilkerson. Delta ver. 2006.08.03, Aug. 2006. http://delta.tigris.org/.
- C. Zhao Automated test program generation for an industrial optimizing compiler . In Proc. ICSE Workshop on Automation of Software Test , pages 36 -- 43 , May 2009 . C. Zhao et al. Automated test program generation for an industrial optimizing compiler. In Proc. ICSE Workshop on Automation of Software Test, pages 36--43, May 2009.
Cited by
Items 801–833 of 833
Testing IFrame Rendering in Web Browsers
Park, D., Jeon, Y., Wi, S., & Kim, M. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834417
Scribe: Practical Static Binary Patching via Binary-Aware Recompilation of Decompiled Code
Dai, H., Priyadarshan, S., Imran, A., Wang, R., & Bianchi, A. (2026) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp68448.2026.00079
“Liu et al. [63] analyze type, variable, and control-flow recovery issues but rely on manual inspection and synthetic datasets (e.g., from Csmith [64]) with simplified features, which limits the real-world applicability of their findings.”
TMLM: A Taxonomy of Type-Driven Lowering Bugs in WebAssembly JIT CompilersPreprint
Huo, Y., You, S., & Wu, B. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7207446
Learning Compiler Fuzzing Mutators from Historical Bugs
Liu, L., Qin, F., Legunsen, O., & d’Amorim, M. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793374 · Cited by 2
Closing the Loop: Program-Interoperable Large Language Models Fuzzing for JavaScript Engines
Mastropaolo, A., & Cheng, Z. (2026) · Computer · doi:10.1109/mc.2026.3701431
Zhou, Z., Mishra, A., Delaware, B., & Jagannathan, S. (2026) · Journal of Functional Programming · doi:10.46298/jfp.17755
RL–LLMfuzzer: reinforcement learning–guided dual–model differential fuzzing for compiler testing
Li, D., Li, Y., Zhao, B., Xu, J., Yao, J., Wang, N., & Liu, L. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01115-5
“9) was constrained to a mere 1.35%, standing in stark contrast to the severe redundancy exhibited by rule-based templates like Csmith (65.70”
InfluentialBDS-BCB: Branch Delay Slot Bounds Check BypassPreprint
Huo, Y., Wu, B., & You, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7327459
Harnessing Large Language Models for Seed Generation in Greybox Fuzzing
Zhang, Y., Shi, W., Xing, X., & Xu, J. (2026) · IEEE/IFIP International Conference on Dependable Systems and Networks - Supplemental Volume (DSN-S) · doi:10.1109/dsn-s70715.2026.00031
Debugging Functionality-Twisting Translations by LLMs via Differential Testing with Bayesian Prior
Wu, S., Sun, X., Wang, X., & Zhou, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3820377
“Differential testing has been widely adopted in defects uncovering of modern software systems [6, 13–15, 17, 85, 101, 106].”
Polaris: MIR-Level Obfuscation in LLVM for Efficient and Robust Decompiler Resistance
Zhou, A., Yu, J., Zhang, Y., Zhao, Z., Li, Z., & Li, T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787619
“The evaluated programs are generated using Csmith [16], and the resulting binaries are analyzed using three mainstream decompilers, specifically for IDA, Ghidra, and Binary Ninja.”
Validating Mixed-Integer Programming Solvers
Zhou, X., Xu, Z., & Sun, C. P. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787827
Parse this! Summoning Context-Sensitive Inputs with Goblin
Lorch, R., Dar, M. D., Tinelli, C., & Chowdhury, O. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773270
ProofCoop: Collaborative Automated Formal Verification
Kaufman, Z., First, E., Sanchez-Stern, A., Thompson, K., Lerner, S., & Brun, Y. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787783 · Cited by 1
WITFuzz: Validity-Preserving Greybox Fuzzing for WebAssembly Interface Type Binding Generators
Guan, H., He, N., Cao, S., Cai, Y., Guo, Y., & Li, D. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832281
“…require models of downstream toolchains and build environments Static techniques such as formal verification or symbolic reasoning over compiler transformations can provide strong guarantees when the relevant semantics, toolchain behavior, and build environment are modeled [31, 36, 38, 40, 56].”
BackSmith: A Systematic Approach to Testing Compiler Backends
Chen, H., Wang, Y., Zhao, J., & Wang, K. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3839529
Zhang, Y., Wang, Y., Wang, K., & Wang, L. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3839454
“Tools such as Csmith [49] and YARPGen [24, 25] automatically generate random programs and compare outputs across multiple compilers or configurations.”
WGSLsmith: Randomised Testing for the WebGPU Shading Language
Andryskowski, M., Gorzynski, A., Mohsin, H., & Donaldson, A. F. (2026) · ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH) · doi:10.1145/3837729.3840486
Dredd: Scalable Source-Level Mutation Testing for Large C/C++ Codebases
Donaldson, A. F., Lee-Jones, J., & Foo, J. (2026) · ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH) · doi:10.1145/3837729.3840482
RICE: Harnessing LLMs and Historical Issues to Discover Internal Rust Compiler Errors
Lu, L., You, W., Liang, B., & Huang, J. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832277
“In C, tools like Csmith [54] generate large UB-free programs, while PolyGlot [9] adds coverage guidance to reveal deeper defects, though complex optimizations and semantic logic remain underexplored.”
RuSMT: An Executable Semantics as Conformance Oracle and Test Suite Synthesizer
Haghshenas, M., & Xu, M. (2026) · doi:10.1145/3842652.3843196
Lesbre, D., & Lemerre, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-40374-2_5
Tracy, T. J., Seneviratne, Y., Nandagopal, K., Rani, D. C., Dwyer, M. B., Bingham, B., & Kevin Skadron, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41160-0_21
LLMCloc: Compiler bug localization via large language model-based root cause analysis
Shi, B., Lu, W., Tang, J., Huang, Z., & Li, W. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108352
Hybrid Fuzzing in Input Generation Logic
Wei, S., Zhu, S., Meng, R., & Cai, Y. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834412 · Cited by 1
“These approaches [11, 55] typically rely on manually designed generators derived from domain knowledge to produce well-formed inputs.”
BanditFuzz: A Synergistic Differential Fuzzing Framework for Compilers
Li, D., Li, Y., Xu, J., Zhao, B., Wang, N., Yao, J., & Liu, L. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834548
Predicate-Guided Synthesis of Input Generators via Gentoo
Vikram, V., & Padhye, R. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837441 · Cited by 3
“Tools such as CSmith [52, 65] for C compilers and SQLancer [8] for database engines generate inputs that are not merely syntactically valid but semantically rich by construction: CSmith emits C programs free of undefined behavior, while SQLancer produces queries that satisfy the relational schemas…”
FPSieve: LLM-Powered Cascade for Filtering False Positives in OS Migration Testing
Dong, J., Fu, Y., Hu, C., Chen, Y., Xiao, Q., & Shi, H. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834460
Self-citationAugmenting Symbolic Execution with Artificial Intelligence: A Systematic Review and Research Agenda
Iavich, M., Kuchukhidze, T., & Lopata, A. (2026) · Applied Sciences · doi:10.3390/app16199969
Diversified Compilation in the LLM Era: Program-Feature-Driven Adaptive Pass SelectionPreprint
Wang, D., Wang, J., Zhang, F., Xie, G., & Sun, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7661555
Formal Analysis of Authentication Security for the TLS 1.3 Protocol
Yu, T., Jiang, R., Pei, B., & Chen, G. (2026) · doi:10.1109/icsip70099.2026.11744361
“As relying solely on program crashes to detect vulnerabilities is not sufficiently rigorous, Yang et al.[5]first proposed a differential testing approach with fuzz testing, named Csmith, to detect vulnerabilities on programs.”
Detecting Error Diagnostic Defects in C Compiler via Invalid Program Mutation
Peng, Y., Li, B., Xu, Z., Wang, L., Liao, L., & Zhou, Y. (2026) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme67971.2026.00023
Metadata
- Authors
- Yang, X., Chen, Y., Eide, E., & Regehr, J.
- Affiliations
- University of Utah (US)
- Venue
- Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation
- Publisher
- ACM
- Published
- 2011-06-04
- Pages
- 283-294
- ISSN
- 1531-7102 (ISSN-L)
- OpenAlex
- W4238083723
- Cited by
- 825
- References
- 23
- Citations by year
- 2026: 71 · 2025: 115 · 2024: 93 · 2023: 88 · 2022: 59 · 2021: 50 · 2020: 58 · 2019: 45 · 2018: 50 · 2017: 40 · 2016: 42 · 2015: 45 · 2014: 24 · 2013: 23 · 2012: 19
- FWCI
- 15.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Keywords
- compiler bugs, C compiler, compiler testing, undefined behavior, bug reports
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.