Finding and understanding bugs in C compilers
The DOI matches its Crossref record; 22 of the 23 references it cites were found in a registry or an index.
Abstract
Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to compiler developers. Every compiler we tested was found to crash and also to silently generate wrong code when presented with valid input. In this paper we present our compiler-testing tool and the results of our bug-hunting study. Our first contribution is to advance the state of the art in compiler testing. Unlike previous tools, Csmith generates programs that cover a large subset of C while avoiding the undefined and unspecified behaviors that would destroy its ability to automatically find wrong-code bugs. Our second contribution is a collection of qualitative and quantitative results about the bugs we have found in open-source C compilers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This paper demonstrates that modern titanic LLMs can be leveraged to directly perform both generation-based and mutation-based fuzzing studied for decades, while being fully automated, generalizable, and applicable to domains challenging for traditional approaches (such as DL systems).”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [80]
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Klein, G., Elphinstone, K., Heiser, G., Andronick, J., Cock, D., Derrin, P., Elkaduwe, D., Engelhardt, K., Kolanski, R., Norrish, M., et al. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629596 · Cited by 1,633 · cited 1×
“As measured by the responses to our bug reports, the defects discovered by Csmith are important.”
MethodAn empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,281
Formal verification of a realistic compiler
Leroy, X. (2009) · Communications of the ACM · doi:10.1145/1538788.1538814 · Cited by 1,187 · cited 1×
“Wolfe [30], talk ing about independent software vendors (ISVs) says: An ISV with a complex code can work around correctness, turn off the optimizer in one or two .les, and usually they have to do that for any of the compilers they use (emphasis ours).”
BackgroundEncyclopedia of Software Engineering
(2002) · Wiley eBooks · doi:10.1002/0471028959 · Cited by 1,161
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079
An experimental evaluation of the assumption of independence in multiversion programming
Knight, J. C., & Leveson, N. G. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312924 · Cited by 774 · cited 3×
“In summary, despite the fact that Knight and Leveson [13] found a substantial number of correlated errors in an experiment on Nversion programming, Csmith has yielded no evidence of correlated failures among unrelated C compilers.”
BackgroundInfluential(1998) · Cited by 469
Grammar-based whitebox fuzzing
Godefroid, P., Kiezun, A., & Levin, M. Y. (2008) · Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1375581.1375607 · Cited by 462 · cited 1×
“[8] showed a way to solve this problem by integrating a constraint solver with a grammar for the language being generated.”
BackgroundHDD: hierarchical delta debugging
Misherghi, G., & Su, Z. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134307 · Cited by 274 · cited 1×
“Delta debugging [31] automates test-case reduction, but all existing variants that are intended for reducing C programs—such as hierarchical delta debugging [18] and Wilkerson’s implementation [29]—introduce undefined behavior.”
Background
Research works printed but not linked: 7
- F. Bellard . TCC: Tiny C compiler, ver. 0.9.25 , May 2009 . http://bellard.org/tcc/. F. Bellard. TCC: Tiny C compiler, ver. 0.9.25, May 2009. http://bellard.org/tcc/.
- C. L. Biffle . Undefined behavior in Google NaCl , Jan. 2010 . http://code.google.com/p/nativeclient/issues/detail?id=245. C. L. Biffle. Undefined behavior in Google NaCl, Jan. 2010. http://code.google.com/p/nativeclient/issues/detail?id=245.
- 10.1002/spe.v37:14 · doi:10.1002/spe.v37:14
- B. Turner . Random Program Generator , Jan. 2007 . http://sites.google.com/site/brturn2/randomcprogramgenerator. B. Turner. Random Program Generator, Jan. 2007. http://sites.google.com/site/brturn2/randomcprogramgenerator.
- 10.5555/1060289.1060313 · doi:10.5555/1060289.1060313
- D. S. Wilkerson . Delta ver . 2006 .08.03, Aug. 2006. http://delta.tigris.org/. D. S. Wilkerson. Delta ver. 2006.08.03, Aug. 2006. http://delta.tigris.org/.
- C. Zhao Automated test program generation for an industrial optimizing compiler . In Proc. ICSE Workshop on Automation of Software Test , pages 36 -- 43 , May 2009 . C. Zhao et al. Automated test program generation for an industrial optimizing compiler. In Proc. ICSE Workshop on Automation of Software Test, pages 36--43, May 2009.
Cited by
Items 401–500 of 833
Debugger Toolchain Validation via Cross-Level Debugging
Yang, Y., Sun, M., Wu, J., Li, Q., & Zhou, Y. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3669940.3707271 · Cited by 9
Runtime verification of microcontroller binary code
Reinbacher, T., Bräuer, J., Horauer, M., Steininger, A., & Kowalewski, S. (2012) · Science of Computer Programming · doi:10.1016/j.scico.2012.10.015 · Cited by 8
JATTACK: Java JIT Testing using Template Programs
Zang, Z., Yu, F.-Y., Wiatrek, N., Gligoric, M., & Shi, A. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion58688.2023.00014 · Cited by 8
“Grammar-based generation [2], [18]–[20] and mutation-based fuzzing [3], [4], [21]–[26] are two common approaches to obtaining programs as test inputs to compilers.”
BackgroundFLUX: Finding Bugs with LLVM IR Based Unit Test Crossovers
Liu, E. X., Xu, S., & Lie, D. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00100 · Cited by 11
“Some generate tests cases with the aid of a grammar [9], [10], heuristics [11], [12], deep learning [13], [14], and by mutating existing tests to form new valid inputs [15]–[18].”
BackgroundMethodZhou, C., Qian, B., Go, G., Zhang, Q., Li, S., & Jiang, Y. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689757 · Cited by 20
Towards Understanding Refactoring Engine Bugs
Wang, H., Xu, Z., Zhang, H., Tsantalis, N., & Tan, S. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747289 · Cited by 22
On the use of formal methods to model and verify neuronal archetypes
MARIA, E. D., Bahrami, A., L’Yvonnet, T., Felty, A., Gaffé, D., Ressouche, A., & Grammont, F. (2021) · Frontiers of Computer Science · doi:10.1007/s11704-020-0029-6 · Cited by 11
“The CompCert compiler is only slightly slower than other well-known C compilers, which are known to have many bugs [41].”
BackgroundA semantic approach for automated test oracle generation
Guo, H.-F. (2016) · Computer Languages · doi:10.1016/j.cl.2016.01.006 · Cited by 9
New, M. S., Fetscher, B., Findler, R. B., & McCarthy, J. A. (2017) · Journal of Functional Programming · doi:10.1017/s0956796817000107 · Cited by 11
A calculus for Esterel: if can, can. if no can, no can.
Florence, S. P., You, S.-H., Tov, J. A., & Findler, R. B. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3290374 · Cited by 8
“Perhaps unsurprisingly, we also discovered bugs in both of the implementations during this process (as random testing can be extremely effective (Yang et al. 2011)).”
BackgroundProgram Ingredients Abstraction and Instantiation for Synthesis-based JVM Testing
Zhao, Y., Wang, Z., Chen, J., Fu, R., Lu, Y., Gao, T., & Ye, H. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690366 · Cited by 8
An Empirical Study of Bugs in the rustc Compiler
Liu, Z., Feng, Y., Ni, Y., Li, S., Yin, X., Shi, Q., Xu, B., & Su, Z. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763800 · Cited by 12
“Research on testing, debugging, and analyzing C/C++ compilers often relies on CSmith [Yang et al. 2011], a random generator that produces valid C programs covering a wide range of syntax features.”
Covering All the Bases: Type-Based Verification of Test Input Generators
Zhou, Z., Mishra, A., Delaware, B., & Jagannathan, S. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591271 · Cited by 15
“Expressing coverage as part of a type system also allows us to be agnostic to (a) how generators are constructed, (b) the particulars of the application domain [Fetscher et al. 2015; Pałka et al. 2011; Yang et al. 2011], and (c) the specific structure of the properties being tested [Lampropoulos et al.”
InfluentialEvolutionary Generative Fuzzing for Differential Testing of the Kotlin Compiler
Calin, G., Olsthoorn, M., Derakhshanfar, P., Akhin, M., & Panichella, A. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663864 · Cited by 10
Testing Gremlin-Based Graph Database Systems via Query Disassembling
Zheng, Y., Dou, W., Tang, L., Cui, Z., Gao, Y., Song, J., Xu, L., Zhu, J., Wang, W., Wei, J., et al. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680392 · Cited by 11
DarthShader: Fuzzing WebGPU Shader Translators & Compilers
Bernhard, L., Schiller, N., Schloegel, M., Bars, N., & Holz, T. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690209 · Cited by 10
Grammar Mutation for Testing Input Parsers
Bendrissou, B., Cadar, C., & Donaldson, A. F. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708517 · Cited by 7
Finding Missed Code Size Optimizations in Compilers using Large Language Models
Italiano, D., & Cummins, C. (2025) · ACM SIGPLAN International Conference on Compiler Construction (CC) · doi:10.1145/3708493.3712686 · Cited by 21
A Tale of Two DL Cities: When Library Tests Meet Compiler
Shen, Q., Tian, Y., Ma, H., Chen, J., Huang, L., Fu, R., Cheung, S.-C., & Wang, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00025 · Cited by 18
“Crash has been widely used in compiler testing [14], [26], which refers to an unexpected termination of compilation process.”
Chiu, A., Garvey, J. D., & Abdelrahman, T. S. (2015) · ACM International Conference on Computing Frontiers (CF) · doi:10.1145/2742854.2742883 · Cited by 7
“Our work also relates to program generators, such as CSmith [21], which generates random C programs to find bugs in compilers, Code-Smith Generator [4] that creates Visual Basic code using templates, TestMake [12], which generates test harnesses for code, and AutoRT [15], which automatically…”
MethodDebugging Debug Information With Neural Networks
Artuso, F., Luna, G. A. D., & Querzoni, L. (2022) · IEEE Access · doi:10.1109/access.2022.3176617 · Cited by 8
“Compiler testing techniques [16] are mainly based on fuzzing: compilers are fed with random programs either generated from scratch [17], [18] or obtained by mutating existing ones [19], [20].”
BackgroundMethodRandom testing of a higher-order blockchain language (experience report)
Hoang, T., Trunov, A., Lampropoulos, L., & Sergey, I. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3547653 · Cited by 10
“While unable prove the absence of errors, this approach has been shown extremely efficient for discovering multiple bugs in industry-strength compilers of both imperative (Yang et al. 2011) and functional languages (Midtgaard et al. 2017; Pałka et al. 2011).”
Generating Conforming Programs with Xsmith
Hatch, W., Darragh, P., Porncharoenwase, S., Watson, G., & Eide, E. N. (2023) · ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE) · doi:10.1145/3624007.3624056 · Cited by 7
Self-citationFaster bug detection for software product lines with incomplete feature models
Souto, S., Gopinath, D., d’Amorim, M., Marinov, D., Sarfraz Khurshid, D., & Batory, D. S. (2015) · International Software Product Line Conference (SPLC) · doi:10.1145/2791060.2791093 · Cited by 7
“These bad cases ranked lower, at positions 23 and 38 respectively.”
BackgroundVerified Translation Validation of Static Analyses
Barthe, G., Blazy, S., Laporte, V., Pichardie, D., & Trieu, A. (2017) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf.2017.16 · Cited by 8
“The fourth block gathers C programs that were randomly generated by the Csmith tool [16].”
MethodiDEV: exploring and exploiting semantic deviations in ARM instruction processing
Qin, S., Zhang, C., Chen, K., & Li, Z. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464842 · Cited by 12
“In addition to test emulator and disassembly tools we mentioned before, differential testing is also widely used on the testing of various types of software, including C compilers [28, 44, 49], JVMs implementations [10, 11], program analyzers [26], SSL/TLS certification validation [6, 39, 43, 47].”
MethodAutomatic Equivalence Checking of UF+IA Programs
Lopes, N. P., & Monteiro, J. C. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39176-7_18 · Cited by 7
An End-to-End Toolchain: From Automated Cost Modeling to Static WCET and WCEC Analysis
Sieh, V., Burlacu, R., Hönig, T., Janker, H., Raffeck, P., Wägemann, P., & Schröder‐Preikschat, W. (2017) · IEEE International Symposium on Real-Time Distributed Computing (ISORC) · doi:10.1109/isorc.2017.10 · Cited by 7
“CSMITH was configured with incrementing seed numbers starting with the number 1.”
MethodInfluentialÇelik, A., Palmskog, K., Parović, M., Arias, E. J. G., & Gligoric, M. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00057 · Cited by 7
“Nevertheless, just as test suites may be inadequate, formal specifications can fail to account for unwanted program behavior [6], [7], potentially compromising the ability of formal verification to rule out bugs and leading to lower trust in verified code.”
BackgroundMuehlboeck, F., & Henzinger, T. A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88494-9_12 · Cited by 7
“This technique has been fruitfully applied to finding bugs in Javascript debuggers [31], C compilers [42], and SQL databases [40,38,37].”
MethodLarge Language Models for C Test Case Generation: A Comparative Analysis
Guzu, A., Nicolae, G., Cucu, H., & Burileanu, C. (2025) · Electronics · doi:10.3390/electronics14112284 · Cited by 7
Interleaving Large Language Models for Compiler Testing
Ni, Y., & Li, S. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763079 · Cited by 15
Fuzzing C++ Compilers via Type-Driven Mutation
Wang, B., Chen, C., Deng, M., Chen, J., Zhang, X., Lin, Y., Hao, D., & Sun, J. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763094 · Cited by 6
Uncovering Bugs in Code Coverage Profilers via Control Flow Constraint Solving
Wang, Y., Zhang, P., Sun, M., Lu, Z., Yang, Y., Tang, Y., Qian, J., Li, Z., & Zhou, Y. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3321381 · Cited by 6
“In addition, following previous studies [22], [23], we use Csmith [52] to generate 1000 random programs as a supplement.”
MethodSyntax-Aware Mutation for Testing the Solidity Compiler
Mitropoulos, C., Sotiropoulos, T., Ioannidis, S., & Mitropoulos, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-51479-1_17 · Cited by 6
Predictive Context-sensitive Fuzzing
Borrello, P., Fioraldi, A., D’Elia, D. C., Balzarotti, D., Querzoni, L., & Giuffrida, C. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24113 · Cited by 6
“The amount of information that a modern fuzzer acquires during the (many) executions of the program under test can vary, leading to a distinction between black-box [21], [22], white-box [23], [24], and grey-box [25], [26] fuzzers.”
BackgroundUPBEAT: Test Input Checks of Q# Quantum Libraries
Hu, T., Ye, G., Tang, Z., Tan, S. H., Wang, H., Li, M., & Wang, Z. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652120 · Cited by 12
Source-Level Debugging of Compiler-Optimised Code: Ill-Posed, but Not Impossible
Kell, S., & Stinnett, J. R. (2024) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3689492.3690047 · Cited by 7
“Unlike miscompilation bugs of the kind caught with the help of Csmith [34] and similar tools, which are usually the result of many compute-hours spent hammering the compiler with unusual programs, incorrect or incomplete debug info is simply the norm.”
Enhancing Black-box Compiler Option Fuzzing with LLM through Command Feedback
Wang, T., Wang, R., Chen, Y., Yu, L., Pan, Z., Zhang, M., Ma, H., & Zheng, J. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00039 · Cited by 9
“There are many researchers focusing on this topic, employing a range of techniques to generate diverse test case, including formal verification [9], [10], translation validation, differential testing [11], equivalence modulo inputs [12]–[14] and fuzzing [15]–[20].”
Self-citationTesting Refactoring Engine via Historical Bug Report driven LLM
Wang, H., Xu, Z., & Tan, S. H. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00020 · Cited by 10
“To reduce this threat, we referred to the previous studies [31], [50], and then adopted an open-coding scheme.”
Type-Centric Kotlin Compiler Fuzzing: Preserving Test Program Correctness by Preserving Types
Stepanov, D., Akhin, M., & Беляев, М. А. (2021) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst49551.2021.00044 · Cited by 6
“, incorporating deep knowledge of the target programming language; CSmith [6] is the most wellknown tool of this kind.”
BackgroundInfluentialRemgen: Remanufacturing a Random Program Generator for Compiler Testing
Tu, H., Jiang, H., Li, X., Ren, Z., Zhou, Z., & Jiang, L. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00057 · Cited by 6
“Only a few constructed test programs can trigger bugs [2], [6], [15], [17].”
BackgroundResultInfluentialSimulink Compiler Testing via Configuration Diversification With Reinforcement Learning
Li, X., Guo, S., Cheng, H., & Jiang, H. (2023) · IEEE Transactions on Reliability · doi:10.1109/tr.2023.3317643 · Cited by 6
“Existing general-purpose compiler testing approaches can be classified into three categories, including, randomized differential testing (RDT), different optimization levels (DOL), and EMI [23], [34], [35], [36], [37].”
BackgroundMulti-class vulnerability prediction using value flow and graph neural networks
McLaughlin, C., & Lu, Y. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09819-3 · Cited by 9
“We utilized the Csmith [15] test generation tool to generate these synthetic programs.”
Emulation in post-silicon validation: It's not just for functionality anymore
Balston, K., Hu, A. J., Wilton, S. J. E., & Nahir, A. (2012) · Proceedings · doi:10.1109/hldvt.2012.6418252 · Cited by 6
“Random A series of 10,000 random programs generated by Csmith [18]”
MethodAn abstract memory functor for verified C static analyzers
Blazy, S., Laporte, V., & Pichardie, D. (2016) · ACM SIGPLAN International Conference on Functional Programming (ICFP) · doi:10.1145/2951913.2951937 · Cited by 6
“Then there are three interesting programs that have been randomly generated by the Csmith tool [26].”
MethodFuzzinator: An Open-Source Modular Random Testing Framework
Hodován, R., & Kiss, Á. (2018) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2018.00050 · Cited by 6
“, to web engines [2], operating system internals [3], compilers [4], interpreters [5], [6], user interfaces [7], etc.”
BackgroundLeveraging Models to Reduce Test Cases in Software Repositories
Gharachorlu, G., & Sumner, W. N. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00035 · Cited by 6
Formalizing Stack Safety as a Security Property
Anderson, S. N., Blanco, R., Lampropoulos, L., Pierce, B. C., & Tolmach, A. P. (2023) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf57540.2023.00037 · Cited by 6
“A better choice might be to generate high-level code using a tool like CSmith [36], or prove the properties instead.”
MethodA new verified compiler backend for CakeML
Tan, Y. K., Myreen, M. O., Kumar, R., Fox, A. C. J., Owens, S., & Norrish, M. (2016) · ACM SIGPLAN Notices · doi:10.1145/3022670.2951924 · Cited by 6
Testing a Formally Verified Compiler
Monniaux, D. P., Gourdin, L., Boulmé, S., & Lebeltel, O. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-38828-6_3 · Cited by 6
“com/csmith-project/csmith [26] with packed structures (gcc extension) disabled.”
Understanding and improving cyber-physical system models and development tools
Chowdhury, S. A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3183455 · Cited by 6
“rich model generation to effective bug discovery [4, 6].”
BackgroundMethodAn Analysis of x86-64 Inline Assembly in C Programs
Rigger, M., Marr, S., Kell, S., Leopoldseder, D., & Mössenböck, H. (2018) · ACM SIGPLAN Notices · doi:10.1145/3296975.3186418 · Cited by 6
Language-Agnostic Generation of Compilable Test Programs
Kreutzer, P., Kraus, S., & Philippsen, M. (2020) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst46399.2020.00015 · Cited by 6
“Our first approach ( SQL-W ) to still compare different implementations is inspired by Csmith [4] and replaces all arithmetic operations with w rapper functions.”
BackgroundMethodResultInfluentialPractical Automatic Lightweight Nondeterminism and Flaky Test Detection and Debugging for Python
Groce, A., & Holmes, J. (2020) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs51102.2020.00035 · Cited by 8
“Differential testing is extremely powerful, in that any (properly defined) divergence of behavior indicates a functional correctness fault in (at least) one of the systems under test, and is widely used for systems software such as compilers [21], [31] and file systems [9].”
MethodLAFuzz: Neural Network for Efficient Fuzzing
Wang, X., Hu, C., Ma, R., Li, B., & Wang, X. (2020) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai50040.2020.00098 · Cited by 6
Model of Execution Trace Obfuscation Between Threads
Sha, Z., Shu, H., Xiong, X., & Kang, F. (2021) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3123159 · Cited by 6
“Yang et al. [29]. proved that mature compiler software such as GCC and LLVM may still produce optimization errors under exceptional circumstances, which further illustrate the uncertainty of formal verification.”
MethodAlive-Infer: data-driven precondition inference for peephole optimizations in LLVM
Menendez, D., & Nagarakatte, S. (2017) · ACM SIGPLAN Notices · doi:10.1145/3140587.3062372 · Cited by 6
Integrated language definition testing
Kats, L. C. L., Vermaas, R., & Eelco Visser, B. (2011) · ACM SIGPLAN Notices · doi:10.1145/2076021.2048080 · Cited by 6
Reducing post-silicon coverage monitoring overhead with emulation and Bayesian feature selection
Gallardo, R. O., Huy, A. J., Ivanov, A., & Mirian, M. S. (2015) · International Conference on Computer Aided Design · doi:10.1109/iccad.2015.7372655 · Cited by 6
“For our training tests, we used random C programs generated using CSmith [23], and then compiled the programs to run on the LEON3.”
MethodExtraction and Mutation at a High Level: Template-Based Fuzzing for JavaScript Engines
Wong, W. K., Xiao, D., Lai, C. T., Peng, Y., Wu, D., & Wang, S. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763154 · Cited by 9
PAClab: a program analysis collaboratory
Brunner, R., Dyer, R., Paquin, M., & Sherman, E. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417936 · Cited by 8
“marks are instrumental to test verification tools [3, 9], researchers tend to prefer realistic benchmarks comprised of real-world pro-”
BackgroundTaking Back Control in an Intermediate Representation for GPU Computing
Klimis, V., Clark, J., Baker, A., Neto, D., Wickerson, J., & Donaldson, A. F. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3571253 · Cited by 11
“Differential and metamorphic compiler-testing techniques have been successful in a range of domains over the last decade or so (see e.g. [Le et al. 2014; Yang et al. 2011] and a recent survey [Chen et al. 2020]).”
Boosting Compiler Testing via Eliminating Test Programs with Long-Execution-Time
Wu, J., Yang, Y., & Zhou, Y. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00061 · Cited by 6
“Randomized differential testing is a widely-used black-box differential testing technique in which the inputs are randomly generated [1], [22].”
BackgroundMethodInfluentialFeature-Sensitive Coverage for Conformance Testing of Programming Language Implementations
Park, J., Youn, D., Lee, K., & Ryu, S. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591240 · Cited by 9
Mix Testing: Specifying and Testing ABI Compatibility of C/C++ Atomics Implementations
Geeson, L., Brotherston, J., Dijkstra, W., Donaldson, A. F., Smith, L., Sorensen, T., & Wickerson, J. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689727 · Cited by 5
First-Class Verification Dialects for MLIR
Fehr, M., Fan, Y., Pompougnac, H., Regehr, J. D., & Grosser, T. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3729309 · Cited by 5
“Peephole optimizers are a common source of compiler bugs; for example, the Csmith paper reports that InstCombine, LLVM’s primary peephole optimization pass, was its single buggiest component [39].”
Self-citationType Batched Program Reduction
Gharachorlu, G., & Sumner, W. N. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598065 · Cited by 5
“For training data across the languages, we used 40 non-buggy CSmith [49] generated programs for C and sampled les from Github trending projects for Rust and Go [11].”
MethodDon’t Look UB: Exposing Sanitizer-Eliding Compiler Optimizations
Isemann, R., Giuffrida, C., Bos, H., van der Kouwe, E., & Gleissenthall, K. V. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591257 · Cited by 5
“While previous work [Le et al. 2014; Livinskii et al. 2020; Yang et al. 2011] proposed differential testing of compilers using different optimization levels, their test oracles relied on the generated programs being free of undefined behavior.”
BackgroundMethodAccurate Coverage Metrics for Compiler-Generated Debugging Information
Stinnett, J. R., & Kell, S. (2024) · ACM SIGPLAN International Conference on Compiler Construction (CC) · doi:10.1145/3640537.3641578 · Cited by 5
“Metrics computed using the same 5,000 Csmith-generated [31] programs as used by Assaiante et al. Of the two, only our metric is capable of measuring coverage at O0 .”
MethodHigh-Throughput, Formal-Methods-Assisted Fuzzing for LLVM
Fan, Y., & Regehr, J. D. (2024) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo57630.2024.10444854 · Cited by 6
“Second, we evaluate our claim that it is “high throughput.””
MethodSelf-citationFinding XPath Bugs in XML Document Processors via Differential Testing
Li, S., & Rigger, M. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639208 · Cited by 5
CoqPyt: Proof Navigation in Python in the Era of LLMs
Carrott, P., Saavedra, N., Thompson, K., Lerner, S., Ferreira, J. F., & First, E. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663814 · Cited by 14
Security Analysis and Formal Verification on Blockchain and its Applications
Li, K., Gu, R., Xu, J., Chen, Z., Wu, S., Zhou, Y., Zhang, M., Luo, X., Tang, Y., Li, Y., et al. (2025) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000044 · Cited by 6
Parallel runtime verification of temporal properties for embedded software
Reinbacher, T., Geist, J., Moosbrugger, P., Horauer, M., & Steininger, A. (2012) · IEEE/ASME International Conference on Mechatronic and Embedded Systems and Applications · doi:10.1109/mesa.2012.6275566 · Cited by 5
“Moreover, applying runtime verification to high-level code does exclude the possibility to detect errors introduced by the translation into native binary code, i.e., by the compiler itself [6]–[8].”
MethodRuntime Verification for Hybrid Analysis Tools
Nguyen, L. V., Schilling, C., Bogomolov, S., & Johnson, T. T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23820-3_19 · Cited by 5
“In this paper, we describe preliminary work towards a randomized differential testing framework [5] that may be used as a runtime monitor for various components (from parsers to analysis algorithms) in hybrid and cyberphysical systems (CPS) analysis tools such as SpaceEx, dReach, Flow*, and Mathworks’ Simulink/Statefl…”
MethodTan, J., Tay, H. J., Drolia, U., Gandhi, R., & Narasimhan, P. (2016) · ACM International Conference on Embedded Software (EMSOFT) · doi:10.1145/2968478.2968492 · Cited by 5
“[36] found many bugs in mainstream C compilers (e.”
BackgroundGuide Me to Exploit: Assisted ROP Exploit Generation for ActionScript Virtual Machine
Yılmaz, F., Sridhar, M., & Choi, W. (2020) · Annual Computer Security Applications Conference · doi:10.1145/3427228.3427568 · Cited by 5
Testing Your (Static Analysis) Truths
Casso, I., Morales, J. F., López-García, P., & Hermenegildo, M. V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68446-4_14 · Cited by 6
“Most of these testing approaches use programs in the target language as test cases and and apply testing techniques like fuzzing (e.g., [3,27,50]) or differ-ential testing [35], (e.g., [3,27–29 , 50 In [49] the actual pointer aliasing in concrete executions is cross-checked with the pointer aliasing…”
MethodTowards Understanding Tool-chain Bugs in the LLVM Compiler Infrastructure
Xie, X., Yang, H., He, Q., & Chen, L. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00010 · Cited by 5
“Yang et al. [14] created a randomized test cases generation tool Csmith , and found over 325 bugs in C compilers.”
MethodHigh‐coverage metamorphic testing of concurrency support in C compilers
Windsor, M., Donaldson, A. F., & Wickerson, J. (2022) · Software Testing Verification and Reliability · doi:10.1002/stvr.1812 · Cited by 6
Fast PokeEMU: Scaling Generated Instruction Tests Using Aggregation and State Chaining
Yan, Q., & McCamant, S. (2018) · ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE) · doi:10.1145/3186411.3186417 · Cited by 8
“Though simple, random or fuzz testing has a long history of finding bugs [15, 16, 20].”
BackgroundA change-aware per-file analysis to compile configurable systems with #ifdefs
Braz, L., Gheyi, R., Mongiovi, M., Ribeiro, M., Medeiros, F., Teixeira, L., & Souto, S. (2018) · Computer Languages · doi:10.1016/j.cl.2018.01.002 · Cited by 5
“The implementation of compilers may vary according to the operating system [19] .”
BackgroundWholly!: A Build System For The Modern Software Stack
Gelle, L., Saı̈di, H., & Gehani, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00244-2_16 · Cited by 5
“To achieve this, we generate a number of random C files using Csmith [21], and measure the time needed to compile all these files sequentially with each compiler.”
MethodAutomatically finding bugs in commercial cyber-physical system development tool chains
Chowdhury, S. A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3190330 · Cited by 5
“Additionally, using a customized Csmith (a powerful random C-program generator [27]), we introduced automated integration of custom block functionalities in the generated models, for the very first time.”
BackgroundMethodSpeeding Up Bug Finding using Focused Fuzzing
Kargén, U., & Shahmehri, N. (2018) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3230833.3230867 · Cited by 5
“Such systems were either based on mutation [14, 30], or generation [2, 15, 31, 38, 42].”
BackgroundAutomatic validation for binary translation
Chen, J., Yang, W., Shen, B.-Y., Li, Y.-J., & Hsu, W. (2015) · Computer Languages · doi:10.1016/j.cl.2015.05.002 · Cited by 5
Concretely Mapped Symbolic Memory Locations for Memory Error Detection
Tu, H., Jiang, L., Hong, J., Ding, X., & Jiang, H. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3395412 · Cited by 4
“Worse still, the higher optimization may even aggregate the problem because compiler optimization can be too aggressive [68], [69], so this is the reason why KLEE(O PT ) and KLEE(D ET +O PT ) miss nearly half of Dof memory errors.”
BackgroundCross-Language Differential Testing of JSON Parsers
Möller, J., Weißberg, F., Pirch, L., Eisenhofer, T., & Rieck, K. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3657003 · Cited by 7
“CSmith [35] and T-Reqs [20] generate inputs from a random seed for C compiler and HTTP parser testing, respectively.”
MethodShoot Yourself in the Foot — Efficient Code Causes Inefficiency in Compiler Optimizations
Gao, F., Chen, H., Zhou, Y., & Wang, K. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695548 · Cited by 6
Unveiling Cross-checking Opportunities in Verilog Compilers
Zhou, Y., Jiang, Y., & Lu, J. (2025) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3715325 · Cited by 5
“In the context of C/C++ compilers, related work includes generation-based [43, 76] and mutation-based [28, 42] fuzzers.”
Validating JIT Compilers via Compilation Space Exploration
Li, C., Jiang, Y., Xu, C., & Su, Z. (2025) · ACM Transactions on Computer Systems · doi:10.1145/3715102 · Cited by 7
“Program generators like Csmith [59] and YARP-Gen [32, 33] can produce random C programs.”
Janus: Detecting Rendering Bugs in Web Browsers via Visual Delta Consistency
Zhou, C., Zhang, Q., Qian, B., & Jiang, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00013 · Cited by 8
“A common approach to detecting such logic bugs in other testing research domains is to perform differential testing [43], [19], [23], [11], which compares the behavior of different implementations of the same applications when given the same inputs.”
CompilerDream: Learning a Compiler World Model for General Code Optimization
Deng, C., Wu, J., Feng, N., Wang, J., & Long, M. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736887 · Cited by 19
“During our preliminary experiments, we found that those generated code datasets like Csmith [59] and llvm-stress [34] provide no benefits or even hurt the generalization to real-world scenarios.”
Translation Validation for LLVM’s AArch64 Backend
Berger, R., Briles, M., Boushehrinejadmoradi, N., Coughlin, N., Lam, K., Lopes, N. P., Mada, S., Tirpankar, T., & Regehr, J. D. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763147 · Cited by 5
“In contrast, most prior compiler bug-finding work uses differential testing—executing two compilers’ outputs, and comparing their effects—as an oracle [6, 12, 26].”
Self-citationSRS: Detecting Logic Bugs of Join Implementation in DBMSs via Set Relation Synthesis
Lai, J., Zhang, C., Li, B., Liang, C., Liang, J., Wu, Z., Fu, J., Jiang, Y., & Xu, Z. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3769828 · Cited by 4
“SQLsmith [1], inspired by Csmith [30], is an open-source tool for randomly generating SQL queries.”
Creating a Language for Writing Real-Time Applications for the Internet of Things
Krook, R., Hui, J., Svensson, B. J., Edwards, S. A., & Claessen, K. (2022) · ACM/IEEE International Conference on Formal Methods and Models for Co-Design (MEMOCODE) · doi:10.1109/memocode57689.2022.9954383 · Cited by 8
“, CSmith for C [21], or multiple optimization levels, e.”
BackgroundGrammar Mutation for Testing Input Parsers (Registered Report)
Bendrissou, B., Cadar, C., & Donaldson, A. F. (2023) · International Fuzzing Workshop (FUZZING) · doi:10.1145/3605157.3605170 · Cited by 4
“Our approach builds upon grammar-based fuzzing [2, 14, 23, 25, 35, 37] and the Grammarinator [13] tool in particular.”
BackgroundMethodStatic Test Case Prioritization Strategies for Grammar-Based Testing
Raselimo, M., Grunske, L., & Fischer, B. (2023) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew60602.2023.00025 · Cited by 4
“However, the simplicity and speed with which these test inputs can be generated often leads to very large test suites whose execution requires substantial resources; for example, grammar-based fuzzers routinely generate millions of test inputs [1].”
BackgroundHow Uncertainty Affects Test Results for Driving Automation
Reisgys, F. M., Steinhauser, C., Schwarzhaupt, A., & Sax, E. (2023) · IEEE International Automated Vehicle Validation Conference (IAVVC) · doi:10.1109/iavvc57316.2023.10328078 · Cited by 4
“[22] U3 SuT Code Simplification/Execution : In case not all parts of the SuT software are represented by real code or if the SuT only covers a part of the full software of an ECU, this may lead to uncertainty.”
Metadata
- Authors
- Yang, X., Chen, Y., Eide, E., & Regehr, J.
- Affiliations
- University of Utah (US)
- Venue
- Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation
- Publisher
- ACM
- Published
- 2011-06-04
- Pages
- 283-294
- ISSN
- 1531-7102 (ISSN-L)
- OpenAlex
- W4238083723
- Cited by
- 825
- References
- 23
- Citations by year
- 2026: 71 · 2025: 115 · 2024: 93 · 2023: 88 · 2022: 59 · 2021: 50 · 2020: 58 · 2019: 45 · 2018: 50 · 2017: 40 · 2016: 42 · 2015: 45 · 2014: 24 · 2013: 23 · 2012: 19
- FWCI
- 15.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Keywords
- compiler bugs, C compiler, compiler testing, undefined behavior, bug reports
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.