Finding and understanding bugs in C compilers
The DOI matches its Crossref record; 22 of the 23 references it cites were found in a registry or an index.
Abstract
Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to compiler developers. Every compiler we tested was found to crash and also to silently generate wrong code when presented with valid input. In this paper we present our compiler-testing tool and the results of our bug-hunting study. Our first contribution is to advance the state of the art in compiler testing. Unlike previous tools, Csmith generates programs that cover a large subset of C while avoiding the undefined and unspecified behaviors that would destroy its ability to automatically find wrong-code bugs. Our second contribution is a collection of qualitative and quantitative results about the bugs we have found in open-source C compilers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This paper demonstrates that modern titanic LLMs can be leveraged to directly perform both generation-based and mutation-based fuzzing studied for decades, while being fully automated, generalizable, and applicable to domains challenging for traditional approaches (such as DL systems).”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [80]
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Klein, G., Elphinstone, K., Heiser, G., Andronick, J., Cock, D., Derrin, P., Elkaduwe, D., Engelhardt, K., Kolanski, R., Norrish, M., et al. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629596 · Cited by 1,633 · cited 1×
“As measured by the responses to our bug reports, the defects discovered by Csmith are important.”
MethodAn empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,281
Formal verification of a realistic compiler
Leroy, X. (2009) · Communications of the ACM · doi:10.1145/1538788.1538814 · Cited by 1,187 · cited 1×
“Wolfe [30], talk ing about independent software vendors (ISVs) says: An ISV with a complex code can work around correctness, turn off the optimizer in one or two .les, and usually they have to do that for any of the compilers they use (emphasis ours).”
BackgroundEncyclopedia of Software Engineering
(2002) · Wiley eBooks · doi:10.1002/0471028959 · Cited by 1,161
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079
An experimental evaluation of the assumption of independence in multiversion programming
Knight, J. C., & Leveson, N. G. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312924 · Cited by 774 · cited 3×
“In summary, despite the fact that Knight and Leveson [13] found a substantial number of correlated errors in an experiment on Nversion programming, Csmith has yielded no evidence of correlated failures among unrelated C compilers.”
BackgroundInfluential(1998) · Cited by 469
Grammar-based whitebox fuzzing
Godefroid, P., Kiezun, A., & Levin, M. Y. (2008) · Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1375581.1375607 · Cited by 462 · cited 1×
“[8] showed a way to solve this problem by integrating a constraint solver with a grammar for the language being generated.”
BackgroundHDD: hierarchical delta debugging
Misherghi, G., & Su, Z. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134307 · Cited by 274 · cited 1×
“Delta debugging [31] automates test-case reduction, but all existing variants that are intended for reducing C programs—such as hierarchical delta debugging [18] and Wilkerson’s implementation [29]—introduce undefined behavior.”
Background
Research works printed but not linked: 7
- F. Bellard . TCC: Tiny C compiler, ver. 0.9.25 , May 2009 . http://bellard.org/tcc/. F. Bellard. TCC: Tiny C compiler, ver. 0.9.25, May 2009. http://bellard.org/tcc/.
- C. L. Biffle . Undefined behavior in Google NaCl , Jan. 2010 . http://code.google.com/p/nativeclient/issues/detail?id=245. C. L. Biffle. Undefined behavior in Google NaCl, Jan. 2010. http://code.google.com/p/nativeclient/issues/detail?id=245.
- 10.1002/spe.v37:14 · doi:10.1002/spe.v37:14
- B. Turner . Random Program Generator , Jan. 2007 . http://sites.google.com/site/brturn2/randomcprogramgenerator. B. Turner. Random Program Generator, Jan. 2007. http://sites.google.com/site/brturn2/randomcprogramgenerator.
- 10.5555/1060289.1060313 · doi:10.5555/1060289.1060313
- D. S. Wilkerson . Delta ver . 2006 .08.03, Aug. 2006. http://delta.tigris.org/. D. S. Wilkerson. Delta ver. 2006.08.03, Aug. 2006. http://delta.tigris.org/.
- C. Zhao Automated test program generation for an industrial optimizing compiler . In Proc. ICSE Workshop on Automation of Software Test , pages 36 -- 43 , May 2009 . C. Zhao et al. Automated test program generation for an industrial optimizing compiler. In Proc. ICSE Workshop on Automation of Software Test, pages 36--43, May 2009.
Cited by
Items 201–300 of 833
Hunting for Bugs in Code Coverage Tools via Randomized Differential Testing
Yang, Y., Zhou, Y., Sun, H., Su, Z., Zuo, Z., Xu, L., & Xu, B. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00061 · Cited by 30
“Randomized differential testing is a widely-used black-box differential testing technique in which the inputs are randomly generated [19], [54].”
BackgroundMethodInfluentialExploring Missed Optimizations in WebAssembly Optimizers
Liu, Z., Xiao, D., Li, Z., Wang, S., & Meng, W. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598068 · Cited by 35
“Note that our test C programs (generated by Csmith [88]; see Sec.”
MethodInfluentialTesting Cross-Platform Mobile App Development Frameworks (T)
Boushehrinejadmoradi, N., Ganapathy, V., Nagarakatte, S., & Iftode, L. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.21 · Cited by 45
“This method has previously been used successfully, for instance, to find bugs in compilers [41].”
MethodPractical verification of peephole optimizations with Alive
Lopes, N. P., Menendez, D., Nagarakatte, S., & Regehr, J. D. (2018) · Communications of the ACM · doi:10.1145/3166064 · Cited by 29
“Random testing tools [20, 15, 7] have discovered numerous bugs in LLVM optimizations both for sequential programs and concurrent programs.”
BackgroundSelf-citationSynthesizing JIT Compilers for In-Kernel DSLs
Geffen, J. V., Nelson, L., Dillig, I., Wang, X., & Torlak, E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-53291-8_29 · Cited by 26
“Recent advances in compiler testing, such as Csmith [41] and EMI [42], have found hundreds of bugs in GCC and LLVM compilers.”
MethodGuiding Greybox Fuzzing with Mutation Testing
Vikram, V., Laybourn, I., Li, A., Nair, N., OBrien, K., Sanna, R., & Padhye, R. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598107 · Cited by 34
“Our interface is general enough to support complex differential testing oracles such as the ones used in CSmith [80].”
MethodAn Empirical Study of the Reliability of High-Level Synthesis Tools
Herklotz, Y., Du, Z., Ramanathan, N., & Wickerson, J. (2021) · IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM) · doi:10.1109/fccm51124.2021.00034 · Cited by 25
“1 static volatile int a[9][1][7]; 2 int main() { 3 int tmp = 1; 4 for (int b = 0; b < 2; b++) {”
MethodInfluentialData-Oriented Differential Testing of Object-Relational Mapping Systems
Sotiropoulos, T., Chaliasos, S., Atlidakis, V., Mitropoulos, D., & Spinellis, D. D. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00137 · Cited by 25
“Differential testing has been successfully applied to various domains, most notably compilers and runtime systems [16], [19], [17], [18], [20].”
MethodSecurity risks of porting C programs to webassembly
Stiévenart, Q., Roover, C. D., & Ghafari, M. (2022) · ACM Symposium on Applied Computing · doi:10.1145/3477314.3507308 · Cited by 31
“Csmith [31] performs fuzzing of compilers by generating C programs, compiling them with several compilers, and observing their result: any difference in output is likely a bug.”
BackgroundHDiff: A Semi-automatic Framework for Discovering Semantic Gap Attack in HTTP Implementations
Shen, K., Lu, J., Yang, Y., Chen, J., Zhang, M., Duan, H., Zhang, J., & Zheng, X. (2022) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn53405.2022.00014 · Cited by 24
Compiler Testing using Template Java Programs
Zang, Z., Wiatrek, N., Gligoric, M., & Shi, A. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556958 · Cited by 30
A formally certified end-to-end implementation of Shor’s factorization algorithm
Peng, Y., Hietala, K., Tao, R., Li, L., Rand, R., Hicks, M., & Wu, X. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218775120 · Cited by 45
“For the latter, the benefits of formal methods have been demonstrated empirically: Using sophisticated testing techniques, researchers found hundreds of bugs in the popular mainstream C compilers gcc and clang, but none in CompCert’s verified core (9).”
BackgroundFinding Correctness Bugs in eBPF Verifier with Structured and Sanitized Program
Sun, H., Xu, Y., Liu, J., Shen, Y., Guan, N., & Jiang, Y. (2024) · European Conference on Computer Systems · doi:10.1145/3627703.3629562 · Cited by 24
“Take CSmith [47] for instance.”
BackgroundA Compositional Semantics for Verified Separate Compilation and Linking
Ramananandro, T., Shao, Z., Weng, S.-C., Koenig, J., & Fu, Y. (2015) · ACM SIGPLAN International Conference on Certified Programs and Proofs (CPP) · doi:10.1145/2676724.2693167 · Cited by 24
“Recent work on CompCert [13, 12] has shown that mechanized verification of the compiler correctness for C is both viable and practical, and the resulting compiler is indeed empirically much more reliable than traditional (unverified) ones [23].”
BackgroundLearning to accelerate compiler testing
Chen, J. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3183456 · Cited by 24
“Randomized differential testing (RDT) [14] is one of the most widely-used compiler testing techniques in practice [12, 19].”
BackgroundMethodInfluentialChen, Y., Groce, A., Zhang, C., Wong, W.-K., Fern, X. Z., Eide, E. N., & Regehr, J. D. (2013) · ACM SIGPLAN Notices · doi:10.1145/2499370.2462173 · Cited by 30
Self-citationK-ST: A Formal Executable Semantics of the Structured Text Language for PLCs
Wang, K., Wang, J., POSKITT, C. M., Chen, X., Sun, J., & Cheng, P. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3315292 · Cited by 39
“While extensive research has been conducted on testing more ’traditional’compilers (e.g. vulnerability detection for GCC and Clang [33], [34], [35]), compilers for PLC languages such as ST have received much less attention.”
BackgroundDifferentially Testing Database Transactions for Fun and Profit
Cui, Z., Dou, W., Dai, Q., Song, J., Wang, W., Wei, J., & Ye, D. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556924 · Cited by 34
“Differential testing [55] has been widely applied in many domains, such as compilers [32, 65], runtime systems [44, 45], symbolic execution engines [51], and software libraries [46].”
MethodPortable Software Fault Isolation
Kroll, J. A., Stewart, G., & Appel, A. W. (2014) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf.2014.10 · Cited by 25
“CompCert was the only compiler that did not exhibit bugs when subjected to arbitrary input; no issues were found in CompCert’s proved-correct components.”
BackgroundAlive-FP: Automated Verification of Floating Point Based Peephole Optimizations in LLVM
Menendez, D., Nagarakatte, S., & Gupta, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-662-53413-7_16 · Cited by 28
“Among them, peephole optimizations are persistent source of compiler bugs [21, 43].”
BackgroundSkeletal program enumeration for rigorous compiler testing
Zhang, Q., Sun, C. P., & Su, Z. (2017) · ACM SIGPLAN Notices · doi:10.1145/3140587.3062379 · Cited by 28
Morrisett, G., Tan, G., Tassarotti, J., Tristan, J.-B., & Gan, E. (2012) · ACM SIGPLAN Notices · doi:10.1145/2345156.2254111 · Cited by 23
HACLxN: Verified Generic SIMD Crypto (for all your favourite platforms)
Polubelova, M., Bhargavan, K., Protzenko, J., Beurdouche, B., Fromherz, A., Kulatova, N., & Zanella-Béguelin, S. (2020) · IACR Cryptology ePrint Archive · doi:10.1145/3372297.3423352 · Cited by 28
“Adding large unverified compilers like GCC into the trusted computing base (TCB) for a cryptographic library is risky, since theymay be buggy [43] andmay introduce side channels that were not present in the C code [40].”
BackgroundDetecting C++ Compiler Front-End Bugs via Grammar Mutation and Differential Testing
Tu, H., Jiang, H., Zhou, Z., Tang, Y., Ren, Z., Qiao, L., & Jiang, L. (2022) · IEEE Transactions on Reliability · doi:10.1109/tr.2022.3171220 · Cited by 26
“Csmith [19] and Yarpgen [25] are two widely used C++ program generators to test C++ compilers.”
BackgroundMethodPushing the Limit of 1-Minimality of Language-Agnostic Program Reduction
Xu, Z., Tian, Y., Zhang, M., Zhao, G., Jiang, Y., & Sun, C. P. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586049 · Cited by 25
“2018], which contains 20 real-world complex bug-triggering C programs generated by fuzzing techniques, including CSmith [Yang et al. 2011] and EMI [Le et al.”
Compiler Test-Program Generation via Memoized Configuration Search
Chen, J., Suo, C., Jiang, J., Chen, P., & Li, X. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00172 · Cited by 32
“Due to its important role, plenty of efforts have been devoted to developing various test-program generators, such as Csmith [11], CLsmith [12], and DeepSmith [26].”
BackgroundMethodInfluentialUnderstanding the Security of Linux eBPF Subsystem
Mohamed, M. H. N., Wang, X., & Ravindran, B. (2023) · ACM SIGOPS Asia-Pacific Workshop on Systems (APSys) · doi:10.1145/3609510.3609822 · Cited by 22
“Csmith [25] is an opensource test-case generator tool that can generate random C programs which conform C99 standard.”
Integrated language definition testing
Kats, L. C. L., Vermaas, R., & Eelco Visser, B. (2011) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2048066.2048080 · Cited by 22
“In some cases, sophis ticated, hand-tailored program generators are used, such as Csmith [40], a successful 40,000-line C++ generator pro gram for randomly generating C programs.”
MethodProvably correct peephole optimizations with alive
Lopes, N. P., Menendez, D., Nagarakatte, S., & Regehr, J. D. (2015) · ACM SIGPLAN Notices · doi:10.1145/2813885.2737965 · Cited by 24
Self-citationA Concrete Memory Model for CompCert
Besson, F., Blazy, S., & Wilke, P. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-22102-1_5 · Cited by 21
“With this respect, the CompCert C semantics successfully run hundreds of random test programs generated by CSmith [23].”
MLIRSmith: Random Program Generation for Fuzzing MLIR Compiler Infrastructure
Wang, H., Chen, J., Xie, C., Liu, S., Wang, Z., Shen, Q., & Zhao, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00120 · Cited by 45
“For example, Yang et al. [12] proposed Csmith, which randomly generates C programs with different code features.”
BackgroundMethodResultInfluentialIsolating Compiler Bugs by Generating Effective Witness Programs With Large Language Models
Tu, H., Zhou, Z., Jiang, H., Yusuf, I. N. B., Li, Y., & Jiang, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3397822 · Cited by 35
“This is reasonable, as demonstrated in previous studies [2], [52], [63], compilers can have unexpected consequences if the program is semantically invalid.”
BackgroundMethodResultInfluentialLPR: Large Language Models-Aided Program Reduction
Zhang, M., Tian, Y., Xu, Z., Dong, Y., Tan, S. H., & Sun, C. P. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652126 · Cited by 36
“The original bug-triggering code [17, 21, 28, 43] may have thousands of lines, whereas the distilled version from program reduction tools only contains dozens of lines of code [29].”
BackgroundOpenMP 4.5 Validation and Verification Suite for Device Offload
Diaz, J. M. M., Pophale, S., Hernández, Ó., Bernholdt, D. E., & Chandrasekaran, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-98521-3_6 · Cited by 21
CUDAsmith: A Fuzzer for CUDA Compilers
Jiang, B., Wang, X., Chan, W. K., Tse, T. H., Li, N., Yin, Y., & Zhang, Z. (2020) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac48688.2020.0-156 · Cited by 29
FormatFuzzer : Effective Fuzzing of Binary File Formats
Dutra, R., Gopinath, R., & Zeller, A. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3628157 · Cited by 20
“In order to fuzz binary inputs, the most powerful solution is to create a dedicated fuzzer.”
BackgroundVectorizing Program Ingredients for Better JVM Testing
Gao, T., Chen, J., Zhao, Y., Zhang, Y., & Zhang, L. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598075 · Cited by 20
“of print statements for recording the results of intermediate variables (which can incur extra I/O overheads), VECT calculates the checksum of intermediate variables in the synthesized test program as the outputs inspired by the existing work [54].”
BackgroundMethodRevealing Performance Issues in Server-Side WebAssembly Runtimes Via Differential Testing
Jiang, S., Zeng, R., Rao, Z., Gu, J., Zhou, Y., & Lyu, M. R.-T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00088 · Cited by 30
“Differential testing has been widely used to detect semantic bugs in diverse domains, such as C compilers [52]–[55], JVM implementations [56]–[59], SSL/TLS implementations [60]–[62], and even deep learning systems [63]–[65].”
MethodQuickChecking static analysis properties
Midtgaard, J., & Møller, A. (2017) · Software Testing Verification and Reliability · doi:10.1002/stvr.1640 · Cited by 25
“[24]: a tool that uses randomized differential testing to generate random C programs free of C’s notorious undefined behavior, yet capable of finding numerous errors in production compilers.”
MethodFailure Avoidance in Configurable Systems through Feature Locality
Garvin, B. J., Cohen, M. B., & Dwyer, M. B. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-36249-1_10 · Cited by 20
FANDANGO: Evolving Language-Based Testing
Amaya, J. A. Z., Smytzek, M., & Zeller, A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728915 · Cited by 33
“For such complex inputs, domain-specific generators such as CSMITH [61] (for C/C++ programs), TLS -Attacker [53] (for TLS interactions), MORPHEUS [59] (for PKCS signature schemes) encompass several person-years of development.”
Component-based Distributed Software Reconfiguration:A Verification-oriented Survey
Coullon, H., Henrio, L., Loulergue, F., & Robillard, S. (2023) · ACM Computing Surveys · doi:10.1145/3595376 · Cited by 19
Jiang, Y., Liu, J., Ba, J., Yap, R. H. C., Liang, Z., & Rigger, M. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623307 · Cited by 32
Testing noninterference, quickly
Hriţcu, C., LAMPROPOULOS, L., SPECTOR-ZABUSKY, A., AMORIM, A. A. D., DÉNÈS, M., HUGHES, J., Pierce, B. C., & Vytiniotis, D. (2016) · Journal of Functional Programming · doi:10.1017/s0956796816000058 · Cited by 30
Random testing of C compilers based on test program generation by equivalence transformation
Kazuhiro, N., & Ishiura, N. (2016) · IEEE Asia Pacific Conference on Circuits and Systems (APCCAS) · doi:10.1109/apccas.2016.7804063 · Cited by 19
“Csmith [7] is one of the most successful random test system for C compilers, which reported 79 bugs in GCCs and 202 bugs in LLVMs.”
BackgroundCyFuzz: A Differential Testing Framework for Cyber-Physical Systems Development Environments
Chowdhury, S. A., Johnson, T. T., & Csallner, C. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-51738-4_4 · Cited by 22
“As an example, various fuzzing schemes have collectively found over 1,000 bugs in widely used compilation tools such as GCC [10, 11, 14].”
BackgroundMethodInfluentialDiffStream: differential output testing for stream processing programs
Καλλάς, Κ., Niksic, F., Stanford, C., & Alur, R. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428221 · Cited by 25
“2007; McKeeman 1998] is a well-established, lightweight, and scalable way to detect bugs in complex programs (for instance, in C compilers [Yang et al. 2011]), by simply comparing two programs that are supposed to be equivalent.”
BackgroundLearning to make compiler optimizations more effective
Mammadli, R., Selakovic, M., Wolf, F., & Pradel, M. (2021) · MAPS@PLDI · doi:10.1145/3460945.3464952 · Cited by 24
“Since many compiler bugs are triggered by optimizations [14], several techniques search for optimization-related compiler bugs via differential testing [35, 56].”
Saying ‘Hi!’ is not enough: Mining inputs for effective test generation
Toffola, L. D., Staicu, C.-A., & Pradel, M. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115617 · Cited by 23
“Finally, some test generators rely on a grammar that describes expected string values [20], [54].”
BackgroundExposing bugs in JavaScript engines through test transplantation and differential testing
Lima, I., da Silva, J. B. B., Miranda, B., Pinto, G., & d’Amorim, M. (2021) · Software Quality Journal · doi:10.1007/s11219-020-09537-8 · Cited by 27
Towards verification of hybrid systems in a foundational proof assistant
Ricketts, D., Malecha, G., Alvarez, M. M., Gowda, V., & Lerner, S. (2015) · ACM/IEEE International Conference on Formal Methods and Models for Co-Design (MEMOCODE) · doi:10.1109/memcod.2015.7340492 · Cited by 18
“Experimental studies have shown that foundationally verified software is far more reliable than software written in a traditional way [3].”
BackgroundEnd-to-end verification of stack-space bounds for C programs
Carbonneaux, Q., Hoffmann, J., Ramananandro, T., & Shao, Z. (2014) · ACM SIGPLAN Notices · doi:10.1145/2666356.2594301 · Cited by 18
Discovering discrepancies in numerical libraries
Vanover, J., Deng, X., & Rubio-González, C. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397380 · Cited by 18
“, [13, 14, 18, 31, 34, 40, 46]) such testing has not been systematically applied in the context of numerical software.”
MethodDSmith: Compiler Fuzzing through Generative Deep Learning Model with Attention
Xu, H., Wang, Y., Fan, S., Xie, P., & Liu, A. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206911 · Cited by 21
Even-Mendoza, K., Cadar, C., & Donaldson, A. F. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3418933 · Cited by 20
“Csmith [2, 20] generates deterministic C programs that are free from undefined and unspecified behaviour.”
BackgroundMethodThe Security Risk of Lacking Compiler Protection in WebAssembly
Stiévenart, Q., Roover, C. D., & Ghafari, M. (2021) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs54544.2021.00024 · Cited by 18
Hashimoto, A., & Ishiura, N. (2016) · IPSJ Transactions on System and LSI Design Methodology · doi:10.2197/ipsjtsldm.9.21 · Cited by 19
“Csmith [7] achieved comprehensive testing of C compilers, which detected 79 bugs in GCCs and 202 bugs in LLVMs over three years and made great contribution to improve the reliability of those open source compilers.”
BackgroundMethodAn Analysis of x86-64 Inline Assembly in C Programs
Rigger, M., Marr, S., Kell, S., Leopoldseder, D., & Mössenböck, H. (2018) · ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE) · doi:10.1145/3186411.3186418 · Cited by 27
“In recent years, random program generators for testing compilers [50, 60, 71] and other tools [14, 33] have been successful in identifying bugs.”
BackgroundBishop, S., Fairbairn, M., Mehnert, H., Norrish, M., Ridge, T., Sewell, P., Smith, M. B., & Wansbrough, K. (2018) · Journal of the ACM · doi:10.1145/3243650 · Cited by 18
“We have handwritten a library of small test cases for the memory object semantics, and also use established test suites and Csmith-generated tests [110].”
MethodFormal verification of SSA-based optimizations for LLVM
Zhao, J., Nagarakatte, S., Martin, M. M. K., & Zdancewic, S. (2013) · ACM SIGPLAN Notices · doi:10.1145/2499370.2462164 · Cited by 18
Type-safe runtime code generation: accelerate to LLVM
McDonell, T. L., Chakravarty, M. M. T., Grover, V., & Newton, R. (2015) · International Symposium on Haskell · doi:10.1145/2804302.2804313 · Cited by 20
“Thus, bugs can lurk undetected [11, 47].”
How Verified is My Code? Falsification-Driven Verification (T)
Groce, A., Ahmed, I., Jensen, C., & McKenney, P. E. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.40 · Cited by 21
“The problem of checking verification results has concerned the model checking community for some time, and resulted in efforts to define coverage metrics for model checking [12], [13].”
FOT: a versatile, configurable, extensible fuzzing framework
Chen, H., Li, Y., Chen, B., Xue, Y., & Liu, Y. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3264593 · Cited by 17
“4) Skyfire [14], Radamsa [8], Csmith [15] can be used in the preprocessor to generate seeds for the external seeds and a structureaware mutator assigned by mutation manager.”
MethodFormally verified superblock scheduling
Six, C., Gourdin, L., Boulmé, S., Monniaux, D. P., Fasse, J., & Nardino, N. (2022) · ACM SIGPLAN International Conference on Certified Programs and Proofs (CPP) · doi:10.1145/3497775.3503679 · Cited by 17
Detecting Simulink compiler bugs via controllable zombie blocks mutation
Guo, S., Jiang, H., Xu, Z., Li, X., Ren, Z., Zhou, Z., & Chen, R. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549159 · Cited by 17
Program Reconditioning: Avoiding Undefined Behaviour When Finding and Reducing Compiler Bugs
Lecoeur, B., Mohsin, H., & Donaldson, A. F. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591294 · Cited by 19
“For example, the Csmith [Yang et al. 2011] and YARPGen [Livinskii et al. 2020] tools have used differential testing to find hundreds of miscompilations in GCC and LLVM.”
MethodResultInfluentialSilent Compiler Bug De-duplication via Three-Dimensional Analysis
Yang, C., Chen, J., Fan, X., Jiang, J., & Sun, J. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598087 · Cited by 17
“In the literature, many compiler testing techniques have been proposed [11, 14, 17, 32, 54, 58], which in general run a large number of test programs to detect as many bugs as possible [14, 16].”
BackgroundPyRTFuzz: Detecting Bugs in Python Runtimes via Two-Level Collaborative Fuzzing
Li, W., Yang, H., Luo, X., Cheng, L., & Cai, H. (2023) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3576915.3623166 · Cited by 30
Experience report: a do-it-yourself high-assurance compiler
Pike, L., Wegmann, N., Niller, S., & Goodloe, A. E. (2012) · ACM SIGPLAN International Conference on Functional Programming (ICFP) · doi:10.1145/2364527.2364553 · Cited by 20
“Work at the University of Utah has dispelled this myth [24], having uncovered hundreds of bugs in C compilers like gcc, clang, and even the (unverified) frontend of the CompCert compiler [16].”
BackgroundSargsyan, S., Kurmangaleev, S. F., Mehrabyan, M., Мишечкин, М. В., Ghukasyan, T., & Asryan, S. (2018) · Ivannikov Memorial Workshop (IVMEM) · doi:10.1109/ivmem.2018.00013 · Cited by 17
Automatic and scalable detection of logical errors in functional programming assignments
Song, D., Lee, M., & Oh, H. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360614 · Cited by 26
“As common compilers require a program or a function as their input, we also categorize compiler testing [Pałka et al. 2011; Sun et al. 2016; Yang et al. 2011] as a part of higher-order program testing.”
MethodHansie: Hybrid and consensus regression test prioritization
Mondal, S., & Nasre, R. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110850 · Cited by 18
Growing a Test Corpus with Bonsai Fuzzing
Vikram, V., Padhye, R., & Sen, K. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00072 · Cited by 17
“In particular, we focus on the problem of automatically generating test cases that exercise the typechecker, since generating well-typed programs is known to be a difficult problem [2]– [5].”
BackgroundInfluentialMind the Gap: Studying the Insecurity of Provably Secure Embedded Trusted Execution Architectures
Bognár, M., Bulck, J. V., & Piessens, F. (2022) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46214.2022.9833735 · Cited by 35
“For instance, an extensive effort to uncover compilation bugs [57] in 11 compilers found no bugs in the machine-checked formally verified CompCert [53] middle-end, but found bugs in all other compilers and in the unverified parts of CompCert.”
Verifying Fast and Sparse SSA-Based Optimizations in Coq
Demange, D., Pichardie, D., & Stefanesco, L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-662-46663-6_12 · Cited by 17
“[20] shows that bugs remain frequent in mainstream compilers.”
BackgroundSpectrum-based fault localization for context-free grammars
Raselimo, M., & Fischer, B. (2019) · ACM SIGPLAN International Conference on Software Language Engineering (SLE) · doi:10.1145/3357766.3359538 · Cited by 19
“Such methods have been used to test SQL [50], C [55], and Java [56] processors, and are also applied in some fuzzing tools such as jsfunfuzz [44], the CSS grammar fuzzer [45], or langfuzz [20].”
MethodHow programming languages will co-evolve with software engineering: a bright decade ahead
Murphy-Hill, E. R., & Grossman, D. (2014) · Future of Software Engineering (FOSE) · doi:10.1145/2593882.2593898 · Cited by 17
“For example, random-testing exposed dozens of bugs in common C compilers, but none in CompCert [32].”
BackgroundTesting Dafny (experience paper)
Irfan, A., Porncharoenwase, S., Rakamarić, Z., Rungta, N., & Torlak, E. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534382 · Cited by 24
“Like prior testing efforts [11, 42], we benefited from having good rapport with the developers of the tools we were using and testing.”
BackgroundMethodInfluentialSaha, S., Sarker, L., Shafiuzzaman, M., Shou, C., Li, A., Sankaran, G., & Bultan, T. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598136 · Cited by 16
“There are several fuzzers [15, 24, 32, 40, 43] focus specifically on structured inputs.”
BackgroundGenCoG: A DSL-Based Approach to Generating Computation Graphs for TVM Testing
Wang, Z., Nie, P., Miao, X., Chen, Y., Wan, C., Bu, L., & Zhao, J. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598105 · Cited by 29
“Similar to traditional compilers [41], TVM is also subject to bugs [32].”
ResultSelf-citationYe, G., Hu, T., Tang, Z., Fan, Z., Tan, S. H., Zhang, B., Qian, W., & Wang, Z. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616332 · Cited by 28
“Inspired by these studies, we derived two key intuitions: (1) historical test programs (e.g., test suites or PoCs) can be used to generate bug-exposing tests for quickly covering buggy compiler components, and (2) buggy compiler components can be continuously and intensively tested by mutating the…”
BackgroundMethodResultInfluentialAPI-Driven Program Synthesis for Testing Static Typing Implementations
Sotiropoulos, T., Chaliasos, S., & Su, Z. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3632904 · Cited by 18
“It adopts a generative process similar to that of Csmith [Yang et al. 2011].”
BackgroundMethodResultTowards Understanding the Bugs in Solidity Compiler
Ma, H., Zhang, W., Shen, Q., Tian, Y., Chen, J., & Cheung, S.-C. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680362 · Cited by 19
“Like other compilers[34, 58, 70], the crash symptom is the major issue among all collected bugs in Solidity compiler, occupying 373 out of 533 bugs.”
BackgroundSoftware Security Analysis in 2030 and Beyond: A Research Roadmap
Böhme, M., Bodden, E., Bultan, T., Cadar, C., Liu, Y., & Scanniello, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708533 · Cited by 21
“Outside of the technical aspects, as the field matures, we should work on pedagogical aspects , such as establishing and teaching the foundations of software security, as well as legal aspects , such as questions of liability or accountability in the digital world.”
Weakest Precondition Synthesis for Compiler Optimizations
Lopes, N. P., & Monteiro, J. C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-642-54013-4_12 · Cited by 16
“In fact, a recent study found bugs in all the most used compilers [39].”
BackgroundHussain, I., Csallner, C., Grechanik, M., Xie, Q., Park, S., Taneja, K., & Hossain, B. M. M. (2014) · Software Practice and Experience · doi:10.1002/spe.2290 · Cited by 18
“Several earlier pieces of work have used probabilistic grammar-based random program generation before [16, 17, 35, 36, 50, 51, 52, 53].”
BackgroundMethodA Decision Procedure for Univariate Real Polynomials in Isabelle/HOL
Eberl, M. (2015) · ACM SIGPLAN International Conference on Certified Programs and Proofs (CPP) · doi:10.1145/2676724.2693166 · Cited by 16
Causal Distance-Metric-Based Assistance for Debugging after Compiler Fuzzing
Holmes, J., & Groce, A. (2018) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2018.00027 · Cited by 17
“Of the three data sets examined in that paper, this paper considers two: faults and tests cases for SpiderMonkey 1.6, Mozilla’s JavaScript engine, with tests generated by jsfunfuzz [70], and wrong code faults and test cases for GCC 4.3.0, with tests generated by Csmith [79].”
BackgroundMethodInfluentialAutomatic Self-Validation for Code Coverage Profilers
Yang, Y., Jiang, Y., Zuo, Z., Wang, Y., Sun, H., Lu, H., Zhou, Y., & Xu, B. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00018 · Cited by 17
“0) and 5,000 random programs generated by csmith [19].”
BackgroundMaking no-fuss compiler fuzzing effective
Groce, A., van Tonder, R., Kalburgi, G. T., & Goues, C. L. (2022) · ACM SIGPLAN International Conference on Compiler Construction (CC) · doi:10.1145/3497776.3517765 · Cited by 22
“Csmith [23] devotes a great deal of effort to avoiding generating such code.”
BackgroundMethodFind more bugs with QuickCheck!
Hughes, J., Norell, U., Smallbone, N., & Arts, T. (2016) · International Workshop on Automation of Software Test (AST) · doi:10.1145/2896921.2896928 · Cited by 18
“Keywords Random testing, avoiding bugs, bug slippage”
Compilation Consistency Modulo Debug Information
Wang, T. L., Tian, Y., Dong, Y., Xu, Z., & Sun, C. P. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575740 · Cited by 17
Formally Verifying Optimizations with Block Simulations
Gourdin, L., Bonneau, B., Boulmé, S., Monniaux, D. P., & Bérard, A. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3622799 · Cited by 15
“We checked that Chamois successfully compiled a test suite of thousands of C programs, some from non-regression tests, some from real applications (including large programs such as GCC and OCaml themselves), and others generated from fuzzers (CSmith [Yang et al. 2011], YarpGen [Livinskii et al.”
MethodEtna: An Evaluation Platform for Property-Based Testing (Experience Report)
Shi, J., Keleş, A., Goldstein, H., Pierce, B. C., & Lampropoulos, L. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3607860 · Cited by 21
Towards Better Semantics Exploration for Browser Fuzzing
Zhou, C., Zhang, Q., Guo, L., Wang, M., Jiang, Y., Liao, Q., Wu, Z., Li, S., & Gu, B. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3622819 · Cited by 22
“In addition to generic grammar-based fuzzers, there are also many custom fuzzers, enhanced with domain knowledge, for fuzzing specific domain such as C/C++ compilers [Livinskii et al. 2020; Yang et al. 2011], databases [Rigger and Su 2020; Wang et al.”
BackgroundMethodWapplique: Testing WebAssembly Runtime via Execution Context-Aware Bytecode Mutation
Zhao, W., Zeng, R., & Zhou, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680340 · Cited by 15
“We also use Csmith [47] to generate C programs and compile them by Emscripten with different optimization levels.”
Application of Domain-aware Binary Fuzzing to Aid Android Virtual Machine Testing
Kyle, S. C., Leather, H., Franke, B., Butcher, D., & Monteith, S. (2015) · ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE) · doi:10.1145/2731186.2731198 · Cited by 16
“Additionally, prior generative approaches have found it difficult to produce a system that fully utilises all features of the language that they are generating code for[16].”
BackgroundInfluentialIsolation without taxation: near-zero-cost transitions for WebAssembly and SFI
Kolosick, M., Narayan, S., Johnson, E., Watt, C., LeMay, M., Garg, D., Jhala, R., & Stefan, D. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3498688 · Cited by 15
Fuzzle: Making a Puzzle for Fuzzers
Lee, H., Kim, S., & Kil, S. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556908 · Cited by 26
“For example, we can randomly synthesize our branch conditions similar to the way Csmith [61] generates random C programs.”
MethodReduKtor: How We Stopped Worrying About Bugs in Kotlin Compiler
Stepanov, D., Akhin, M., & Беляев, М. А. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00038 · Cited by 15
“At the moment, the most developed tool for delta debugging is C-Reduce [7], initially created for reduction of C/C ++ compiler tests, generated by Csmith [15].”
Method
Metadata
- Authors
- Yang, X., Chen, Y., Eide, E., & Regehr, J.
- Affiliations
- University of Utah (US)
- Venue
- Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation
- Publisher
- ACM
- Published
- 2011-06-04
- Pages
- 283-294
- ISSN
- 1531-7102 (ISSN-L)
- OpenAlex
- W4238083723
- Cited by
- 825
- References
- 23
- Citations by year
- 2026: 71 · 2025: 115 · 2024: 93 · 2023: 88 · 2022: 59 · 2021: 50 · 2020: 58 · 2019: 45 · 2018: 50 · 2017: 40 · 2016: 42 · 2015: 45 · 2014: 24 · 2013: 23 · 2012: 19
- FWCI
- 15.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Keywords
- compiler bugs, C compiler, compiler testing, undefined behavior, bug reports
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.