The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
On the reliability of coverage-based fuzzer benchmarking
Open accessAbstract
Given a program where none of our fuzzers finds any bugs, how do we know which fuzzer is better? In practice, we often look to code coverage as a proxy measure of fuzzer effectiveness and consider the fuzzer which achieves more coverage as the better one.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Complete line coverage alone does not guarantee effective testing [4].”
From How We Built Cedar: A Verification-Guided Approach (2024) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
Correlation Coefficients: Appropriate Use and Interpretation
Schober, P., Boer, C., & Schwarte, L. A. (2018) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000002864 · Cited by 10,986
Search‐based software test data generation: a survey
McMinn, P. (2004) · Software Testing Verification and Reliability · doi:10.1002/stvr.294 · Cited by 1,457
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
13th USENIX Workshop on Offensive Technologies, WOOT 2019
Gantman, A., & Maurice, C. (2019) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 934
Klees, G., Ruef, A., Cooper, B., Wei, S., & Hicks, M. (2018) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3243734.3243804 · Cited by 659
Arcuri, A., & Briand, L. (2012) · Software Testing Verification and Reliability · doi:10.1002/stvr.1486 · Cited by 628
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635929 · Cited by 602
FairFuzz: a targeted mutation strategy for increasing greybox fuzz testing coverage
Lemieux, C., & Sen, K. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238176 · Cited by 477
Proceedings of the 24th USENIX Conference on Security Symposium
Jung, J. (2015) · Cited by 465
Predicting vulnerable software components
Neuhaus, S., Zimmermann, T., Holler, C., & Zeller, A. (2007) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/1315245.1315311 · Cited by 429
Coverage is not strongly correlated with test suite effectiveness
Inozemtseva, L., & Holmes, R. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568271 · Cited by 413
Interrater Reliability and Agreement
Tinsley, H. E. A., & Weiss, D. J. (2000) · Elsevier eBooks · doi:10.1016/b978-012691360-6/50005-7 · Cited by 411
CollAFL: Path Sensitive Fuzzing
Gan, S., Zhang, C., Qin, X., Tu, X., Li, K., Pei, Z., & Chen, Z. (2018) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2018.00040 · Cited by 397
Coverage-Based Greybox Fuzzing as Markov Chain
Böhme, M., Pham, V.-T., & Roychoudhury, A. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2785841 · Cited by 342
Self-citationSearch-Based Software Testing: Past, Present and Future
McMinn, P. (2011) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2011.100 · Cited by 322
LAVA: Large-Scale Automated Vulnerability Addition
Dolan-Gavitt, B., Hulin, P., Kirda, E., Leek, T., Mambretti, A., Robertson, W., Ulrich, F., & Whelan, R. (2016) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2016.15 · Cited by 318
Pham, V.-T., Boehme, M., Santosa, A. E., Căciulescu, A. R., & Roychoudhury, A. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2941681 · Cited by 215
FuzzBench: an open fuzzer benchmarking platform and service
Metzman, J., Szekeres, L., Simon, L., Sprabery, R., & Arya, A. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473932 · Cited by 201
Self-citationHazimeh, A., Herrera, A., & Payer, M. (2020) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3428334 · Cited by 177
Code coverage for suite evaluation by developers
Gopinath, R., Jensen, C., & Groce, A. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2568225.2568278 · Cited by 175
Fuzzing: Challenges and Reflections
Böhme, M., Cadar, C., & Roychoudhury, A. (2020) · IEEE Software · doi:10.1109/ms.2020.3016773 · Cited by 161
Self-citationChekam, T. T., Papadakis, M., Traon, Y. L., & Harman, M. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.61 · Cited by 154
Where is the bug and how is it fixed? an experiment with practitioners
Böhme, M., Soremekun, E., Chattopadhyay, S., Ugherughe, E., & Zeller, A. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106255 · Cited by 136
Self-citationAre mutation scores correlated with real fault detection?
Papadakis, M., Shin, D., Yoo, S., & Bae, D. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180183 · Cited by 130
Boosting fuzzer efficiency: an information theoretic perspective
Böhme, M., Manès, V. J. M., & Kil, S. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409748 · Cited by 115
Self-citationAssertions are strongly correlated with test suite effectiveness
Zhang, Y., & Mesbah, A. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2786858 · Cited by 110
Seed selection for successful fuzzing
Herrera, A., Gunadi, H., Magrath, S., Norrish, M., Payer, M., & Hosking, A. L. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464795 · Cited by 108
BugsJS: a Benchmark of JavaScript Bugs
Gyimesi, P., Vancsics, B., Stocco, A., Mazinanian, D., Beszédes, Á., Ferenć, R., & Mesbah, A. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00019 · Cited by 99
CoREBench: studying complexity of regression errors
Böhme, M., & Roychoudhury, A. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628058 · Cited by 99
Self-citationSound empirical evidence in software testing
Fraser, G., & Arcuri, A. (2012) · doi:10.5555/2337223.2337245 · Cited by 96
Ivanković, M., Petrović, G., Just, R., & Fraser, G. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3340459 · Cited by 92
BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixes
Tomassi, D. A., Dmeiri, N., Wang, Y., Bhowmick, A., Liu, Y.-C., Dévanbu, P., Vasilescu, B., & Rubio-González, C. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00048 · Cited by 91
Grey-Box Concolic Testing on Binary Code
Choi, J., Jang, J., Han, C., & Kil, S. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00082 · Cited by 90
Code coverage and test suite effectiveness: Empirical study with real bugs in large systems
Kochhar, P. S., Thung, F., & Lo, D. (2015) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2015.7081877 · Cited by 88
Semantic bug seeding: a learning-based approach for creating realistic bugs
Patra, J., & Pradel, M. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468623 · Cited by 66
Guidelines for Coverage-Based Comparisons of Non-Adequate Test Suites
Gligoric, M., Groce, A., Zhang, C., Sharma, R., Alipour, M. A., & Marinov, D. (2015) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2660767 · Cited by 62
Bug synthesis: challenging bug-finding tools with deep faults
Roy, S., Pandey, A., Dolan-Gavitt, B., & Hu, Y. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236084 · Cited by 59
Revisiting the relationship between fault detection, test adequacy criteria, and test set size
Chen, Y. T., Gopinath, R., Tadakamalla, A., Ernst, M. D., Holmes, R., Fraser, G., Ammann, P., & Just, R. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416667 · Cited by 55
Controlling factors in evaluating path-sensitive error detection techniques
Dwyer, M. B., Person, S., & Elbaum, S. G. (2006) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1181775.1181787 · Cited by 53
Pewny, J., & Holz, T. (2016) · doi:10.1145/2991079.2991103 · Cited by 44
Community expectations for research artifacts and evaluation processes
Hermann, B., Winter, S., & Siegmund, J. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409767 · Cited by 40
Is Branch Coverage a Good Measure of Testing Effectiveness?
Wei, Y., Meyer, B., & Manuel Oriol, B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-25231-0_5 · Cited by 39
Generating Effective Test Suites by Combining Coverage Criteria
Gay, G. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66299-2_5 · Cited by 36
Bundt, J., Fasano, A., Dolan-Gavitt, B., Robertson, W., & Leek, T. (2021) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3433210.3453096 · Cited by 20
Lipp, S., Elsner, D., Hutzelmann, T., Bănescu, S., Pretschner, A., & Böhme, M. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510454.3516847 · Cited by 9
Self-citationA Practical, Principled Measure of Fuzzer Appeal: A Preliminary Study
Gavrilov, M., Dewey, K. T., Groce, A., Zamanzadeh, D. J., & Hardekopf, B. (2020) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs51102.2020.00071 · Cited by 4
Flagged first, then by citations
Research works printed but not linked: 8
- Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. 2020. AFL++ : Combining Incremental Steps of Fuzzing Research. In 14th USENIX Workshop on Offensive Technologies (WOOT 20). USENIX Association. https://www.usenix.org/conference/woot20/presentation/fioraldi
- Shuitao Gan, Chao Zhang, Peng Chen, Bodong Zhao, Xiaojun Qin, Dong Wu, and Zuoning Chen. 2020. GREYONE: Data Flow Sensitive Fuzzing. In 29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 2577--2594.
- Sijia Geng, Yuekang Li, Yunlan Du, Jun Xu, Yang Liu, and Bing Mao. 2020. An Empirical Study on Benchmarks of Artificial Software Vulnerabilities. (March 2020). https://arxiv.org/abs/2003.09561v1 · arXiv
- Rahul Gopinath, Philipp Görz, and Alex Groce. 2022. Mutation Analysis: Answering the Fuzzing Challenge. arXiv:2201.11303 [cs.SE] · arXiv
- Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276--282.
- Sebastian Österlund, Kaveh Razavi, Herbert Bos, and Cristiano Giuffrida. 2020. ParmeSan: Sanitizer-guided Greybox Fuzzing. In 29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 2289--2306.
- Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitry Vyukov. 2012. AddressSanitizer: A Fast Address Sanity Checker. In Proceedings of the 2012 USENIX Conference on Annual Technical Conference (Boston, MA) (USENIX ATC'12). USENIX Association, USA, 28.
- Xiaogang Zhu, Shigang Liu, Xian Li, Sheng Wen, Jun Zhang, Seyit Ahmet Çamtepe, and Yang Xiang. 2020. DeFuzz: Deep Learning Guided Directed Fuzzing. CoRR abs/2010.12149 (2020). · arXiv
Cited by
Sorted by citations
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306
Large Language Model guided Protocol Fuzzing
Meng, R., Mirchev, M., Böhme, M., & Roychoudhury, A. (2024) · Proceedings 2024 Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24556 · Cited by 246
Self-citationFuzz4ALL: Universal Fuzzing with Large Language Models
Xia, C. S., Paltenghi, M., Le Tian, J., Pradel, M., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639121 · Cited by 229
Effective test generation using pre-trained Large Language Models and mutation testing
Dakhel, A. M., Nikanjam, A., Majdinasab, V., Khomh, F., & Desmarais, M. C. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107468 · Cited by 157
Su, J., Dai, H., Zhao, L., Zheng, Z., & Luo, X. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3560429 · Cited by 78
Metadata
- Authors
- Böhme, M., Szekeres, L., & Metzman, J.
- Affiliations
- Monash University (AU)
- Google (United States) (US)
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 1621-1633
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4284706927
- Cited by
- 104
- References
- 47
- Citations by year
- 2026: 18 · 2025: 25 · 2024: 32 · 2023: 25 · 2022: 4
- FWCI
- 16.22
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, other-oa)
- Funders
- École Polytechnique Fédérale de Lausanne
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- code coverage, fuzzing, software testing, bug detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
