The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
On the reliability of coverage-based fuzzer benchmarking
Open accessAbstract
Given a program where none of our fuzzers finds any bugs, how do we know which fuzzer is better? In practice, we often look to code coverage as a proxy measure of fuzzer effectiveness and consider the fuzzer which achieves more coverage as the better one.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Complete line coverage alone does not guarantee effective testing [4].”
From How We Built Cedar: A Verification-Guided Approach (2024) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
Correlation Coefficients: Appropriate Use and Interpretation
Schober, P., Boer, C., & Schwarte, L. A. (2018) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000002864 · Cited by 10,986
Search‐based software test data generation: a survey
McMinn, P. (2004) · Software Testing Verification and Reliability · doi:10.1002/stvr.294 · Cited by 1,457
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
13th USENIX Workshop on Offensive Technologies, WOOT 2019
Gantman, A., & Maurice, C. (2019) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 934
Klees, G., Ruef, A., Cooper, B., Wei, S., & Hicks, M. (2018) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3243734.3243804 · Cited by 659
Arcuri, A., & Briand, L. (2012) · Software Testing Verification and Reliability · doi:10.1002/stvr.1486 · Cited by 628
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635929 · Cited by 602
FairFuzz: a targeted mutation strategy for increasing greybox fuzz testing coverage
Lemieux, C., & Sen, K. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238176 · Cited by 477
Proceedings of the 24th USENIX Conference on Security Symposium
Jung, J. (2015) · Cited by 465
Flagged first, then by citations
Research works printed but not linked: 8
- Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. 2020. AFL++ : Combining Incremental Steps of Fuzzing Research. In 14th USENIX Workshop on Offensive Technologies (WOOT 20). USENIX Association. https://www.usenix.org/conference/woot20/presentation/fioraldi
- Shuitao Gan, Chao Zhang, Peng Chen, Bodong Zhao, Xiaojun Qin, Dong Wu, and Zuoning Chen. 2020. GREYONE: Data Flow Sensitive Fuzzing. In 29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 2577--2594.
- Sijia Geng, Yuekang Li, Yunlan Du, Jun Xu, Yang Liu, and Bing Mao. 2020. An Empirical Study on Benchmarks of Artificial Software Vulnerabilities. (March 2020). https://arxiv.org/abs/2003.09561v1 · arXiv
- Rahul Gopinath, Philipp Görz, and Alex Groce. 2022. Mutation Analysis: Answering the Fuzzing Challenge. arXiv:2201.11303 [cs.SE] · arXiv
- Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276--282.
- Sebastian Österlund, Kaveh Razavi, Herbert Bos, and Cristiano Giuffrida. 2020. ParmeSan: Sanitizer-guided Greybox Fuzzing. In 29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 2289--2306.
- Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitry Vyukov. 2012. AddressSanitizer: A Fast Address Sanity Checker. In Proceedings of the 2012 USENIX Conference on Annual Technical Conference (Boston, MA) (USENIX ATC'12). USENIX Association, USA, 28.
- Xiaogang Zhu, Shigang Liu, Xian Li, Sheng Wen, Jun Zhang, Seyit Ahmet Çamtepe, and Yang Xiang. 2020. DeFuzz: Deep Learning Guided Directed Fuzzing. CoRR abs/2010.12149 (2020). · arXiv
Cited by
Sorted by citations
Items 101–104 of 104
MioHint: LLM-Assisted Request Mutation for Whitebox REST API Testing
Li, J., Shen, J., Su, Y., & Lyu, M. R.-T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773228
Research on Adaptive Fuzzing Test Case Generation and Optimization Based on Reinforcement Learning
Wu, N. (2026) · International Conference on Advanced Algorithms and Neural Networks (AANN) · doi:10.1109/aann71145.2026.11709378
Fuzzing FPGA Synthesis and Simulation Tools via LLM-Generated Syntax-Valid HDL Codes
Jiang, H., Zhao, W., Guo, S., Xu, Z., Li, X., & Huang, R. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832160
Predicate-Guided Synthesis of Input Generators via Gentoo
Vikram, V., & Padhye, R. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837441 · Cited by 1
Metadata
- Authors
- Böhme, M., Szekeres, L., & Metzman, J.
- Affiliations
- Monash University (AU)
- Google (United States) (US)
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 1621-1633
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4284706927
- Cited by
- 104
- References
- 47
- Citations by year
- 2026: 18 · 2025: 25 · 2024: 32 · 2023: 25 · 2022: 4
- FWCI
- 16.22
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, other-oa)
- Funders
- École Polytechnique Fédérale de Lausanne
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- code coverage, fuzzing, software testing, bug detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
