On the reliability of coverage-based fuzzer benchmarking
Open accessAbstract
Given a program where none of our fuzzers finds any bugs, how do we know which fuzzer is better? In practice, we often look to code coverage as a proxy measure of fuzzer effectiveness and consider the fuzzer which achieves more coverage as the better one.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Complete line coverage alone does not guarantee effective testing.
“Complete line coverage alone does not guarantee effective testing [4].”
From How We Built Cedar: A Verification-Guided Approach (2024) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965 · see citing works
Correlation Coefficients: Appropriate Use and Interpretation
Schober, P., Boer, C., & Schwarte, L. A. (2018) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000002864 · Cited by 10,986 · see citing works
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447 · see citing works
Search‐based software test data generation: a survey
McMinn, P. (2004) · Software Testing Verification and Reliability · doi:10.1002/stvr.294 · Cited by 1,376 · see citing works
13th USENIX Workshop on Offensive Technologies, WOOT 2019
Gantman, A., & Maurice, C. (2019) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 934 · see citing works
Klees, G., Ruef, A., Cooper, B., Wei, S., & Hicks, M. (2018) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3243734.3243804 · Cited by 659 · see citing works
Arcuri, A., & Briand, L. (2012) · Software Testing Verification and Reliability · doi:10.1002/stvr.1486 · Cited by 628 · see citing works
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635929 · Cited by 602 · see citing works
FairFuzz: a targeted mutation strategy for increasing greybox fuzz testing coverage
Lemieux, C., & Sen, K. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238176 · Cited by 477 · see citing works
Proceedings of the 24th USENIX Conference on Security Symposium
Jung, J. (2015) · Cited by 465 · see citing works
Predicting vulnerable software components
Neuhaus, S., Zimmermann, T., Holler, C., & Zeller, A. (2007) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/1315245.1315311 · Cited by 429 · see citing works
Coverage is not strongly correlated with test suite effectiveness
Inozemtseva, L., & Holmes, R. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568271 · Cited by 413 · see citing works
Interrater Reliability and Agreement
Tinsley, H. E. A., & Weiss, D. J. (2000) · Elsevier eBooks · doi:10.1016/b978-012691360-6/50005-7 · Cited by 401 · see citing works
CollAFL: Path Sensitive Fuzzing
Gan, S., Zhang, C., Qin, X., Tu, X., Li, K., Pei, Z., & Chen, Z. (2018) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2018.00040 · Cited by 397 · see citing works
Coverage-Based Greybox Fuzzing as Markov Chain
Böhme, M., Pham, V.-T., & Roychoudhury, A. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2785841 · Cited by 342 · see citing works
Self-citationSearch-Based Software Testing: Past, Present and Future
McMinn, P. (2011) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2011.100 · Cited by 322 · see citing works
LAVA: Large-Scale Automated Vulnerability Addition
Dolan-Gavitt, B., Hulin, P., Kirda, E., Leek, T., Mambretti, A., Robertson, W., Ulrich, F., & Whelan, R. (2016) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2016.15 · Cited by 318 · see citing works
Pham, V.-T., Boehme, M., Santosa, A. E., Căciulescu, A. R., & Roychoudhury, A. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2941681 · Cited by 215 · see citing works
FuzzBench: an open fuzzer benchmarking platform and service
Metzman, J., Szekeres, L., Simon, L., Sprabery, R., & Arya, A. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473932 · Cited by 201 · see citing works
Self-citationHazimeh, A., Herrera, A., & Payer, M. (2020) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3428334 · Cited by 177 · see citing works
Code coverage for suite evaluation by developers
Gopinath, R., Jensen, C., & Groce, A. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2568225.2568278 · Cited by 175 · see citing works
Fuzzing: Challenges and Reflections
Böhme, M., Cadar, C., & Roychoudhury, A. (2020) · IEEE Software · doi:10.1109/ms.2020.3016773 · Cited by 161 · see citing works
Self-citationChekam, T. T., Papadakis, M., Traon, Y. L., & Harman, M. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.61 · Cited by 154 · see citing works
Where is the bug and how is it fixed? an experiment with practitioners
Böhme, M., Soremekun, E., Chattopadhyay, S., Ugherughe, E., & Zeller, A. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106255 · Cited by 122 · see citing works
Self-citationBoosting fuzzer efficiency: an information theoretic perspective
Böhme, M., Manès, V. J. M., & Kil, S. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409748 · Cited by 115 · see citing works
Self-citationAssertions are strongly correlated with test suite effectiveness
Zhang, Y., & Mesbah, A. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2786858 · Cited by 110 · see citing works
Seed selection for successful fuzzing
Herrera, A., Gunadi, H., Magrath, S., Norrish, M., Payer, M., & Hosking, A. L. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464795 · Cited by 108 · see citing works
Are mutation scores correlated with real fault detection?
Papadakis, M., Shin, D., Yoo, S., & Bae, D. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180183 · Cited by 106 · see citing works
CoREBench: studying complexity of regression errors
Böhme, M., & Roychoudhury, A. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628058 · Cited by 99 · see citing works
Self-citationSound empirical evidence in software testing
Fraser, G., & Arcuri, A. (2012) · doi:10.5555/2337223.2337245 · Cited by 96 · see citing works
BugsJS: a Benchmark of JavaScript Bugs
Gyimesi, P., Vancsics, B., Stocco, A., Mazinanian, D., Beszédes, Á., Ferenć, R., & Mesbah, A. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00019 · Cited by 95 · see citing works
Ivanković, M., Petrović, G., Just, R., & Fraser, G. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3340459 · Cited by 92 · see citing works
BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixes
Tomassi, D. A., Dmeiri, N., Wang, Y., Bhowmick, A., Liu, Y.-C., Dévanbu, P., Vasilescu, B., & Rubio-González, C. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00048 · Cited by 91 · see citing works
Grey-Box Concolic Testing on Binary Code
Choi, J., Jang, J., Han, C., & Kil, S. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00082 · Cited by 90 · see citing works
Code coverage and test suite effectiveness: Empirical study with real bugs in large systems
Kochhar, P. S., Thung, F., & Lo, D. (2015) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2015.7081877 · Cited by 88 · see citing works
Semantic bug seeding: a learning-based approach for creating realistic bugs
Patra, J., & Pradel, M. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468623 · Cited by 66 · see citing works
Guidelines for Coverage-Based Comparisons of Non-Adequate Test Suites
Gligoric, M., Groce, A., Zhang, C., Sharma, R., Alipour, M. A., & Marinov, D. (2015) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2660767 · Cited by 62 · see citing works
Bug synthesis: challenging bug-finding tools with deep faults
Roy, S., Pandey, A., Dolan-Gavitt, B., & Hu, Y. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236084 · Cited by 59 · see citing works
Revisiting the relationship between fault detection, test adequacy criteria, and test set size
Chen, Y. T., Gopinath, R., Tadakamalla, A., Ernst, M. D., Holmes, R., Fraser, G., Ammann, P., & Just, R. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416667 · Cited by 55 · see citing works
Controlling factors in evaluating path-sensitive error detection techniques
Dwyer, M. B., Person, S., & Elbaum, S. G. (2006) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1181775.1181787 · Cited by 53 · see citing works
Pewny, J., & Holz, T. (2016) · doi:10.1145/2991079.2991103 · Cited by 44 · see citing works
Community expectations for research artifacts and evaluation processes
Hermann, B., Winter, S., & Siegmund, J. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409767 · Cited by 40 · see citing works
Is Branch Coverage a Good Measure of Testing Effectiveness?
Wei, Y., Meyer, B., & Manuel Oriol, B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-25231-0_5 · Cited by 39 · see citing works
Generating Effective Test Suites by Combining Coverage Criteria
Gay, G. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66299-2_5 · Cited by 36 · see citing works
Bundt, J., Fasano, A., Dolan-Gavitt, B., Robertson, W., & Leek, T. (2021) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3433210.3453096 · Cited by 20 · see citing works
Lipp, S., Elsner, D., Hutzelmann, T., Bănescu, S., Pretschner, A., & Böhme, M. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510454.3516847 · Cited by 9 · see citing works
Self-citationA Practical, Principled Measure of Fuzzer Appeal: A Preliminary Study
Gavrilov, M., Dewey, K. T., Groce, A., Zamanzadeh, D. J., & Hardekopf, B. (2020) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs51102.2020.00071 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. 2020. AFL++ : Combining Incremental Steps of Fuzzing Research. In 14th USENIX Workshop on Offensive Technologies (WOOT 20). USENIX Association. https://www.usenix. org/conference/woot20/presentation/fioraldi
- Shuitao Gan, Chao Zhang, Peng Chen, Bodong Zhao, Xiaojun Qin, Dong Wu, and Zuoning Chen. 2020. GREYONE: Data Flow Sensitive Fuzzing. In 29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 2577-2594.
- Sijia Geng, Yuekang Li, Yunlan Du, Jun Xu, Yang Liu, and Bing Mao. 2020. An Empirical Study on Benchmarks of Artificial Software Vulnerabilities. (March 2020). https://arxiv.org/abs/2003.09561v1 · arXiv
- Rahul Gopinath, Philipp Görz, and Alex Groce. 2022. Mutation Analysis: An- swering the Fuzzing Challenge. arXiv:2201.11303 [cs.SE] · arXiv
- Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276-282.
- Sebastian Österlund, Kaveh Razavi, Herbert Bos, and Cristiano Giuffrida. 2020. ParmeSan: Sanitizer-guided Greybox Fuzzing. In 29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 2289-2306.
- Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitry Vyukov. 2012. AddressSanitizer: A Fast Address Sanity Checker. In Proceed- ings of the 2012 USENIX Conference on Annual Technical Conference (Boston, MA) (USENIX ATC'12). USENIX Association, USA, 28.
- Xiaogang Zhu, Shigang Liu, Xian Li, Sheng Wen, Jun Zhang, Seyit Ahmet Çamtepe, and Yang Xiang. 2020. DeFuzz: Deep Learning Guided Directed Fuzzing. CoRR abs/2010.12149 (2020). · arXiv
Cited by
Sorted by citations
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306 · see citing works
Large Language Model guided Protocol Fuzzing
Meng, R., Mirchev, M., Böhme, M., & Roychoudhury, A. (2024) · Proceedings 2024 Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24556 · Cited by 246 · see citing works
Self-citationFuzz4ALL: Universal Fuzzing with Large Language Models
Xia, C. S., Paltenghi, M., Le Tian, J., Pradel, M., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639121 · Cited by 229 · see citing works
Effective test generation using pre-trained Large Language Models and mutation testing
Dakhel, A. M., Nikanjam, A., Majdinasab, V., Khomh, F., & Desmarais, M. C. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107468 · Cited by 157 · see citing works
Su, J., Dai, H., Zhao, L., Zheng, Z., & Luo, X. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3560429 · Cited by 78 · see citing works
SoK: Prudent Evaluation Practices for Fuzzing
Schloegel, M., Bars, N., Schiller, N., Bernhard, L., Scharnowski, T., Crump, A., Ebrahim, A. A., Bissantz, N., Muench, M., & Holz, T. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00137 · Cited by 76 · see citing works
Vulseye: Detect Smart Contract Vulnerabilities via Stateful Directed Graybox Fuzzing
Liang, R., Chen, J., Wu, C., He, K., Wu, Y., Cao, R., Du, R., Zhao, Z., & Liu, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3537827 · Cited by 72 · see citing works
Test smells 20 years later: detectability, validity, and reliability
Panichella, A., Panichella, S., Fraser, G., Sawant, A. A., & Hellendoorn, V. J. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10207-5 · Cited by 50 · see citing works
Open Problems in Fuzzing RESTful APIs: A Comparison of Tools
Zhang, M., & Arcuri, A. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3597205 · Cited by 48 · see citing works
Sequence-Oriented DBMS Fuzzing
Liang, J., Chen, Y., Wu, Z., Fu, J., Wang, M., Jiang, Y., Huang, X., Chen, T., Wang, J., & Li, J. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00057 · Cited by 29 · see citing works
Guiding Greybox Fuzzing with Mutation Testing
Vikram, V., Laybourn, I., Li, A., Nair, N., OBrien, K., Sanna, R., & Padhye, R. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598107 · Cited by 25 · see citing works
Reachable Coverage: Estimating Saturation in Fuzzing
Liyanage, D., Böhme, M., Tantithamthavorn, C. K., & Lipp, S. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00042 · Cited by 18 · see citing works
Self-citationShapFuzz: Efficient Fuzzing via Shapley-Guided Byte Selection
Zhang, K., Zhu, X., Xiao, X., Xue, M., Zhang, C., & Wen, S. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.23134 · Cited by 17 · see citing works
Icicle: A Re-designed Emulator for Grey-Box Firmware Fuzzing
Chesser, M., Nepal, ., & Ranasinghe, D. C. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598039 · Cited by 16 · see citing works
AFLNet Five Years Later: On Coverage-Guided Protocol Fuzzing
Meng, R., Pham, V.-T., Böhme, M., & Roychoudhury, A. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3535925 · Cited by 14 · see citing works
Self-citationDatAFLow : Toward a Data-Flow-Guided Fuzzer
Herrera, A., Payer, M., & Hosking, A. L. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3587156 · Cited by 14 · see citing works
Rustlantis: Randomized Differential Testing of the Rust Compiler
Wang, Q., & Jung, R. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689780 · Cited by 14 · see citing works
SBFT Tool Competition 2023 - Fuzzing Track
Liu, D., Metzman, J., Böhme, M., Chang, O., & Arya, A. K. (2023) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1109/sbft59156.2023.00016 · Cited by 13 · see citing works
Self-citationGreen Fuzzing: A Saturation-Based Stopping Criterion using Vulnerability Prediction
Lipp, S., Elsner, D., Kacianka, S., Pretschner, A., Böhme, M., & Bănescu, S. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598043 · Cited by 9 · see citing works
Self-citationMeng, R., Duck, G. J., & Roychoudhury, A. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690229 · Cited by 9 · see citing works
A Usability Evaluation of AFL and libFuzzer with CS Students
Plöger, S., Meier, M., & Smith, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581178 · Cited by 9 · see citing works
Planning to Guide LLM for Code Coverage Prediction
Dhulipala, H., Yadavally, A., & Nguyen, T. N. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652292 · Cited by 9 · see citing works
FOX: Coverage-guided Fuzzing as Online Stochastic Control
She, D., Storek, A., Xie, Y., Kweon, S., Srivastava, P., & Jana, S. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3670362 · Cited by 8 · see citing works
No Peer, no Cry: Network Application Fuzzing via Fault Injection
Bars, N., Schloegel, M., Schiller, N., Bernhard, L., & Holz, T. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690274 · Cited by 7 · see citing works
Security Evaluation of Smart Contracts based on Code and Transaction - A Survey
Su, J., Liu, J., Nan, Y., & Li, Y. (2022) · International Conference on Service Science (ICSS) · doi:10.1109/icss55994.2022.00016 · Cited by 7 · see citing works
DiPri : Distance-Based Seed Prioritization for Greybox Fuzzing
Qian, R., Zhang, Q., Fang, C., Yang, D., Li, S., Li, B., & Chen, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3654440 · Cited by 7 · see citing works
Predictive Context-sensitive Fuzzing
Borrello, P., Fioraldi, A., D’Elia, D. C., Balzarotti, D., Querzoni, L., & Giuffrida, C. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24113 · Cited by 6 · see citing works
SoK: Where to Fuzz? Assessing Target Selection Methods in Directed Fuzzing
Weißberg, F., Möller, J., Ganz, T., Imgrund, E., Pirch, L., Seidel, L., Schloegel, M., Eisenhofer, T., & Rieck, K. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3661141 · Cited by 6 · see citing works
SLOPT: Bandit Optimization Framework for Mutation-Based Fuzzing
Koike, Y., Katsura, H., Yakura, H., & Kurogome, Y. (2022) · Annual Computer Security Applications Conference · doi:10.1145/3564625.3564659 · Cited by 6 · see citing works
Ounjai, J., Wüstholz, V., & Christakis, M. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598144 · Cited by 6 · see citing works
Laaber, C., Yue, T., Ali, S., Schwitalla, T., & Nygård, J. F. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3613882 · Cited by 6 · see citing works
Revisiting Neural Program Smoothing for Fuzzing
Nicolae, M.-I., Eisele, M., & Zeller, A. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616308 · Cited by 6 · see citing works
An Empirical Examination of Fuzzer Mutator Performance
Kukucka, J., Pina, L., Ammann, P., & Bell, J. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680387 · Cited by 5 · see citing works
No-Fuzz: Efficient Anti-fuzzing Techniques
Zhou, Z., Wang, C., & Zhao, Q. (2023) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-25538-0_38 · Cited by 5 · see citing works
DiPri: Distance-Based Seed Prioritization for Greybox Fuzzing (Registered Report)
Qian, R., Zhang, Q., Fang, C., & Chen, Z. (2023) · International Fuzzing Workshop (FUZZING) · doi:10.1145/3605157.3605172 · Cited by 4 · see citing works
Ahmadi-Pour, S., Logemann, M., Herdt, V., & Drechsler, R. (2023) · Chips · doi:10.3390/chips2030012 · Cited by 4 · see citing works
Forward-porting and its limitations in fuzzer evaluation
Elahi, H., & Wang, G. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120142 · Cited by 4 · see citing works
DeepDiffer: Find Deep Learning Compiler Bugs via Priority-guided Differential Fuzzing
Lin, K., Song, X., Zeng, Y., & Guo, S. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00066 · Cited by 5 · see citing works
Make out like a (Multi-Armed) Bandit: Improving the Odds of Fuzzer Seed Scheduling with T-Scheduler
Luo, S., Herrera, A., Quirk, P., Chase, M. K., Ranasinghe, D. C., & Kanhere, S. S. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3637639 · Cited by 4 · see citing works
How We Built Cedar: A Verification-Guided Approach
Disselkoen, C., Eline, A., He, S., Headley, K., Hicks, M., Hietala, K., Kastner, J., Mamat, A., McCutchen, M., Rungta, N., et al. (2024) · Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3663529.3663854 · Cited by 3 · see citing works
Model-Guided Fuzzing of Distributed Systems
Gulcan, B., Ozkan, B. K., Majumdar, R., & Nagendra, S. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763060 · Cited by 3 · see citing works
Metamong: Detecting Render-Update Bugs in Web Browsers through Fuzzing
Song, S., & Lee, B. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616336 · Cited by 3 · see citing works
FunFuzz: Greybox Fuzzing with Function Significance
Qian, R., Zhang, Q., Fang, C., Guo, L., & Chen, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3702974 · Cited by 3 · see citing works
FuzzTheREST: An Intelligent Automated Black-Box RESTful API Fuzzer
Dias, T., Maia, E., & Praça, I. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-82073-1_16 · Cited by 3 · see citing works
The Havoc Paradox in Generator-Based Fuzzing
Li, A., Huang, M., Vikram, V., Lemieux, C., & Padhye, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3742894 · Cited by 3 · see citing works
ZTaint-Havoc: From Havoc Mode to Zero-Execution Fuzzing-Driven Taint Inference
Xie, Y., Zhang, W., & She, D. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728916 · Cited by 3 · see citing works
Practical Anti-Fuzzing Techniques With Performance Optimization
Zhou, Z., & Wang, C. (2023) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2023.3301883 · Cited by 3 · see citing works
BazzAFL: Moving Fuzzing Campaigns Towards Bugs via Grouping Bug-Oriented Seeds
Ye, K., Zhu, X., Xiao, X., Wen, S., Xue, M., & Xiang, Y. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3391795 · Cited by 3 · see citing works
DiPri : Distance-Based Seed Prioritization for Greybox Fuzzing—RCR Report
Qian, R., Zhang, Q., Fang, C., Yang, D., Li, S., Li, B., & Chen, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3701298 · Cited by 3 · see citing works
AutoFuzz: automatic fuzzer-sanitizer scheduling with multi-armed bandit
Gao, Y., Zeng, W., Liu, S., & Zeng, Y. (2025) · Software Quality Journal · doi:10.1007/s11219-025-09707-6 · Cited by 3 · see citing works
Program Feature-Based Benchmarking for Fuzz Testing
Miao, M., Kummita, S., Bodden, E., & Wei, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728899 · Cited by 2 · see citing works
On Evaluating Fuzzers with Context-Sensitive Fuzzed Inputs: A Case Study on PKCS#1-v1.5
Hasan, S. M., Козырева, П., & Hoque, E. (2025) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1109/sbft66712.2025.00013 · Cited by 2 · see citing works
In Bugs We Trust? On Measuring the Randomness of a Fuzzer Benchmarking Outcome
Madadi, A., Lee, S., Aschermann, C., & Böhme, M. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797112 · Cited by 2 · see citing works
Self-citationRCABench: Open Benchmarking Platform for Root Cause Analysis
Nishimura, K., Sugiyama, Y., Koike, Y., Motoda, M., Kitagawa, T., Takatera, T., & Kurogome, Y. (2023) · Workshop on Binary Analysis Research (BAR) · doi:10.14722/bar.2023.23004 · Cited by 2 · see citing works
Triereme: Speeding up hybrid fuzzing through efficient query scheduling
Geretto, E., Hohnerlein, J., Giuffrida, C., Bos, H., van der Kouwe, E., & Gleissenthall, K. V. (2023) · Annual Computer Security Applications Conference · doi:10.1145/3627106.3627173 · Cited by 2 · see citing works
TR-Fuzz: A syntax valid tool for fuzzing C compilers
Zhang, C., Chen, J., Cai, S., Wen, Z., Sosu, R. N. A., & Chen, H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103155 · Cited by 2 · see citing works
On the Analysis of Coverage Feedback in a Fuzzing Proprietary System
Jang, D., Kim, J., Kim, J., Im, W., Jeong, M., Choi, B.-C., & Kil, C. (2024) · Applied Sciences · doi:10.3390/app14135939 · Cited by 2 · see citing works
Evaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study
Harzevili, N. S., Wei, M., Mohajer, M. M., Pham, H. V., & Wang, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3729533 · Cited by 3 · see citing works
An Empirical Study on AST-level mutation-based fuzzing techniques for JavaScript Engines
Tang, S., Liu, S., Wang, J., & Zhang, X. (2023) · Asia-Pacific Symposium on Internetware · doi:10.1145/3609437.3609440 · Cited by 1 · see citing works
CatchFuzz: Reliable active anti-fuzzing techniques against coverage-guided fuzzer
Kim, H. Y., & Lee, D. H. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103904 · Cited by 1 · see citing works
Fuzzing: On Benchmarking Outcome as a Function of Benchmark Properties
Wolff, D., Böhme, M., & Roychoudhury, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3732936 · Cited by 1 · see citing works
Self-citationSoK: From Systematization to Best Practices in Fuzz Driver Generation
Yan, Q., Huang, M., Cao, H., & Lu, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9101-2_18 · Cited by 1 · see citing works
fFuzz: A State-Aware Function-Level Fuzzing Framework for Smart Contract Vulnerabilities Detection
Li, C., Lu, B., Zhang, W., Yang, K., & Zhu, H. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3543-9_2 · Cited by 1 · see citing works
Amusuo, P. C., Cochell, O., Lievre, T. L., Patil, P. V., Machiry, A., & Davis, J. C. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764525 · Cited by 1 · see citing works
RandSet: Randomized Corpus Reduction for Fuzzing Seed Scheduling
Xie, Y., Zhang, K. F., Liu, Y., Yang, R., Chen, P., Wang, S., & She, D. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798257 · Cited by 1 · see citing works
Botacin, M. F. (2024) · Digital Threats Research and Practice · doi:10.1145/3700147 · Cited by 1 · see citing works
Effective Unit Test Generation for Java Null Pointer Exceptions
Lee, M.-H., Bak, J., Moon, S., Jhi, Y., & Oh, H. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695484 · Cited by 1 · see citing works
On Understanding and Forecasting Fuzzers Performance with Static Analysis
Zhang, D., Fioraldi, A., & Balzarotti, D. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3670348 · Cited by 1 · see citing works
Efficient Fuzzing Infrastructure for Pointer-to-Object Association
Ling, H., Huang, H., Cai, Y., & Zhang, C. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3730580 · Cited by 1 · see citing works
Tumbling Down the Rabbit Hole: How do Assisting Exploration Strategies Facilitate Grey-Box Fuzzing?
Wu, M., Xiang, J., Chen, K., Di, P., Tan, S. H., Cui, H., & Zhang, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00044 · Cited by 1 · see citing works
Structures of the data and the information reuse based in-memory fuzzing
Álvarez-Pérez, D., Fernández‐Veiga, M., & Vilas, A. F. (2023) · doi:10.23919/jnic58574.2023.10205765
Kim, J., & Hong, S. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112119
Design and Implementation of a Coverage-Guided Ruby Fuzzer
Schwager, M., Klemba, D., & Dykstra, J. (2024) · Cyber Security Experimentation and Test Workshop (CSET) · doi:10.1145/3675741.3675749
Inconsistencies in TeX-Produced Documents
Tan, J., & Rigger, M. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680370
LOOL: Low-Overhead, Optimization-Log-Guided Compiler Fuzzing (Registered Report)
Schwarcz, F., Berlakovich, F., Barany, G., & Mössenböck, H. (2024) · doi:10.1145/3678722.3685533
Poster: YFuzz: Data-Driven Fuzzing
Chang, Y., Huang, C.-C., Mori, T., & Hsiao, H. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3691420
A Metric for Measuring the Impact of Rare Paths on Program Coverage
Amour, L. S., Tilevich, E., & Gulzar, M. A. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00042
HSPFuzzer: High-Speed Network Protocol Fuzzing With Connection Reuse
Xia, Z., Zeng, Y., Song, X., Guo, S., & Wu, T. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3590178
AceCov: Auxiliary Composite Edge Coverage for Fuzzing
Yoshida, H., Sugiyama, Y., & Shioya, R. (2025) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp63326.2025.00062
Understanding the Potentially Confounding Effect of Test Suite Size in Test Effectiveness Evaluation
Lu, Z., Wang, Y., Rong, Y., Huang, Y., Wang, X., Zhang, P., Gao, S., Sun, M., Yang, Y., Li, Y., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748504
SBFT’25 Competition Report — Fuzzing Track
Crump, A., Sihag, S., & Leonelli, M. (2025) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1109/sbft66712.2025.00008
From Coverage to Causes: Data-Centric Fuzzing for JavaScript EnginesPreprint
Ganguly, K. K., & Menzies, T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5686732
From Coverage to Causes: Data-Centric Fuzzing for JavaScript EnginesPreprint
Ganguly, K. K., & Menzies, T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5686730
Function Clustering-Based Fuzzing Termination: Toward Smarter Early Stopping
Ding, L., Yang, W., & Xue, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00145
Towards Path-Aware Coverage-Guided Fuzzing
Priamo, G., D’Elia, D. C., Payer, M., & Querzoni, L. (2026) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo68049.2026.11395191
Sagitta: Facilitating Post-Fuzzing Root Cause Analysis via Data Flow Differencing
Aoki, K., & Shinagawa, T. (2025) · Annual Computer Security Applications Conference · doi:10.1109/acsac67867.2025.00017
Optimized mutation scheduling for fuzzing based on estimation of distribution algorithm
Lv, G., Chen, J., Chen, H., Cai, S., & Zhang, W. (2026) · Cybersecurity · doi:10.1186/s42400-025-00461-7
SeedAIchemy: LLM-Driven Seed Corpus Generation for Fuzzing
Wen, A., Alzahrani, N. A., Jiang, J., Joe, A., Shieh, K., Zhang, A. K., Alomair, B., & Wagner, D. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00149
LOOL: Low-Overhead, Optimization-Log-Guided Compiler Fuzzing
Berlakovich, F., Schwarcz, F., Barany, G., Mössenböck, H., & Brunthaler, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3807508
Nyzhnyk, A., & Kotliarov, O. Y. (2026) · Komp'ûternì sistemi ta merežì · doi:10.23939/csn2026.01.193
From coverage to causes: Data-centric fuzzing for Javascript engines
Ganguly, K. K., & Menzies, T. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108211
SemuFuzz: An efficient fuzzing approach for deep learning libraries based on semantic representation
Chen, Z., Chen, J., Cai, S., Chen, J., & Gu, W. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112999
pPatch: Automated Vulnerability Unpatching
Jing, T., Ding, P., Xu, M., Hu, Y., Yu, Z., & Mu, D. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808201
Reality Check: Independent Evaluation of Modern Grey-Box Fuzzing Techniques
Frolikov, P. (2026) · IEEE Secure Development Conference (SecDev) · doi:10.1145/3805773.3806004
Böhme, M. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00097
Self-citationSoK: A Systematic Review of Integration and Reproducibility of Fuzzing Research into AFL++
Schiller, N., Bars, N., Schloegel, M., & Holz, T. (2026) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp68448.2026.00074
FrameShift: Resizing Fuzzer Inputs Without Breaking Them
Green, H., Goues, C. L., & Brown, F. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787765
- Authors
- Böhme, M., Szekeres, L., & Metzman, J.
- Affiliations
- Monash University (AU)
- Google (United States) (US)
- Venue
- Proceedings of the 44th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 1621-1633
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4284706927
- Cited by
- 104
- References
- 47
- Citations by year
- 2026: 18 · 2025: 25 · 2024: 32 · 2023: 25 · 2022: 4
- FWCI
- 16.22
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, other-oa)
- Funders
- École Polytechnique Fédérale de Lausanne
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- code coverage, fuzzing, software testing, bug detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review