The Relevance of Classic Fuzz Testing: Have We Solved This One?
Open accessAbstract
As fuzz testing has passed its 30th anniversary, and in the face of the incredible progress in fuzz testing techniques and tools, the question arises if the classic, basic fuzz technique is still useful and applicable? In that tradition, we have updated the basic fuzz tools and testing scripts and applied them to a large collection of Unix utilities on Linux, FreeBSD, and MacOS. As before, our failure criteria was whether the program crashed or hung. We found that 9 crash or hang out of 74 utilities on Linux, 15 out of 78 utilities on FreeBSD, and 12 out of 76 utilities on MacOS. A total of 24 different utilities failed across the three platforms. We note that these failure rates are somewhat higher than our in previous 1995, 2000, and 2006 studies of the reliability of command line utilities. In the basic fuzz tradition, we debugged each failed utility and categorized the causes the failures. Classic categories of failures, such as pointer and array errors and not checking return codes, were still broadly present in the current results. In addition, we found a couple of new categories of failures appearing. We present examples of these failures to illustrate the programming practices that allowed them to happen. As a side note, we tested the limited number of utilities available in a modern programming language (Rust) and found them to be of no better reliability than the standard ones.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Fuzzing remains a remarkably effective testing technique.
“Fuzzing remains to be a remarkably effective testing technique.”
From Zero-Trust Principles for Legacy Components: 12 Rules for Legacy Devices: An Antidote to Chaos (2021) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,295 · see citing works
Self-citationMcKusick, M. K., Joy, W. N., Leffler, S. J., & Fabry, R. S. (1984) · ACM Transactions on Computer Systems · doi:10.1145/989.990 · Cited by 754 · see citing works
Klees, G., Ruef, A., Cooper, B., Wei, S., & Hicks, M. (2018) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3243734.3243804 · Cited by 659 · see citing works
(2021) · doi:10.1109/tse.2019.2946563 · Cited by 574 · see citing works
Fuzzing for Software Security Testing and Quality Assurance
Takanen, A., DeMott, J. D., & Miller, C. (2008) · Cited by 289 · see citing works
Li, J., Zhao, B., & Zhang, C. (2018) · Cybersecurity · doi:10.1186/s42400-018-0002-y · Cited by 279 · see citing works
An empirical study of the robustness of Windows NT applications using random testing
Forrester, J. E., & Miller, B. P. (2000) · Cited by 266 · see citing works
Self-citationFuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and Services
Miller, B. P., Koski, D., Lee, C. P., et al. (1995) · Cited by 250 · see citing works
Self-citationLiang, H., Pei, X., Jia, X., Shen, W., & Zhang, J. (2018) · IEEE Transactions on Reliability · doi:10.1109/tr.2018.2834476 · Cited by 245 · see citing works
Donovan, A. A. A., & Kernighan, B. W. (2015) · CERN Document Server (European Organization for Nuclear Research) · Cited by 195 · see citing works
Meyerson, J. (2014) · IEEE Software · doi:10.1109/ms.2014.127 · Cited by 176 · see citing works
Spafford, E. H. (1989) · Communications of the ACM · doi:10.1145/63526.63527 · Cited by 172 · see citing works
An empirical study of the robustness of MacOS applications using random testing
Miller, B. P., Cooksey, G., & Moore, F. H. (2006) · doi:10.1145/1145735.1145743 · Cited by 91 · see citing works
Self-citationAn In-Depth Security Assessment of Maritime Container Terminal Software Systems
Eichenhofer, J. O., Heymann, E., Miller, B. P., et al. (2020) · IEEE Access · doi:10.1109/access.2020.3008395 · Cited by 23 · see citing works
Self-citationAn evaluation of free fuzzing tools
Vimpari, M. (2015) · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [10][10] S. Klabnik and C. Nichols. The Rust Programming Language . No Starch Press, 2019. -
- [17][17] V. J. M. Man` es, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz, and M. Woo. The Art, Science, and Engineering of Fuzzing: A Survey. IEEE Transactions on Software Engineering , October 2019. -
- [25][25] R. Pike, D. Presotto, S. Dorward, B. Flandrena, K. Thompson, H. Trickey, and P. Winterbottom. Plan 9 from Bell Labs. Computer Systems , 8(3):221-254, Summer 1995. -
Cited by
Sorted by citations
Zero-Trust Principles for Legacy Components: 12 Rules for Legacy Devices: An Antidote to Chaos
Køien, G. M. (2021) · Wireless Personal Communications · doi:10.1007/s11277-021-09055-1 · Cited by 24 · see citing works
- Authors
- Miller, B. P., Zhang, M., & Heymann, E. R.
- Affiliations
- University of Wisconsin–Madison (US)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2022-06-01
- Volume
- 48
- Issue
- 6
- Pages
- 2028-2039
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W3049661990
- Cited by
- 26
- References
- 15
- Citations by year
- 2026: 1 · 2025: 8 · 2024: 5 · 2023: 4 · 2022: 3 · 2021: 5
- FWCI
- 5.51
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- fuzzing, software reliability, failure causes, Rust, programming practice, Linux, FreeBSD, macOS
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
