Cited by 229Software Testing and Debugging TechniquesSoftware Engineering ResearchFormal Methods in Verification

Fuzz4ALL: Universal Fuzzing with Large Language Models

Open access
Xia, Chunqiu Steven, Paltenghi, Matteo, Le Tian, Jia, Pradel, Michael, Zhang, Lingming·2024·Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

Abstract

Fuzzing has achieved tremendous success in discovering bugs and vulnerabilities in various software systems. Systems under test (SUTs) that take in programming or formal language as inputs, e.g., compilers, runtime engines, constraint solvers, and software libraries with accessible APIs, are especially important as they are fundamental building blocks of software development. However, existing fuzzers for such systems often target a specific language, and thus cannot be easily applied to other languages or even other versions of the same language. Moreover, the inputs generated by existing fuzzers are often limited to specific features of the input language, and thus can hardly reveal bugs related to other or new features. This paper presents Fuzz4All, the first fuzzer that is universal in the sense that it can target many different input languages and many different features of these languages. The key idea behind Fuzz4All is to leverage large language models (LLMs) as an input generation and mutation engine, which enables the approach to produce diverse and realistic inputs for any practically relevant language. To realize this potential, we present a novel autoprompting technique, which creates LLM prompts that are well-suited for fuzzing, and a novel LLM-powered fuzzing loop, which iteratively updates the prompt to create new fuzzing inputs. We evaluate Fuzz4All on nine systems under test that take in six different languages (C, C++, Go, SMT2, Java, and Python) as inputs. The evaluation shows, across all six languages, that universal fuzzing achieves higher coverage than existing, language-specific fuzzers. Furthermore, Fuzz4All has identified 98 bugs in widely used systems, such as GCC, Clang, Z3, CVC5, OpenJDK, and the Qiskit quantum computing platform, with 64 bugs already confirmed by developers as previously unknown.

Integrity signals

Registry

Crossref · 10.1145/3597503.3639121

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SWE-bench unites many separate software engineering tasks under a single task formulation that faithfully mirrors practical software engineering.

    “We use SWE-bench because it unites many separate SE tasks, such as automated program repair [10, 40, 55], bug localization [4, 58], and testing [22, 46, 56] under a single task formulation that faithfully mirrors practical SE.”
  • LLMs have a strong understanding of programming languages, enabling them to perform testing tasks in multiple languages.

    “Another factor is that LLMs have a strong understanding of programming languages, enabling them to perform testing tasks in multiple languages.”

    From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Xia et al. (2024)

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Xia, C. S., Paltenghi, M., Le Tian, J., Pradel, M., & Zhang, L.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • University of Stuttgart (DE)
  • UIUC, Urbana, Illinois, USA
Venue
Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
Publisher
ACM
Published
2024-04-12
Pages
1-13
ISSN
0270-5257 (ISSN-L)
OpenAlex
W4394769544
Cited by
229
References
42
Citations by year
2026: 93 · 2025: 103 · 2024: 31 · 2023: 2
FWCI
61.97
Type
conference paper
Language
en
Open access
Locations
  • ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (CCF-2141474, CCF-2131943)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Formal Methods in Verification
Keywords
large language models, input generation, input mutation, compiler bugs, SMT solvers, software vulnerability detection, coverage-guided fuzzing, Qiskit

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review