Fuzz4ALL: Universal Fuzzing with Large Language Models
Open accessAbstract
Fuzzing has achieved tremendous success in discovering bugs and vulnerabilities in various software systems. Systems under test (SUTs) that take in programming or formal language as inputs, e.g., compilers, runtime engines, constraint solvers, and software libraries with accessible APIs, are especially important as they are fundamental building blocks of software development. However, existing fuzzers for such systems often target a specific language, and thus cannot be easily applied to other languages or even other versions of the same language. Moreover, the inputs generated by existing fuzzers are often limited to specific features of the input language, and thus can hardly reveal bugs related to other or new features. This paper presents Fuzz4All, the first fuzzer that is universal in the sense that it can target many different input languages and many different features of these languages. The key idea behind Fuzz4All is to leverage large language models (LLMs) as an input generation and mutation engine, which enables the approach to produce diverse and realistic inputs for any practically relevant language. To realize this potential, we present a novel autoprompting technique, which creates LLM prompts that are well-suited for fuzzing, and a novel LLM-powered fuzzing loop, which iteratively updates the prompt to create new fuzzing inputs. We evaluate Fuzz4All on nine systems under test that take in six different languages (C, C++, Go, SMT2, Java, and Python) as inputs. The evaluation shows, across all six languages, that universal fuzzing achieves higher coverage than existing, language-specific fuzzers. Furthermore, Fuzz4All has identified 98 bugs in widely used systems, such as GCC, Clang, Z3, CVC5, OpenJDK, and the Qiskit quantum computing platform, with 64 bugs already confirmed by developers as previously unknown.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SWE-bench unites many separate software engineering tasks under a single task formulation that faithfully mirrors practical software engineering.
“We use SWE-bench because it unites many separate SE tasks, such as automated program repair [10, 40, 55], bug localization [4, 58], and testing [22, 46, 56] under a single task formulation that faithfully mirrors practical SE.”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [56]
LLMs have a strong understanding of programming languages, enabling them to perform testing tasks in multiple languages.
“Another factor is that LLMs have a strong understanding of programming languages, enabling them to perform testing tasks in multiple languages.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Xia et al. (2024)
Cites
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · see citing works
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · see citing works
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305 · see citing works
Finding and understanding bugs in C compilers
Yang, X., Chen, Y., Eide, E., & Regehr, J. (2011) · Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1993498.1993532 · Cited by 825 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · see citing works
Klees, G., Ruef, A., Cooper, B., Wei, S., & Hicks, M. (2018) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3243734.3243804 · Cited by 659 · see citing works
Fuzzing: Brute Force Vulnerability Discovery
Sutton, M. J. D., Greene, A., & Amini, P. (2007) · Medical Entomology and Zoology · Cited by 545 · see citing works
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540 · see citing works
A Survey on Automated Fact-Checking
Guo, Z., Schlichtkrull, M., & Vlachos, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00454 · Cited by 500 · see citing works
Learn&Fuzz: Machine learning for input fuzzing
Godefroid, P., Peleg, H., & Singh, R. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115618 · Cited by 375 · see citing works
Gulwani, S., Polozov, O., & Singh, R. (2017) · Foundations and Trends® in Programming Languages · doi:10.1561/2500000010 · Cited by 357 · see citing works
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342 · see citing works
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326 · see citing works
Self-citationChen, J., Patra, J., Pradel, M., Xiong, Y., Zhang, H., Hao, D., & Zhang, L. (2020) · ACM Computing Surveys · doi:10.1145/3363562 · Cited by 222 · see citing works
Self-citationOpen source software in quantum computing
Fingerhuth, M., Babej, T., & Wittek, P. (2018) · PLoS ONE · doi:10.1371/journal.pone.0208561 · Cited by 188 · see citing works
Fuzzing: Challenges and Reflections
Böhme, M., Cadar, C., & Roychoudhury, A. (2020) · IEEE Software · doi:10.1109/ms.2020.3016773 · Cited by 161 · see citing works
Compiler fuzzing through deep learning
Cummins, C., Petoumenos, P., Murray, A. C., & Leather, H. (2018) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3213846.3213848 · Cited by 157 · see citing works
No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence
Wang, C., Yang, Y., Gao, C., Peng, Y. F., Zhang, H., & Lyu, M. R.-T. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549113 · Cited by 153 · see citing works
Random testing for C and C++ compilers with YARPGen
Livinskii, V., Babokin, D., & Regehr, J. D. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428264 · Cited by 144 · see citing works
Free lunch for testing: fuzzing deep-learning libraries from open source
Wei, A., Deng, Y., Yang, C., & Zhang, L. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510041 · Cited by 106 · see citing works
Self-citationOn the reliability of coverage-based fuzzer benchmarking
Böhme, M., Szekeres, L., & Metzman, J. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510230 · Cited by 104 · see citing works
DeepFuzz: Automatic Generation of Syntax Valid C Programs for Fuzz Testing
Liu, X., Li, X., Prajapati, R., & Wu, D. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011044 · Cited by 101 · see citing works
NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers
Liu, J., Lin, J., Ruffy, F., Tan, C., Li, J., Panda, A., & Zhang, L. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575707 · Cited by 99 · see citing works
Self-citationValidating SMT solvers via semantic fusion
Winterer, D., Zhang, C., & Su, Z. (2020) · Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/3385412.3385985 · Cited by 82 · see citing works
GrayC: Greybox Fuzzing of Compilers and Analysers for C
Even-Mendoza, K., Sharma, A., Donaldson, A. F., & Cadar, C. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598130 · Cited by 74 · see citing works
SeqFuzzer: An Industrial Protocol Fuzzing Framework from a Deep Learning Perspective
Zhao, H., Li, Z., Wei, H., Shi, J., & Huang, Y. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00016 · Cited by 73 · see citing works
Bugs in Quantum computing platforms: an empirical study
Paltenghi, M., & Pradel, M. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3527330 · Cited by 73 · see citing works
Self-citationLidbury, C., Lascu, A., Chong, N., & Donaldson, A. F. (2015) · ACM SIGPLAN Notices · doi:10.1145/2813885.2737986 · Cited by 66 · see citing works
History-driven test program synthesis for JVM testing
Zhao, Y., Wang, Z., Chen, J., Liu, M., Wu, M., Zhang, Y., & Zhang, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510059 · Cited by 61 · see citing works
Self-citationMorphQ: Metamorphic Testing of the Qiskit Quantum Computing Platform
Paltenghi, M., & Pradel, M. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00202 · Cited by 57 · see citing works
Self-citationOn the unusual effectiveness of type-aware operator mutations for testing SMT solvers
Winterer, D., Zhang, C., & Su, Z. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428261 · Cited by 56 · see citing works
One Engine to Fuzz ’em All: Generic Language Processor Testing with Semantic Validation
Chen, Y., Zhong, R., Hu, H. P., Zhang, H., Yang, Y., Wu, D., & Lee, W. (2021) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40001.2021.00071 · Cited by 55 · see citing works
Coverage-guided tensor compiler fuzzing with joint IR-pass mutation
Liu, J., Wei, Y., Yang, S., Deng, Y., & Zhang, L. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3527317 · Cited by 53 · see citing works
Self-citationChaliasos, S., Sotiropoulos, T., Spinellis, D. D., Gervais, A., Livshits, B., & Mitropoulos, D. (2022) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3519939.3523427 · Cited by 37 · see citing works
Generative type-aware mutation for testing SMT solvers
Park, J., Winterer, D., Zhang, C., & Su, Z. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485529 · Cited by 33 · see citing works
Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on CodePreprint
Bareiß, P., Souza, B., d’Amorim, M., & Pradel, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.01335 · Cited by 31 · see citing works
Self-citationNo Grammar, No Problem: Towards Fuzzing the Linux Kernel without System-Call Descriptions
Bulekov, A., Das, B., Hajnoczi, S., & Egele, M. (2023) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2023.24688 · Cited by 31 · see citing works
CsmithEdge: more effective compiler testing by handling undefined behaviour less conservatively
Even-Mendoza, K., Cadar, C., & Donaldson, A. F. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10146-1 · Cited by 29 · see citing works
TF-Coder: Program Synthesis for Tensor Manipulations
Shi, K., Bieber, D., & Singh, R. (2022) · ACM Transactions on Programming Languages and Systems · doi:10.1145/3517034 · Cited by 27 · see citing works
Metamorphic testing of Datalog engines
Mansur, M. N., Christakis, M., & Wüstholz, V. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468573 · Cited by 26 · see citing works
Making no-fuss compiler fuzzing effective
Groce, A., van Tonder, R., Kalburgi, G. T., & Goues, C. L. (2022) · ACM SIGPLAN International Conference on Compiler Construction (CC) · doi:10.1145/3497776.3517765 · Cited by 16 · see citing works
Feedback-Directed Differential Testing of Interactive Debuggers (Artifact)
Lehmann, D., & Pradel, M. (2018) · Artifact Digital Object Group · doi:10.1145/3277000 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
He, J., Treude, C., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712003 · Cited by 246 · see citing works
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 220 · see citing works
“Xia et al. (2024) tests 6 languages code (i.e., C, C ++ , Go, SMT2, Java, and Python) with a method named Fuzz-Loop, which automatically mutates test cases based on LLMs.”
Enhancing Autonomous System Security and Resilience With Generative AI: A Comprehensive Survey
Lopez, M. A., Lunardi, W. T., Lawton, G., & Thakkar, S. (2024) · IEEE Access · doi:10.1109/access.2024.3439363 · Cited by 115 · see citing works
Demystifying LLM-Based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715754 · Cited by 87 · see citing works
Self-citationGenerative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77 · see citing works
LLMIF: Augmented Large Language Model for Fuzzing IoT Devices
Wang, J., Yu, L., & Luo, X. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00211 · Cited by 68 · see citing works
SpecGen: Automated Generation of Formal Program Specifications via Large Language Models
Ma, L., Liu, S., Li, Y., Xie, X., & Bu, L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00129 · Cited by 98 · see citing works
CoverUp: Effective High Coverage Test Generation for Python
Pizzorno, J. A., & Berger, E. D. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729398 · Cited by 84 · see citing works
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72 · see citing works
“(2018), testing (Kang et al., 2023; Xia et al., 2023; Wang et al., 2023), and program repair (Monperrus, 2018; Gupta et al.”
BackgroundCan LLMs Replace Manual Annotation of Software Engineering Artifacts?
Ahmed, T., Dévanbu, P., Treude, C., & Pradel, M. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00086 · Cited by 41 · see citing works
Self-citationKnowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs
Cassano, F., Gouwar, J., Lucchetti, F., Schlesinger, C., Freeman, A., Anderson, C. J., Feldman, M. Q., Greenberg, M., Jangda, A., & Guha, A. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689735 · Cited by 39 · see citing works
You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects
Bouzenia, I., & Pradel, M. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3728922 · Cited by 35 · see citing works
Self-citationSWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31 · see citing works
The Evolution of Large Language Model: Models, Applications and Challenges
Sindhu, B., Prathamesh, R. P., Sameera, M. B., & Kumaraswamy, S. (2024) · International Conference on Current Trends in Advanced Computing (ICCTAC) · doi:10.1109/icctac61556.2024.10581180 · Cited by 34 · see citing works
When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges
Wang, C., Zhao, J., Jiao, L., Li, L., Liu, F., & Yang, S. (2025) · Research · doi:10.34133/research.0646 · Cited by 38 · see citing works
Analyzing Quantum Programs with LintQ: A Static Analysis Framework for Qiskit
Paltenghi, M., & Pradel, M. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660802 · Cited by 29 · see citing works
Self-citationBeyond Random Inputs: A Novel ML-Based Hardware Fuzzing
Rostami, M., Chilese, M., Zeitouni, S., Kande, R., Rajendran, J., & Sadeghi, A. (2024) · Design, Automation, and Test in Europe · doi:10.23919/date58400.2024.10546625 · Cited by 28 · see citing works
The Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators
Ou, X., Li, C., Jiang, Y., & Xu, C. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3622781.3674171 · Cited by 28 · see citing works
SecureQwen: Leveraging LLMs for vulnerability detection in python codebases
Mechri, A., Ferrag, M. A., & Debbah, M. A. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104151 · Cited by 27 · see citing works
Enhancing Genetic Improvement Mutations Using Large Language Models
Brownlee, A. E. I., Callan, J., Even-Mendoza, K., Geiger, A., Hanna, C., Petke, J., Sarro, F., & Sobania, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48796-5_13 · Cited by 24 · see citing works
Wang, D., Zhou, G., Chen, L., Li, D., & Miao, Y. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690231 · Cited by 23 · see citing works
ChatHTTPFuzz: large language model-assisted IoT HTTP fuzzing
Yang, Z., Hao, P., Jiang, Y., Li, X., Du, H., Wang, S., & Liu, J. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02527-3 · Cited by 22 · see citing works
Alhanahnah, M., Hasan, M. R., Xu, L., & Bagheri, H. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10687-1 · Cited by 20 · see citing works
Towards Understanding the Effectiveness of Large Language Models on Directed Test Input Generation
Jiang, Z., Wen, M., Cao, J., Shi, X., & Jin, H. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695513 · Cited by 22 · see citing works
Clear, T., Cajander, Å., Clear, A., McDermott, R., Daniels, M., Divitini, M., Forshaw, M., Humble, N., Kasinidou, M., Kleanthous, S., et al. (2025) · 2024 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3689187.3709607 · Cited by 19 · see citing works
An LLM-Based Agent-Oriented Approach for Automated Code Design Issue Localization
Batole, F., O'Brien, D. P., Nguyen, T. N., Dyer, R., & Rajan, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00100 · Cited by 20 · see citing works
Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories
Bouzenia, I., & Pradel, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00234 · Cited by 19 · see citing works
Self-citationAgentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24 · see citing works
Self-citationDemonstration-Free: Towards More Practical Log Parsing with Large Language Models
Xiao, Y., Le, V.-H., & Zhang, H. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694994 · Cited by 33 · see citing works
Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration
Ma, Y., Yang, Q., Cao, R., Li, B., Huang, F., & Li, Y. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728549 · Cited by 49 · see citing works
Rug: Turbo Llm for Rust Unit Test Generation
Cheng, X. C., Sang, F., Zhai, Y., Zhang, X., & Kim, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00097 · Cited by 17 · see citing works
Cheng, Y., Bajaber, O., Tsegai, S. A., Song, D. X., & Gao, P. (2025) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp63326.2025.00057 · Cited by 25 · see citing works
Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models
Vishwamitra, N., Guo, K., Romit, F. T., Ondracek, I., Cheng, L., Zhao, Z., & Hu, H. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00181 · Cited by 16 · see citing works
Hui, M., Wang, L., Li, H., Ren, Y., Song, Y., Zhuang, H., Cui, D., & Li, Q. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112232 · Cited by 22 · see citing works
Black, G., Vaidyan, V. M., & Comert, G. (2024) · IEEE Access · doi:10.1109/access.2024.3484947 · Cited by 15 · see citing works
Adversarial generation method for smart contract fuzz testing seeds guided by chain-based LLM
Sun, J., Yin, Z., Zhang, H., Chen, X., & Wei, Z. M. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00483-4 · Cited by 15 · see citing works
On the Challenges of Fuzzing Techniques via Large Language Models
Huang, L., Zhao, P., Ma, L., & Chen, H. (2025) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse67621.2025.00028 · Cited by 18 · see citing works
Validating Network Protocol Parsers with Traceable RFC Document Interpretation
Zheng, M., Xie, D., Shi, Q., Wang, C., & Zhang, X. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728955 · Cited by 13 · see citing works
Clozemaster: Fuzzing Rust Compiler by Harnessing Llms for Infilling Masked Real Programs
Gao, H., Yang, Y., Sun, M., Wu, J., Zhou, Y., & Xu, B. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00175 · Cited by 12 · see citing works
Are LLMs Correctly Integrated into Software Systems?
Shao, Y., Huang, Y., Shen, J., Ma, L., Su, T., & Wan, C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00204 · Cited by 20 · see citing works
Mokav: Execution-driven differential testing with LLMs
Etemadi, K., Mohammadi, B., Su, Z., & Monperrus, M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112571 · Cited by 11 · see citing works
Guan, H., Bai, G., & Liu, Y. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680383 · Cited by 10 · see citing works
Enhancing search-based testing with LLMs for finding bugs in system simulators
Dakhama, A., Even-Mendoza, K., Langdon, W. B., Menéndez, H. D., & Petke, J. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00531-7 · Cited by 10 · see citing works
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 24 · see citing works
MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing
Xu, C., Chen, S., Wu, J., Cheung, S.-C., Terragni, V., Zhu, H., & Cao, J. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3696020 · Cited by 9 · see citing works
Large language model based mutations in genetic improvement
Brownlee, A. E. I., Callan, J., Even-Mendoza, K., Geiger, A., Hanna, C., Petke, J., Sarro, F., & Sobania, D. (2025) · Automated Software Engineering · doi:10.1007/s10515-024-00473-6 · Cited by 9 · see citing works
An MLLM-Assisted Web Crawler Approach for Web Application Fuzzing
Yang, W., Wang, E., Gui, Z., Yuan, Z., Wang, B., & Xie, W. (2025) · Applied Sciences · doi:10.3390/app15020962 · Cited by 9 · see citing works
LLM4TDG: test-driven generation of large language models based on enhanced constraint reasoning
Liu, J., Liang, R., Zhu, X., Zhang, Y., Liu, Y., & Liu, Q. (2025) · Cybersecurity · doi:10.1186/s42400-024-00335-4 · Cited by 9 · see citing works
Kitten: A Simple Yet Effective Baseline for Evaluating LLM-Based Compiler Testing Techniques
Xie, Y., Xu, Z., Tian, Y., Zhou, M. Q., Zhou, X., & Sun, C. P. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731731 · Cited by 9 · see citing works
FuzzCoder: Code Large Language Model-Based Fuzz Testing for Industrial IoT Programs
Yang, L., Wei, C., Yang, J., Xia, W., Yang, Y., Luo, Y., Niyato, D. T., Sun, L., & Liu, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3577602 · Cited by 11 · see citing works
Bringing Rust to Safety-Critical Systems in Space
Seidel, L., & Beier, J. (2024) · doi:10.23919/3s60530.2024.10592287 · Cited by 8 · see citing works
MSFuzz: Augmenting Protocol Fuzzing with Message Syntax Comprehension via Large Language Models
Cheng, M., Zhu, K.-L., Chen, Y., Yang, G., Lu, Y., & Lu, C. (2024) · Electronics · doi:10.3390/electronics13132632 · Cited by 8 · see citing works
Resource-Efficient & Effective Code Summarization
Afrin, S., Call, J., Nguyen, K.-N., Chaparro, O., & Mastropaolo, A. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00032 · Cited by 11 · see citing works
Unlocking Low Frequency Syscalls in Kernel Fuzzing with Dependency-Based RAG
Zhang, Z., Li, L., Liang, R., & Chen, K. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728913 · Cited by 6 · see citing works
Enhancing Protocol Fuzzing via Diverse Seed Corpus Generation
Luo, Z., Du, Q., Wang, Y., Roychoudhury, A., & Jiang, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3595396 · Cited by 8 · see citing works
An Automated Blackbox Noncompliance Checker for QUIC Server Implementations
Ang, K. K., Farrelly, G., Pope, C., & Ranasinghe, D. C. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3736222 · Cited by 6 · see citing works
Fuzzing C++ Compilers via Type-Driven Mutation
Wang, B., Chen, C., Deng, M., Chen, J., Zhang, X., Lin, Y., Hao, D., & Sun, J. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763094 · Cited by 6 · see citing works
CodeDoctor: multi-category code review comment generation
Li, Y., Wu, Y., Wang, Z., Huang, L., Wang, J., Li, J. K., & Huang, M. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00491-y · Cited by 6 · see citing works
A survey of coverage-guided greybox fuzzing with deep neural models
Qiu, J., Jiang, Y., Miao, Y., Luo, W., Pan, L., & Zheng, X. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107797 · Cited by 6 · see citing works
Lin, L. G., Zhu, Q., Chen, H., Wang, Z., Wu, R., & Xie, X. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728908 · Cited by 5 · see citing works
Unleashing the Power of LLM to Infer State Machine From the Protocol Implementation
Wei, H., Chen, L., Du, Z., Wu, Y., Huang, H., Liu, Y., Cheng, G., Xu, F., Wang, L., & Mao, B. (2025) · International Workshop on Quality of Service · doi:10.1109/iwqos65803.2025.11143461 · Cited by 5 · see citing works
LLM-based Satisfiability Checking of String Requirements by Consistent Data and Checker Generation
Chen, B.-Q., Babikian, A. A., Feng, S., Varró, D., & Mussbacher, G. (2025) · IEEE International Requirements Engineering Conference · doi:10.1109/re63999.2025.00030 · Cited by 7 · see citing works
Interleaving Large Language Models for Compiler Testing
Ni, Y., & Li, S. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763079 · Cited by 6 · see citing works
P rome F uzz : A Knowledge-Driven Approach to Fuzzing Harness Generation with Large Language Models
Liu, Y., Deng, J., Jia, X., Wang, Y., Wang, M., Huang, L., Wei, T., & Su, P. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765222 · Cited by 5 · see citing works
MCP-enabled agentic AI workflow for building energy modelling: framework and use cases
Li, H., Zhang, L., Zhou, H., & Hong, T. (2026) · Journal of Building Performance Simulation · doi:10.1080/19401493.2026.2653969 · Cited by 5 · see citing works
Self-citationEnhancing Black-box Compiler Option Fuzzing with LLM through Command Feedback
Wang, T., Wang, R., Chen, Y., Yu, L., Pan, Z., Zhang, M., Ma, H., & Zheng, J. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00039 · Cited by 6 · see citing works
A Holistic Review of Fuzzing for Vulnerability Assessment in Industrial Network Protocols
Reda, A. M., Moustafa, N., & Lakshika, E. (2025) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2025.3569291 · Cited by 5 · see citing works
Oracle-Guided Program Selection from Large Language Models
Fan, Z., Ruan, H., Mechtaev, S., & Roychoudhury, A. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680308 · Cited by 5 · see citing works
Optimization for threat classification of various data types-based on ML model and LLM
Hong, C., & Oh, T. (2025) · Scientific Reports · doi:10.1038/s41598-025-05182-y · Cited by 5 · see citing works
SWE-GPT: A Process-Centric Language Model for Automated Software Improvement
Ma, Y., Cao, R., Cao, Y., Zhang, Y., Chen, J., Liu, Y., Liu, Y., Li, B., Huang, F., & Li, Y. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728981 · Cited by 22 · see citing works
Shaking Up Quantum Simulators with Fuzzing and Rigour
Klimis, V., Bensoussan, A., Chachkarova, E., Even-Mendoza, K., Fortz, S., & Lenihan, C. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763100 · Cited by 4 · see citing works
Jiang, J., Wen, C., Fu, Z., & Qin, S. (2025) · Science of Computer Programming · doi:10.1016/j.scico.2025.103395 · Cited by 4 · see citing works
Bare-Metal Firmware Fuzzing: A Survey of Techniques and Approaches
Asmita, A., Tsang, R., Ghimire, S., Salehi, S., & Homayoun, H. (2025) · IEEE Access · doi:10.1109/access.2025.3575691 · Cited by 4 · see citing works
Cao, D., Hong, Y., Pan, Q., & Wu, J. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3581213 · Cited by 4 · see citing works
One Mutation Fits All: Exploring Universal Library Fuzzing Based on Exogenous Mutation
Dong, R., Tong, F., Huang, H., Zhu, X., Xiao, X., Wang, S., Wen, S., & Xiang, Y. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3588878 · Cited by 4 · see citing works
Search+LLM-Based Testing for ARM Simulators
Even-Mendoza, K., Menéndez, H. D., Langdon, W. B., Dakhama, A., Petke, J., & Bruce, B. R. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00047 · Cited by 4 · see citing works
Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review
Zhang, J., Hu, X., Gao, C., Xia, X., & Li, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3802827 · Cited by 6 · see citing works
Pașca, E. M., Erdei, R., Delinschi, D., & Matei, O. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-75010-6_23 · Cited by 4 · see citing works
Large Language Model guided State Selection Approach for Fuzzing Network Protocol
Yu, B., Song, Q., & Cai, C. (2024) · IEEE International Performance, Computing, and Communications Conference · doi:10.1109/ipccc59868.2024.10850039 · Cited by 4 · see citing works
RGFuzz: Rule-Guided Fuzzer for WebAssembly Runtimes
Park, J., Kim, Y., & Yun, I. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00003 · Cited by 4 · see citing works
Large Language Models-Aided Program Debloating
Lin, B., Wang, S., Qin, Y., Chen, L., & Mao, X. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3594673 · Cited by 5 · see citing works
Beyond pip Install: Evaluating LLM Agents for the Automated Installation of Python Projects
Milliken, L., Kang, S., & Yoo, S. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00009 · Cited by 5 · see citing works
CrossGuard : Runtime‐Adaptive LLM Fuzzing for Cross‐Contract Vulnerabilities Detection
Ali, G. M. A., Chen, H., Wang, Z., & Du, C. (2025) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70421 · Cited by 3 · see citing works
LLM-Guided Genetic Improvement: Envisioning Semantic Aware Automated Software Evolution
Even-Mendoza, K., Brownlee, A., Geiger, A., Hanna, C., Petke, J., Sarro, F., & Sobania, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00339 · Cited by 3 · see citing works
Less Is More: Failing Test Generation with Large Language Models
Li, T.-O., Yan, X., Zong, W., Wang, Y., Tian, H., Xu, C., Wang, Y., Cheung, S.-C., & Kramer, J. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3793675 · Cited by 3 · see citing works
Chen, J., Chen, Y., Pan, Z., Chen, Y., Li, Y., Li, Y., Zhang, M., & Shen, Y. (2024) · Electronics · doi:10.3390/electronics13173476 · Cited by 3 · see citing works
MorphQ++: A Reproducibility Study of Metamorphic Testing on Quantum Compilers
Kitt, L., & Cohen, M. B. (2024) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1145/3691621.3694959 · Cited by 3 · see citing works
Large Language Model-enabled Vulnerability Investigation: A Review
Pan, Z., Liu, J., Dai, Y., & Fan, W. (2024) · International Conference on Intelligent Computing and Next Generation Networks (ICNGN) · doi:10.1109/icngn63705.2024.10871716 · Cited by 3 · see citing works
A Comprehensive Study of OOP-Related Bugs in C++ Compilers
Wang, B., Chen, C., Chen, J., Xu, B., Ye, C., Lin, Y., Dong, G., & Sun, J. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3566490 · Cited by 3 · see citing works
Can Large Language Models Discover Metamorphic Relations? A Large-Scale Empirical Study
Zhang, J., Sun, C., Liu, H., & Dong, S. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00011 · Cited by 3 · see citing works
IRFuzzer: Specialized Fuzzing for LLVM Backend Code Generation
Rong, Y., Yu, Z., Weng, Z., Neuendorffer, S., & Chen, H. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00130 · Cited by 3 · see citing works
Wang, E., Xie, W., Li, S., Liu, R., Zhou, Y. W., Wang, Z., Ma, S., Yang, W., & Wang, B. (2025) · Sensors · doi:10.3390/s25092913 · Cited by 3 · see citing works
Dhulipala, H., Yadavally, A., Patel, S. S., & Nguyen, T. N. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729401 · Cited by 2 · see citing works
FreeWavm: Enhanced WebAssembly Runtime Fuzzing Guided by Parse Tree Mutation and Snapshot
Qian, P., Ying, X., Wang, J., Liu, L., Zhang, L., Chen, J., & He, Q. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728877 · Cited by 2 · see citing works
Self-citationTreefix: Enabling Execution with a Tree of Prefixes
Souza, B., & Pradel, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00215 · Cited by 2 · see citing works
Self-citationFinding Compiler Bugs through Cross-Language Code Generator and Differential Testing
Feng, Q., Ma, X., Feng, Z., Akhin, M., Song, W., & Liang, P. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763152 · Cited by 2 · see citing works
Reference-Based Retrieval-Augmented Unit Test Generation
Zhang, Z., Liu, X., Lin, Y., Gao, X., Sun, H., & Yuan, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3765758 · Cited by 2 · see citing works
SATORI: Static Test Oracle Generation for REST APIs
Alonso, J. C. G., Martin-Lopez, A., Segura, S., Bavota, G., & Ruiz–Cortés, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00116 · Cited by 3 · see citing works
LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework
Lops, A., Narducci, F., Ragone, A., Trizio, M., & Bartolini, C. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00198 · Cited by 2 · see citing works
Once4All: Skeleton-Guided SMT Solver Fuzzing with LLM-Synthesized Generators
Sun, M., Yang, Y., & Zhou, Y. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790195 · Cited by 2 · see citing works
- Authors
- Xia, C. S., Paltenghi, M., Le Tian, J., Pradel, M., & Zhang, L.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- University of Stuttgart (DE)
- UIUC, Urbana, Illinois, USA
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-04-12
- Pages
- 1-13
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4394769544
- Cited by
- 229
- References
- 42
- Citations by year
- 2026: 93 · 2025: 103 · 2024: 31 · 2023: 2
- FWCI
- 61.97
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (CCF-2141474, CCF-2131943)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Formal Methods in Verification
- Keywords
- large language models, input generation, input mutation, compiler bugs, SMT solvers, software vulnerability detection, coverage-guided fuzzing, Qiskit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review