The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index; 5 of the 29 sources its claims cite were checked for support.
Demystifying LLM-Based Software Engineering Agents
Open accessAbstract
Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless – an agentless approach to automatically resolve software development issues. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents at the time of paper submission! Agentless also achieves more than 50% solve rate when using Claude 3.5 Sonnet on the new SWE-bench Verified benchmark. In fact, Agentless has already been adopted by OpenAI as the go-to approach to showcase the real-world coding performance of both GPT-4o and the new o1 models; more recently, Agentless has also been used by DeepSeek to evaluate their newest DeepSeek V3 and R1 models . Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patches or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite- S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the currently overlooked potential of a simplistic, cost-effective technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction. We have open-sourced Agentless at: https://github.com/OpenAutoCoder/Agentless
Overview
CoChat Summary
TL;DR
Agentless, a deliberately non-agentic three-phase pipeline for localization, repair, and patch validation, resolves 96/300 SWE-bench Lite issues (32.00%) at $0.70 average cost, the highest among open-source approaches at submission time. The paper also manually classifies SWE-bench Lite problems and constructs a filtered SWE-bench Lite-S benchmark.
Key findings
- On SWE-bench Lite, Agentless with GPT-4o resolves 96/300 problems (32.00%) at average $0.70 and 78,165 tokens; correct-location rates are 20.7% line, 50.3% function, and 69.7% file.
- On SWE-bench Verified, Agentless resolves 194/500 problems (38.80%) with GPT-4o and achieves more than 50% solve rate with Claude 3.5 Sonnet.
- Ablations show patch selection improves from 77 fixes with majority voting to 81 with regression tests and 96 with reproduction tests; repair with 4×10 multi-samples reaches 96 fixes versus 88 for greedy locations and 85 for merged multi-samples.
- Manual classification finds 4.3% of SWE-bench Lite issues contain the exact ground-truth patch, 9.7% contain complete solution steps, 5.0% contain misleading solutions, and 10.0% lack enough information; the filtered SWE-bench Lite-S has 249 problems, where Agentless resolves 84 (33.73%).
Methods
- Agentless uses hierarchical localization: top-N suspicious files from repository structure plus embedding retrieval, then classes/functions via skeleton format, then fine-grained edit locations; repair samples Search/Replace diff patches; patch validation generates reproduction tests and uses regression tests plus majority voting.
- Evaluation uses SWE-bench Lite (300 problems) against 26 agent-based baselines, with GPT-4o (gpt-4o-2024-05-13), text-embedding-3-small, top 3 files, 4 edit-location samples, 10 patches per sample (40 patches per issue), and 40 reproduction tests; metrics are % Resolved, average cost, average tokens, and correct location at line/function/file.
- The paper manually classifies SWE-bench Lite problems and constructs SWE-bench Lite-S (249 problems), and also reports results on SWE-bench Verified (500 problems).
Limitations
- Stated internal threat: possible data leakage because SWE-bench Lite ground-truth developer patches may be in GPT-4o’s training data; since GPT-4o is closed-source, this cannot be checked, though the authors note prior work used similar closed LLMs and SWE-bench authors found no significant pre/post-cutoff difference.
- Stated external threat: SWE-bench Lite may not generalize to other datasets, and it mostly focuses on bug-fixing issues; future work is planned on other benchmarks.
- Evident from the paper’s own results: reproduction tests are imperfect—213/300 tests reproduce the original issue, but only 94 correctly output “Issue resolved” after applying the ground-truth patch; issue descriptions can lack sufficient information, and Agentless underperforms closed-source agents when no location clue is provided.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 4 Not valid · 14 Not checked · 106 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“That is to say they tend to take all information/feedback and do not know how to filter out or correct irrelevant, incorrect, or misleading information [Shi et al. 2023;Zhang et al. 2023].”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
What the sources say
- Not matchedcites Shi et al. 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Zhang et al. 2023 sourceExperimental — accuracy not yet measured
Section 4, “Sources of LLM Hallucination” (page not available), says LLMs can overestimate their ability to assess their own answers and may favor a user’s perspective over accuracy. Section 5.4.2, “Resorting to External Knowledge” (page not available), calls verification of retrieved information an open challenge. These passages support difficulties with self-assessment and unreliable information, but not the blanket claim that agents take all feedback or cannot filter or correct it. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, we observed that SWE-bench Lite contains problems with exact ground truth patch in the description (4.3%), problems with missing critical information needed to solve the issue (10.0%), and problems that include misleading solutions in the issue description (5.0%).”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As such, we construct SWE-bench Lite-𝑆 by excluding such problematic issues to perform more rigorous evaluation and comparison.”
- Also stated in: § Abstract · plain
“Recognizing these issues, we built SWE-bench Lite-𝑆, which removes such problematic problems, and serves as a more rigorous benchmark to evaluate the ability to solve real-world software development challenges.”
- Also stated in: § Abstract · plain
“To address these issues, we constructed a filtered dataset of SWE-bench Lite-𝑆 that excludes such problematic issues, enabling more rigorous evaluation and comparison.”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, we observed that SWE-bench Lite contains problems with exact ground truth patch in the description (4.3%), problems with missing critical information needed to solve the issue (10.0%), and problems that include misleading solutions in the issue description (5.0%).”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, we observed that SWE-bench Lite contains problems with exact ground truth patch in the description (4.3%), problems with missing critical information needed to solve the issue (10.0%), and problems that include misleading solutions in the issue description (5.0%).”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
64 more claims
Create a free account to see all 129 claims and to verify the 4 references still open.
- Probably not a substantial claim. 28
- It may state two separate claims as one. 39
- Borderline: one of the tests for a substantial claim is unclear. 1
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unlike single-turn code completion, practical coding scenarios require an agent to sustain a sequence of interdependent steps across many rounds of interaction, such as requirement understanding, code localization, evidence retrieval, multi-file editing, test execution, bug fixing, and regression verification [176, 177].”
From Code as Agent Harness (2026) · cites this paper as [176]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Demystifying LLM-Based Software Engineering Agents > 5.2.2 Repair Ablation.
“Starting with using the greedy location set generated in the edit location generation stage, we observe that we can already achieve very high performance of more than 88 issues fixed.”
§ Demystifying LLM-Based Software Engineering Agents > 8 Conclusion
“| Greedy location (40 samples) | 88 (29.33%) | $0.22 |”
Citations
Cites
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Self-citationThe probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374
Modern Information Retrieval : A Brief Overview
Singhal, A. (2001) · IEEE Data(base) Engineering Bulletin · Cited by 1,556
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Empirical evaluation of the tarantula automatic fault-localization technique
Jones, J. A., & Harrold, M. J. (2005) · Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/1101908.1101949 · Cited by 1,238 · cited 1×
“Dynamic FL techniques mainly include spectrum-based FL (SBFL) [Abreu et al. 2009, 2007; Jones and Harrold 2005] and mutation-based FL (MBFL) [Lou et al. 2020; Moon et al. 2014; Papadakis and Le Traon 2015].”
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110 · cited 1×
“Fault localization (FL) [Wong et al. 2016] techniques aim to identify the suspicious locations (e.g., statements or methods) in source code related to bugs.”
GenProg: A Generic Method for Automatic Software Repair
Le Goues, C., Nguyen, T., Forrest, S., & Weimer, W. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,085 · cited 1×
“Traditional APR techniques can be categorized as template-based [Ghanbari et al. 2019; Liu et al. 2019], heuristic-based [Le et al. 2016; Le Goues et al. 2012], and constraint-based [Long and Rinard 2015; Mechtaev et al. 2016] tools.”
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · cited 1×
“With the emergence and popularity of agent-based frameworks [Xi et al. 2023], recently researchers and industry practitioners have begun developing agent-based approaches to solve software engineering tasks.”
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
On the Accuracy of Spectrum-based Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2007) · Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007) · doi:10.1109/taic.part.2007.13 · Cited by 736
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 712
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Xu, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.01219 · Cited by 544
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534
Angelix: scalable multiline program patch synthesis via symbolic analysis
Mechtaev, S., Yi, J., & Roychoudhury, A. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884807 · Cited by 515 · cited 1×
“Traditional APR techniques can be categorized as template-based [Ghanbari et al. 2019; Liu et al. 2019], heuristic-based [Le et al. 2016; Le Goues et al. 2012], and constraint-based [Long and Rinard 2015; Mechtaev et al. 2016] tools.”
Zhou, J., Zhang, H., & Lo, D. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227210 · Cited by 507
A practical evaluation of spectrum-based fault localization
Abreu, R., Zoeteweij, P., Golsteijn, R., & van Gemund, A. J. C. (2009) · Journal of Systems and Software · doi:10.1016/j.jss.2009.06.035 · Cited by 495 · cited 1×
“Dynamic FL techniques mainly include spectrum-based FL (SBFL) [Abreu et al. 2009, 2007; Jones and Harrold 2005] and mutation-based FL (MBFL) [Lou et al. 2020; Moon et al. 2014; Papadakis and Le Traon 2015].”
Automated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433
Self-citationAn Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3334955 · Cited by 431
Mutation Testing Advances: An Analysis and Survey
(2019) · Advances in Computers · doi:10.1016/bs.adcom.2018.03.015 · Cited by 425
Spectrum-Based Multiple Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2009) · 2009 IEEE/ACM International Conference on Automated Software Engineering · doi:10.1109/ase.2009.25 · Cited by 401
SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z., Kommrusch, S. J., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., & Monperrus, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 396 · cited 1×
“As such, researchers have proposed learning-based APR tools either by training NMT (neural machine translation) models [Chen et al. 2019; Jiang et al. 2021; Li et al. 2020; Zhu et al. 2021] or using pre-trained LLMs to perform repair [Kolak et al. 2022; Xia et al. 2023a; Xia and Zhang 2022; Zhang…”
Improving bug localization using structured information retrieval
Saha, R. K., Lease, M., Khurshid, S., & Perry, D. E. (2013) · 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693093 · Cited by 390 · cited 1×
“Such IR-based techniques [Saha et al. 2013; Wang et al. 2015; Wang and Lo 2014] formulate FL as a search problem and compare the textual similarity between code elements and the bug report (i.e., query).”
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
Metallaxis‐FL: mutation‐based fault localization
Papadakis, M., & Le Traon, Y. (2015) · Software Testing, Verification and Reliability · doi:10.1002/stvr.1509 · Cited by 371 · cited 1×
“Dynamic FL techniques mainly include spectrum-based FL (SBFL) [Abreu et al. 2009, 2007; Jones and Harrold 2005] and mutation-based FL (MBFL) [Lou et al. 2020; Moon et al. 2014; Papadakis and Le Traon 2015].”
Staged program repair with condition synthesis
Long, F., & Rinard, M. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786811 · Cited by 348 · cited 1×
“Traditional APR techniques can be categorized as template-based [Ghanbari et al. 2019; Liu et al. 2019], heuristic-based [Le et al. 2016; Le Goues et al. 2012], and constraint-based [Long and Rinard 2015; Mechtaev et al. 2016] tools.”
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336 · cited 1×
“Moreover, learning-based techniques have also been proposed to leverage machine learning to combine multiple sources of dynamic/static information, including DeepFL [Li et al. 2019], FLUCCS [Sohn and Yoo 2017], and TRANSFER [Meng et al. 2022].”
Self-citationLe, X. B. D., Lo, D., & Le Goues, C. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.76 · Cited by 332 · cited 1×
“Traditional APR techniques can be categorized as template-based [Ghanbari et al. 2019; Liu et al. 2019], heuristic-based [Le et al. 2016; Le Goues et al. 2012], and constraint-based [Long and Rinard 2015; Mechtaev et al. 2016] tools.”
Automatic Software Repair: A Survey
Gazzola, L., Micucci, D., & Mariani, L. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2755013 · Cited by 327
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
Self-citationLarge Language Model-Based Agents for Software Engineering: A Survey
Jun-wei, L., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., & Lou, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3796507 · Cited by 323
Self-citationTBar: revisiting template-based automated program repair
Liu, K., Koyuncu, A., Kim, D., & Bissyandé, T. F. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330577 · Cited by 321 · cited 1×
“Traditional APR techniques can be categorized as template-based [Ghanbari et al. 2019; Liu et al. 2019], heuristic-based [Le et al. 2016; Le Goues et al. 2012], and constraint-based [Long and Rinard 2015; Mechtaev et al. 2016] tools.”
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306
DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.19437 · Cited by 294
Self-citationAsk the Mutants: Mutating Faulty Programs for Fault Localization
Moon, S., Kim, Y., Kim, M., & Yoo, S. (2014) · 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation · doi:10.1109/icst.2014.28 · Cited by 291 · cited 1×
“Dynamic FL techniques mainly include spectrum-based FL (SBFL) [Abreu et al. 2009, 2007; Jones and Harrold 2005] and mutation-based FL (MBFL) [Lou et al. 2020; Moon et al. 2014; Papadakis and Le Traon 2015].”
CURE: Code-Aware Neural Machine Translation for Automatic Program Repair
Jiang, N., Lutellier, T., & Tan, L. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00107 · Cited by 290 · cited 1×
“As such, researchers have proposed learning-based APR tools either by training NMT (neural machine translation) models [Chen et al. 2019; Jiang et al. 2021; Li et al. 2020; Zhu et al. 2021] or using pre-trained LLMs to perform repair [Kolak et al. 2022; Xia et al. 2023a; Xia and Zhang 2022; Zhang…”
2012 34th International Conference on Software Engineering (ICSE)
(2012) · doi:10.1109/icse19354.2012 · Cited by 269
Liu, Z., Chen, C., Wang, J., Chen, M., Wu, B., Che, X., Wang, D., & Wang, Q. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639180 · Cited by 267
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264
Large Language Model guided Protocol Fuzzing
Meng, R., Mirchev, M., Böhme, M., & Roychoudhury, A. (2024) · Proceedings 2024 Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24556 · Cited by 246
FLUCCS: using code and change metrics to improve fault localization
Sohn, J., & Yoo, S. (2017) · Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3092703.3092717 · Cited by 242 · cited 1×
“Moreover, learning-based techniques have also been proposed to leverage machine learning to combine multiple sources of dynamic/static information, including DeepFL [Li et al. 2019], FLUCCS [Sohn and Yoo 2017], and TRANSFER [Meng et al. 2022].”
Less training, more repairing please: revisiting automated program repair via zero-shot learning
Xia, C. S., & Zhang, L. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549101 · Cited by 231 · cited 3×
“The intuition behind adding the additional code before and after the identified location is to provide the LLM with relevant contextual information for better program repair [Xia and Zhang 2022].”
Self-citationFuzz4ALL: Universal Fuzzing with Large Language Models
Xia, C. S., Paltenghi, M., Le Tian, J., Pradel, M., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639121 · Cited by 229
Self-citationVersion history, similar report, and structure: putting them together for improved bug localization
Wang, S., & Lo, D. (2014) · Proceedings of the 22nd International Conference on Program Comprehension · doi:10.1145/2597008.2597148 · Cited by 218 · cited 1×
“Such IR-based techniques [Saha et al. 2013; Wang et al. 2015; Wang and Lo 2014] formulate FL as a search problem and compare the textual similarity between code elements and the bug report (i.e., query).”
A syntax-guided edit decoder for neural program repair
Zhu, Q., Sun, Z., Xiao, Y.-A., Zhang, W., Yuan, K., Xiong, Y., & Zhang, L. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468544 · Cited by 216 · cited 1×
“…have proposed learning-based APR tools either by training NMT (neural machine translation) models [Chen et al. 2019; Jiang et al. 2021; Li et al. 2020; Zhu et al. 2021] or using pre-trained LLMs to perform repair [Kolak et al. 2022; Xia et al. 2023a; Xia and Zhang 2022; Zhang et al. 2024a].”
Self-citationDLFix: context-based code transformation learning for automated program repair
Li, Y., Wang, S., & Nguyen, T. N. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380345 · Cited by 209 · cited 1×
“…researchers have proposed learning-based APR tools either by training NMT (neural machine translation) models [Chen et al. 2019; Jiang et al. 2021; Li et al. 2020; Zhu et al. 2021] or using pre-trained LLMs to perform repair [Kolak et al. 2022; Xia et al. 2023a; Xia and Zhang 2022; Zhang et al.…”
Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT
Xia, C. S., & Zhang, L. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680323 · Cited by 203
Self-citationImpact of Code Language Models on Automated Program Repair
Jiang, N., Liu, K., Lutellier, T., & Tan, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00125 · Cited by 202
AutoCodeRover: Autonomous Program Improvement
Zhang, Y., Ruan, H., Fan, Z., & Roychoudhury, A. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680384 · Cited by 202 · cited 4×
“More recently, agent-based APR techniques have also been proposed [Bouzenia et al. 2024; Chen 2024; Hidvégi et al. 2024; Xia and Zhang 2023; Zhang et al. 2024b].”
InfluentialAutoCodeRover: Autonomous Program Improvement
Zhang, Y., Ruan, H., Fan, Z., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2404.05427 · Cited by 202
A Systematic Literature Review on Large Language Models for Automated Program Repair
Zhang, Q., Fang, C., Xie, Y., Ma, Y., Sun, W., Yang, Y. J., & Chen, Z. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3799693 · Cited by 188
WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Yang, C., Deng, Y., Lu, R., Yao, J., Liu, J., Jabbarvand, R., & Zhang, L. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689736 · Cited by 180
Self-citationCoverUp: Effective High Coverage Test Generation for Python
Pizzorno, J. A., & Berger, E. D. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729398 · Cited by 168
An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems
Long, F., & Rinard, M. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884872 · Cited by 167
Evaluating and Improving ChatGPT for Unit Test Generation
Yuan, Z., Liu, M., Ding, S., Wang, K., Chen, Y., Peng, X., & Lou, Y. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660783 · Cited by 163
Automatic software repair: a survey
Gazzola, L., Micucci, D., & Mariani, L. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3182526 · Cited by 163
Practical program repair via bytecode mutation
Ghanbari, A., Benton, S., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330559 · Cited by 162 · cited 1×
“Traditional APR techniques can be categorized as template-based [Ghanbari et al. 2019; Liu et al. 2019], heuristic-based [Le et al. 2016; Le Goues et al. 2012], and constraint-based [Long and Rinard 2015; Mechtaev et al. 2016] tools.”
Self-citation(2025) · doi:10.1109/icse55347.2025.00157 · Cited by 157
SpecRover: Code Intent Extraction via LLMs
Ruan, H.-F., Zhang, Y.-T., & Roychoudhury, A. (2024) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00080 · Cited by 152
SoapFL: A Standard Operating Procedure for LLM-Based Method-Level Fault Localization
Qin, Y., Wang, S., Lou, Y., Dong, J., Wang, K., Li, X., & Mao, X. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3543187 · Cited by 151
Evaluating the usefulness of IR-based fault localization techniques
Wang, Q., Parnin, C., & Orso, A. (2015) · International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771797 · Cited by 144 · cited 1×
“Such IR-based techniques [Saha et al. 2013; Wang et al. 2015; Wang and Lo 2014] formulate FL as a search problem and compare the textual similarity between code elements and the bug report (i.e., query).”
Large Language Models for Test-Free Fault Localization
Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623342 · Cited by 144 · cited 2×
“Recently, researchers have proposed LLM-based FL [Kang et al. 2024; Qin et al. 2024; Wu et al. 2023; Yang et al. 2024b], which leverages the powerful code and natural language understanding of modern LLMs to directly localize bugs.”
KernelGPT: Enhanced Kernel Fuzzing via Large Language Models
Yang, C., Zhao, Z., & Zhang, L. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716022 · Cited by 130
Self-citationLarge Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTPreprint
Deng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02014 · Cited by 130
Self-citationA Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
Kang, S., An, G., & Yoo, S. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660771 · Cited by 128 · cited 2×
“Recently, researchers have proposed LLM-based FL [Kang et al. 2024; Qin et al. 2024; Wu et al. 2023; Yang et al. 2024b], which leverages the powerful code and natural language understanding of modern LLMs to directly localize bugs.”
BugRedux: Reproducing field failures for in-house debugging
Jin, W., & Orso, A. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227168 · Cited by 122 · cited 1×
“Lambdify misinterprets some matrix expressions Using lambdify on an ... Bug reproduction is a critical step in investigating bug reports [Jin and Orso 2012], and is integrated into many recent software engineering agents [Arora et al. 2024; Factory 2024; Hon-eycomb 2024; MentatBot 2024; OpenDevin…”
Can automated program repair refine fault localization? a unified debugging approach
Lou, Y., Ghanbari, A., Li, X., Zhang, L., Zhang, H., Hao, D., & Zhang, L. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397351 · Cited by 110 · cited 1×
“Dynamic FL techniques mainly include spectrum-based FL (SBFL) [Abreu et al. 2009, 2007; Jones and Harrold 2005] and mutation-based FL (MBFL) [Lou et al. 2020; Moon et al. 2014; Papadakis and Le Traon 2015].”
Self-citationMeng, X., Wang, X., Zhang, H., Sun, H., & Liu, X. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510147 · Cited by 104 · cited 1×
“Moreover, learning-based techniques have also been proposed to leverage machine learning to combine multiple sources of dynamic/static information, including DeepFL [Li et al. 2019], FLUCCS [Sohn and Yoo 2017], and TRANSFER [Meng et al. 2022].”
Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration
Ma, Y., Yang, Q., Cao, R., Li, B.-H., Huang, F., & Li, Y.-B. (2024) · SIGSOFT FSE Companion · doi:10.1145/3696630.3728549 · Cited by 91
Adaptive Test Generation Using a Large Language ModelPreprint
Schäfer, M., Nadi, S., Eghbali, A., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2302.06527 · Cited by 74
How to Understand Whole Software Repository?Preprint
Ma, Y., Yang, Q., Cao, R., Li, B.-H., Huang, F., & Li, Y.-B. (2024) · arXiv.org · doi:10.48550/arxiv.2406.01422 · Cited by 73
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
Zhang, L., Li, K., Sun, K., Wu, D., Liu, Y., Tian, H., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3590108 · Cited by 71 · cited 1×
“More recently, agent-based APR techniques have also been proposed [Bouzenia et al. 2024; Chen 2024; Hidvégi et al. 2024; Xia and Zhang 2023; Zhang et al. 2024b].”
Self-citationFuzzing BusyBox: Leveraging LLM and Crash Reuse for Embedded Bug UnearthingPreprint
Asmita, Oliinyk, Y., Scott, M., et al. (2024) · USENIX Security Symposium · doi:10.48550/arxiv.2403.03897 · Cited by 69
OpenAI, :, Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., et al. (2024) · arXiv · Cited by 50
Large Language Model-Based Agents for Software Engineering: A Survey
Liu, J., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., & Lou, Y. (2024) · arXiv · doi:10.48550/arxiv.2409.02977 · Cited by 49
Self-citationPraPR: Practical Program Repair via Bytecode Mutation
Ghanbari, A., & Zhang, L. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00116 · Cited by 41
Self-citationSWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31
Large Language Models in Fault Localisation
Wu, Y., Li, Z., Zhang, J. M., Papadakis, M., Harman, M., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.15276 · Cited by 28
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2408.02232 · Cited by 28
A Syntax-Guided Edit Decoder for Neural Program RepairPreprint
Zhu, Q., Sun, Z., Xiao, Y.-A., Zhang, W., Yuan, K., Xiong, Y., & Zhang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08253 · Cited by 21
Self-citationA Systematic Literature Review on Large Language Models for Automated Program Repair
Zhang, Q., Fang, C., Xie, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.01466 · Cited by 17
RepairAgent: An Autonomous, LLM-Based Agent for Program Repair
(2024) · doi:10.48550/arxiv.2403.17134 · Cited by 14
CigaR: Cost-efficient Program Repair with LLMs
Hidvégi, D., Etemadi, K., Bobadilla, S., & Monperrus, M. (2024) · arXiv · doi:10.48550/arxiv.2402.06598 · Cited by 12
AgentFL: Scaling LLM-based Fault Localization to Project-Level Context
Qin, Y., Wang, S., Lou, Y., Dong, J., Wang, K., Li, X., & Mao, X. (2024) · arXiv · doi:10.48550/arxiv.2403.16362 · Cited by 8
CodeR: Issue Resolving with Multi-Agent and Task Graphs
Chen, D., Lin, S., Zeng, M., Zan, D., Wang, J.-G., Cheshkov, A., Sun, J., Yu, H., Dong, G., Aliev, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.01304 · Cited by 8
MarsCode Agent: AI-native Automated Bug Fixing
Liu, Y., Gao, P., Wang, X., Liu, J., Shi, Y., Zhang, Z., & Peng, C. (2024) · arXiv · doi:10.48550/arxiv.2409.00899 · Cited by 5
Zhang, L., Li, K., Sun, K., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.06838 · Cited by 5
Self-citationFlakiness Repair in the Era of Large Language Models
Chen, Y. (2024) · Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3639478.3641227 · Cited by 4
MASAI: Modular Architecture for Software-engineering AI Agents
Arora, D., Sonwane, A., Wadhwa, N., Mehrotra, A., Utpala, S., Bairi, R., Kanade, A., & Natarajan, N. (2024) · arXiv · doi:10.48550/arxiv.2406.11638 · Cited by 2
Reproducing and debugging field failures in house
Jin, W. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606738 · Cited by 2
Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents
Zhang, K., Yao, W., Liu, Z., Feng, Y., Liu, Z., Murthy, R., Lan, T., Li, L., Lou, R., Xu, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2408.07060 · Cited by 1
Impact of Code Language Models on Automated Program Repair (Dataset)
Jiang, N., Liu, K., Lutellier, T., & Tan, L. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7559207
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
CoverUp: Effective High Coverage Test Generation for Python
Pizzorno, J. A., & Berger, E. D. (2024) · arXiv · doi:10.48550/arxiv.2403.16218
Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT
Xia, C. S., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2304.00385
Self-citationWhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Yang, C., Deng, Y., Lu, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.15991
Self-citationKernelGPT: Enhanced Kernel Fuzzing via Large Language Models
Yang, C., Zhao, Z., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2401.00563
Self-citation
Research works printed but not linked: 9
- 10.5555/1308173.1308264 · doi:10.5555/1308173.1308264
- Jie Huang Xinyun Chen Swaroop Mishra Huaixiu Steven Zheng Adams Wei Yu Xinying Song and Denny Zhou. 2024. Large Language Models Cannot Self-Correct ReasoningYet. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=IkmD3fKBPQ
- Carlos E Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik R Narasimhan. 2024b. SWE-bench Leaderboard. https://www.swebench.com/.
- Sophia D Kolak Ruben Martins Claire Le Goues and Vincent Josua Hellendoorn. 2022. Patch Generation with Language Models: Feasibility and Scaling Behavior. In Deep Learning for Code Workshop.
- Theo X Olausson Jeevana Priya Inala Chenglong Wang Jianfeng Gao and Armando Solar-Lezama. 2023. Is Self-Repair a Silver Bullet for Code Generation?. In The Twelfth International Conference on Learning Representations.
- Xianfei Ou Cong Li Yanyan Jiang and Chang Xu. 2024. The Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators. In ASPLOS.
- Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning. PMLR, 31210–31227.
- Chunqiu Steven Xia Matteo Paltenghi Jia Le Tian Michael Pradel and Lingming Zhang. 2024. Universal Fuzzing via Large Language Models. In 46th International Conference on Software Engineering (ICSE).
- Chunqiu Steven Xia Yuxiang Wei and Lingming Zhang. 2023b. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery.
Cited by
Why Do Multi-Agent LLM Systems Fail?
Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 630
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.16941 · Cited by 484
Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
Zhang, J., Hu, S., Lu, C., Lange, R., & Clune, J. (2025) · arXiv · doi:10.48550/arxiv.2505.22954 · Cited by 344
Game-theoretic LLM: Agent Workflow for Negotiation Games
Hua, W., Liu, O., Li, L., Amayuelas, A., Chen, J., Jiang, L., Jin, M., Fan, L., Sun, F., Wang, W., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.05990 · Cited by 101
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
Le, T., Thai, M. V. T., Manh, D. N., Nhat, H. P., & Bui, N. D. Q. (2025) · arXiv · doi:10.48550/arxiv.2512.18470 · Cited by 79
Metadata
- Authors
- Xia, C. S., Deng, Y., Dunn, S., & Zhang, L.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Venue
- Proceedings of the ACM on Software Engineering
- Published
- 2025-06-19
- Volume
- 2
- Issue
- FSE
- Pages
- 801-824
- ISSN
- 2994-970X (ISSN-L)
- OpenAlex
- W4411449940
- Cited by
- 849
- References
- 34
- Citations by year
- 2026: 617 · 2025: 206
- FWCI
- 490.90
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the ACM on software engineering. (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (CCF-2131943)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- autonomous software agents, large language models, program repair, software development automation, SWE-bench Verified, patch validation, Claude 3.5 Sonnet, GPT-4o, DeepSeek-V3
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.