The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index; 5 of the 29 sources its claims cite were checked for support.
Demystifying LLM-Based Software Engineering Agents
Open accessAbstract
Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless – an agentless approach to automatically resolve software development issues. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents at the time of paper submission! Agentless also achieves more than 50% solve rate when using Claude 3.5 Sonnet on the new SWE-bench Verified benchmark. In fact, Agentless has already been adopted by OpenAI as the go-to approach to showcase the real-world coding performance of both GPT-4o and the new o1 models; more recently, Agentless has also been used by DeepSeek to evaluate their newest DeepSeek V3 and R1 models . Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patches or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite- S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the currently overlooked potential of a simplistic, cost-effective technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction. We have open-sourced Agentless at: https://github.com/OpenAutoCoder/Agentless
Overview
CoChat Summary
TL;DR
Agentless, a deliberately non-agentic three-phase pipeline for localization, repair, and patch validation, resolves 96/300 SWE-bench Lite issues (32.00%) at $0.70 average cost, the highest among open-source approaches at submission time. The paper also manually classifies SWE-bench Lite problems and constructs a filtered SWE-bench Lite-S benchmark.
Key findings
- On SWE-bench Lite, Agentless with GPT-4o resolves 96/300 problems (32.00%) at average $0.70 and 78,165 tokens; correct-location rates are 20.7% line, 50.3% function, and 69.7% file.
- On SWE-bench Verified, Agentless resolves 194/500 problems (38.80%) with GPT-4o and achieves more than 50% solve rate with Claude 3.5 Sonnet.
- Ablations show patch selection improves from 77 fixes with majority voting to 81 with regression tests and 96 with reproduction tests; repair with 4×10 multi-samples reaches 96 fixes versus 88 for greedy locations and 85 for merged multi-samples.
- Manual classification finds 4.3% of SWE-bench Lite issues contain the exact ground-truth patch, 9.7% contain complete solution steps, 5.0% contain misleading solutions, and 10.0% lack enough information; the filtered SWE-bench Lite-S has 249 problems, where Agentless resolves 84 (33.73%).
Methods
- Agentless uses hierarchical localization: top-N suspicious files from repository structure plus embedding retrieval, then classes/functions via skeleton format, then fine-grained edit locations; repair samples Search/Replace diff patches; patch validation generates reproduction tests and uses regression tests plus majority voting.
- Evaluation uses SWE-bench Lite (300 problems) against 26 agent-based baselines, with GPT-4o (gpt-4o-2024-05-13), text-embedding-3-small, top 3 files, 4 edit-location samples, 10 patches per sample (40 patches per issue), and 40 reproduction tests; metrics are % Resolved, average cost, average tokens, and correct location at line/function/file.
- The paper manually classifies SWE-bench Lite problems and constructs SWE-bench Lite-S (249 problems), and also reports results on SWE-bench Verified (500 problems).
Limitations
- Stated internal threat: possible data leakage because SWE-bench Lite ground-truth developer patches may be in GPT-4o’s training data; since GPT-4o is closed-source, this cannot be checked, though the authors note prior work used similar closed LLMs and SWE-bench authors found no significant pre/post-cutoff difference.
- Stated external threat: SWE-bench Lite may not generalize to other datasets, and it mostly focuses on bug-fixing issues; future work is planned on other benchmarks.
- Evident from the paper’s own results: reproduction tests are imperfect—213/300 tests reproduce the original issue, but only 94 correctly output “Issue resolved” after applying the ground-truth patch; issue descriptions can lack sufficient information, and Agentless underperforms closed-source agents when no location clue is provided.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 4 Not valid · 14 Not checked · 106 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“That is to say they tend to take all information/feedback and do not know how to filter out or correct irrelevant, incorrect, or misleading information [Shi et al. 2023;Zhang et al. 2023].”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
What the sources say
- Not matchedcites Shi et al. 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Zhang et al. 2023 sourceExperimental — accuracy not yet measured
Section 4, “Sources of LLM Hallucination” (page not available), says LLMs can overestimate their ability to assess their own answers and may favor a user’s perspective over accuracy. Section 5.4.2, “Resorting to External Knowledge” (page not available), calls verification of retrieved information an open challenge. These passages support difficulties with self-assessment and unreliable information, but not the blanket claim that agents take all feedback or cannot filter or correct it. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, we observed that SWE-bench Lite contains problems with exact ground truth patch in the description (4.3%), problems with missing critical information needed to solve the issue (10.0%), and problems that include misleading solutions in the issue description (5.0%).”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As such, we construct SWE-bench Lite-𝑆 by excluding such problematic issues to perform more rigorous evaluation and comparison.”
- Also stated in: § Abstract · plain
“Recognizing these issues, we built SWE-bench Lite-𝑆, which removes such problematic problems, and serves as a more rigorous benchmark to evaluate the ability to solve real-world software development challenges.”
- Also stated in: § Abstract · plain
“To address these issues, we constructed a filtered dataset of SWE-bench Lite-𝑆 that excludes such problematic issues, enabling more rigorous evaluation and comparison.”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, we observed that SWE-bench Lite contains problems with exact ground truth patch in the description (4.3%), problems with missing critical information needed to solve the issue (10.0%), and problems that include misleading solutions in the issue description (5.0%).”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, we observed that SWE-bench Lite contains problems with exact ground truth patch in the description (4.3%), problems with missing critical information needed to solve the issue (10.0%), and problems that include misleading solutions in the issue description (5.0%).”
§ Demystifying LLM-Based Software Engineering Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
64 more claims
Create a free account to see all 129 claims and to verify the 4 references still open.
- Probably not a substantial claim. 28
- It may state two separate claims as one. 39
- Borderline: one of the tests for a substantial claim is unclear. 1
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unlike single-turn code completion, practical coding scenarios require an agent to sustain a sequence of interdependent steps across many rounds of interaction, such as requirement understanding, code localization, evidence retrieval, multi-file editing, test execution, bug fixing, and regression verification [176, 177].”
From Code as Agent Harness (2026) · cites this paper as [176]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Demystifying LLM-Based Software Engineering Agents > 5.2.2 Repair Ablation.
“Starting with using the greedy location set generated in the edit location generation stage, we observe that we can already achieve very high performance of more than 88 issues fixed.”
§ Demystifying LLM-Based Software Engineering Agents > 8 Conclusion
“| Greedy location (40 samples) | 88 (29.33%) | $0.22 |”
Citations
Cites
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Self-citationThe probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374
Modern Information Retrieval : A Brief Overview
Singhal, A. (2001) · IEEE Data(base) Engineering Bulletin · Cited by 1,556
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Empirical evaluation of the tarantula automatic fault-localization technique
Jones, J. A., & Harrold, M. J. (2005) · Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/1101908.1101949 · Cited by 1,238 · cited 1×
“Dynamic FL techniques mainly include spectrum-based FL (SBFL) [Abreu et al. 2009, 2007; Jones and Harrold 2005] and mutation-based FL (MBFL) [Lou et al. 2020; Moon et al. 2014; Papadakis and Le Traon 2015].”
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110 · cited 1×
“Fault localization (FL) [Wong et al. 2016] techniques aim to identify the suspicious locations (e.g., statements or methods) in source code related to bugs.”
GenProg: A Generic Method for Automatic Software Repair
Le Goues, C., Nguyen, T., Forrest, S., & Weimer, W. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,085 · cited 1×
“Traditional APR techniques can be categorized as template-based [Ghanbari et al. 2019; Liu et al. 2019], heuristic-based [Le et al. 2016; Le Goues et al. 2012], and constraint-based [Long and Rinard 2015; Mechtaev et al. 2016] tools.”
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · cited 1×
“With the emergence and popularity of agent-based frameworks [Xi et al. 2023], recently researchers and industry practitioners have begun developing agent-based approaches to solve software engineering tasks.”
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
On the Accuracy of Spectrum-based Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2007) · Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007) · doi:10.1109/taic.part.2007.13 · Cited by 736
Research works printed but not linked: 9
- 10.5555/1308173.1308264 · doi:10.5555/1308173.1308264
- Jie Huang Xinyun Chen Swaroop Mishra Huaixiu Steven Zheng Adams Wei Yu Xinying Song and Denny Zhou. 2024. Large Language Models Cannot Self-Correct ReasoningYet. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=IkmD3fKBPQ
- Carlos E Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik R Narasimhan. 2024b. SWE-bench Leaderboard. https://www.swebench.com/.
- Sophia D Kolak Ruben Martins Claire Le Goues and Vincent Josua Hellendoorn. 2022. Patch Generation with Language Models: Feasibility and Scaling Behavior. In Deep Learning for Code Workshop.
- Theo X Olausson Jeevana Priya Inala Chenglong Wang Jianfeng Gao and Armando Solar-Lezama. 2023. Is Self-Repair a Silver Bullet for Code Generation?. In The Twelfth International Conference on Learning Representations.
- Xianfei Ou Cong Li Yanyan Jiang and Chang Xu. 2024. The Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators. In ASPLOS.
- Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning. PMLR, 31210–31227.
- Chunqiu Steven Xia Matteo Paltenghi Jia Le Tian Michael Pradel and Lingming Zhang. 2024. Universal Fuzzing via Large Language Models. In 46th International Conference on Software Engineering (ICSE).
- Chunqiu Steven Xia Yuxiang Wei and Lingming Zhang. 2023b. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery.
Cited by
Why Do Multi-Agent LLM Systems Fail?
Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 630
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.16941 · Cited by 484
Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
Zhang, J., Hu, S., Lu, C., Lange, R., & Clune, J. (2025) · arXiv · doi:10.48550/arxiv.2505.22954 · Cited by 344
Game-theoretic LLM: Agent Workflow for Negotiation Games
Hua, W., Liu, O., Li, L., Amayuelas, A., Chen, J., Jiang, L., Jin, M., Fan, L., Sun, F., Wang, W., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.05990 · Cited by 101
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
Le, T., Thai, M. V. T., Manh, D. N., Nhat, H. P., & Bui, N. D. Q. (2025) · arXiv · doi:10.48550/arxiv.2512.18470 · Cited by 79
CellVoyager: AI CompBio agent generates new insights by autonomously analyzing biological data
Alber, S., Chen, B., Sun, E., Isakova, A., Wilk, A. J., & Zou, J. (2026) · Nature Methods · doi:10.1038/s41592-026-03029-6 · Cited by 79
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2408.02232 · Cited by 28
SpecRover: Code Intent Extraction via LLMs
Ruan, H.-F., Zhang, Y.-T., & Roychoudhury, A. (2024) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00080 · Cited by 152
Human-In-the-Loop Software Development Agents
Takerngsaksiri, W., Pasuksmit, J., Thongtanunam, P., Tantithamthavorn, C., Zhang, R., Jiang, F., Li, J., Cook, E., Chen, K., & Wu, M. (2024) · arXiv · doi:10.48550/arxiv.2411.12924 · Cited by 24
Human-In-The-Loop Software Development Agents
Takerngsaksiri, W., Pasuksmit, J., Thongtanunam, P., Tantithamthavorn, C., Zhang, R., Jiang, F., Li, J., Cook, E., Chen, K., & Wu, M. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00036 · Cited by 36
Liu, Z., Zeng, R., Wang, D., Peng, G., Liu, X., Liu, Q., Liu, P., Wang, W., & Wang, J. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3667895 · Cited by 20
An Empirical Study of Harness Design for Coding Agents
Fan, R.-Z., Zhang, Z., Ma, S., Hu, Y., Wang, S., Song, K., Liu, F., Zamani, H., & Wang, X. (2026) · arXiv · doi:10.48550/arxiv.2609.20804 · Cited by 16
Are Large Language Models Memorizing Bug Benchmarks?
Ramos, D., Mamede, C., Jain, K., Canelas, P., Gamboa, C., & Goues, C. L. (2024) · arXiv · doi:10.48550/arxiv.2411.13323 · Cited by 16
DAWN: Designing Distributed Agents in a Worldwide Network
Aminiranjbar, Z., Tang, J., Wang, Q., Pant, S., & Viswanathan, M. (2025) · IEEE Access · doi:10.1109/access.2025.3588425 · Cited by 19
DAWN: Designing Distributed Agents in a Worldwide Network
Aminiranjbar, Z., Tang, J., Wang, Q., Pant, S., & Viswanathan, M. (2024) · arXiv · doi:10.48550/arxiv.2410.22339 · Cited by 8
Lau, S., & Guo, P. J. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00041 · Cited by 15
Khan, A., Masood, F., Iqbal, A., Junaid, A., Arif, S., Al-Naeem, M., Husnain, G., & Alzahrani, A. S. (2026) · Scientific Reports · doi:10.1038/s41598-026-42158-y · Cited by 8
AdaCoder: An Adaptive Planning and Multi-Agent Framework for Function-Level Code Generation
Zhu, Y., Liu, C., He, X., Ren, X., Liu, Z., Pan, R., & Zhang, H. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3642621 · Cited by 19
Issue Localization via LLM-Driven Iterative Code Graph Searching
Jiang, Z., Ren, X., Yan, M. L., Jiang, W., Li, Y., & Liu, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00249 · Cited by 64
Can test cases generated by large language models facilitate automated program repair?
Zhang, C., Wang, H., Xu, C., Liu, J., Liu, K., & Liu, Z. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10802-w · Cited by 8
Chang, J., Zhou, X., Wang, L., Lo, D., & Li, B. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3668601 · Cited by 28
Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks
Lu, R., Li, Y., & Huo, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00330 · Cited by 26
“Recent research has increasingly applied LLM-based agents to software engineering from two directions [10]: developing agents to handle specific SE tasks [11], [12] (e.g., debugging), and improving agent frameworks by enhancing role definitions and collaboration mechanisms [1], [13].”
Coding agents: A comprehensive survey of automated bug fixing systems and benchmarks
Puvvadi, M., Arava, S. K., Santoria, A., Chennupati, S. S. P., & Puvvadi, H. V. (2025) · 2025 IEEE 14th International Conference on Communication Systems and Network Technologies (CSNT) · doi:10.1109/csnt64827.2025.10968728 · Cited by 4
ClearAgent: Agentic Binary Analysis for Effective Vulnerability Detection
Chen, X., Zhou, A., Ye, C., & Zhang, C. (2025) · ACM SIGPLAN International Workshop on Language Models and Programming Languages (LMPL) · doi:10.1145/3759425.3763397 · Cited by 15
Let the Code Speak: Incorporating Program Dynamic State for Better Method-Level Fault Localization
Qin, Y., Wang, S., Lin, B., Peng, X. L., Ouyang, S., Chen, L., & Mao, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00072 · Cited by 3
RFCAudit: AI Agent for Auditing Protocol Implementations Against RFC Specifications
Zheng, M., Wang, C., Liu, X., Guo, J., Feng, S., & Zhang, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00105 · Cited by 20
“Basically, a simple indexing design is to utilize the names of functions, files, and directories as summaries, as they often suggest the intended functionality [31], [32].”
Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study
Wang, Y., Pradel, M., & Liu, Z. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3764576 · Cited by 98
Wölflein, G., Ferber, D., Truhn, D., Arandjelović, O., & Kather, J. N. (2025) · arXiv · doi:10.48550/arxiv.2502.11705 · Cited by 3
Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Repair
Huang, K., Zhang, J., Xie, X., & Chen, C. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00100 · Cited by 28
“While some approaches [17], [18] generate test cases as substitutes, they are not well-suited for visual issues.”
InfluentialBusiness Process Discovery Through Agentic Generative AI
Lindenberg, P. P., Kumara, I., Owotogbe, J., van den Heuvel, W., & Tamburri, D. A. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5015-9_19 · Cited by 2
Banitaan, S., Daoud, M., Alquran, H., & Akour, M. A. (2026) · Information · doi:10.3390/info17010073 · Cited by 8
Vul-R2: A Reasoning LLM for Automated Vulnerability Repair
Wen, X.-C., Lin, Z., Yang, Y., Gao, C., & Ye, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00011 · Cited by 19
MCTS-Refined CoT: High-Quality Fine-Tuning Data for LLM-Based Repository Issue Resolution
Wang, Y., Peng, Z., Wang, Y., Wei, Z., Yu, H., & Zhu, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00154 · Cited by 13
FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale
Li, C., Behrang, F., Shi, A., & Liu, P. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00179 · Cited by 2
FURINA: A Multi-Agent Framework for Automated Patch Correctness Assessment
Li, Y., Ji, Y., Chen, X., Shen, Z., Ge, J., & Luo, B. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3811825 · Cited by 5
“Recently, LLMs have supplanted custom neural networks and pre-trained models in APR methods, significantly improving repair rates [61, 76, 78, 93, 94].”
BackportBench: A Multilingual Benchmark for Automated Patch Backporting
Zhong, Z., Huang, J., & He, P. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797127 · Cited by 8
Kang, S., Yun, S., Hong, J., Yoo, S., & An, G. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786874 · Cited by 2
Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection
Ahmed, T., Ganhotra, J., Shinnar, A., & Hirzel, M. J. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787837 · Cited by 29
“To further mitigate risk, one could use separate systems for surrogate patches and the proposed final patch (of course, with increased cost).”
InfluentialGenerative AI Efficiency and Effectiveness in Software Project Documentation Review Process
Demir, K. A., Muppalla, T. N. V., & Liu, B. (2025) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa65407.2025.11165833 · Cited by 1
Toward Non-Expert Customized Congestion Control
Zhang, M., Bagheri, H., & Xu, L. (2025) · International Conference on Communications · doi:10.1109/icc52391.2025.11160790 · Cited by 1
Ramirez, O., & Ekler, P. (2025) · IEEE International Conference on Cognitive Infocommunications · doi:10.1109/coginfocom66819.2025.11200869 · Cited by 3
Designing for Inclusion: Human-Centered Multi-Agent Accessibility Repair in Modern Web Applications
Wieland, J. P., & Werner, C. (2025) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew67781.2025.00067 · Cited by 1
“Unlike monolithic or agentless approaches [16], [17], our architecture supports flexible and explainable repair strategies tailored to React applications and WCAG standards.”
Review of Large Language Model-Based Software Fault Localization Techniques
Tian, P., Liu, P., & Li, Y. (2025) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa66321.2025.00021 · Cited by 2
“Xia et al. [53] analyze patterns in agent design including tool selection, planning, memory, and reflection with case studies that include debugging and localization.”
Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute
Ma, Y., Li, Y.-B., Dong, Y., Jiang, X., Li, Y.-H., Liu, Y., Cao, R., Chen, J., Huang, F., & Li, B.-H. (2025) · International Conference on Automated Software Engineering · doi:10.1109/ase63991.2025.00309 · Cited by 28
PaVeRL-SQL: Text–to–SQL via Partial–Match Rewards and Verbal Reinforcement Learning
Hao, H., Hu, W., Verkholyak, O., Tarzanagh, D. A., Gutow, B., Didari, S., Faraki, M., Moon, H., & Min, S. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402412 · Cited by 4
Are They All Good? Evaluating the Quality of CoTs in LLM-Based Code Generation
Zhang, B., Zhang, L., Luo, Z., Du, Y.-X., Liu, F., Wang, S., & Shi, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3676295 · Cited by 10
“By using LLM-generated CoT, an LLM can approach problem-solving like a human programmer [49], [51], incrementally building solutions step by step, which leads to improvements in the correctness and reliability of the generated code.”
Self-citationATLAS: Agentic Taxonomy of LArge-Scale Software Ecosystems
Lu, J., Lyu, M., Wu, J., Yu, L., Liu, C., Zhang, F., Yang, L., Zuo, C., & Liu, Y. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834361 · Cited by 1
Reformulate, Retrieve, Localize: Agents for Repository-Level Bug Localization
Caumartin, G., & Melo, G. (2026) · doi:10.1145/3786161.3788460 · Cited by 3
Detecting Privilege Escalation in Polyglot Microservices via Agentic Program Analysis
Li, P., Chong, H. Y., Cao, Y., & Yang, J. (2026) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp63933.2026.00121 · Cited by 1
“Recent work has applied this paradigm to code analysis tasks [24]–[26], [29], [30], where agents interact with codebases through tools like file system operations and command execution.”
BLAgent : Agentic RAG for File-Level Bug Localization
Mamun, M. A. A., & Uddin, G. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3830238 · Cited by 4
He, J., Wang, S., Zhao, Y., Hou, X., Liu, Z., Zou, Q., & Wang, H. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773199 · Cited by 13
Bridging Feature Models and Users: LLM-Generated Low-Code Configurators for Software Product Lines
Wegerer, P., Schenkenfelder, B., Ramler, R., Rabiser, R., & Wimmer, M. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3822455.3838767 · Cited by 1
From Seed to Scope: Reasoning to Identify Change Impact Sets
Yadavally, A., & Nguyen, T. N. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773265 · Cited by 1
From Code to Low-Code: An LLM-Driven Pipeline for Lifting Code Clones into Reusable Abstractions
Zefferer, R., Schenkenfelder, B., & Wagner, Ş. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3837062.3839368 · Cited by 1
Dialogagent: An Auto-Engagement Agent for Code Question Answering Data Production
Liang, X., Ren, J., Qi, J., Peng, C., & Jiang, B. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00012 · Cited by 1
RustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust
Ou, G., Liu, M., Chen, Y., Wang, Y., Peng, X., & Zheng, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00057 · Cited by 42
“LLM have also been applied to the translation of entire repository [37]–[39].”
Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
Zhao, X., Li, H., Li, S., Zhao, T., Barr, E. T., Sarro, F., & Ye, H. (2026) · arXiv · doi:10.48550/arxiv.2607.09510 · Cited by 17
Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
Yan, H., Su, M.-L., Zhang, H., Li, Z., Zhang, C., Zhang, S., Chen, Y., Bai, L., & Hu, S. (2026) · arXiv · doi:10.48550/arxiv.2609.01481 · Cited by 5
Wang, W., Piękos, P., Nanbo, L., Laakom, F., Chen, Y., Ostaszewski, M., Zhuge, M., & Schmidhuber, J. (2025) · arXiv · doi:10.48550/arxiv.2510.21614 · Cited by 40
“ChatDev (Qian et al., 2023) first illustrated this idea in the context of automated bug fixing, and similar frameworks were later explored in SWE Yang et al. (2024), OpenHands (Wang et al., 2024), MetaGPT (Hong et al., 2024), and AgentLess (Xia et al., 2025).”
BitsAI-Fix: LLM-Driven Approach for Automated Lint Error Resolution in Practice
Li, Y., Long, Q., Yao, Z., Xu, J., Xie, L., He, X., Geng, L., Han, X., Chen, Y., & Duan, W. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00288 · Cited by 2
Liang, C., Zhao, Y., & Mo, R. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7081-2_16
LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review ⋆Preprint
Rasheed, Z., Muhammad, W., Kemell, K., Saari, M., & Abrahamsson, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6332149 · Cited by 3
Generative AI solutions for software quality: Assessing industrial readiness
Gheventer, A., Gurgel, P. D. A., Brito, C. H., de Mello, R., Rocha, S., Gonçalves, R. F., & Travassos, G. B. S. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09754-7 · Cited by 4
“While they include installation and configuration instructions, they still demand specialized technical knowledge for replication, including specific pro - gramming environments such as Python (Li et al., 2023; Phan et al., 2024 ; Xia et al., 2025 , 2024) and Java (Kessel, 2025; Patel et al., 2024).”
Chen, Z., Lin, L., Lin, X., Chen, Y., & Wang, L. (2026) · Applied Sciences · doi:10.3390/app16104788
Towards Structured, State-Aware, and Execution-Grounded Reasoning for Software Engineering Agents
Chen, T.-H. P. (2026) · BoatSE@ICSE · doi:10.1145/3786161.3788456 · Cited by 2
Evaluating Large Language Models in Cybersecurity: A Systematic Taxonomy and Empirical Analysis
Chen, M., Cheng, H., Su, T., Chen, M., Cai, W., & Zou, H. (2026) · Electronics · doi:10.3390/electronics15102222 · Cited by 2
kAPR: A coverage-guided, context-aware agent for automated repair of Linux kernel bugs
Li, B., Yin, X., Zhang, Y., Liu, S., & Ji, S. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108198
Gudimetla, S., Gangula, S., Adinarayana, R., Challa, C., Vunnam, S., & Buhyavarapu, S. (2026) · International Conference on Artificial Intelligence, Systems, and Emerging Technologies (ICAISET) · doi:10.1109/icaiset66439.2026.11542057 · Cited by 1
Streamlining Repository Tasks with Effective Snippet Retrieval
Xu, T., Li, C., Xu, Z., Jiang, Y., Yao, Y., Zhu, X., Xu, F., Di, P., Xu, C., & Su, Z. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3821433
iCoRe: An Iterative Correlation-Aware Retriever for Bug Reproduction Test Generation
Wang, J., Cao, J., & Liu, Z. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808193 · Cited by 7
“For automated issue-solving agents, many of them incorporate BRT generation as a key component, relying on the resulting tests to validate whether their proposed fixes have correctly resolved the bug [8, 33, 36, 38].”
Still Manual? Automated Linter Configuration via DSL-Based LLM Compilation of Coding Standards
Zhang, Z., Gan, Y., Xing, Z., Zhang, T., Li, Y., Lu, Q., Xu, X., & Zhu, L. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797064 · Cited by 2
Large Language Models for UML Class Diagram Modeling: A Preliminary Empirical Evaluation
Cheng, Y., Huang, Y. R., Yao, S., Qian, Y., Zhang, L., Fang, X., & Wu, T. (2026) · Applied Sciences · doi:10.3390/app16136540
“…language models (LLMs) [11,12], including GPT-4, Claude, Gemini, and open-source alternatives such as LLaMA and Qwen, has demonstrated remarkable capabilities across a wide range of software engineering tasks, including code generation [13], bug fixing [14], test generation, and documentation [15].”
Self-citationMulti-CoLoR: Context-Aware Localization and Reasoning Across Multi-Language Codebases
Vats, I., De, S., Roy, S., Bodhe, S., Varghese, L., Kiehn, M., Bedasso, Y., & Chechik, M. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00107 · Cited by 2
SWE-Bench+: Enhanced LLM Coding Benchmark
Xue, H., Aleithan, R., Enan, N., Uddin, G., & Wang, S. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814924 · Cited by 7
Intelligent Analysis of Software Team Performance: An Agentic Approach
Bishop, V., & Krause, L. (2026) · Structural integrity · doi:10.1007/978-3-032-19744-3_3
Tool Context Adaptation for Agentic Retrieval-Augmented Generation
Li, H., & Jin, J. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582519
“However, developers always understand the tool from coding and common sense [13], which is unavailable to agents.”
From Automation to Autonomy: A Survey of Agentic Workflows in CI/CD Orchestration
Amar, A., Qaiser, I., & Kanwal, A. (2026) · International Journal of Innovations in Science and Technology · doi:10.33411/ijist/1813
Fixing Security Vulnerabilities with Agentic AI in OSS-Fuzz
Zhang, Y., Wang, J., Berzin, D., Мирчев, М., & Roychoudhury, A. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786880 · Cited by 3
LLM-Oriented Information Retrieval: A Denoising-First Perspective
Dai, L., Sun, L., Cao, F., Rao, Z., Yang, C., Liu, H., & Xiong, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808544 · Cited by 2
How Effectively Do Large Language Models Help with Build Conflict Resolution?
Towqir, S. S., Zang, X., Mytkowicz, T., & Meng, N. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00081
“Recently, large language models (LLMs) have demonstrated remarkable abilities to reason across the syntactic, semantic, and contextual aspects of source code [15], [23], [32]–[34], [36], [53], [54], [54], [55], [57].”
AutoCodeRover: Agentic Program Repair for SonarQube Issues
Мирчев, М., Shariffdeen, R., Ruan, H., Zhang, Y., & Roychoudhury, A. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805209
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
Li, J., & Storhaug, A. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805548
Benchmarking Open-Weight LLMs for Security Backporting
Sghaier, S., Daoud, M. A., Chaieb, M., & Chouchen, M. (2026) · Intelligence in Business and Industry (IBI) · doi:10.1109/ibi68858.2026.11604072
Boukhlif, M., Kharmoum, N., Rhalem, W., & Hanine, M. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-30563-3_49
IssueExec: A Test-Driven Approach for Localizing Software Engineering Issues
Liu, J., Lin, Y., Liu, C., Qian, Y., Liu, Y., Chang, J., Zhang, W., & Huang, L. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832290 · Cited by 5
Planning as graphs: Adaptive agentic planning for LLM agents
Zhuang, Y., Liu, Y., Hou, W., Zhao, W., & Liu, X. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133777 · Cited by 1
A Multi-agent AI System for Automated Patent Deep Research and Analysis
Sofean, M., & Aras, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-34374-1_20
Reassessing One-Round Test-Time Refinement for Code Generation
Kim, J. (2026) · AI · doi:10.3390/ai7090353
Ju, X., Yu, X., Wang, R., Li, C., & Chen, X. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7275148
Cao, Y., Keung, J., Wang, Y., Sun, Y., & Mao, Z. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00062 · Cited by 1
ReviewTestFuse: Fusing review-guided and direct test generation for deployable regression testing
Zhen, H., & Zhang, Z. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00669-y
“Recent patch-first software-engineering agents such as AutoCodeRover and Agentless continue this trajectory toward autonomous repository repair (Zhang et al. 2024; Xia et al. 2025).”
CGARF: a causality-guided framework for reliable automated program repair
Yuan, L., Liu, S., Wang, Y., Yao, Y.-W., Liu, Y., Wang, Y., Wang, T., & Mao, T. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10940-1
Diagnosing Candidate Quality Bottlenecks in RL-Guided LLM Code Repair
Zhang, J., & Noh, G. (2026) · IEEE Access · doi:10.1109/access.2026.3727158
“T5APR uses checkpoint ensembles across multiple programming languages [17], ThinkRepair uses self-directed conversational feedback [18], RepairA-gent combines an LLM with autonomous repair tools [19], and Agentless separates localization, repair, and validation for repository-level issues [20].”
Multi-Agent Change Impact Analysis and Test Optimization for AI-Enabled Software Systems
Mani, N., Abdelfattah, A. S., Weerasinghe, S., Li, X., & Cerny, T. (2026) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest70988.2026.00044
“At the same time, recent FSE work on Agentless warns that complex autonomous agents are not always necessary and that simpler, interpretable workflows can be competitive for software-engineering tasks [24].”
Complexity Backpressure for AI Coding Agents: A Cross-Model Study on Substantive Software Tasks
Shaw, A., & Beheshti, A. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00047
CI-Bench: A Framework for Evaluating Large Language Model Tools on CI Failures
Nabil, R. L., Zhu, H., & Rubio-González, C. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787606
Enhancing Issue Localization Agent with Tool-Interactive Training
Ma, Z., Peng, C., Zeng, Q., Gao, P., Zou, Y., & Xie, B. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773137 · Cited by 1
Agent-Based Ensemble Reasoning for Repository-Level Issue Resolution
Tian, Z., Gao, P., Chen, J., & Peng, C. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787761 · Cited by 8
Jonnakuti, S. R. (2026) · International Conference on Modern Sustainable Systems (CMSS) · doi:10.1109/cmss69636.2026.11688408
Messarhi, O., Kerdoudi, M. L., & Tibermacine, O. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113101
Metadata
- Authors
- Xia, C. S., Deng, Y., Dunn, S., & Zhang, L.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Venue
- Proceedings of the ACM on Software Engineering
- Published
- 2025-06-19
- Volume
- 2
- Issue
- FSE
- Pages
- 801-824
- ISSN
- 2994-970X (ISSN-L)
- OpenAlex
- W4411449940
- Cited by
- 849
- References
- 34
- Citations by year
- 2026: 617 · 2025: 206
- FWCI
- 490.90
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the ACM on software engineering. (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (CCF-2131943)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- autonomous software agents, large language models, program repair, software development automation, SWE-bench Verified, patch validation, Claude 3.5 Sonnet, GPT-4o, DeepSeek-V3
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.