Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index; 5 of the 29 sources its claims cite were checked for support.

Demystifying LLM-Based Software Engineering Agents

Open access
Xia, Chunqiu Steven, Deng, Yinlin, Dunn, Soren, Zhang, Lingming · 2025 · Proceedings of the ACM on Software Engineering

Abstract

Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless – an agentless approach to automatically resolve software development issues. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents at the time of paper submission! Agentless also achieves more than 50% solve rate when using Claude 3.5 Sonnet on the new SWE-bench Verified benchmark. In fact, Agentless has already been adopted by OpenAI as the go-to approach to showcase the real-world coding performance of both GPT-4o and the new o1 models; more recently, Agentless has also been used by DeepSeek to evaluate their newest DeepSeek V3 and R1 models . Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patches or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite- S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the currently overlooked potential of a simplistic, cost-effective technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction. We have open-sourced Agentless at: https://github.com/OpenAutoCoder/Agentless

Overview

CoChat Summary

TL;DR

Agentless, a deliberately non-agentic three-phase pipeline for localization, repair, and patch validation, resolves 96/300 SWE-bench Lite issues (32.00%) at $0.70 average cost, the highest among open-source approaches at submission time. The paper also manually classifies SWE-bench Lite problems and constructs a filtered SWE-bench Lite-S benchmark.

Key findings

  • On SWE-bench Lite, Agentless with GPT-4o resolves 96/300 problems (32.00%) at average $0.70 and 78,165 tokens; correct-location rates are 20.7% line, 50.3% function, and 69.7% file.
  • On SWE-bench Verified, Agentless resolves 194/500 problems (38.80%) with GPT-4o and achieves more than 50% solve rate with Claude 3.5 Sonnet.
  • Ablations show patch selection improves from 77 fixes with majority voting to 81 with regression tests and 96 with reproduction tests; repair with 4×10 multi-samples reaches 96 fixes versus 88 for greedy locations and 85 for merged multi-samples.
  • Manual classification finds 4.3% of SWE-bench Lite issues contain the exact ground-truth patch, 9.7% contain complete solution steps, 5.0% contain misleading solutions, and 10.0% lack enough information; the filtered SWE-bench Lite-S has 249 problems, where Agentless resolves 84 (33.73%).

Methods

  • Agentless uses hierarchical localization: top-N suspicious files from repository structure plus embedding retrieval, then classes/functions via skeleton format, then fine-grained edit locations; repair samples Search/Replace diff patches; patch validation generates reproduction tests and uses regression tests plus majority voting.
  • Evaluation uses SWE-bench Lite (300 problems) against 26 agent-based baselines, with GPT-4o (gpt-4o-2024-05-13), text-embedding-3-small, top 3 files, 4 edit-location samples, 10 patches per sample (40 patches per issue), and 40 reproduction tests; metrics are % Resolved, average cost, average tokens, and correct location at line/function/file.
  • The paper manually classifies SWE-bench Lite problems and constructs SWE-bench Lite-S (249 problems), and also reports results on SWE-bench Verified (500 problems).

Limitations

  • Stated internal threat: possible data leakage because SWE-bench Lite ground-truth developer patches may be in GPT-4o’s training data; since GPT-4o is closed-source, this cannot be checked, though the authors note prior work used similar closed LLMs and SWE-bench authors found no significant pre/post-cutoff difference.
  • Stated external threat: SWE-bench Lite may not generalize to other datasets, and it mostly focuses on bug-fixing issues; future work is planned on other benchmarks.
  • Evident from the paper’s own results: reproduction tests are imperfect—213/300 tests reproduce the original issue, but only 94 correctly output “Issue resolved” after applying the ground-truth patch; issue descriptions can lack sufficient information, and Agentless underperforms closed-source agents when no location clue is provided.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3715754

Cites

99 of 99 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

5 Valid · 4 Not valid · 14 Not checked · 106 Own results and interpretations

64 more claims

Create a free account to see all 129 claims and to verify the 4 references still open.

See all claims — free account
  • Probably not a substantial claim. 28
  • It may state two separate claims as one. 39
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 34 references

Research works printed but not linked: 9

  1. 10.5555/1308173.1308264 · doi:10.5555/1308173.1308264
  2. Jie Huang Xinyun Chen Swaroop Mishra Huaixiu Steven Zheng Adams Wei Yu Xinying Song and Denny Zhou. 2024. Large Language Models Cannot Self-Correct ReasoningYet. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=IkmD3fKBPQ
  3. Carlos E Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press and Karthik R Narasimhan. 2024b. SWE-bench Leaderboard. https://www.swebench.com/.
  4. Sophia D Kolak Ruben Martins Claire Le Goues and Vincent Josua Hellendoorn. 2022. Patch Generation with Language Models: Feasibility and Scaling Behavior. In Deep Learning for Code Workshop.
  5. Theo X Olausson Jeevana Priya Inala Chenglong Wang Jianfeng Gao and Armando Solar-Lezama. 2023. Is Self-Repair a Silver Bullet for Code Generation?. In The Twelfth International Conference on Learning Representations.
  6. Xianfei Ou Cong Li Yanyan Jiang and Chang Xu. 2024. The Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators. In ASPLOS.
  7. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning. PMLR, 31210–31227.
  8. Chunqiu Steven Xia Matteo Paltenghi Jia Le Tian Michael Pradel and Lingming Zhang. 2024. Universal Fuzzing via Large Language Models. In 46th International Conference on Software Engineering (ICSE).
  9. Chunqiu Steven Xia Yuxiang Wei and Lingming Zhang. 2023b. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery.

Cited by

Metadata

Authors
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L.
Affiliations
University of Illinois Urbana-Champaign (US)
Venue
Proceedings of the ACM on Software Engineering
Published
2025-06-19
Volume
2
Issue
FSE
Pages
801-824
ISSN
2994-970X (ISSN-L)
OpenAlex
W4411449940
Cited by
849
References
34
Citations by year
2026: 617 · 2025: 206
FWCI
490.90
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the ACM on software engineering. (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
NSF (CCF-2131943)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
autonomous software agents, large language models, program repair, software development automation, SWE-bench Verified, patch validation, Claude 3.5 Sonnet, GPT-4o, DeepSeek-V3

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.